SoundExpert

Opus, AAC, Vorbis, MP3 at 96 kbit/s

2013-04-27T12:13:48Z

This listening test follows the similar @64 one, finished not so far ago. Codec contenders are the same with a few exceptions:

Opus version was updated from 1.0.1 to 1.0.2
Lame mp3 encoder added (3.99.5)
QuikTime encoder was updated but without any quality improvements; in this test it is used in True VBR mode.

Final list of contenders:

AAC VBR@89.8 (Winamp 5.63) - VBR, AAC LC

AAC Encoder v1.04 (Fraunhofer IIS) from Winamp 5.63: Variable Bitrate, Preset 3
The encoder is also available as CLI utility and can be downloaded from https://github.com/tmkk/fhgaacenc

AAC VBR@92.1 (QTime 7.7.3) - TrueVBR, AAC LC

QuickTime (7.7.3) AAC Encoder via qaac 2.18 (CoreAudioToolbox 7.9.8.2): qaac -V45 ref.wav
The encoder is included in such Apple products as iTunes, QuickTime, Safari; though not all of its features can be reached through these applications. For greater flexibility the CLI utility qaac can be used
download from https://sites.google.com/site/qaacpage/

AAC VBR@90.0 (NeroRef 1540) - CVBR, AAC LC

Nero AAC Encoder 1.5.4.0 (build 2010-02-18): neroAacEnc.exe -q 0.34 -if ref.wav -of out.mp4
download from http://www.nero.com/eng/downloads-nerodigital-nero-aac-codec.php

Vorbis VBR@90.4 (Xiph 1.3.3)
OggEnc v2.87 (libVorbis 1.3.3): oggenc2 -q2.2 ref.wav

download from http://vorbis.com/

Opus VBR@90.7 (libopus 1.0.2)

opusenc --bitrate 90 ref48.wav (44.1/16 -> 48/24 by Audition CS6)
The encoder operates at 48kHz only, so for other sample rates (as in our case) resampling is required. Some good quality resampler is already integrated into the codec but we used the external one from Audition CS6; the complete test files production chain: reference test files 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> decode by foobar2000(48/24) -->> 44.1/32(Audition CS6) -->> test files production
download from http://opus-codec.org/downloads/

mp3 VBR@90.2 (Lame 3.99.5) - MPEG-1 Layer 3, VBR

encode: lame -V6.9 ref.wav
For this quality parameter (-V6.9) Lame down-samples the input from 44.1 kHz to 32 kHz. Decoder's output was up-sampled back to 44.1 with Audition CS6; the complete test files production chain: reference test files 44.1/16 -->> Lame -->> decode by MAD(32/24) -->> 44.1/32(Audition CS6) -->> test files production
download from http://www.rarewares.org/mp3-lame-bundle.php

In addition to usual live ratings on 96kbit/s page there will be detailed results of this test when corresponding ratings reach 1% accuracy (example of the detailed report is in the previous 64 kbit/s listening test).

***

SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after finishing corresponding listening test. So we encourage you to take part in testing. SE ratings exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading test files from SoundExpert you will not necessarily get the ones for those newly added codecs. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Opus, AAC and Vorbis in 64 Kbit/s section

2013-04-11T21:41:15Z

In September 2012 new, open and royalty-free audio codec Opus was standardized by the IETF. First stable release is available on Opus site - www.opus-codec.org. SE starts testing of the codec from 64kbit/s section. Other contenders are AAC (three versions from different developers) and Vorbis (Xiph version). Settings for all codecs were adjusted to get the target bitrate as close as possible to each other. As usually the target bitrate (FBR) is calculated on the basis of nine SE sound samples concatenated. Details of the codecs added:

AAC+ VBR@59.2 (Winamp 5.63) - CVBR, HE-AAC

AAC Encoder v1.04 (Fraunhofer IIS) from Winamp 5.63: Variable Bitrate, Preset 2
The encoder is also available as CLI utility and can be downloaded from https://github.com/tmkk/fhgaacenc

AAC+ VBR@59.6 (QTime 7.7.2) - CVBR, HE-AAC

QuickTime (7.7.2) AAC Encoder via qaac 1.45 (CoreAudioToolbox 7.9.8.1): qaac --he -v56 ref.wav
The encoder is included in such Apple products as iTunes, QuickTime, Safari; though not all of its features can be reached through these applications. For greater flexibility the CLI utility qaac can be used
download from https://sites.google.com/site/qaacpage/

AAC+ VBR@60.1 (NeroRef 1540) - CVBR, HE-AAC

Nero AAC Encoder 1.5.4.0 (build 2010-02-18): neroAacEnc.exe -q 0.25 -if ref.wav -of out.mp4
download from http://www.nero.com/eng/downloads-nerodigital-nero-aac-codec.php

Vorbis VBR@60.3 (Xiph 1.3.3)

OggEnc v2.87 (libVorbis 1.3.3): oggenc2 -q0.3 ref.wav
download from http://vorbis.com/

Opus VBR@59.9 (libopus 1.0.1)

opusenc --bitrate 59 ref48.wav (44.1/16 -> 48/24 by Audition CS6)
The encoder operates at 48kHz only, so for other sample rates (as in our case) resampling is required. Some good quality resampler is already integrated into the codec but we used the external one from Audition CS6; the complete test files production chain: reference test files 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> decode by foobar2000(48/24) -->> 44.1/32(Audition CS6) -->> test files production
download from http://opus-codec.org/downloads/

All of the codecs are in the system already. Shortly after first grades have been received an incorrect calculation of ratings' reliability in 64kbit/s section was spotted. In order to recalculate all reliability values, all codecs in the section including newly added ones were put on rotation. Also testing of all other devices at SE was paused. After achieving reliable ratings in 64kbit/s section normal operation of SE testing engine will be resumed (such technical details of SE operation are announced in SE twitter now).

***

SE thanks participants of discussion on rutracker.org forum (in Russian) and specially G-Man90 and MindWin for their help on choosing these particular codecs and settings. G-man90's personal web site audiophilesoft.ru (in Russian) also helped a lot with the latest info about audio codecs and other PC audio issues.

As usually SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after completing of SE listening tests. So we encourage you to take part. SE ratings exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading test files from SoundExpert you will not necessarily get the ones for those newly added codecs. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Update 04.2013

Results of the test

Although the ratings of the above codecs had been available just from the beginning of the listening test with gradually increasing reliability (1% accuracy now), more detailed results could be helpful and interesting as well. Below is the outcome of this test in more traditional form with means and confidence intervals.

Full Res PDF (23KB): Results of SE listening test of Opus, AAC and Vorbis @64 kbit/s (11.2012)

Numbers of collected grades for each codec and sample are in the tables below. Numbers in brackets indicate that those grades were discarded by SE automated post-screening mechanism (Median +/- 1.5) and were not used for further calculations. Sound samples (BAH, BAS, CST, ...) can be found here.

AAC+ VBR@59.2 (Winamp 5.63) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | [1] *7* | [1] [1] *6* | [2] [1] [2] [2] [2] ----+------------------------------------------------------- *5* | 10 3 3 7 6 1 2 *4* | 6 12 8 12 8 9 7 3 5 *3* | 2 4 5 9 3 *2* | [1] *1* | ----+------------------------------------------------------- NN | 16 17 11 16 15 15 13 12 10 Mean| 4.63 4.06 4.27 3.75 4.47 4.40 3.69 3.25 3.90 AAC+ VBR@60.1 (NeroRef 1540) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | [1] *7* | *6* | [2] [1] [1] [1] [2] [2] ----+------------------------------------------------------- *5* | 7 11 6 1 [1] [1] 3 *4* | 10 6 7 11 5 7 4 5 *3* | 1 [2] 1 7 9 14 8 11 7 *2* | [1] 1 1 *1* | ----+------------------------------------------------------- NN | 18 17 14 19 10 19 15 15 16 Mean| 4.33 4.65 4.36 3.68 2.90 3.26 3.47 3.27 3.63 AAC+ VBR@59.6 (QTime 7.7.2) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | *7* | [1] *6* | [1] [1] [1] [1] ----+------------------------------------------------------- *5* | 4 5 [1] 3 3 [2] [2] *4* | 14 10 2 15 5 8 [1] 4 *3* | 2 1 11 12 2 3 1 3 14 *2* | 1 13 1 *1* | ----+------------------------------------------------------- NN | 20 16 13 13 17 11 12 16 19 Mean| 4.10 4.25 3.15 2.92 3.88 4.00 4.17 2.19 3.16 Opus VBR@59.9 (libopus 1.0.1) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | [1] [1] [1] *7* | [2] *6* | [3] [1] [2] [3] [2] [2] ----+------------------------------------------------------- *5* | [2] [5] [4] 2 [3] [3] *4* | 1 1 1 11 [1] 1 1 2 *3* | 13 10 13 1 4 9 14 4 *2* | 3 1 9 11 7 1 6 *1* | 4 7 ----+------------------------------------------------------- NN | 17 12 14 14 17 18 17 16 12 Mean| 2.88 3.00 3.07 4.07 2.00 1.61 2.65 3.00 2.67 Vorbis VBR@60.3 (Xiph 1.3.3) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | [1] *8* | [1] *7* | [1] [1] [1] *6* | [3] [1] [1] ----+------------------------------------------------------- *5* | [2] [1] [3] [1] [1] [3] [1] *4* | 1 [2] 2 1 4 [1] [2] 1 *3* | 5 7 10 14 8 6 6 10 *2* | 8 11 9 3 10 13 8 16 7 *1* | 2 1 [1] ----+------------------------------------------------------- NN | 14 18 21 18 22 21 14 17 18 Mean| 2.50 2.39 2.67 2.89 2.73 2.19 2.43 1.94 2.67

Audio quality of iPod Video

2012-06-23T17:23:39Z

iPod Video

iPod Video which belongs to fifth generation of iPods was introduced in 2005 and still loved by many demanding listeners of portable players. Powerful output with excellent sound quality (Wolfson audio chip on-board) made the player iconic. Not surprising that a lot of them are still in use. Adding iPod Video to SE rating system we place an anchor point on SE quality scale making the whole scale a bit more meaningful. And this is the first non-smartphone in the section "Portable audio players". Should be the one at least.

Preliminary conclusions in short:

Technical measurements of distortions are good and usual for premium audio players
Predominance of odd harmonics in sine wave reproduction may theoretically lead to some harshness of the sound
As many players iPod Video smears slightly music passages with fast transients
The above technical features can't be used for judging final audio quality of iPod Video. Please, refer to the results of listening tests for the purpose. Participation is welcome!

Difference levels for SE standard set of samples and signals are in the Table 1.

**Table 1**: Difference levels [dB] of signals and samples transferred through iPod Video. The values show how much output waveform differs from input one (-∞dB: perfectly similar, 0dB: totally different).
Sine@1kHz 88dB SPL	White noise 88dB SPL	Bass 83dB SPL	Castanets 80dB SPL	French male speech 84dB SPL	Glockenspiel 77dB SPL	Harpsichord 79dB SPL	J.S.Bach 82dB SPL	Postscriptum 80dB SPL	Mike Oldfield 83dB SPL	Quartet 85dB SPL
-68.72 (L)	-24.58 (L)	-26.22	-30.39	-30.03	-32.90	-18.51	-27.24	-20.88	-11.80	-30.60
-68.83 (R)	-24.59 (R)

The values are quite usual. The nature of sine wave distortion is unpleasant - odd harmonics prevail over the even ones (see Audio 1). Usualy this makes the sound harsh and causes fatigue when listening long. Though the level of such distortion in iPod Video is relatively low (-69dB) to cause any dramatic consequences for its sound.

Preview this Audio in lossy format
(vorbis@160 or mp3@165 depending on your browser)

Preview is not available in your browser (HTML5 support is required). Use the option below instead

Download this Audio in lossless format
(24sec, mono, FLAC:927Kb

Audio 1: Sinusoidal signals: reference and output from iPod Video with sound artifacts amplified to +69dB

More serious problems could arise from some weakness in reproduction of fast transients in sound. SE sample "Mike Oldfield" clearly shows this even at small level (+4dB) of artifacts amplification. It looks like it is a common problem for all portable players - sound bursts are smeared. Probably this is a feature of energy-saving design of integrated headphones amplifiers.

***

SoundExpert thanks Ivasha for his still in use iPod Video and possibility to record SE test samples on his device.

As usually SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after completing of SE listening tests and any technical parameters could be misleading when evaluating perceived sound quality. So we encourage you to take part in our listening tests. All SE ratings, including iPod Video one, exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading a test file from SoundExpert you will not necessarily get a test file of iPod Video. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Audio quality of Galaxy SII

2012-06-12T21:55:25Z

Galaxy SII
smartphone by Samsung

The second iPhone-killer from Samsung - Galaxy SII (GT-I9100, Android "Gingerbread") was added to SE rating system, "Portable players" section. Unlike its predecessor it uses Yamaha audio chip (Galaxy S features Walfson one) which is considered to be of inferior audio quality. Our main conclusions are:

Distortions of various audio signals are low (see Table 1), they are comparable with the values for other players of this class
There is some oddity in reproduction of pure sine wave (see Audio 1) but the level of the distortion is perfectly low
Music passages with high macro-dynamics and pronounced stereo effects are a problem for Galaxy SII integrated headphones amplifier (see Audio 2)
Final conclusion about Galaxy SII audio quality can be made only after finishing of listening tests which are integral part of SE audio metric; consider taking part in SoundExpert tests

Objective measurements

Difference levels for our standard set of signals and samples are in the Table 1.

**Table 1**: Difference levels of signals and samples transferred through Galaxy SII. The values show how much output waveforms differ from input ones (-∞dB: perfectly similar, 0dB: totally different).
Sine@1kHz 88dB SPL	White noise 88dB SPL	Bass 83dB SPL	Castanets 80dB SPL	French male speech 84dB SPL	Glockenspiel 77dB SPL	Harpsichord 79dB SPL	J.S.Bach 82dB SPL	Postscriptum 80dB SPL	Mike Oldfield 83dB SPL	Quartet 85dB SPL
-67.38 (L)	-20.83 (L)	-26.58	-28.10	-27.85	-33.31	-19.49	-24.87	-20.41	-11.85	-29.59
-67.54 (R)	-20.88 (R)

The values look promising especially in case of white noise. The level of sine signal distortion is low but the nature of its distortion is unexpectedly strange (see Audio 1).

Preview this Audio in lossy format
(vorbis@160 or mp3@165 depending on your browser)

Preview is not available in your browser (HTML5 support is required). Use the option below instead

Download this Audio in lossless format
(10sec, mono, FLAC:750Kb

Audio 1: Sinusoidal signal (@1kHz, 88dB SPL unweighted) at the output of Galaxy SII with sound artifacts amplified to +67dB

The origin of that modulation is unclear but its level is pretty low to provoke any serious audible consequences on real life audio material.

Sound flavoring

One of SE test samples revealed another weakness of Galaxy SII audio. In the Table 1 "Mike Oldfield" sample has relatively high level of distortion -11.85dB. The sample contains independent "sound bursts" in each stereo channel which easily penetrate into neighboring channel with different phase shifts mostly inverted. To demonstrate the results of such occasional cross-talk we prepared "artificial stereo" recording of "Mike Oldfield" sample as we did that previously for Galaxy S. Left and right channels were played back and recorded separately (sequentially) and combined later in stereo track. Resulting "artificial stereo" doesn't suffer from any cross-talks and can be compared with the usual stereo recording. For the ease of comparison sound artifacts of both recordings were amplified to +15dB and the reference sample is added at the beginning (see Audio 2).

Preview this Audio in lossy format
(vorbis@160 or mp3@165 depending on your browser)

Preview is not available in your browser (HTML5 support is required). Use the option below instead

Download this Audio in lossless format
(38sec, stereo, FLAC:2.68Mb)

Audio 2: Three sound samples show the consequences of occasional cross-talk in Galaxy SII. First sample is a reference one. Second – played back by Galaxy SII without cross-talk (left and right channels were recorded separately). Third sample was played back and recorded in stereo normally thus it is affected by cross-talk. Sound artifacts of two last samples were amplified to +15dB making the difference between samples more audible.

The "artificial stereo" sample shows only lack of mid-high frequencies which is normal at this level of artifacts amplification - almost all audio equipment loaded with real load has some gradual cut of highs. The third "normal stereo" sample has a lot of highs penetrated from neighboring channel with varying phase shifts making the whole sound picture look like being processed by low bitrate encoder. Additional research revealed that such cross-talk occurs only in rare cases. Most audio material sounds the same in "normal stereo" and "artificial stereo" recordings. So the only problem is the music with high macro-dynamics and pronounced stereo effects. Such combination is common for electronic music. Taking into account that the above weakness become noticeable at relatively small artifacts amplification (+5dB) we can conclude that most dramatic parts of such music will be slightly smeared for sure.

As the above cross-talk occurs only during fast and loud sound transients, the most probable origin of this is headphones amplifier.

***

SoundExpert kindly thanks Galina for possibility to make all necessary recordings on her Galaxy SII smartphone.

As usually SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after completing of SE listening tests and any technical parameters could be misleading when evaluating perceived sound quality. So we encourage you to take part in our listening tests. All SE ratings, including Galaxy SII one, exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading a test file from SoundExpert you will not necessarily get a test file of Galaxy SII. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Mastered For iTunes vs. Mastered For People

2012-04-18T18:04:57Z

Not so far ago Apple announced the initiative aimed to form a kind of standard for mastering of audio content intended for distribution through iTunes Store. Mastering guidelines for audio engineers and independent musicians as well as a set of special audio tools are available on Apple site. The guidelines could be summarized as follows:

If possible master recordings should be submitted to iTunes Store as 96/24 audio files; remastering is advised in cases when it could provide discernible improvements.
These high resolution files must be free of clipping; it is recommended to have at least 1dB headroom for peak levels.
Apple highly recommends to preserve full dynamic range of recordings which is usually well defined by creative purposes of the author. Over-compression of dynamic range (“loudness war”) is no longer a way to attract listeners because almost all distribution and broadcasting channels now have technologies to control playback volume regardless of actual loudness of a track. Itunes has them for sure.
As almost all tracks in iTunes Store are distributed now in iTunes Plus format (44.1/16, stereo, AAC, VBR-constrained, 256 kbit/s) it is recommended before submitting to preview audio material in the above format and to listen it on various consumer devices using both headphones and speakers.

In order to ease following these guidelines Apple provided a set of audio utilities for Mac OS:

afconvert performs sample rate and bit depth conversions as well as AAC encoding exactly the way iTunes Store does. So any author has possibility to preview quality of final AAC file which will be distributed. Conversion from high resolution files into AAC is made through intermediate 32bit .caf file which helps to avoid dithering thus the whole process of encoding preserves the maximum possible quality.

afclip finds all possible clippings in audio file including the ones between samples; it is recommended to test audio material for clipping both before and after encoding as the latter could introduce additional clipping sometimes.

AURoundTripAAC Audio Unit is intended mostly for blind ABX testing of encoded audio material. With this tool an author can test critical parts of the track for encoding artifacts audibility. No doubt this utility will be helpful also for those listeners who would like to test their ability to hear coding artifacts at various bit rates.

Master for iTunes Droplet automates some of the above procedures.

First AAC@256 encoded audio tracks appeared in iTunes Store in 2007. Soon after that SoundExpert added AAC@256 encoder from the latest at that moment iTunes to the rating system, though there was no confidence that audio material for iTunes Store prepared with that exact encoder. Now Apple disclosed the whole procedure of encoding. The encoder itself was improved since 2007. Taking all this into account we decided to encode SE test files into iTunes Plus format using Apple mastering guidelines and add them to the 256 kbit/s section.

Encoding was performed on Mac OS X 10.7.3 (11D50b) using two recommended commands:

afconvert test.wav intermediate.caf -d 0 -f caff –soundcheck-generate

afconvert intermediate.caf -d aac -f m4af -u pgcm 2 --soundcheck-read -b 256000 -q 127 -s 2 final.m4a

Decoding:

afconvert final.m4a decode.wav -d LEI24 -f WAVE

Resulting 24bit .wav file was used to prepare SE test files as usually, thus preserving the highest possible audio quality the iTunes Plus format can offer for 44.1/16 source material.

Out of curiosity SE test files were also encoded with the latest iTunes for Windows (10.6.1.7) using iTunes Plus setting. Difference levels for both encodings are in the Table 1.

Table 1: Difference levels [dB] of SE test files encoded into iTunes Plus format on Mac OS and Windows.

	Bass	Castanets	French male speech	Glockenspiel	Harpsichord	J.S.Bach	Postscriptum	Mike Oldfield	Quartet
Mac OS X	-39.21	-39.95	-41.56	-43.22	-43.68	-37.09	-36.02	-35.96	-35.89
Windows	-39.21	-39.96	-41.56	-43.23	-43.68	-37.09	-36.02	-35.96	-35.89

Similar values with small variance between encoders mean that they use the same psycho-acoustic model but differ in computation procedures. Usually such small variance of Df levels is imperceptible and it can be safely assumed that both encoders produce audibly identical files. So anyone with the latest iTunes installed can encode their music with the exact quality established in iTunes Store.

Mastered For People

Apple's initiative to create some framework standard for digitally distributed music seems actual and praiseworthy. Exactly poor mastered recordings is the main source of “bad sound” problem nowadays. Using of psycho-acoustic encoding on the final stage of distribution just worsens the problem not originates it. Even taking into consideration that iTunes is only a small part of global music distribution, Apple is capable to influence on mastering practices worldwide as the company is well known trend-setter today. Not to mention the sweet marketing goal for the company to give the name – Mastered For iTunes – to this new trend of more careful mastering practices.

On the other hand iTunes Store is only a tiny, almost insignificant part of global music sharing on the internet. P2Pnetworks, cyberlocking and cloud services, social networks and author's personal web sites rapidly become effective channels of music distribution. Distinction between distribution and sharing become less and less obvious. Definitely some mastering guidelines are necessary for authors and musicians who get on in the world this new way. Apple's guidelines with a few remarks could serve the purpose and such “Mastered For People” recommendations may look like follows:

1. If possible retain your audio tracks as multi-track projects for

further improvement and development
producing various master records in 2.0, 5.1, 7.1, 3D … formats
sharing your work as multi-track mixes for “creative listening” - making by listeners their own versions of your tracks

2. Retain at least 96/24 stereo mix with full dynamic range and 1dB headroom for peak levels (-1 dBFS)

3. While making equalization, limiting, dynamic range compression ... of the 96/24 master for final distribution:

use 32bit mode
avoid clipping (-1 dBFS)
lo-fi versions of your track may require special post-mastering
resulting master record should be 96/24 as well

4. Variants of conversions for final distribution/sharing:

[96/24 master] → downsampling → [48/32] → dithering → [48/16] → FLAC | ALAC | RAR | …
[96/24 master] → downsampling → [48/32] → Vorbis | AAC | MP3 | … @256-320 kbit/s
[96/24 master] → downsampling → [44.1/32] → dithering → [44.1/16] → Audio CD

Sample frequency 44.1kHz seems become more and more obsolete/senseless as it helps only to solve some compatibility issues such as creation of Red Book Audio CDs and using of old audio equipment. Sampling frequency 48kHz serves well both professional and consumer audio today making 44.1kHz unnecessary.

The draft is open for discussion, drop us a line if you are interested.

SoundExpert thanks James Orlov for encoding test files on his perfectly updated MacBook Pro and for technical assistance with Mac OS X.

As usually SoundExpert encourages you to take part in listening tests. All SE ratings, including iTunes Plus 2012 ones, exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading a test file from SoundExpert you will not necessarily get a test file of iTunes Plus 2012. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Vorbis 1.3.3 in 256 kbit/s section

2012-03-29T00:05:12Z

According to Xiph.Org new release of popular Vorbis audio codec was out on 2012.02.03. It contains only bug, doc and security fixes (no quality improvements). Previous version – 1.3.2 – was released on 2010.11.02.

The latest Vorbis encoder was added to 256 kbit/s section with the following settings:

Vorbis VBR@256.4 (Xiph 1.3.3) - Ogg Vorbis, 256.4 kbit/s FBR
CODER: OggEnc v2.87 (libVorbis 1.3.3)
- usage: oggenc2 -q8.2 ref.wav
- 44100 Hz Stereo
DECODER: OggDec v1.9.7 (libVorbis 1.3.1)
- oggdec ref.ogg

As usually encoding was made with OggEnc utility and decoding – with OggDec utility without dithering.

What is Vorbis codec

Ogg Vorbis is a fully open, non-proprietary, patent-and-royalty-free, general-purpose compressed audio format for mid to high quality audio and music at fixed and variable bitrates. It means that you can freely encode and distribute your music in high quality without paying to anyone.

The Vorbis bitstream format was frozen in 2000 and all bitstreams encoded since will remain compatible with all future releases of Vorbis.

Extensive FAQ about Vorbis format is on Xiph.Org.

Where to download Vorbis codec

Vorbis encoder, decoder and other tools are available for all major platforms. Visit Vorbis.com to get them.

SoundExpert thanks David for this codec suggestion. We also received other suggestions on the codec and plan to add it to different SE bitrate sections. Your proposals are still welcome.

SoundExpert encourages you to take part in listening tests. All SE ratings including Vorbis ones exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading a test file from SoundExpert you will not necessarily get a test file of Vorbis 1.3.3. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Thank you in advance!
SE Team.

Test files of iPhone 4 were replaced

2011-10-22T21:27:14Z

Extensive research experiments with Galaxy S revealed that accuracy of test files preparaton for analog audio devices must be higher than we used for iPhone 4. As a result comparison of players could be unfair. We decided to prepare test files for iPhone 4 from scratch using our latest recordings of white model with iOS 4.3.5. Unfortunately there is no possibility to account the grades submitted with old test files. They were discarded, sorry!

New test files are alredy in the system. Article about iPhone 4 was corrected accordingly.

Best regards,
SE Team.

Audio quality of Galaxy S

2012-06-09T10:46:53Z

Figure 1: Galaxy S smartphone by Samsung

This smartphone by Samsung is considered by many as a competitor of iPhone 4. So we desided to add it to our new "Portable players" section side by side with its rival. The model we tested is GT-I9000 with Android "Gingerbread" operating system. Our main conclusions in short:

Objective measurements show that audio tract of Galaxy S is on a par with similar devices of this class. Final conclusion about its quality can be made only after finishing listening tests when its rating become reliable. Consider taking part in SoundExpert tests.
If you listen it with wired headphones and satisfied with the loudness, you don't need to upgrade original audio drivers to Voodoo ones. If you upgrade though, you'll get additional output power without quality sacrifice.
Besides common for all audio players noise and gradual cutting of high frequences Galaxy S has some special flavour of sound – spatial distortion of stereo image. But the level of the degradation is too subtle to be noticed even in perfect cans.

Objective measurements

First of all we failed to get our standard required output level from headphones OUT of Galaxy S, connected to Koss Porta Pro. Even at max volume available with original sound drivers we fell short of 2dB SPL. Actually it's not a big problem for the objective measurements but limitation of output level is obvious and can be a problem for high impedance or low sensitivity headphones.

Difference levels for our standard set of signals and samples are in the Table 1:

Sine@1kHz 86dB SPL	White noise 86dB SPL	Bass 81dB SPL	Castanets 78dB SPL	French male speech 82dB SPL	Glockenspiel 76dB SPL	Harpsichord 77dB SPL	J.S.Bach 80dB SPL	Postscriptum 78dB SPL	Mike Oldfield 81dB SPL	Quartet 83dB SPL
-70.85 (L)	-6.15 (L)	-32.26	-22.17	-34.50	-30.59	-18.60	-32.00	-24.82	-16.69	-36.52
-70.84 (R)	-6.15 (R)

Table 1: Difference levels of various signals and samples transfered through Galaxy S with naitive sound drivers. The values show to what extent the waveformes change their shapes passing through device:

-∞ dB - signals passed unchanged, perfectly exact shapes
0 dB – input and output signals have nothing in common

For sinusoidal signals and small Df values Difference level can be easily converted to THD+Noise:

THD+Noise [dB] ≈ Df [dB] + 3 [dB].

Sound pressure levels of samples and signals are unweighted and averaged over 400ms.

The values look good and depend on complexity of signals; nothing extraodinary. If you like to listen the sine wave @1kHz at the output of Galaxy S with artifacts amplification +71dB refer to Audio 1.

Preview this Audio in lossy format
(vorbis@160 or mp3@165 depending on your browser)

Preview is not available in your browser (HTML5 support is required). Use the option below instead

Download this Audio in lossless format
(10sec, FLAC:1.24Mb)

Audio 1: Sinusoidal signal (@1kHz, 86dB SPL unweighted) at the output of Galaxy S with sound artifacts amplified to +71dB

The sine signal is deeply buried in noise. The origin of its audible fluctuations is unclear. Our guess – it is headphones amplifier of class W at work. Galaxy S uses Walfson WM8994 audio hub with integrated power amplifier of this class. Such amplifires dinamicly adjust voltage level for their output stages maximizing efficiency and reducing heat dissipation this way. The adjustments are made discretely depending on power required at each moment. So what we hear is probably some modulation of signal by those adjustments fluctuating around optimum voltage value (as the signal level is costant). If somebody can suggest better explanation, please, drop us a line. And yes, this pecularity doesn't depend on audio drivers, software players, input files (WAV or FLAC) and specific to Galaxy S only.

In any case this oddity at the level of -70dB doesn't affect real-life listening experience but can be interesting for developers of amplifires.

Also, despite heavy isolation of headphones from background noise during recording, some of our voices are noticable at this level of artifacts amplification. We'll find who cried louder than others during that particular recording and punish demonstratively . All future recordings will be made in silence. Promise.

Voodoo audio drivers

Opensource project Voodoo originated by François Simond aka supercurio from xda-developers forums developed alternative and very popular audio drivers. They unlock full power of headphones amplifier and make possible some low level tunings of WM8994 chip which improve sound quality. For assessment of those improvements we installed Voodoo Control Plus 3.0 (WM8994 v10) with the following settings (Figure 2):

Figure 2: Settings of Voodoo sound drivers we used

Now we could easily achieve required sound pressure level. Results of our standard measurements are in the Table 2.

Sine@1kHz 88dB SPL	White noise 88dB SPL	Bass 83dB SPL	Castanets 80dB SPL	French male speech 84dB SPL	Glockenspiel 77dB SPL	Harpsichord 79dB SPL	J.S.Bach 82dB SPL	Postscriptum 80dB SPL	Mike Oldfield 83dB SPL	Quartet 85dB SPL
-72.87 (L)	-6.15 (L)	-32.48	-22.19	-34.58	-30.65	-18.67	-32.15	-24.97	-16.97	-36.68
-72.90 (R)	-6.15 (R)

Table 2: Difference levels of various signals and samples transfered through Galaxy S with Voodoo sound drivers. The values show to what extent the waveformes change their shapes passing through device:

-∞ dB - signals passed unchanged, perfectly exact shapes
0 dB – input and output signals have nothing in common

For sinusoidal signals and small Df values Difference level can be easily converted to THD+Noise:

THD+Noise [dB] ≈ Df [dB] + 3 [dB].

Sound pressure levels of samples and signals are unweighted and averaged over 400ms.

The improvements are not dramatic. Audible difference is also very subtle and illustrated by the piano excerpt (Audio 2).

Preview this Audio in lossy format
(vorbis@160 or mp3@165 depending on your browser)

Preview is not available in your browser (HTML5 support is required). Use the option below instead

Download this Audio in lossless format
(15sec, FLAC:1.21Mb)

Audio 2: Piano excerpt played back on Galaxy S using Voodoo (1) and original (2) sound drivers. In both samples sound artifacts are amplified to +32dB. The second one (with original sound drivers) has a bit more noise and more "dirty" high frequences because they are reproduced less accurately. Though the difference is very subtle even with this huge artifacts amplification.

Taking into account that quantative improvements, measured with Diff. Level, are within 0.1dB for real-life signals and audible difference is faintly discernible, we can safely conclude that such improvements are far beyond the threshold of human perception. Original sound drivers in this sense are similar to Voodoo ones. So if there is no need for additional output power from your Galaxy S (very reasonable self-restriction nowadays), there is no necessity to install Voodoo drivers. In case of using some headphones with low sensitivity or high impedance or both Voodoo drivers will help to drive those cans without loosing audio quality.

Sound flavoring

Another feature of Galaxy S sound is some subtle flavoring caused by small and unusual cross-talk between stereo channels. It is unusual because amplitude and phase of signal penetrating into neighboring channel are frequency dependent. In most cases penetrating signal consists of middle-to-high frequencies, has inverted polarity and amplitude -40dB (and lower) below level of main signal in channel. In order to assess consequences of such cross-talk we prepared two stereo recordings – one usual with cross-talk and one without. For the purpose the piano sample, played back by Galaxy S, was recorded three times – left and right channels separately and in stereo as usual. Then left and right channels were combined in stereo thus completely eliminating any cross-talk in this stereo recording. In both stereo recordings sound artifacts were amplified to +16dB. Audio 3 presents result of the experiment. It consists of three samples – original piano excerpt, played back version without cross-talk and version with cross-talk.

Preview this Audio in lossy format
(vorbis@160 or mp3@165 depending on your browser)

Preview is not available in your browser (HTML5 support is required). Use the option below instead

Download this Audio in lossless format
(23sec, FLAC:1.42Mb)

Audio 3: Three sound samples show the influence of cross-talk in Galaxy S on resulting sound quality. First sample is a reference piano sample. Second – the same sample played back by Galaxy S without cross-talk (left and right channels were recorded separately). Third sample was played back and recorded in stereo normally thus it is affected by cross-talk. Sound artifacts of two last samples were amplified to +16dB making the difference between samples more audible.

In the third sample (affected by cross-talk) sound image is spread among channels as if it would been processed by spatializer. If it were not for so low levels of this cross-talk we would suppose it was made in purpose. Cross-mixing of stereo channels with inverted polarity is often used for widening of stereo base in various sound enhancers. But in this case we rather suppose it is a side effect of analog circuitry design, again the headphones amplifier most likely. The effect becomes audible only with pretty high levels of artifacts amplification (+12dB and more). At these levels increased noise and absence of high frequencies are already revealed vividly in sound, so the results of this small cross-talk can hardly be heard in real-life.

All changes a sound undergoes passing through Galaxy S can be summarized as follows:

Sound artifacts amplification applied, dB	Revealed artifacts
+7 ... +10	Usual broadband noise
+10 ... +12	Lack of high frequencies
+12 ... +15	Diluting of stereo image
+70 and more	Discrete switching peculiarity

Table 3: Various sound artifacts revealed at different levels of artifacts amplification

All these types of sound degradation in various combinations (except the last one which is reliably masked by the three first) are present in test files of Galaxy S uploaded to SE testing engine. Listening tests will show how far these sound drawbacks lay beyond threshold of human perception.

As usually we would like to thank some people. First of all Kirill who placed his smartphone to our disposal and patiently performed all reconfigurations and modifications necessary for more than dozen recordings we made using different settings. His Galaxy S was not hurt during those inhuman tests. When most of the work was finished we received suggestion from Alex to include Galaxy S and S II to SE testing. It was encouraging, thanks.

We in our turn encourage You to take part in SoundExpert listening tests. All SE ratings, including newly added Galaxy S, exist only thanks to visitors like you. Testing is short and easy, visit our listening room for simple 1-2-3 instruction. Thank you in advance!

Please, remember, downloading a test file from SoundExpert you will not necessarily get a test file of Galaxy S. SE testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

And some bad news at the end. Bad news for iPhone 4. No, it is not a looser in comparison with Galaxy S. Our extensive research testing of the latter revealed that test files of iPhone 4 that already in the system were prepared with not sufficient accuracy. For fair comparison of portable players they must be replaced. This will be done within a few days followed by corresponding notification in SE news. Sorry for that.

SE Team.

Sound quality of iPhone 4

2012-08-03T09:02:14Z

iPhone 4/4S
smartphone by Apple

It finally happened, new section "Portable Players" has its first contender – iPhone 4 (white GSM model with iOS 4.3.5 and 8Gb of storage). This “player” was chosen due to a few reasons:

it is popular
it has the same audio circuitry as iPod Touch 4 and therefore produces very similar level of audio quality
convergence of portable players and means of communication seems to be inevitable

This is a beginning, all other players will be chosen according to your suggestions as usual. Procedure of recording SE test signal for portable players is pretty solid now, so we are not excluding possibility to ask help from people outside SE to record SE test signal for some players which we don't have at our disposal, especially as we already had successful experience of this kind with MD-recorders (ATRAC) a year ago.

Here is some basic points of testing conditions for portable players at SE:

nine SE sound samples are uploaded to player as uncompressed WAV file
all audio enhancements are off
headphone's out of player is loaded with headphones Koss Porta Pro
playback level is adjusted to have some predefined and equal for all players sound pressure level
output signal (with headphones load) is recorded by hi-res recorder (MicroTrack II at the moment)
after precise time warping of recorded signal it is treated like any other output signal at SE – sound artifacts amplification, preparing of test files, usual SE listening tests …

(full specs of testing conditions will be available in “Portable players” section)

With these testing conditions we tried to achieve two goals:

to make sound quality testing as close as possible to real-life usage of portable players
to make resulting ratings directly comparable to existing SE ratings (of audio codecs)

Testing details specific to iPhone 4

Along with usual nine SE samples we also used sinusoidal signal (1kHz, -12dB RMS) and white noise (Gaussian, -12dB RMS) for some technical measurements. In particular the most interesting are the levels of degradation of initial shapes of those signals. Being transferred through player the signals are distorted to some level which can be conveniently measured with Difference level parameter. Lower Df values correspond to more accurate (less distorted) transfer of signals. The utmost limits of Df parameter are:

-∞ [db] – signal shapes are perfectly identical
0 [dB] – signals are completely different (nothing in common at all)

Table 1. Difference levels [db] of various signals transferred through iPhone 4.

Sine@1kHz 88dB SPL	White noise 88dB SPL	Bass 83dB SPL	Castanets 80dB SPL	French male speech 84dB SPL	Glockenspiel 77dB SPL	Harpsichord 79dB SPL	J.S.Bach 82dB SPL	Postscriptum 80dB SPL	Mike Oldfield 83dB SPL	Quartet 85dB SPL
-74.05 (L)	-4.58 (L)	-32.21	-19.59	-31.76	-27.94	-16.71	-32.60	-23.41	-17.69	-35.71
-73.94 (R)	-4.58 (R)

According to the measurements sinusoidal signal is transferred rather accurately (Df ~ -74 dB). For sinusoidal signals and small Df values Difference level can be easily converted to THD+Noise:

THD+Noise (dB) ≈ Df [dB] + 3 [dB].

Resulting value of THD+Noise -69 dB (0.04%) looks good for a portable player loaded with headphones. But if we replace sine with white noise Df drops to -5 dB. In this case we can't convert Df to THD as the latter is defined for pure tones only. Keeping that in mind we can still formally convert Df = -5.6 dB to THD+Noise = -0.9 dB (90%) and say that white noise being transferred through the player changes its initial shape dramatically. All other real-world sound signals are transferred with Df values between -74 dB and -5 dB (SE nine samples are in the table). Such accuracy spread of reproducing pure tones and more complicated signals is not surprising. This is more or less common for all audio equipment and iPhone 4 actually looks pretty good in comparison with other players that we tested.

Specific feature of iPhone 4 sound is less accurate reproduction of highs in comparison with other frequency regions. This is mostly caused by phase distortions. SE testing procedure doesn't allow to discern origin of the distortions - hardware circuitry or software processing but it allows to hear them (Audio 1).

Preview this Audio in lossy format
(vorbis@160 or mp3@165 depending on your browser)

Preview is not available in your browser (HTML5 support is required). Use the option below instead

Download this Audio in lossless format
(16sec, FLAC:1.09Mb)

Audio 1: Phase distortions of highs in iPhone 4 amplified. The first sample is reference piano excerpt, the second - output of iPhone 4 with artifacts amplified to +32dB. Outstanding highs reveal phase inaccuracy in that frequency region.

Such inaccuracy could lead to less transparent reproduction of highs, in particular - to blurring of their spatial positions. Resulting effect will depend on how far such inaccuracy is beyond threshold of human perception. No measurements (including artifacts gain) can reveal that, results of SE listening tests are more reliable in this sense as they take into account how any particular bunch of distortions is assessed by human hearing system.

UPDATE: iPhone 4S has the same audio quality as iPhone 4

Despite the fact that audio chip in iPhone 4S has different labeling:

iPhone 4: Apple 338S0589 / Cirrus Logic CLI1495B0 42L61
iPhone 4S: Apple 338S0987 / Cirrus Logic CLI1560B0 42L63

the overall audio performance is almost similar to iPhone 4. Results of Df measurements are in the Table 2

**Table 2**: Difference levels (dB) for signals and samples transferred through iPhone 4S. The values show how much output waveforms differ from input ones (-∞dB: perfectly similar, 0dB: totally different).
Sine@1kHz 86dB SPL	White noise 86dB SPL	Bass 81dB SPL	Castanets 78dB SPL	French male speech 82dB SPL	Glockenspiel 76dB SPL	Harpsichord 77dB SPL	J.S.Bach 80dB SPL	Postscriptum 78dB SPL	Mike Oldfield 81dB SPL	Quartet 83dB SPL
-70.08 (L)	-4.64 (L)	-31.22	-19.39	-31.27	-27.69	-16.48	-31.57	-22.72	-16.51	-34.93
-70.05 (R)	-4.64 (R)

The values are very close to the ones from Table 1. What is more important – audible distortions revealed by sound artifacts amplification are identical. It could be safely concluded that both devices sound exactly the same. The only difference is loudness limit in iPhone 4S, which prevented achieving usual SPL levels while measuring Df levels of SE samples.

For our testing we used white iPhone 4S 32Gb (MD245KH) with iOS 5.1 (9B179). SoundExpert thanks Ivan for the device.

We encourage You to take part in SoundExpert listening tests. All SE ratings, as well as iPhone 4 one, exist only thanks to visitors like you. Testing is short and easy, visit our listening room for simple 1-2-3 instruction. Thank you in advance!

Please, remember, downloading a test file from SoundExpert you will not necessarily get a test file of iPhone 4. SE testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Post-screening of grades implemented

2011-04-03T15:02:38Z

After a short period of testing the post-screening mechanism was successfully implemented at SE engine. As a result about 10% of all ratings will change their values slightly in the nearest future (at least one grade should be submitted for a device to recalculate its rating with post-screening). This adjustment will not be dramatic though to change relative positions of devices. Low's and high's of ratings will be affected to a grater extent. Screen-shorts of the ratings as of 2011.04.03 are saved in Appendix for future references and comparisons.

What will be changed substantially is reliability of ratings. Now it will take less grades to achieve 5% reliability and as a consequence - reliable ratings will be available sooner. This is the main purpose of SE post-screening mechanism.

Among other news - relocation to a faster server (probably you noticed some downtime during last week). This will also help to achieve better and faster user experience while interacting with SE service.

Best regards,
The SE Team.

Appendix

SE ratings before implementation of post-screening (as of 2011.04.03).

Coders 320+

Coders 256

Coders 224

Coders 192

Coders 160

Coders 128

Coders 105

Coders 96

Coders 80

Coders 64

Coders 48

Coders 32

DSP

Post-screening of grades in SE listening tests

2011-02-16T18:25:27Z

Simple and unrestricted participation in SE listening tests neccessitates some post-screening procedure for rejecting evidently deviant grades submitted either by chance or intentionally or due to whatever reasons.

Some caution should be applied here as there is a danger of inappropriate shaping of collected data which can lead to invalid inference of their means. So there is a trade off between contamination of data with outlying grades and violation of data integrity through inappropriate shaping. Normally this kind of data shaping is not recommended for analysis of results of traditional listening tests (ITU-R BS.1116-1), but in case of SE testing there is a serious uncertanty about listeners' level of expertise, therefore some rejection mechanism is desirable.

In case of SE listening tests all submitted grades are annonymous because they belong to different people, who can't be identified reliably by recorded IP addresses. So the procedure has to analyze only grades but not subjects as recommended for traditional listening tests.

Data set

Below are descriptive statistics of 1626 test files and 13738 grades submitted to SE as of the end of 2009. Pairs of test files that differ in position of reference sample have been merged together. This data set is moderately contaminated with outliers as only severe ones were removed manually from time to time during the period of collection.


Fig.1 Number of grades distribution over test files.	Fig.2 Mean score distribution over test files.

Fig.3 Std. deviation of grades distribution over test files.	Fig.4 Range of grades distribution over test files.

Fig.5 Skewness of grades distribution over test files.	Fig.6 Kurtosis of grades distribution over test files.

Note Grades “6” and “7” appear when listeners confuse reference and processed samples (“4” and “3” respectively); it was an idea to retain them for possible accounting in the future.

Rejection approaches

a) There are no methods to reliably separate submitted “6”s and “7”s into three groups:

true grades when listeners prefer processed variant over reference
inverted grades when listeners confused samples by mistake
random grades

Actually this can be done with some probability but the whole system of post-screening will be more complicated, less transparent and more dependent on separation method. So it was decided to reject "6" and "7" grades completely.

b) Taking into account the nature of listening test experiment we could assume that submitted grades for a test file can't differ from each other too greatly. It's highly unlikely that a test file that has slightly annoying artifacts could be graded as imperceptible (“5”) or very annoying (“1”) the same time. In other words we could suppose the following rough relationship between “true” scores and possible grades for test files:

"True" mean score	1.0	1.5	2.0	2.5	3.0	3.5	4.0	4.5	5.0
Possible grades	1, 2	1, 2, 3	1, 2, 3	1, 2, 3, 4	2, 3, 4	2, 3, 4, 5	3, 4, 5	3, 4, 5	4, 5

Table 1 Supposed relationship between "true" score of a test file and possible grades submitted for it.

In spite of speculative nature of that relationship we can safely assume that all grades are groupped around "true" mean within some radius. In the example above the radius is 1.5 and maximum range of grades is 3.0. Thus the second rejection rule could be formalized as:

M ± Rr, where

M - some parameter of central tendency of grades
Rr - radius of rejection which is supposed to be around 1.5 or higher; all grades beyond this radius are to be rejected.

c) All collected grades should be retained in DB and rejection should be applied on-the-fly each time new grade being received.

Testing rejection approaches on data set

While rules (a) and (c) are pretty obvious, rule (b) is questionable:

What parameter of central tendency should be used?
Is rejection radius 1.5 adequate for the case? How it will affect mean scores of test files?

Parameter of central tendency

First we need to remove all “6”s and “7”s from our data set. All further operations will be applied to remaining grades only. Let this reduced data set be a control data set.

Arithmetic mean, trimmed mean and median are usual choices for estimator of central tendency. More sophisticated iterative M-estimators could lead to more efficient inference in the presence of outliers but adjusting of different parameters that control their performance turned out to be not trivial and needs more research. So at the moment we decided to limit our choice with more predictable traditional ones:

Arithmetic mean
Trimmed mean 20%
Trimmed mean 30%
Trimmed mean 40%
Trimmed mean 50%
Median

Let's assign constant value 1.5 to the rejection radius Rr at this stage. Then different estimators of location will cause slightly different rejection of grades. As a result we have six alternative data sets which can be analyzed and compared. Below are distributions of grades – in-liers and outliers - before and after rejection with different estimators.

Fig.7 Distribution of inlying grades before and after applying of rejection rule with different estimators of central tendency (Rr = 1.5).
Fig.8 Distribution of outlying grades after applying of rejection rule with different estimators of central tendency (Rr = 1.5).

For sure there is some difference between estimators and now some test files in our six data sets have different mean scores comparing to the control data set. Let's estimate how many files were affected and how much they were affected in each of six cases. For the purpose we shall calculate three parameters for each data set:

number of affected test files
RMS and Max. deviations of mean scores of affected test files from corresponding mean scores in control set:

	Arithmetic mean	Trimmed mean 20%	Trimmed mean 30%	Trimmed mean 40%	Trimmed mean 50%	Median
Number of affected test files (files that changed their mean scores)	95	103	113	125	131	144
RMS deviation of the mean scores from control ones	0.3362	0.3285	0.3251	0.3297	0.3389	0.3861
Max deviation of the mean scores from control ones	1.0	1.0	1.0	1.0	1.0	1.0

Table 2 Number of test files affected by different rejection rules (M±1.5) as well as RMS/Max deviations of mean scores for these affected files.

Use of median leads to higher number of rejected grades and consequently higher number of affected test files. Unfortunately there is no way to find out is it good or bad in our case, - whether these higher values of rejection are caused by median's higher robustness to outliers or by its lower efficiency as an estimator of central tendency (it can possess discrete values only - 1.0, 1.5, 2.0, …, 5.0). But if we separate all test files into groups according to number of grades per test file and compute those three parameters for each group, then we'll see the following regularity:

Number of grades per file	Number of files	Arithmetic mean	Trimmed mean 20%	Trimmed mean 30%	Trimmed mean 40%	Trimmed mean 50%	Median
3	133	3 0.8924 1.0000	3 0.8924 1.0000	3 0.8924 1.0000	3 0.8924 1.0000	3 0.8924 1.0000	14 0.7210 1.0000
4	186	6 0.7312 0.8333	6 0.7312 0.8333	6 0.7312 0.8333	6 0.7312 0.8333	11 0.6366 0.8333	11 0.6366 0.8333
5	166	4 0.4131 0.4500	4 0.4131 0.4500	4 0.4131 0.4500	12 0.3722 0.4500	12 0.3722 0.4500	13 0.4209 0.8000
6	167	7 0.3842 0.4667	7 0.3842 0.4667	7 0.3842 0.4667	9 0.3672 0.4667	9 0.3672 0.4667	8 0.3712 0.4667
7	104	4 0.3179 0.3810	4 0.3179 0.3810	6 0.3623 0.5714	6 0.3623 0.5714	6 0.3623 0.5714	8 0.4726 0.8571
8	100	8 0.2892 0.3750	8 0.2892 0.3750	8 0.2892 0.3750	8 0.2892 0.3750	9 0.3125 0.4583	9 0.3125 0.4583
9	90	9 0.2153 0.2500	9 0.2153 0.2500	14 0.2036 0.2500	14 0.2036 0.2500	14 0.2036 0.2500	14 0.2036 0.2500
10	84	6 0.2102 0.2444	7 0.2045 0.2444	7 0.2045 0.2444	8 0.2279 0.3500	8 0.2279 0.3500	8 0.2279 0.3500
11	102	13 0.2374 0.5253	13 0.2374 0.5253	13 0.2374 0.5253	13 0.2782 0.5455	13 0.2782 0.5455	13 0.2782 0.5455
12	70	2 0.3883 0.5278	2 0.3883 0.5278	2 0.3883 0.5278	3 0.3613 0.5278	3 0.3613 0.5278	3 0.3613 0.5278
13	84	10 0.1708 0.2937	13 0.2093 0.4385	13 0.2093 0.4385	13 0.2093 0.4385	13 0.2093 0.4385	13 0.2093 0.4385
14	50	3 0.1563 0.1703	3 0.1563 0.1703	6 0.2014 0.2381	6 0.2014 0.2381	6 0.2014 0.2381	6 0.2014 0.2381
15	50	3 0.1327 0.1476	4 0.1263 0.1476	4 0.1263 0.1476	4 0.1263 0.1476	4 0.1263 0.1476	4 0.1263 0.1476
16	40	1 0.1042 0.1042	2 0.1021 0.1042	2 0.1021 0.1042	2 0.1021 0.1042	2 0.1021 0.1042	2 0.1021 0.1042
17	27	5 0.1177 0.1471	5 0.1177 0.1471	5 0.1888 0.3613	5 0.1888 0.3613	5 0.1888 0.3613	5 0.1888 0.3613
18	15	3 0.0804 0.1013	3 0.0804 0.1013	3 0.0804 0.1013	3 0.0804 0.1013	3 0.0804 0.1013	3 0.0804 0.1013
19	12	2 0.1611 0.1920	3 0.1653 0.1920	3 0.1653 0.1920	3 0.1653 0.1920	3 0.1653 0.1920	3 0.1653 0.1920
20	9	2 0.1443 0.1833	2 0.1443 0.1833	2 0.1443 0.1833	2 0.1443 0.1833	2 0.1443 0.1833	2 0.1443 0.1833

Table 3 Number of test files affected by different rejection rules (M±1.5) as well as RMS/Max deviations of mean scores for these affected files calculated for different groups of files; each group contains test files with equal number of collected grades.

For convenience numbers of affected test files from Table 3 are also presented on Fig.9.

Fig.9 Number of test files affected by different rejection rules (M±1.5) and calculated for different groups of files; each group contains test files with equal number of collected grades.

With increase of number of grades per test file the difference between the estimators decreases. For small number of grades median results in higher rates of rejection. Starting from 13-14 grades all trimmed means and median result in rejection of the same grades (number of grades and RMS/Max deviations are the same). Starting from 18-19 grades all estimators operate equally.

Considering that:

median is the most robust to outliers among all other estimators of central tendency
median is better estimator of central tendency in case of deviations from normality of sample data and in case of small size of sample data
gathering of 13-15 grades per test file is a common practice for listening tests and is not a problem for SE ones

it is preferable to use median as an estimator of central tendency for rejection of outlying grades in SE listening tests. For eliminating potential problems with low efficiency of median as an estimator of location the number of grades per test file should be around 15 or more.

Radius of rejection

First let's look at grade's standard deviations and ranges for test files from our control set (without “6”s and “7”s).

Fig.10 Distribution of std. deviation of grades over files
in control data set.

Fig.11 Distribution of range of grades over files
in control data set.

And here is representation of these distributions by percentiles:

Fig.12 Std.deviation of grades distribution by percentiles.

Fig.13 Range of grades distribution by percentiles.

Almost all (99.7%) test files have grade's ranges below 3.0 and standard deviations below 1.5. Now supposed radius of rejection Rr=1.5 looks quite reasonable. In order to understand whether this rejection radius is acceptable or not, it's worth to research surrounding values as well. For larger perspective let's choose relatively wide interval [0.5, 2.5]. As median can possess discrete values only - 1.0, 1.5, 2.0, … , 5.0, we'll use in our research special case of Mean, which operates almost like Median but can possess arbitrary values. For simplisity let us intriduce the new estimator of location by example. For order statistic {X1, X2, X3, X4, X5} this special Mean – pfMean - is calculated by averaging auxiliary means M1...M7:

The parameter could be considered as mean, weighted towards median or average of trimmed means. It operates very similar to bootstrapped median but has less computational complexity and provides repetitive results for a given sample statistic. Also one can see other statistical hooks in calculating this parameter like Jackknifing, aggregating, sampling with/without replacement; the closest known statistical parameter with similar meaning is Trimean, but it has some restrictions on sample size. Considering all these as well as the method of the new parameter calculation a funny nickname “Pilfering Mean” well stuck to this estimator. We'll use it as a substitute to Median to test arbitrary values of rejection radius Rr.

Fig.14 Number of affected test files in control data set depending on radius of rejection.

For example relationship between rejection radius and number of affected test files from control set is on Fig.14.

Let's have a closer look at how characteristics of those affected files change as rejection radius decreases from 2.5 to 0.5. Fig.15 and Fig.16 show averages of std. deviations and ranges of grades in affected files depending on rejection radius.

Fig.15 Average of std.deviations of affected test files
in control data set depending on radius of rejection.

Fig.16 Average of ranges of affected test files
in control data set depending on radius of rejection.

Steeper section above Rr~1.8 means that first cluster of affected files (~90 out of 1626) has substantially higher values of std. deviation and range. Further shortening of Rr below 1.8 affects another and bigger cluster of test files.

Skewness and kurtosis of affected files relate to rejection radius as follows:

Fig.17 Average of skewnesses of affected test files
in control data set depending on radius of rejection.

Fig.18 Average of kurtosises of affected test files
in control data set depending on radius of rejection.

Border between clusters is less obvious here, may be Rr~1.7 on skewness plot has some sense.

Fig.19 RMS deviation of mean scores of affected files from the ones in control data set depending on radius of rejection.

RMS deviation of mean scores of affected files from corresponding ones in control data set is more revealing (see Fig.19). Distinctive point Rr~1.85 clearly delimits first cluster of affected test files. These files dramatically change their mean scores after rejection of some grades.

Considering all these plots another supposed value for rejection radius is 1.8.

Now we have two estimators of central tendency and two values for rejection radius. Total number of resulting rules of rejection is three:

pfMean ± 1.8
pfMean ± 1.5
Median ± 1.5

Results of their application to our control data set are summerized in Table 4.

	pfMean ± 1.8	pfMean ± 1.5	Median ± 1.5
Number of affected test files (files that changed their mean scores)	83	143	144
RMS deviation of the mean scores from the ones in control set	0.8807	0.8998	0.9002
Max deviation of the mean scores from the ones in control set	1.0	1.0	1.0

Table 4 Number of test files affected by different rejection rules as well as RMS/Max deviations of mean scores for these affected files.

Total number of cases where at least one of the rejection rules triggers removal of grades is 145. It is not hard to compare them all "by hand". The complete list with comments is in Appendix 1. Manual peer reviewing of the cases reveals:

1) that performance of the rules “pfMean±1.5” and “Median±1.5” is almost identical, - only three cases out of 145 are different, in all of them “Median±1.5” performes better (see Appendix 2).

2) scores of appropriateness for each rule are in Table 5

	pfMean±1.8	pfMean±1.5	Median±1.5
Score of appropriateness relating to number of files affected by particular rule	61/83 (73.5%)	99/143 (69.2%)	101/144 (70.1%)
Score of appropriateness relating to total number of affected files (145)	82/145 (56.6%)	99/145 (68.3%)	102/145 (70.3%)

Table 5 Scores of appropriateness for each rule.

Shortening the radius from 1.8 to 1.5 leads to higher number of affected files - 143 against 83, while percent of "better" cases stays the same - around 70%. Being calculated for all 145 cases scores of "pfMean±1.5" and "Median±1.5" are noticeably higher. It means that the value 1.5 could be safely used for rejection radius, it performs at least non the worse than 1.8 value.

Fig.20 shows results of contamination of reference data set with random grades (reference data set is the control data set cleaned with “Median ± 1.5” rule). Grades are added to random test files; number of added grades is proportional to number of existing grades in each test file; 50% contamination means that number of added and existing grades are equal. After each step of gradual contamination eight different rules of rejection are applied to contaminated data and RMS deviation of mean scores is calculated for each resulting (“cleaned”) data set.

Fig.20 Results of contaminating reference data set with random grades and subsequent application of different rejection rules to contaminated data; for each case of "cleaning" the plot shows RMS deviation of mean scores of all (not only affected) test files.

Difference between contaminated and cleaned data is obvious. It is not surprising that median-based rule of rejection prevents changing of mean scores better than others, pfMean-based is the second. Increasing radius of rejection from 1.5 to 1.8 (dashed line) makes rejection mechanism slightly less efficient.

Thus, the recommended rejection rule for SE listening tests is Median ± 1.5 with min. 15 grades per test file (Median1.5/15).

As all collected grades are retained in SE data base it is recommended to repeat such research from time to time to reveal potential problems with rejection of grades. Being discovered they could be fixed increasing radius of rejection and using pfMean instead of median.

Appendix 1

List of all cases (145 out of 1626) where applying of at least one of rejection rules (pfMean ± 1.8, pfMean ± 1.5, Median ± 1.5) removes grades. In other words each test file in the list is affected by at least one of the rules.

Each case consists of four rows:

original grades from control set
grades after rejection with “pfMean ± 1.8” rule
grades after rejection with “pfMean ± 1.5” rule
grades after rejection with “Median ± 1.5” rule

For convenience the list is sorted by number of grades and has color marking:

first row - original grades - are always black
second row (pfMean ± 1.8) is black if unchanged (equal to first row) and magenta if some grades were rejected
third row (pfMean ± 1.5) is magenta if unchanged (equal to second row) and orange if some grades were rejected
fourth row (Median ± 1.5) is orange if unchanged (equal to third row) and red if some grades were rejected

Each case was peer reviewed for the appropriatness (A) of the rules application. If there is any kind of uncertanty about nesessity of particular rule application, such rule is marked with "0". Rule is marked with "1" only if there is high degry of confidence that application of particular rule is nesessary.

1 2 3 4 5 A
--------------
0 0 2 0 1
0 0 2 0 1 0
0 0 2 0 0 1
0 0 2 0 0 1

0 2 0 0 1
0 2 0 0 0 1
0 2 0 0 0 1
0 2 0 0 0 1

0 2 0 1 0
0 2 0 1 0 0
0 2 0 0 0 1
0 2 0 0 0 1

0 1 0 2 0
0 1 0 2 0 0
0 0 0 2 0 1
0 0 0 2 0 1

2 0 1 0 0
2 0 1 0 0 0
2 0 0 0 0 1
2 0 0 0 0 1

0 1 0 2 0
0 1 0 2 0 0
0 0 0 2 0 1
0 0 0 2 0 1

0 0 2 0 1
0 0 2 0 1 0
0 0 2 0 0 1
0 0 2 0 0 1

0 2 0 1 0
0 2 0 1 0 0
0 2 0 0 0 1
0 2 0 0 0 1

1 1 0 1 0
1 1 0 1 0 0
1 1 0 0 0 1
1 1 0 0 0 1

1 0 2 0 0
1 0 2 0 0 0
0 0 2 0 0 1
0 0 2 0 0 1

0 0 2 0 1
0 0 2 0 1 0
0 0 2 0 0 1
0 0 2 0 0 1

0 1 2 0 1
0 1 2 0 0 1
0 1 2 0 0 1
0 1 2 0 0 1

0 1 1 0 1
0 1 1 0 1 0
0 1 1 0 0 1
0 1 1 0 0 1

0 0 3 0 1
0 0 3 0 0 1
0 0 3 0 0 1
0 0 3 0 0 1

0 2 1 0 1
0 2 1 0 0 1
0 2 1 0 0 1
0 2 1 0 0 1

1 2 0 0 1
1 2 0 0 0 1
1 2 0 0 0 1
1 2 0 0 0 1

2 0 1 0 1
2 0 1 0 0 1
2 0 1 0 0 1
2 0 1 0 0 1

1 2 0 0 1
1 2 0 0 0 1
1 2 0 0 0 1
1 2 0 0 0 1

3 0 1 0 0
3 0 0 0 0 1
3 0 0 0 0 1
3 0 0 0 0 1

1 0 3 0 0
0 0 3 0 0 1
0 0 3 0 0 1
0 0 3 0 0 1

0 3 0 1 0
0 3 0 0 0 1
0 3 0 0 0 1
0 3 0 0 0 1

0 0 2 0 1
0 0 2 0 1 0
0 0 2 0 0 1
0 0 2 0 0 1

0 0 3 1 1
0 0 3 1 1 0
0 0 3 1 0 1
0 0 3 1 0 1

0 0 3 0 2
0 0 3 0 2 0
0 0 3 0 2 0
0 0 3 0 0 1

0 0 1 0 3
0 0 0 0 3 1
0 0 0 0 3 1
0 0 0 0 3 1

1 2 1 1 0
1 2 1 1 0 0
1 2 1 0 0 1
1 2 1 0 0 1

0 1 2 0 1
0 1 2 0 0 1
0 1 2 0 0 1
0 1 2 0 0 1

0 1 2 1 1
0 1 2 1 1 0
0 1 2 1 0 1
0 1 2 1 0 1

0 3 1 1 0
0 3 1 1 0 0
0 3 1 0 0 1
0 3 1 0 0 1

0 0 3 1 1
0 0 3 1 1 0
0 0 3 1 0 1
0 0 3 1 0 1

0 3 1 1 0
0 3 1 1 0 0
0 3 1 0 0 1
0 3 1 0 0 1

0 1 3 0 1
0 1 3 0 0 1
0 1 3 0 0 1
0 1 3 0 0 1

0 1 1 3 1
0 0 1 3 1 1
0 0 1 3 1 1
0 0 1 3 1 1

0 0 1 1 3
0 0 1 1 3 0
0 0 0 1 3 1
0 0 0 1 3 1

0 0 4 1 11
0 0 4 1 1 0
0 0 4 1 0 1
0 0 4 1 0 1

0 0 5 0 1
0 0 5 0 0 1
0 0 5 0 0 1
0 0 5 0 0 1

0 1 1 4 0
0 1 1 4 0 0
0 0 1 4 0 1
0 0 1 4 0 1

1 4 0 1 0
1 4 0 0 0 1
1 4 0 0 0 1
1 4 0 0 0 1

3 2 0 1 0
3 2 0 0 0 1
3 2 0 0 0 1
3 2 0 0 0 1

2 3 0 1 0
2 3 0 0 0 1
2 3 0 0 0 1
2 3 0 0 0 1

0 1 5 0 1
0 1 5 0 0 1
0 1 5 0 0 1
0 1 5 0 0 1

0 2 2 2 1
0 2 2 2 1 0
0 2 2 2 0 1
0 2 2 2 0 1

0 0 3 1 1
0 0 3 1 1 0
0 0 3 1 0 1
0 0 3 1 0 1

0 0 4 0 1
0 0 4 0 0 1
0 0 4 0 0 1
0 0 4 0 0 1

0 2 1 2 1
0 2 1 2 1 1
0 2 1 2 0 0
0 2 1 2 1 1

0 0 4 1 2
0 0 4 1 2 0
0 0 4 1 0 1
0 0 4 1 0 1

0 4 2 1 0
0 4 2 1 0 1
0 4 2 0 0 0
0 4 2 0 0 0

0 4 2 0 1
0 4 2 0 0 1
0 4 2 0 0 1
0 4 2 0 0 1

0 1 5 1 1
0 1 5 1 0 1
0 1 5 1 0 1
0 1 5 1 0 1

0 2 5 0 1
0 2 5 0 0 1
0 2 5 0 0 1
0 2 5 0 0 1

2 1 5 0 0
2 1 5 0 0 0
0 1 5 0 0 1
0 1 5 0 0 1

0 3 4 0 1
0 3 4 0 0 1
0 3 4 0 0 1
0 3 4 0 0 1

1 5 1 0 1
1 5 1 0 0 1
1 5 1 0 0 1
1 5 1 0 0 1

1 4 1 1 0
1 4 1 0 0 1
1 4 1 0 0 1
1 4 1 0 0 1

0 0 6 1 1
0 0 6 1 0 1
0 0 6 1 0 1
0 0 6 1 0 1

0 0 5 0 2
0 0 5 0 2 0
0 0 5 0 0 1
0 0 5 0 0 1

0 0 4 0 3
0 0 4 0 3 0
0 0 4 0 3 0
0 0 4 0 0 1

1 0 4 3 0
0 0 4 3 0 1
0 0 4 3 0 1
0 0 4 3 0 1

0 1 0 1 4
0 0 0 1 4 1
0 0 0 1 4 1
0 0 0 1 4 1

0 5 3 1 0
0 5 3 1 0 1
0 5 3 0 0 0
0 5 3 0 0 0

0 6 2 1 0
0 6 2 0 0 0
0 6 2 0 0 0
0 6 2 0 0 0

0 0 5 3 1
0 0 5 3 1 1
0 0 5 3 0 0
0 0 5 3 0 0

0 5 3 1 0
0 5 3 1 0 1
0 5 3 0 0 0
0 5 3 0 0 0

0 7 1 1 0
0 7 1 0 0 1
0 7 1 0 0 1
0 7 1 0 0 1

0 6 2 1 0
0 6 2 0 0 0
0 6 2 0 0 0
0 6 2 0 0 0

0 1 2 6 0
0 0 2 6 0 0
0 0 2 6 0 0
0 0 2 6 0 0

0 2 3 3 1
0 2 3 3 1 0
0 2 3 3 0 1
0 2 3 3 0 1

0 3 2 3 1
0 3 2 3 1 0
0 3 2 3 0 1
0 3 2 3 0 1

0 0 5 3 1
0 0 5 3 1 1
0 0 5 3 0 0
0 0 5 3 0 0

0 1 0 4 3
0 0 0 4 3 1
0 0 0 4 3 1
0 0 0 4 3 1

0 1 3 6 0
0 1 3 6 0 1
0 0 3 6 0 0
0 0 3 6 0 0

0 6 2 2 0
0 6 2 2 0 0
0 6 2 0 0 1
0 6 2 0 0 1

0 4 5 0 1
0 4 5 0 0 1
0 4 5 0 0 1
0 4 5 0 0 1

0 1 1 5 2
0 0 1 5 2 1
0 0 1 5 2 1
0 0 1 5 2 1

0 2 6 1 1
0 2 6 1 0 1
0 2 6 1 0 1
0 2 6 1 0 1

3 4 1 1 0
3 4 1 0 0 1
3 4 1 0 0 1
3 4 1 0 0 1

2 6 1 1 0
2 6 1 0 0 1
2 6 1 0 0 1
2 6 1 0 0 1

0 0 8 0 1
0 0 8 0 0 1
0 0 8 0 0 1
0 0 8 0 0 1

0 0 6 1 1
0 0 6 1 0 1
0 0 6 1 0 1
0 0 6 1 0 1

0 0 1 3 5
0 0 1 3 5 1
0 0 0 3 5 0
0 0 0 3 5 0

1 3 5 2 0
0 3 5 2 0 0
0 3 5 2 0 0
0 3 5 2 0 0

0 9 1 1 0
0 9 1 0 0 1
0 9 1 0 0 1
0 9 1 0 0 1

0 0 7 3 1
0 0 7 3 1 1
0 0 7 3 0 0
0 0 7 3 0 0

0 8 2 1 0
0 8 2 0 0 0
0 8 2 0 0 0
0 8 2 0 0 0

0 0 9 0 1
0 0 9 0 0 1
0 0 9 0 0 1
0 0 9 0 0 1

0 1 0 7 3
0 0 0 7 3 1
0 0 0 7 3 1
0 0 0 7 3 1

0 0 8 2 1
0 0 8 2 0 0
0 0 8 2 0 0
0 0 8 2 0 0

0 0 9 1 1
0 0 9 1 0 1
0 0 9 1 0 1
0 0 9 1 0 1

0 0 7 3 1
0 0 7 3 1 1
0 0 7 3 0 0
0 0 7 3 0 0

1 8 0 1 0
1 8 0 0 0 1
1 8 0 0 0 1
1 8 0 0 0 1

2 2 5 2 0
2 2 5 2 0 0
0 2 5 2 0 1
0 2 5 2 0 1

0 1 6 1 3
0 1 6 1 3 0
0 1 6 1 0 1
0 1 6 1 0 1

2 6 1 3 0
2 6 1 3 0 0
2 6 1 0 0 1
2 6 1 0 0 1

0 0 7 2 1
0 0 7 2 0 0
0 0 7 2 0 0
0 0 7 2 0 0

0 1 2 5 3
0 0 2 5 3 1
0 0 2 5 3 1
0 0 2 5 3 1

0 0 9 1 1
0 0 9 1 0 1
0 0 9 1 0 1
0 0 9 1 0 1

0 2 3 6 1
0 2 3 6 1 1
0 0 3 6 1 0
0 0 3 6 1 0

0 8 1 0 2
0 8 1 0 0 1
0 8 1 0 0 1
0 8 1 0 0 1

0 10 2 1 0
0 10 2 0 0 0
0 10 2 0 0 0
0 10 2 0 0 0

0 1 2 9 1
0 0 2 9 1 1
0 0 2 9 1 1
0 0 2 9 1 1

0 0 8 3 2
0 0 8 3 2 0
0 0 8 3 0 1
0 0 8 3 0 1

0 1 3 8 1
0 0 3 8 1 1
0 0 3 8 1 1
0 0 3 8 1 1

1 2 10 0 0
0 2 10 0 0 0
0 2 10 0 0 0
0 2 10 0 0 0

7 5 1 0 0
7 5 1 0 0 1
7 5 0 0 0 0
7 5 0 0 0 0

1 0 11 1 0
0 0 11 1 0 1
0 0 11 1 0 1
0 0 11 1 0 1

0 0 9 3 1
0 0 9 3 0 0
0 0 9 3 0 0
0 0 9 3 0 0

0 9 2 1 0
0 9 2 0 0 0
0 9 2 0 0 0
0 9 2 0 0 0

0 2 6 3 2
0 2 6 3 2 0
0 2 6 3 0 1
0 2 6 3 0 1

0 6 4 2 1
0 6 4 2 0 1
0 6 4 2 0 1
0 6 4 2 0 1

0 3 9 0 1
0 3 9 0 0 1
0 3 9 0 0 1
0 3 9 0 0 1

0 8 4 2 0
0 8 4 2 0 1
0 8 4 0 0 0
0 8 4 0 0 0

0 0 8 4 2
0 0 8 4 2 1
0 0 8 4 0 0
0 0 8 4 0 0

0 0 9 1 3
0 0 9 1 3 0
0 0 9 1 0 1
0 0 9 1 0 1

0 0 8 4 2
0 0 8 4 2 1
0 0 8 4 0 0
0 0 8 4 0 0

0 2 8 3 1
0 2 8 3 0 0
0 2 8 3 0 0
0 2 8 3 0 0

1 0 8 5 0
0 0 8 5 0 1
0 0 8 5 0 1
0 0 8 5 0 1

0 1 1 8 4
0 0 1 8 4 1
0 0 1 8 4 1
0 0 1 8 4 1

1 5 9 0 0
1 5 9 0 0 1
0 5 9 0 0 0
0 5 9 0 0 0

0 0 8 6 1
0 0 8 6 1 1
0 0 8 6 0 0
0 0 8 6 0 0

0 1 13 0 1
0 1 13 0 0 1
0 1 13 0 0 1
0 1 13 0 0 1

5 7 2 1 0
5 7 2 0 0 1
5 7 2 0 0 1
5 7 2 0 0 1

0 0 1 3 9
0 0 0 3 9 0
0 0 0 3 9 0
0 0 0 3 9 0

0 0 9 6 1
0 0 9 6 1 1
0 0 9 6 0 0
0 0 9 6 0 0

0 0 10 5 1
0 0 10 5 1 1
0 0 10 5 0 0
0 0 10 5 0 0

0 10 4 2 1
0 10 4 2 0 1
0 10 4 0 0 1
0 10 4 0 0 1

0 1 5 11 0
0 0 5 11 0 0
0 0 5 11 0 0
0 0 5 11 0 0

1 5 11 0 0
0 5 11 0 0 0
0 5 11 0 0 0
0 5 11 0 0 0

0 3 13 0 1
0 3 13 0 0 1
0 3 13 0 0 1
0 3 13 0 0 1

0 0 13 3 1
0 0 13 3 0 0
0 0 13 3 0 0
0 0 13 3 0 0

1 0 12 4 1
0 0 12 4 0 1
0 0 12 4 0 1
0 0 12 4 0 1

0 0 14 3 1
0 0 14 3 0 0
0 0 14 3 0 0
0 0 14 3 0 0

0 0 1 6 11
0 0 1 6 11 1
0 0 0 6 11 0
0 0 0 6 11 0

2 6 11 0 0
2 6 11 0 0 1
0 6 11 0 0 0
0 6 11 0 0 0

0 0 14 3 2
0 0 14 3 0 1
0 0 14 3 0 1
0 0 14 3 0 1

0 0 15 3 2
0 0 15 3 0 1
0 0 15 3 0 1
0 0 15 3 0 1

0 8 8 2 1
0 8 8 2 0 1
0 8 8 2 0 1
0 8 8 2 0 1

0 0 16 4 1
0 0 16 4 0 0
0 0 16 4 0 0
0 0 16 4 0 0

1 5 14 1 0
0 5 14 1 0 0
0 5 14 1 0 0
0 5 14 1 0 0

0 0 15 4 1
0 0 15 4 0 0
0 0 15 4 0 0
0 0 15 4 0 0

1 6 14 0 0
0 6 14 0 0 0
0 6 14 0 0 0
0 6 14 0 0 0

0 0 12 9 2
0 0 12 9 2 1
0 0 12 9 0 0
0 0 12 9 0 0

1 8 16 0 0
0 8 16 0 0 0
0 8 16 0 0 0
0 8 16 0 0 0

Appendix 2

Three cases from Appendix 1 where two rules - "pfMean±1.5" and "Median±1.5" - produce different output:

0 0 3 0 2
0 0 3 0 2 0
0 0 3 0 2 0
0 0 3 0 0 1

0 2 1 2 1
0 2 1 2 1 1
0 2 1 2 0 0
0 2 1 2 1 1

0 0 4 0 3
0 0 4 0 3 0
0 0 4 0 3 0
0 0 4 0 0 1

Spotify uses Vorbis -q5 and -q9 for audio streaming

2010-10-13T23:06:35Z

One more web project from Sweden becomes popular – music service Spotify. Founded at the end of 2008 it offers almost the same functionality as iTunes but with one and very important difference – all music is available for “free” - adverts in Spotify player cover all expenses. Payed ad-free subscription is also available. More than ten million tracks and contracts with all major music labels make it a serious competitor on the market. A very reasonable description of the service is on Wikipedia – Spotify.

Spotify uses Vorbis encoder for streaming its content. Free users get ~160 kbit/s audio stream (Vorbis, -q5) and premium users - ~320 kbit/s (Vorbis, -q9). Exactly these two presets were added to SoundExpert rating system in 160 kbit/s and 320+ kbit/s groups respectively. Now their test files are ready for downloading and testing along with other encoders.

We thank Øyvin Eikeland for this suggestion.

lossyWAV in new DSP section

2010-09-06T22:39:16Z

lossyWAV is a bit reduction processor which carefully removes some lower significant bits from digital audio signal by setting them to zero. Resulting audio file is compressed more efficiently with any lossless encoder, FLAC for example. The algorithm was designed to be audibly transparent with high degree of confidence.

More about this bit reduction scheme is on lossyWAV – Hydrogenaudio Wiki page.

As the process is not mathematically transparent (lossy), some small degradation of signal is introduced inevitably. How small is it? In order to answer the question lossyWAV was added to SE. New DSP section was created for the purpose as lossyWAV is not encoder.

Also it is a good contender to test (and tune if necessary) SE ability to rate devices/technologies with extremely small degradation of signal, inaudible in most cases.

The testing is open now – lossyWAV test files have been added to rotation. Volunteers are welcome!

Rating bar graphs improved

2010-07-25T07:34:57Z

Some time ago a visitor of SE site by the name of Uhl, who accidentally turned out a human factor engineer, pointed out some shortcomings of our rating bar graphs – extending left beyond the first score and disproportional lengths of bars relative to their values. He suggested a few variants which eliminate those drawbacks. We found the arguments are reasonable and corrected our bar graphs according to the recommendations.

Old rating bar design

New rating bar design

Thank you, Uhl, for your efforts; new bars are visually more correct and understandable indeed.

Micropayments by Flattr

2010-07-22T19:48:50Z

Another micropayment system - Flattr appeared on February 2010. It was founded by ex-PirateBay Peter Sunde and entrepreneur Linus Olson. The main idea is to give people the possibility to say “Thank You!” for any thing they like on the internet with 1-2 or even a fraction of a cent by simply pressing a button. The “thing” can be a single blog post, a song, a photo, a whole web site – any piece of creative work. Much like the “Like” button on Facebook but supported with money.

In order to thank people this way you have to register with Flattr and upload minimum 2EUR to your account. Each time you press the Flattr button on a web page a fraction of this amount will be transferred to the author of the thing you like. In order to get your own Flattr button you have to register with flattr and upload … - exactly the same way because Flattr doesn't discriminate between creators and consumers of content. In modern life we are both the same time. Then you register your content in your account and get the button code. Details are explained on Flattr web site.

All things already registered on Flattr are available for discovering in the catalog. On the 15-th of July it contains 17907 things. They are divided into 6 sections:

Written text 81.3% (14469)
Images 5.8% (1031)
Audio 4.9% (866)
Video 2.7% (487)
Software 2.1% (370)
The rest 3.3% (584)

Let's have a closer look at Audio section. It numbers 866 things. Each day 10 more audio things are registered on the average. As told before, a thing can be either a single creative work or a bunch of them. Both tactics are used now by Flattr participants (we at SE decided to register the whole site). Please, bear this in mind looking at how things fall apart in Audio section:

Podcasts 65.1% (564)
Indie/Recognized musicians 21.8/0.5% (189/4)
Web radio 5.3% (46)
Music blogs 2.1% (18)
Net labels 1.6% (14)
Videocasts <1% (9)
Non-audio <1% (7)
Music pieces with unclear authoring <1% (6)
Artist aggregators <1% (5)
About Audio <1% (3)
Audio books <1% (1)

Predominance of podcasts, covering various topics from love stories to software engineering, seems a bit surprising. Top 10 most flattered things are also podcasts. But if we think of podcasting as a next step in personal blogging and a very convenient form of “broadcasting” both for program authors and listeners, those figures become more understandable. Conventional radio is trying to be more convenient for listeners and listeners convert their blog posts into podcasts.

Indie musicians, Net labels and Aggregators (offering web space and services to artists) is another big group which benefit from Flattr micropayments. Almost all musicians published their works under Creative Commons licences. Among recognized musicians are:

Carbon Based Lifeforms
Sian Chandler and Ray Hughes
Pandemonium
Ian Simmonds

Only first two of them have Flattr buttons on their official web sites. Carbon Based Lifeforms accompanied donation buttons with the following: “If you downloaded our music on p2p or otherwise and still want to pay us, feel free to donate”. Now with Flattr button it is extremely easy to do this. Ian Simmonds and Pandemonium don't mention Flattr on their official web sites, so the question - who will receive microdonations for them – is open. Seems like the only solution for such uncertain situations is using Flattr buttons on official web sites or other places reliably controlled by authors.

Web radios presented in Flattr catalog mostly rotate music under CC licenses by indie-artists. One discovery is worth to mention – SceneSat Radio playing “the best music from every corner of the scene and related stuff” as they say about themselves. We could add that streaming content has surprisingly good artistic and technical quality (professional pre-processing, four streams: aac+ vbr@48, mp3 cbr@128, aac+ vbr@128, mp3 cbr@320), not to say that the station has very convenient way to say “Thank You!” with Flattr button.

Six things point to music pieces which are difficult to understand whom they belong to and who will receive the revenue. Eighteen things are from music blogs. They raise the question - is it ethical to ask for donations for discovering new music and promoting artists? Well … why not?! Presence of Videocasts and ordinary blog posts (16 in total) in Audio section looks like mistake. Three things belong to web resources devoted to Audio itself as a technology (SE – one of them). While numbers of things submitted to Audio section changes rapidly and when you read this article actual figures will differ from those above, the structure of submissions will stay actual for longer time as it changes more slowly if at all.

At the moment Flattr is still closed beta – you need invitation to join. Hope when it goes public there will be substantial increase in content submissions which will help Flattr to get critical mass – a key factor of any micropayment service success. By the way, we have three more invitations; drop us a line if you need one.

ATRAC SP (Type-R) in 320 kbit/s group

2010-07-17T13:02:49Z

First hardware audio codec - ATRAC1 SP (Type-R) has been just added to SE. Some of the reasons why:

For relatively long period of time MiniDisc and ATRAC SP were a standard of hi-fi digital recordings at home and in semi-professional area.
The format still has substantial user base consisting mostly of demanding listeners.
Its perceived audio quality is well known, discussed and compared a lot with other formats.
Development of ATRAC stopped and now it could be considered as Latin among lossy audio compression schemes.

Taking all these into account ATRAC SP will be a perfect anchor point on SE quality scale, helping other ratings of upper end to be more meaningful.

ATRAC SP is also a landmark in the history of SoundExpert. For the first time output signal for a codec was obtained during real-time recording-playback on hardware device - MD recorder. Usually real-time recordings have small time inconsistency, which is insignificant for most tasks of processing/listening but easily destroying difference signal - the core of SE artifacts amplification technology. Thanks to precise time warping procedure developed inside SE those time inconsistencies can be effectively corrected now with predefined accuracy. This special time-treatment opens SE for wider class of devices/technologies which now can be rated. Also for the first time test recordings for a codec were made outside SE by volunteers. The whole story is in two news articles:

As usually SE thanks several people who contributed their time and efforts to make all this possible. Lee Keywood for this codec proposal and invaluable help during development of time warping procedure. Numinos and Mike Rofone for making test recordings on their MD equipment.

Now ATRAC SP is in the system along with a hundred other codecs. First reliable results are in two-three weeks depending on testers' activity. Participation is simple and easy as usually.

Cheers,
SE Team.

Analysis of submitted ATRAC SP (Type-R) recordings

2010-06-23T22:19:14Z

1. Collection of submitted recordings

All submitted recordings were processed with time warping algorithm and listed in the table below.

Author of submitted recording	Difference levels (dB) of time warped SE samples (Left and Right channels)																		Download
	BAH		BAS		CST		FMS		GLK		HRP		LOB		MOF		QRT
	L	R	L	R	L	R	L	R	L	R	L	R	L	R	L	R	L	R
Lee1	-38.1	-38.7	-41.1	-40.0	-19.4	-19.7	-35.0	-34.9	-30.3	-30.7	-19.9	-19.9	-26.2	-22.6	-27.3	-27.0	-37.1	-36.2
Lee2	-38.5	-39.2	-40.9	-39.4	-19.3	-19.8	-34.7	-34.9	-31.6	-31.2	-19.8	-20.1	-25.7	-22.0	-26.6	-26.7	-36.8	-36.3	RAR 65.5Mb
Lee3	-38.5	-39.0	-41.3	-40.5	-19.5	-19.5	-34.8	-35.2	-31.1	-31.2	-19.7	-19.8	-26.0	-22.3	-26.7	-26.6	-36.6	-36.2

Numinos1	-37.3	-37.7	-39.2	-38.8	-18.9	-19.2	-34.2	-34.3	-29.5	-29.3	-19.3	-19.5	-25.1	-21.7	-26.0	-26.0	-36.3	-35.5
Numinos2	-37.8	-38.0	-39.8	-38.9	-19.2	-19.1	-34.6	-34.4	-30.1	-29.9	-18.6	-19.7	-25.0	-21.8	-27.0	-26.6	-36.3	-35.5
Numinos3	-37.6	-38.3	-39.4	-38.7	-18.6	-19.3	-34.5	-34.6	-29.2	-28.8	-19.7	-19.3	-25.9	-22.5	-27.0	-26.7	-36.4	-35.6	RAR 65.6Mb

MikeRofone1	-31.6	-31.7	-32.8	-32.7	-18.1	-18.4	-30.9	-30.9	-30.0	-28.7	-19.3	-19.4	-24.1	-21.2	-25.1	-24.9	-31.7	-31.5	RAR 65.0Mb
MikeRofone2	-31.6	-31.7	-32.7	-32.6	-18.3	-18.6	-30.9	-30.8	-26.5	-28.1	-19.1	-19.5	-24.5	-21.3	-25.1	-24.9	-31.6	-31.5
MikeRofone3	-31.6	-31.7	-32.9	-32.7	-18.2	-18.5	-30.9	-30.7	-25.8	-24.9	-19.2	-19.3	-23.8	-20.5	-25.1	-24.4	-31.7	-31.4

MikeRofone21	-38.6	-38.9	-41.1	-39.9	-19.2	-19.4	-34.6	-34.8	-33.7	-31.9	-20.1	-20.5	-25.5	-22.2	-27.2	-26.7	-36.9	-36.4
MikeRofone22	-38.5	-38.8	-40.6	-39.4	-19.4	-19.6	-34.5	-34.7	-30.0	-30.4	-19.8	-19.9	-25.8	-22.2	-26.7	-26.5	-36.9	-36.3	RAR 65.3Mb
MikeRofone23	-38.2	-39.0	-41.3	-39.5	-19.3	-19.4	-34.7	-34.5	-27.1	-25.9	-19.8	-20.0	-24.8	-21.5	-27.0	-26.8	-36.8	-36.2

Notes:
1. Difference level shows how much does the warped signal differ from reference one.
-Infinity(dB) means identical signal shapes,
0(dB) – completely different shapes.
2. As time warping procedure is computationally extensive fast version of the algorithm was used for processing of all submitted recordings. The candidate chosen for inclusion to SE rating system will be processed with the highest possible accuracy. Accurate time warping gives slightly lower values of Difference levels (around 1dB lower) and lower contamination of processed signal with time warping artifacts (noticeable in the above difference signals as amplitude modulation).
3. Test file used for recording consists of three identical groups of SE test samples. Difference levels are indicated for all of them. Time warped versions are uploaded for one of them only as they are almost similar.

2. Cluster analysis

Here is graphical representation of Diff. levels from the table above:

All recordings except MikeRofone1-3 look consistent with each other. However the graph doesn't show how similar they are. Cluster analysis is more informative in this sense.

This dendrogramm shows how the recordings break up into groups according to similarity to each other. The higher the |¯|-shaped lines the less similar connected recordings. In our analysis all recordings could be divided into three groups:

Lee1-3 and MikeRofone22 are most similar. This is not surprising because Lee did some research to find the setup which provides the lowest number of errors while recording-playback-recording on his equipment. MikeRofone22 also hit the group and MikeRofone21 is the second closest. However MikeRofone21-23 recordings are less similar to each other.
Numinos1-3 recordings are also less similar to each other but pretty close to the first group.
MikeRofone1-3 recordings differ greatly from all others. They have to be excluded from further analysis.

3. Selection of best samples

Recordings from group#1 - Lee1-3 and MikeRofone22 plus MikeRofone21,23 - were time warped with higher accuracy. Results are below:

Author of submitted recording	Difference levels (dB) of time warped SE samples (Left and Right channels mixed)									Download DIFF signals
	BAH	BAS	CST	FMS	GLK	HRP	LOB	MOF	QRT

Lee1	-39.42	-43.25	-22.11	-37.44	-33.60	-21.10	-27.12	-30.81	-37.44	RAR 9.7Mb
Lee2	-39.88	-42.84	-22.06	-37.65	-33.45	-21.01	-26.85	-31.33	-37.40	RAR 9.7Mb
Lee3	-39.80	-43.69	-22.05	-37.57	-34.13	-20.83	-26.58	-31.06	-37.21	RAR 9.7Mb

MikeRofone21	-39.76	-43.22	-22.01	-37.52	-34.01	-21.56	-26.40	-31.19	-37.36	RAR 9.7Mb
MikeRofone22	-39.65	-42.73	-21.97	-37.34	-33.39	-21.48	-26.95	-30.98	-37.36	RAR 9.6Mb
MikeRofone23	-39.64	-43.09	-21.82	-37.40	-32.46	-21.54	-26.50	-31.16	-37.27	RAR 9.7Mb

Note: It turned out that Diff. level of mixed Left and Right channels is more sensitive parameter because it also accounts phase shifts between channels. From now on it will be used instead of separate Diff. levels.

Visual representation of the values:

Audibly similar type of artifacts in diff signals and close values of Diff. levels clearly show that all recordings belong to Type-R compression. The only difference between them is number of recording errors. In other words degradation of the signals caused by two factors - ATRAC compression and recording errors.

Unfortunately each author's recording has only three observations for each sound sample. This doesn't allow us to perform analysis of variance which would reveal distribution type of those observations. So we assume that it is one-sided normal distribution because it is highly unlikely that recording errors could improve Diff. levels over those conditioned by ATRAC. If so we could chose for each sound sample the recording that has the lowest Diff. level of warped signal, which means that the recording is less affected by recording errors. These lowest values are marked with green in the table.

The selected recordings were chosen as the best representatives of ATRAC SP (Type-R) encodings of nine SoundExpert samples. They were then time-warped with accuracy below -100dB - thus hiding artifacts of time-warping beyond resolution of 16 bit audio. Resulting Diff. levels, diff. signals and time-warped samples are in the table:

Author of
submitted
recording

Difference levels (dB) of time warped SE samples
(Left and Right channels mixed)

Download

BAH

BAS

CST

FMS

GLK

HRP

LOB

MOF

QRT

Lee1

-22.28

-27.10

-37.46

diff
(RAR:9.7Mb)
warp
(RAR:25.5Mb)

Lee2

-39.90

-37.69

-31.38

Lee3

-43.82

-34.16

MikeRofone21

-21.62

4. Contributors and equipment

SoundExpert would like to thank people who recorded SE test file on their equipment:

Lee Keywood (www.leekeywood.co.uk)

Recording chain: SE test file(.wav) ==> PC(Linux) ==S/PDIF==> Sony MZ-R410(SP,Type-R) ==MD-disc==> Technics SJ-MD100 ==S/PDIF==> PC(Linux) ==> Lee1-3(.wav)

Numinos from Minidiscforum.de (forum post)

Recording chain: SE test file(.wav) ==> PC ==USB-S/PDIF==> Sony MZ-NH600(SP,Type-R) ==MD-disc==> Kenwood DM-3090 ==S/PDIF==> Sony MZ-NH600(PCM) ==upload as PCM through SonicStage==> PC ==> Numinos1-3(.wav)

Mike Rofone from Audio T-board (forum post)

Recording chain: SE test file(.wav) ==> Audio CD ==> Pioneer CD player ==Toslink==> Sony MZ-RH1(SP,Type-R) ==upload as Hi-SP through SonicStage==> PC ==> MikeRofone1-3(.wav)
Recording chain: SE test file(.wav) ==> Audio CD ==> Pioneer CD player ==Toslink==> Sony MZ-RH1(SP,Type-R) ==upload as PCM through SonicStage==> PC ==> MikeRofone21-23(.wav)

Call for ATRAC SP (Type-R) test recordings

2011-09-12T21:31:53Z

From now on SoundExpert is ready to accept wider class of devices/technologies for testing and rating. Hardware audio equipment for example. This became possible owing to development of precise time warping procedure which eliminates various time distortions of audio signals usually caused by AD/DA conversion, filtering, inaccuracy of clock rate, etc. Being inaudible and practically immeasurable such time/phase inconsistency affects greatly the difference signal used for artifacts amplification in SE testing scheme. Fortunately all software codecs produce time accurate output while most of other devices/processings usually have constant or variable (wow-and-flutter) time shifts. Now we can precisely adjust time grid of an audio signal to time grid of reference signal. Thus difference signal contains only shape differences regardless of time squeezing/shrinking.

The first device that will be added to SE rating system using this kind of “time treatment” is hardware codec ATRAC SP (Type-R). It has no software implementations and used in MD players/recorders/decks. Recording/playback of ATRAC SP streams as many other real-time processes is affected by inaccuracy of clock rates. Resulting digitally recorded streams usually have small variations of their time scales, not audible but sufficient to spoil difference signal completely. So ATRAC SP is a good candidate for testing our new time warping technology.

We thank Lee Keywood who not only suggested inclusion of this codec for benchmarking purposes but also encouraged us to develop the time warping procedure. He carefully made on his MD equipment tens of recordings that we used for testing and tuning of our code.

This successful collaboration convinced us in possibility of preparing testing material outside SoundExpert by volunteers. Corresponding methodology of selection and verification of submitted output samples was also developed and now needs to be tested in practice.

For the purpose we ask volunteers who have MD equipment with ATRAC SP (Type-R) capability to record digitally SE test file and then record it back to PCM format digitally as well. This is in short. In detail:

Download a single test file - [link deleted as outdated] (WAV, 44100, 16, stereo, duration 5:15, size 29.7 MB). It consists of three identical groups of nine SE test samples (with trailing samples). Three groups are necessary to verify the consistency of recoding. The test file can be played back on computer directly or by mounting virtual Audio CD. Also it can be burnt as real Audio CD and played back on CD/DVD player. CUE-sheet for burning/mounting applications is included.
Using S/PDIF, TosLink, AES, USB or other digital interface record the test file to MD (Type-R) as a single track.
Playback the recorded MD track through digital interface and record it back to computer or other recording equipment capable of recording PCM 44100. If you record in 32bit format, leave it as is to avoid additional processing.
Archive or losslessly compress resulting WAV/AIFF 44100 PCM file and upload it to some file hosting service.

As we never did this before we don't know how many such recordings will be sufficient (this is another goal of the experiment). We'll process all received files with our time warping software to eliminate wow-and-flutter and other time distortions. For every file submitted we'll upload corrected (warped) versions along with difference signals. So the whole experiment will be openly covered on SE site and could be discussed on Sony Insider forum.

The most consistent and representative recording will then be added to SE rating system as ATRAC SP (Type-R) device under test. Although this format is not developed anymore it is still used for home and semi-professional recording. It will serve as a benchmark for SE rating scale, let alone the possibility to compare it to other audio compression algorithms at various bit rates. Also if this experiment is successful the above scheme could be used for adding other audio equipment to SE rating system.

Thank you in advance,
SE Team.

New Nero AAC Codec 1.5.3.0 in 192 kbit/s group

2010-03-09T11:24:18Z

The latest version of free Nero encoder has been just added. According to SE site statistics 192 kbit/s group is the most visited now. So it will be populated with contenders first of all. Thank you, Taras, for the proposal.

Cultural evolution @home

2010-01-03T11:52:40Z

One more Distributed Human Project appeared on the internet. DarwinTunes models cultural evolution using music as an example. The project assumes that there is an analogy between darwinian evolution of species and cultural evolution inside human society, that all cultural diversity exists thanks to continuous copying and modifying of previous cultural artifacts and subsequent selection and preserving of good modifications. For the purpose DarwinTunes continuously produces new generations of music excerpts – loops by modifying the best ones from previous generations. Selection of good ones is entrusted to visitors of the site. Everyone, including you, can participate. You will be asked to express your attitude towards generated music excerpts choosing from “I love it!”, “I like it”, “It's OK”, “I don't like it”, “I can't stand it!”. At the moment more than 60 000 visitors took part in the experiment and we can see\hear music loops after 300 generations of evolution.

If you think the goal of the experiment is creation of some musical masterpiece, you are mistaken. DarwinTunes aimed to research the process of evolution itself, how it develops in time, what factors affect it. The music evolving inside the project is pretty interesting, though.

Besides the main goal of the project there is another one, may be even more important – to show that cultural evolution progresses by means of copying and modifying. So in order to support progress and creativeness we must make available as many objects of culture in public domain as possible. It wouldn't be a big exaggeration to say that promoting such seditious ideas DarwinTunes moves cultural evolution further, not just researches it.

Open DarwinTunes in new window >>>