What is SoundExpert

SoundExpert (SE) is an independent non-commercial audio research project, which provides audio quality ratings of sound equipment and technologies such as mp3, aac, wma ... encoders and portable players (to be extended). It is the home for innovative music-based audio metric which works both for objective audio measurements and for listening tests. SE is in beta state because the new audio metric and corresponding listening test design are not widely adopted by audio engineers. So, please, use our ratings with caution.

The new listener-centric audio metric allows to perform objective audio measurements using real-life music material and various technical signals, not only sine waves. Such measurements:
• correlate well to human perception of sound quality
• are valid for digital audio
• do not contradict traditional audio metric but extend it
• are understandable to people without technical background
• can be performed at home using any recording device

The first practical application of the new measurement method can be found on the Portable players page.

The new listening test design helps to hear sound artifacts which are normally beyond threshold of human perception. The core of the new design is sound artifacts amplification technology (SARTAMP), which is backed by the new audio metric. Such amplification is particularly important for testing high-bitrate encoders and high quality audio equipment. This technology is used in SE listening tests since 2005. The tests are blind - participants don't know the particular device they assess. Resulting quality scores are completely unbiased and free from marketing and advertising noise.

The ratings of encoders are computed in real time while participants of the listening tests send their grades to the rating system. The more participants - the more devices and technologies can be tested. The testing procedure is simple and short enough that anyone, including you, can participate. Just download a test file (<3Mb), listen it (<15s) and send back your grade. Details are in the test file. The more accurate grades - the more reliable ratings.

Detailed but still easy to understand overview of SE distributed listening tests is in the paper Live Listening Tests @SoundExpert.

The main idea behind SoundExpert project is consumer control over sound quality of various audio equipment and technologies on the market. Production side deserves clearly defined feedback from consumers.

Articles

« Back

Opus, AAC, Vorbis, MP3 at 96 kbit/s

Published Date 4/27/13 11:18 PM

This listening test follows the similar @64 one, finished not so far ago. Codec contenders are the same with a few exceptions:

Opus version was updated from 1.0.1 to 1.0.2
Lame mp3 encoder added (3.99.5)
QuikTime encoder was updated but without any quality improvements; in this test it is used in True VBR mode.

Final list of contenders:

AAC VBR@89.8 (Winamp 5.63) - VBR, AAC LC

AAC Encoder v1.04 (Fraunhofer IIS) from Winamp 5.63: Variable Bitrate, Preset 3
The encoder is also available as CLI utility and can be downloaded from https://github.com/tmkk/fhgaacenc

AAC VBR@92.1 (QTime 7.7.3) - TrueVBR, AAC LC

QuickTime (7.7.3) AAC Encoder via qaac 2.18 (CoreAudioToolbox 7.9.8.2): qaac -V45 ref.wav
The encoder is included in such Apple products as iTunes, QuickTime, Safari; though not all of its features can be reached through these applications. For greater flexibility the CLI utility qaac can be used
download from https://sites.google.com/site/qaacpage/

AAC VBR@90.0 (NeroRef 1540) - CVBR, AAC LC

Nero AAC Encoder 1.5.4.0 (build 2010-02-18): neroAacEnc.exe -q 0.34 -if ref.wav -of out.mp4
download from http://www.nero.com/eng/downloads-nerodigital-nero-aac-codec.php

Vorbis VBR@90.4 (Xiph 1.3.3)
OggEnc v2.87 (libVorbis 1.3.3): oggenc2 -q2.2 ref.wav

download from http://vorbis.com/

Opus VBR@90.7 (libopus 1.0.2)

opusenc --bitrate 90 ref48.wav (44.1/16 -> 48/24 by Audition CS6)
The encoder operates at 48kHz only, so for other sample rates (as in our case) resampling is required. Some good quality resampler is already integrated into the codec but we used the external one from Audition CS6; the complete test files production chain: reference test files 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> decode by foobar2000(48/24) -->> 44.1/32(Audition CS6) -->> test files production
download from http://opus-codec.org/downloads/

mp3 VBR@90.2 (Lame 3.99.5) - MPEG-1 Layer 3, VBR

encode: lame -V6.9 ref.wav
For this quality parameter (-V6.9) Lame down-samples the input from 44.1 kHz to 32 kHz. Decoder's output was up-sampled back to 44.1 with Audition CS6; the complete test files production chain: reference test files 44.1/16 -->> Lame -->> decode by MAD(32/24) -->> 44.1/32(Audition CS6) -->> test files production
download from http://www.rarewares.org/mp3-lame-bundle.php

In addition to usual live ratings on 96kbit/s page there will be detailed results of this test when corresponding ratings reach 1% accuracy (example of the detailed report is in the previous 64 kbit/s listening test).

***

SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after finishing corresponding listening test. So we encourage you to take part in testing. SE ratings exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading test files from SoundExpert you will not necessarily get the ones of the newly added codecs. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Results of the test

In addition to usual results on 96kbit/s page here is a detailed analysis of submitted grades for these codecs.

SoundExpert listening test of Opus, AAC, Vorbis, MP 3 @96 kbit/s (11.2013)

All grades submitted for the codecs are below. Numbers in brackets indicate that those grades were discarded by SE automated post-screening mechanism (Median +/- 1.5) and were not used for further calculations. Reference sound samples (BAH, BAS, CST, ...) can be found here.

AAC VBR@89.8 (Winamp 5.63) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | [1] [1] *7* | [2] *6* | [2] [4] [2] [1] [3] [2] [1] ----+------------------------------------------------------- *5* | 11 6 8 4 9 4 3 5 4 *4* | 3 6 9 9 4 8 9 8 6 *3* | 1 [1] 1 2 2 *2* | *1* | ----+------------------------------------------------------- NN | 14 13 17 13 13 13 14 15 10 Mean| 4.79 4.38 4.47 4.31 4.69 4.23 4.07 4.20 4.40 AAC VBR@90.0 (NeroRef 1540) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | *7* | [1] [1] *6* | [1] [3] [1] [2] [1] [2] [2] [1] ----+------------------------------------------------------- *5* | 8 3 15 5 10 [1] 6 4 3 *4* | 6 9 3 9 3 2 10 8 6 *3* | [1] [1] 11 3 *2* | [1] *1* | ----+------------------------------------------------------- NN | 14 12 18 14 13 13 16 12 12 Mean| 4.57 4.25 4.83 4.36 4.77 3.15 4.38 4.33 4.00 AAC VBR@92.1 (QTime 7.7.3) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | *7* | [1] [2] *6* | [2] [5] [1] [2] [1] [5] ----+------------------------------------------------------- *5* | 13 4 1 [1] 1 2 6 10 *4* | 8 9 7 11 6 8 11 8 8 *3* | [1] 1 5 4 12 3 3 1 [1] *2* | [1] 1 *1* | ----+------------------------------------------------------- NN | 21 14 12 16 19 12 16 15 18 Mean| 4.62 4.21 3.58 3.81 3.26 3.83 3.94 4.33 4.56 Vorbis VBR@90.4 (Xiph 1.3.3) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | [2] *7* | [1] [1] *6* | [3] [2] [1] [2] [1] [1] ----+------------------------------------------------------- *5* | 4 [2] 2 [1] 1 4 1 11 *4* | 7 4 9 4 9 3 11 7 3 *3* | 1 10 3 10 9 11 1 7 [2] *2* | 4 *1* | ----+------------------------------------------------------- NN | 12 14 14 14 19 18 16 15 14 Mean| 4.25 3.29 3.93 3.29 3.58 2.94 4.19 3.60 4.79 mp3 VBR@90.2 (Lame 3.99.5) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | [1] [1] [1] *7* | [1] [1] [1] *6* | [3] [1] [1] [1] [1] [3] [2] [1] ----+------------------------------------------------------- *5* | 9 2 [1] 1 3 4 6 *4* | 5 9 8 7 2 5 14 6 *3* | 2 13 6 13 4 7 3 4 *2* | 11 *1* | ----+------------------------------------------------------- NN | 14 13 21 14 15 15 15 21 16 Mean| 4.64 4.00 3.38 3.64 3.13 2.27 3.73 4.05 4.13 Opus VBR@90.7 (libopus 1.0.2) | BAH BAS CST FMS GLK HRP LOB MOF QRT ----+------------------------------------------------------- *9* | *8* | *7* | [1] *6* | [4] [1] [1] [1] ----+------------------------------------------------------- *5* | 4 3 2 7 3 2 2 *4* | 7 8 14 7 [1] 6 8 13 *3* | 5 1 2 2 1 8 4 6 *2* | [1] 8 11 1 [1] *1* | 1 [1] ----+------------------------------------------------------- NN | 16 12 18 14 11 12 18 14 21 Mean| 3.94 4.17 4.00 4.50 2.09 2.08 3.61 3.86 3.81

Previous graph of the same results (for archival purposes)

Version 2.0 (11.2013)

bootstrapped confidence intervals of resulting means (black ones) are computed using nine sample means and shows therefore consistency of codec performance with different types of audio material. In case of small number of test samples (9) confidence intervals of resulting means computed this way have little to no meaning (statisticaly incorrect). In final version of the graph standard/usual method was used (using all grades collected for a codec).

SoundExpert listening test of Opus, AAC, Vorbis, MP3 @96 kbit/s (11.2013)