In September 2012 new, open and royalty-free audio codec Opus was standardized by the IETF. First stable release is available on Opus site - www.opus-codec.org. SE starts testing of the codec from 64kbit/s section. Other contenders are AAC (three versions from different developers) and Vorbis (Xiph version). Settings for all codecs were adjusted to get the target bitrate as close as possible to each other. As usually the target bitrate (FBR) is calculated on the basis of nine SE sound samples concatenated. Details of the codecs added:
AAC+ [email protected] (Winamp 5.63) - CVBR, HE-AAC
- AAC Encoder v1.04 (Fraunhofer IIS) from Winamp 5.63: Variable Bitrate, Preset 2
- The encoder is also available as CLI utility and can be downloaded from https://github.com/tmkk/fhgaacenc
AAC+ [email protected] (QTime 7.7.2) - CVBR, HE-AAC
- QuickTime (7.7.2) AAC Encoder via qaac 1.45 (CoreAudioToolbox 7.9.8.1): qaac --he -v56 ref.wav
- The encoder is included in such Apple products as iTunes, QuickTime, Safari; though not all of its features can be reached through these applications. For greater flexibility the CLI utility qaac can be used
- download from https://sites.google.com/site/qaacpage/
AAC+ [email protected] (NeroRef 1540) - CVBR, HE-AAC
Vorbis [email protected] (Xiph 1.3.3)
Opus [email protected] (libopus 1.0.1)
- opusenc --bitrate 59 ref48.wav (44.1/16 -> 48/24 by Audition CS6)
- The encoder operates at 48kHz only, so for other sample rates (as in our case) resampling is required. Some good quality resampler is already integrated into the codec but we used the external one from Audition CS6; the complete test files production chain: reference test files 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> decode by foobar2000(48/24) -->> 44.1/32(Audition CS6) -->> test files production
- download from http://opus-codec.org/downloads/
All of the codecs are in the system already. Shortly after first grades have been received an incorrect calculation of ratings' reliability in 64kbit/s section was spotted. In order to recalculate all reliability values, all codecs in the section including newly added ones were put on rotation. Also testing of all other devices at SE was paused. After achieving reliable ratings in 64kbit/s section normal operation of SE testing engine will be resumed (such technical details of SE operation are announced in SE twitter now).
***
SE thanks participants of discussion on rutracker.org forum (in Russian) and specially G-Man90 and MindWin for their help on choosing these particular codecs and settings. G-man90's personal web site audiophilesoft.ru (in Russian) also helped a lot with the latest info about audio codecs and other PC audio issues.
As usually SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after completing of SE listening tests. So we encourage you to take part. SE ratings exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.
Please, remember, downloading test files from SoundExpert you will not necessarily get the ones of the newly added codecs. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.
Best regards,
SE Team.
Results of the test
Although the ratings of the above codecs had been available just from the beginning of the listening test with gradually increasing reliability (1% accuracy now), more detailed results could be helpful and interesting as well. Below is the outcome of this test in more traditional form with means and confidence intervals.
Numbers of collected grades for each codec and sample are in the tables below. Numbers in brackets indicate that those grades were discarded by SE automated post-screening mechanism (Median +/- 1.5) and were not used for further calculations. Sound samples (BAH, BAS, CST, ...) can be found here.
AAC+ [email protected] (Winamp 5.63)
| BAH BAS CST FMS GLK HRP LOB MOF QRT
----+-------------------------------------------------------
*9* |
*8* | [1]
*7* | [1] [1]
*6* | [2] [1] [2] [2] [2]
----+-------------------------------------------------------
*5* | 10 3 3 7 6 1 2
*4* | 6 12 8 12 8 9 7 3 5
*3* | 2 4 5 9 3
*2* | [1]
*1* |
----+-------------------------------------------------------
NN | 16 17 11 16 15 15 13 12 10
Mean| 4.63 4.06 4.27 3.75 4.47 4.40 3.69 3.25 3.90
AAC+ [email protected] (NeroRef 1540)
| BAH BAS CST FMS GLK HRP LOB MOF QRT
----+-------------------------------------------------------
*9* |
*8* | [1]
*7* |
*6* | [2] [1] [1] [1] [2] [2]
----+-------------------------------------------------------
*5* | 7 11 6 1 [1] [1] 3
*4* | 10 6 7 11 5 7 4 5
*3* | 1 [2] 1 7 9 14 8 11 7
*2* | [1] 1 1
*1* |
----+-------------------------------------------------------
NN | 18 17 14 19 10 19 15 15 16
Mean| 4.33 4.65 4.36 3.68 2.90 3.26 3.47 3.27 3.63
AAC+ [email protected] (QTime 7.7.2)
| BAH BAS CST FMS GLK HRP LOB MOF QRT
----+-------------------------------------------------------
*9* |
*8* |
*7* | [1]
*6* | [1] [1] [1] [1]
----+-------------------------------------------------------
*5* | 4 5 [1] 3 3 [2] [2]
*4* | 14 10 2 15 5 8 [1] 4
*3* | 2 1 11 12 2 3 1 3 14
*2* | 1 13 1
*1* |
----+-------------------------------------------------------
NN | 20 16 13 13 17 11 12 16 19
Mean| 4.10 4.25 3.15 2.92 3.88 4.00 4.17 2.19 3.16
Opus [email protected] (libopus 1.0.1)
| BAH BAS CST FMS GLK HRP LOB MOF QRT
----+-------------------------------------------------------
*9* |
*8* | [1] [1] [1]
*7* | [2]
*6* | [3] [1] [2] [3] [2] [2]
----+-------------------------------------------------------
*5* | [2] [5] [4] 2 [3] [3]
*4* | 1 1 1 11 [1] 1 1 2
*3* | 13 10 13 1 4 9 14 4
*2* | 3 1 9 11 7 1 6
*1* | 4 7
----+-------------------------------------------------------
NN | 17 12 14 14 17 18 17 16 12
Mean| 2.88 3.00 3.07 4.07 2.00 1.61 2.65 3.00 2.67
Vorbis [email protected] (Xiph 1.3.3)
| BAH BAS CST FMS GLK HRP LOB MOF QRT
----+-------------------------------------------------------
*9* | [1]
*8* | [1]
*7* | [1] [1] [1]
*6* | [3] [1] [1]
----+-------------------------------------------------------
*5* | [2] [1] [3] [1] [1] [3] [1]
*4* | 1 [2] 2 1 4 [1] [2] 1
*3* | 5 7 10 14 8 6 6 10
*2* | 8 11 9 3 10 13 8 16 7
*1* | 2 1 [1]
----+-------------------------------------------------------
NN | 14 18 21 18 22 21 14 17 18
Mean| 2.50 2.39 2.67 2.89 2.73 2.19 2.43 1.94 2.67
Previous graphs of the same results (for archival purposes)
Version 1.0 (04.2013)
Initial graph:
- overall confidence intervals of means are computed for all grades of all sound samples (usual/standard approach in listening tests)
- 95% bootstrap confidence intervals of means are computed using bias corrected and accelerated percentile method
- small deviation of overall means from corresponding ratings on 64kbit/s page is due to incorrect calculation of overall means on the graph - they are bootstrapped means, not actual means
Full Res. PDF(23KB): Results of SE listening test of Opus, AAC and Vorbis @64 kbit/s (11.2012)
Version 2.0 (11.2013)
Slightly different graphical presentation of the same results. Changes:
- resulting mean of each codec (black ones) is an average of its nine sample means (previously it was a bootstrapped average of all grades submitted for the codec). Its 95% bootstrapped confidence intervals are also computed using these nine sample means and show therefore consistency of codec performance with different types of audio material. In case of small number of test samples (9) confidence intervals of resulting means computed this way have little to no meaning (statisticaly incorrect). In final version of the graph standard/usual method was used (as in v.1.0).
- all bootstrapped confidence intervals of means are computed using basic percentile method which is more simple and clear (previously bias corrected and accelerated percentile method was used)
- sample means are colored for convenience
Full Res. PDF(14KB): Results of SE listening test of Opus, AAC and Vorbis @64 kbit/s (11.2012)
SoundExpert (SE) is an independent non-commercial audio research project, which provides audio quality ratings of sound equipment and technologies such as mp3, aac, wma ... encoders and portable players (to be extended). It is the home for innovative music-based audio metric which works both for objective audio measurements and for listening tests. SE is in beta state because the new audio metric and corresponding listening test design are not widely adopted by audio engineers. So, please, use our ratings with caution.
The ratings of encoders are computed in real time while participants of the listening tests send their grades to the rating system. The more participants - the more devices and technologies can be tested. The testing procedure is simple and short enough that anyone, including you, can participate. Just download a test file (<3Mb), listen it (<15s) and send back your grade. Details are in the test file. The more accurate grades - the more reliable ratings.
