Recent News and Articles

« Back

Opus, AAC and Vorbis in 64 Kbit/s section

Opus, AAC and Vorbis in 64 Kbit/s section

Opus audio codec logo

In September 2012 new, open and royalty-free audio codec Opus was standardized by the IETF. First stable release is available on Opus site - www.opus-codec.org. SE starts testing of the codec from 64kbit/s section. Other contenders are AAC (three versions from different developers) and Vorbis (Xiph version). Settings for all codecs were adjusted to get the target bitrate as close as possible to each other. As usually the target bitrate (FBR) is calculated on the basis of nine SE sound samples concatenated. Details of the codecs added:

AAC+ VBR@59.2 (Winamp 5.63) - CVBR, HE-AAC

  • AAC Encoder v1.04 (Fraunhofer IIS) from Winamp 5.63: Variable Bitrate, Preset 2
  • The encoder is also available as CLI utility and can be downloaded from https://github.com/tmkk/fhgaacenc

AAC+ VBR@59.6 (QTime 7.7.2) - CVBR, HE-AAC

  • QuickTime (7.7.2) AAC Encoder via qaac 1.45 (CoreAudioToolbox 7.9.8.1): qaac --he -v56 ref.wav
  • The encoder is included in such Apple products as iTunes, QuickTime, Safari; though not all of its features can be reached through these applications. For greater flexibility the CLI utility qaac can be used
  • download from https://sites.google.com/site/qaacpage/

AAC+ VBR@60.1 (NeroRef 1540) - CVBR, HE-AAC

Vorbis VBR@60.3 (Xiph 1.3.3)

Opus VBR@59.9 (libopus 1.0.1)

  • opusenc --bitrate 59 ref48.wav (44.1/16 -> 48/24 by Audition CS6)
  • The encoder operates at 48kHz only, so for other sample rates (as in our case) resampling is required. Some good quality resampler is already integrated into the codec but we used the external one from Audition CS6; the complete test files production chain: reference test files 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> decode by foobar2000(48/24) -->> 44.1/32(Audition CS6) -->> test files production
  • download from http://opus-codec.org/downloads/

All of the codecs are in the system already. Shortly after first grades have been received an incorrect calculation of ratings' reliability in 64kbit/s section was spotted. In order to recalculate all reliability values, all codecs in the section including newly added ones were put on rotation. Also testing of all other devices at SE was paused. After achieving reliable ratings in 64kbit/s section normal operation of SE testing engine will be resumed (such technical details of SE operation are announced in SE twitter now).

***

SE thanks participants of discussion on rutracker.org forum (in Russian) and specially G-Man90 and MindWin for their help on choosing these particular codecs and settings. G-man90's personal web site audiophilesoft.ru (in Russian) also helped a lot with the latest info about audio codecs and other PC audio issues.

As usually SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after completing of SE listening tests. So we encourage you to take part. SE ratings exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading test files from SoundExpert you will not necessarily get the ones of the newly added codecs. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

Results of the test

Although the ratings of the above codecs had been available just from the beginning of the listening test with gradually increasing reliability (1% accuracy now), more detailed results could be helpful and interesting as well. Below is the outcome of this test in more traditional form with means and confidence intervals.

Results of SE listening test of Opus, AAC and Vorbis @64 kbit/s (11.2012)

 

Numbers of collected grades for each codec and sample are in the tables below. Numbers in brackets indicate that those grades were discarded by SE automated post-screening mechanism (Median +/- 1.5) and were not used for further calculations. Sound samples (BAH, BAS, CST, ...) can be found here.

 

                 AAC+ VBR@59.2 (Winamp 5.63)
    |   BAH   BAS   CST   FMS   GLK   HRP   LOB   MOF   QRT
----+-------------------------------------------------------
*9* |                                                       
*8* |                           [1]                         
*7* |                           [1]                     [1]
*6* |   [2]   [1]               [2]   [2]         [2]       
----+-------------------------------------------------------
*5* |    10     3     3           7     6     1           2
*4* |     6    12     8    12     8     9     7     3     5
*3* |           2           4                 5     9     3
*2* |                                                   [1]
*1* |                                                       
----+-------------------------------------------------------
NN  |    16    17    11    16    15    15    13    12    10
Mean|  4.63  4.06  4.27  3.75  4.47  4.40  3.69  3.25  3.90


                 AAC+ VBR@60.1 (NeroRef 1540)
    |   BAH   BAS   CST   FMS   GLK   HRP   LOB   MOF   QRT
----+-------------------------------------------------------
*9* |                                                       
*8* |                                       [1]             
*7* |                                                       
*6* |   [2]   [1]   [1]   [1]               [2]         [2]
----+-------------------------------------------------------
*5* |     7    11     6     1   [1]               [1]     3
*4* |    10     6     7    11           5     7     4     5
*3* |     1   [2]     1     7     9    14     8    11     7
*2* |         [1]                 1                       1
*1* |                                                       
----+-------------------------------------------------------
NN  |    18    17    14    19    10    19    15    15    16
Mean|  4.33  4.65  4.36  3.68  2.90  3.26  3.47  3.27  3.63


                 AAC+ VBR@59.6 (QTime 7.7.2)
    |   BAH   BAS   CST   FMS   GLK   HRP   LOB   MOF   QRT
----+-------------------------------------------------------
*9* |                                                       
*8* |                                                       
*7* |               [1]                                     
*6* |         [1]         [1]   [1]   [1]                   
----+-------------------------------------------------------
*5* |     4     5   [1]                 3     3   [2]   [2]
*4* |    14    10     2          15     5     8   [1]     4
*3* |     2     1    11    12     2     3     1     3    14
*2* |                       1                      13     1
*1* |                                                       
----+-------------------------------------------------------
NN  |    20    16    13    13    17    11    12    16    19
Mean|  4.10  4.25  3.15  2.92  3.88  4.00  4.17  2.19  3.16


                 Opus VBR@59.9 (libopus 1.0.1)
    |   BAH   BAS   CST   FMS   GLK   HRP   LOB   MOF   QRT
----+-------------------------------------------------------
*9* |                                                       
*8* |                                       [1]   [1]   [1]
*7* |                                       [2]             
*6* |   [3]         [1]   [2]               [3]   [2]   [2]
----+-------------------------------------------------------
*5* |   [2]   [5]   [4]     2                     [3]   [3]
*4* |     1     1     1    11   [1]           1     1     2
*3* |    13    10    13     1     4           9    14     4
*2* |     3     1                 9    11     7     1     6
*1* |                             4     7                   
----+-------------------------------------------------------
NN  |    17    12    14    14    17    18    17    16    12
Mean|  2.88  3.00  3.07  4.07  2.00  1.61  2.65  3.00  2.67


                 Vorbis VBR@60.3 (Xiph 1.3.3)
    |   BAH   BAS   CST   FMS   GLK   HRP   LOB   MOF   QRT
----+-------------------------------------------------------
*9* |                                       [1]             
*8* |                                       [1]             
*7* |               [1]   [1]               [1]             
*6* |   [3]   [1]                                 [1]       
----+-------------------------------------------------------
*5* |   [2]   [1]   [3]   [1]               [1]   [3]   [1]
*4* |     1   [2]     2     1     4   [1]         [2]     1
*3* |     5     7    10    14     8     6     6          10
*2* |     8    11     9     3    10    13     8    16     7
*1* |                                   2           1   [1]
----+-------------------------------------------------------
NN  |    14    18    21    18    22    21    14    17    18
Mean|  2.50  2.39  2.67  2.89  2.73  2.19  2.43  1.94  2.67

 

 


Previous graphs of the same results (for archival purposes)

 

Version 1.0 (04.2013)

Initial graph:

  • overall confidence intervals of means are computed for all grades of all sound samples (usual/standard approach in listening tests)
  • 95% bootstrap confidence intervals of means are computed using bias corrected and accelerated percentile method
  • small deviation of overall means from corresponding ratings on 64kbit/s page is due to incorrect calculation of overall means on the graph - they are bootstrapped means, not actual means

 

Results of SE listening test of Opus, AAC and Vorbis @64 kbit/s (11.2012)

Full Res. PDF(23KB): Results of SE listening test of Opus, AAC and Vorbis @64 kbit/s (11.2012)

 

Version 2.0 (11.2013)

Slightly different graphical presentation of the same results. Changes:

  • resulting mean of each codec (black ones) is an average of its nine sample means (previously it was a bootstrapped average of all grades submitted for the codec). Its 95% bootstrapped confidence intervals are also computed using these nine sample means and show therefore consistency of codec performance with different types of audio material. In case of small number of test samples (9) confidence intervals of resulting means computed this way have little to no meaning (statisticaly incorrect). In final version of the graph standard/usual method was used (as in v.1.0).
  • all bootstrapped confidence intervals of means are computed using basic percentile method which is more simple and clear (previously bias corrected and accelerated percentile method was used)
  • sample means are colored for convenience

SoundExpert listening test of Opus, AAC, Vorbis @64 kbit/s (11.2012)

Full Res. PDF(14KB): Results of SE listening test of Opus, AAC and Vorbis @64 kbit/s (11.2012)

 

 

Comments
Trackback URL :

No comments yet. Be the first.

What is SoundExpert

SoundExpert - How it works SoundExpert (SE) is a crowdtesting service that provides audio quality ratings of sound equipment and technologies such as mp3, aac, wma ... encoders, portable players, sound cards, amplifiers ...

The ratings are based solely on results of blind listening tests when listeners don't know the particular device they test. So the values are completely unbiased and free from any marketing and advertising noise.

SoundExpert provides ratings thanks to visitors who take part in testing. The more number of participants - the more devices and technologies could be tested. The ratings are computed in real time while new grades are returned by visitors. The testing procedure is simple and short enough that anyone, including you, can participate. Just download a test file (<3Mb), listen it (<15s) and send back your judgment. Details are in test file. The more accurate grades - the more reliable ratings.

SoundExpert is independent non-commercial research project. It's in beta state because it uses new audio metrics and corresponding listening test design which are not widely adopted by audio engineers. The key feature of this new method is possibility to hear sound artifacts which are normally beyond threshold of human perception. So, please, use these ratings with caution. Detailed but still easy to understand explanation of SE internal mechanics is in the article "SoundExpert - How it works".

The main idea behind this service is consumer control over sound quality measurements of various audio equipment on the market. SE, being a distributed human project, combines tiny efforts of audio enthusiasts for carrying out this task.