What is SoundExpert

Audio metric that makes a differenceSoundExpert (SE) is an independent non-commercial audio research project, which provides audio quality ratings of sound equipment and technologies such as mp3, aac, wma ... encoders and portable players (to be extended). It is the home for innovative music-based audio metric which works both for objective audio measurements and for listening tests. SE is in beta state because the new audio metric and corresponding listening test design are not widely adopted by audio engineers. So, please, use our ratings with caution.

The new listener-centric audio metric allows to perform objective audio measurements using real-life music material and various technical signals, not only sine waves. Such measurements:
• correlate well to human perception of sound quality
• are valid for digital audio
• do not contradict traditional audio metric but extend it
• are understandable to people without technical background
• can be performed at home using any recording device

The first practical application of the new measurement method can be found on the Portable players page.

The new listening test design helps to hear sound artifacts which are normally beyond threshold of human perception. The core of the new design is sound artifacts amplification technology (SARTAMP), which is backed by the new audio metric. Such amplification is particularly important for testing high-bitrate encoders and high quality audio equipment. This technology is used in SE listening tests since 2005. The tests are blind - participants don't know the particular device they assess. Resulting quality scores are completely unbiased and free from marketing and advertising noise.

Live Listening Tests @SoundExpertThe ratings of encoders are computed in real time while participants of the listening tests send their grades to the rating system. The more participants - the more devices and technologies can be tested. The testing procedure is simple and short enough that anyone, including you, can participate. Just download a test file (<3Mb), listen it (<15s) and send back your grade. Details are in the test file. The more accurate grades - the more reliable ratings.

Detailed but still easy to understand overview of SE distributed listening tests is in the paper Live Listening Tests @SoundExpert.

* * *

The main idea behind SoundExpert project is consumer control over sound quality of various audio equipment and technologies on the market. Production side deserves clearly defined feedback from consumers.

 

 

Articles

« Back

Audio quality of iPhone 6

Audio quality of iPhone 6

Published Date 7/7/15 11:27 AM
This article is obsolete and retained for archival purpose. Listening tests of portable players are no-longer performed and substituted by objective measurements which are available on portable players page - http://soundexpert.org/portable-players. The measurements of iPhone 6 can be found here - http://soundexpert.org/portable-players-beta

Four years ago SE started sound quality evaluation of portable media players. Results collected for the first few contenders showed that testing procedure has some serious drawbacks and needs to be corrected. For example, using real headphones as a load (and especially Koss Porta Pro) leads to specific coloring of sound for players with higher output impedance. Also, recent research related to diffrograms revealed that test files for analog audio devices can be and must be processed with higher accuracy. And finally, objective measurements of analog circuits have sense. New method of objective testing, based on Difference level parameter, helps to discover problem signals and to evaluate waveform degradation of real-life sound material for any device under test. So, results of such testing in addition to usual SE listening tests would be helpful.

Taking all this into account it was decided to start testing of portable media players afresh. Previous results will be discarded, sorry.

The first contender for this enhanced testing is iPhone 6. Results of its objective measurements are summarized in compact form - df-slide, shown in Fig.1. All df-slides for portable players which were tested at SoundExpert are collected on this page - http://soundexpert.org/portable-players-beta

 

Results of testing iPhone 6
Figure 1. Results of testing media player of iPhone 6 with various audio signals and single measurement approach based on Difference level parameter.

 

The diagram shows how various signals (including real-life ones) are distorted when they are played back by iPhone 6. We can safely conclude that:

  • pure sine waves both 1kHz and 12.5kHz have low levels of distortion (-79.0dB and -78.9dB)
  • inter-modulation distortion is pretty high (the sum of sine waves 12460Hz and 12540Hz at the output of iPhone 6 has substantially higher level of difference with initial waveform: -45.9dB)
  • serious gap between accuracy of reproduction of triangle and square waves (-38.7dB and -16.3dB) means that iPhone 6 has phase distortion; signals with multiple frequency components are affected the most (white noise -4.6dB)
  • median of 6430 Df values (the whole Pink Floyd album divided into 400ms pieces) sums up all possible types of distortion of the real-life audio signal; the value (-25.7dB) is pretty close to degradation of Program Simulation Noise (-24.9dB); this is very promising and even a bit surprising; level of degradation of this signal could be a single integral and informative objective indicator which says a lot about performance of a device under test; more audio devices should be tested in order to make more confident conclusions about usefulness of such indicator.

Other test signals could be synthesized and used for discovering various aspects of an audio device performance. As the measurement procedure is the same for all signals, their values of degradation can be compared with each other directly; the whole audio metric becomes connected.

Feel free to share and discuss the above diagram.

SoundExpert thanks Rasim for allowing to test his iPhone 6.

***

SoundExpert reminds you that final conclusion about audio quality of any device added can be made only after finishing corresponding listening test. So we encourage you to take part in testing. SE ratings exist only thanks to visitors like you. Testing is short and easy, visit our Testing Room for simple 1-2-3 instruction.

Please, remember, downloading test files from SoundExpert you will not necessarily get the ones of the newly added devices or codecs. SoundExpert testing is blind testing – you don't know the device you test. This is the only reliable way of getting true audio quality ratings, free from any cheating and various human senses other than perception of sound quality. After you have sent your grade you will see the device you tested.

Best regards,
SE Team.

 

Update 2015 Aug 26

After adoption of final set of test signals (http://soundexpert.org/test-signals) it was decided to test iPhone 6 once again. Results are below.

 

Testing iPhone 6 with final set of test signals
Figure 2. Results of testing media player of iPhone 6 with final set of test signals and single measurement approach based on Difference level parameter.

 

What changed:

  • triangular and square waves are band-limited now, so they are reproduced with higher accuracy (lower Df values) by device under test
  • triangular (even) signal is added (ass train); comparing its degradation with the standard triangular (odd) wave we can conclude that even harmonics slightly prevail over odd ones at the output of iPhone 6.

SE thanks Olesya for allowing to test her iPhone 6.

Best regards,
SE Team.

 

By Serge Smirnoff
867299 Views, 4 Comments
Flag
audio quality df-slide diffrogram iphone 6
Average (0 Votes)
The average rating is 0.0 stars out of 5.
Comments
Trackback URL :

Add Comment
audio tester
Please, try it with iPhone S4, since it was the last phone in which Apple did use 3rd party audio analog-digital + dsp components.
In S5 and S6, Apple is using internally design pipeline for audio.
Posted on 10/15/15 3:59 PM.
Serge Smirnoff
http://soundexpert.org/portable-players
Posted on 10/19/15 7:44 PM in reply to audio tester.
Jason
Samples from the only device currently offered, the "iPhone 6", have boosted bass and treble. They are thus easily distinguishable.

I have difficulty rating them on the same scale as codecs. If the iPhone 6 was a codec, such severe deviation in frequency response would deserve a "1" or "2" (too much treble is definitely annoying in mof.wav and hrp.wav). Being able to identify it, I have rated samples from this player as "3", because I am able to compensate for the treble while listening as I do with other real sound sources, and there are no obvious frequency domain artifacts. In direct comparison, the reference even sounds "worse" (less clarity, muffled).

I doubt that the rating for this device has any meaning if the scale is uncertain.
Posted on 3/31/17 1:49 AM.
Serge Smirnoff
To avoid such uncertainty with grading frequency response artifacts it is necessary to include in test sequence the third stimuli – unblinded reference sample. This will make the whole procedure of testing at SE more complicated. May be it's worse it. At the moment all grades with confused reference and processed samples are discarded.
Posted on 7/22/17 12:09 PM in reply to Jason.
Audio-Transparency Initiative
All articles ►
Audio-Transparency Initiative