Methodology of SE listening tests
Actually sound quality evaluation tests through the internet is a very complicated task. Many recommendations by ITU-R/AES/EBU for listening tests could not be fully implemented in the internet testing. Listening sessions have to be easy enough for non-prepared listeners and resulting scores must be firm against various listening environments and skills of those ordinary listeners.
Taking all this into account it was decided to use the traditional testing methodology - double-blind triple-stimulus with hidden reference - with some corrections. For tested devices with clearly audible artifacts the method is used as is. The only difference is absence of the third stimuli - non-hidden reference as it could be picked out from two samples with ease. Thus the resulting methodology could be determined as double-blind double-stimulus with hidden reference. Participants download a sound file with two samples of the same sound excerpt. One of them is a reference sample and the other is processed by device under test. Testers have no information either about order of samples or about device used. Listeners are asked to point to processed sample and to grade its Basic Audio Quality according to standard five-grade impairment scale.
For tested devices with hard to distinguish artifacts this methodology is used in combination with sound artifacts amplification (SARTAMP) – special technology aimed to gain artifacts to some controlled level. In this case the whole testing procedure is completely the same except computing of final ratings. They are calculated considering the gain applied to artifacts. SARTAMP is integral part of new audio metric based on the single parameter - Difference Level (Df, dB). This audio parameter helps to determine relationship between subjective assessments and objective measurements.
Detailed explanation of SE listening test engine and corresponding method can be found in the papers below:
1. Live Listening Tests @SoundExpert. (PDF: 385k)
Easy to understand overview of SE blind tests functionality.
- Supplementary diagram: SE test engine at a glance. (PDF: 581k)
2. Infinite Grade Impairment Scale, Smirnoff S (PDF: 221k)
The paper describes briefly new method of judging audio quality - the one SoundExpert is based on.
3. DIFFERENCE LEVEL. An objective audio parameter. Smirnoff S (PDF:695k)
The paper describes objective parameter, called "Difference Level" that could be considered either as an extension of THD for non-periodic signals, or as one of the estimations of widely used difference signal. It could be used for instrumental measurements of signal degradation in various audio circuits and for psycho-acoustic research. Infinite grade impairment scale and corresponding method for measurements of perceived audio quality, based on this parameter is also proposed.
4. Post-screening of grades in SE listening tests
Choice and justification of post-screening method for rejection of outliers from grades submitted by SE testers.
For almost 20 years SoundExpert is the home for innovative audio measurements and research. Newly developed listener-centric audio metric combines instrumental measurements with listening assessments and helps to perform both more efficiently.
Since 2001 more than 10 000 volunteers participated in distributed blind listening tests on Soundexpert website. Sound quality ratings of 100+ codecs were computed thanks to their time and efforts. You can still take part in them.
Recently we developed a new method for instrumental measurements of audio equipment. It is based on the new audio metric and helps to examine technical performance of an audio device in great detail. Furthermore all audio devices can be grouped according to their sound signatures. The measurement procedure was fine-tuned during two years of beta-testing with 30+ portable devices. Now it is mature enough for production environment. First few tested players can be found on Portable players page.
I hope this new audio metric will attract more audio engineers and enthusiasts as it opens new field of audio research where objective and subjective are not separated anymore. Further advance in the research requires many more different audio devices to be tested - a serious challenge for our non-commercial research project. Fortunately portable players are good test subjects for this research, even the old ones. That's why I need some help from our volunteers again. If you have such old/previous/ex digital portable audio player or a smartphone that was marketed as "HQ audio inside", if it is fully functional but not in use anymore, please, consider donating it to SoundExpert. Its df-measurements will appear on the portable players page. Then it will be either kept for research purposes or sold off supporting the project additionally (or given away to some music lover if it can't be sold for some reason). I'm going to refund your postal expenses but can't confirm this right now. Please, contact me for further arrangements.
Manufacturers of portable devices can order df-measurements for their products. The cost of the work will be equal to retail price of the device. I will buy it and perform standard df-measurements. If you decide I will not publish results during a year.
Today building a comprehensive audio quality map of various audio devices on the market can be done by ourselves - audio consumers. And it's not hard indeed. In the end this will free up some time for listening and discussing Music, not technology )).