Methodology of SE listening tests
Sound quality evaluation tests through the internet is a very complicated task. Many recommendations by ITU-R/AES/EBU for listening tests could not be fully implemented in the internet testing. Listening sessions have to be easy enough for non-prepared listeners and resulting scores must be firm against various listening environments and skills of those ordinary listeners.
Taking all this into account it was decided to use the traditional testing methodology - double-blind triple-stimulus with hidden reference - with some corrections. For tested devices with clearly audible artifacts the method is used as is. The only difference is absence of the third stimuli - non-hidden reference as it could be picked out from two samples with ease. Thus the resulting methodology could be determined as double-blind double-stimulus with hidden reference. Participants download a sound file with two samples of the same sound excerpt. One of them is a reference sample and the other is processed by device under test. Testers have no information either about order of samples or about device used. Listeners are asked to point to processed sample and to grade its Basic Audio Quality according to standard five-grade impairment scale.
For tested devices with hard to distinguish artifacts this methodology is used in combination with sound artifacts amplification (SARTAMP) – special technology aimed to gain artifacts to some controlled level. In this case the whole testing procedure is completely the same except computing of final ratings. They are calculated considering the gain applied to artifacts. SARTAMP is integral part of new audio metric based on the single parameter - Difference Level (Df, dB). This audio parameter helps to determine relationship between subjective assessments and objective measurements.
Detailed explanation of SE listening test engine and corresponding method can be found in the papers below:
1. Live Listening Tests @SoundExpert. (PDF: 385k)
Easy to understand overview of SE blind tests functionality.
- Supplementary diagram: SE test engine at a glance. (PDF: 581k)
2. Infinite Grade Impairment Scale, Smirnoff S (PDF: 221k)
The paper describes briefly new method of judging audio quality - the one SoundExpert is based on.
3. DIFFERENCE LEVEL. An objective audio parameter. Smirnoff S (PDF:695k)
The paper describes objective parameter, called "Difference Level" that could be considered either as an extension of THD for non-periodic signals, or as one of the estimations of widely used difference signal. It could be used for instrumental measurements of signal degradation in various audio circuits and for psycho-acoustic research. Infinite grade impairment scale and corresponding method for measurements of perceived audio quality, based on this parameter is also proposed.
4. Post-screening of grades in SE listening tests
Choice and justification of post-screening method for rejection of outliers from grades submitted by SE testers.