Actually sound quality evaluation test through the internet is a very complicated task. Many recommendations (by ITU-R/AES/EBU) for listening tests could not be fully implemented in the internet testing. Listening sessions have to be easy enough for non-prepared listeners and results must be firm against various listening environments and skills of those ordinary listeners.
Taking all this into account only the core of SoundExpert testing engine was based on traditional testing methodology - double blind, A/B scale with hidden reference or in ITU-R terms - double stimulus / hidden reference / double blind methodology. For tested devices with clearly audible artifacts the method is used as is. The only difference is absence of the third stimuli - non-hidden reference as it could be picked out from two samples with ease. Participants download a sound file with two samples of the same sound excerpt. One of them is a reference sample and the other is processed by device under test. Testers have no information either about order of samples or about device used. Listeners are asked to point to processed sample and to grade its Basic Audio Quality (BAQ) according to standard five-grade impairment scale.
For tested devices with hard to distinguish artifacts this methodology is used in combination with sound artifacts amplification (SARTAMP) – special technology aimed to gain artifacts to some controlled level. In this case the whole testing procedure is completely the same except computing of final ratings. They are calculated considering the gain applied to artifacts.
Detailed explanation of SE testing engine and method can be found in the papers below (in the order of increasing complexity).
1. SoundExpert. How it works. (PDF: 240k)
2. Infinite Grade Impairment Scale, Smirnoff S (PDF: 175k)
3. DIFFERENCE LEVEL. An objective audio parameter. Smirnoff S (PDF:694k)