Test criteria are arbitrary

Hands up, we admit it.

The criteria for any test – not just ours – are inherently arbitrary. You might be using W3C’s validator, for instance, and assume that because it’s got W3C stamped over it, that it’s the perfect way of testing that your HTML is ‘correct’.

And you would be wrong.

Try implementing a secure e-commerce page that passes W3C validation. You’ll be wanting to turn off autocomplete of course, because you can’t have credit card numbers and the like being stored in the user’s browser. Unfortunately, the HTML for that – which works in every browser – will cause you to fail W3C.

Of course W3C has a get-out clause: they’re only testing what they claim to be testing, which is conformance against the HTML spec. Great news for them, bunk-all good to those trying to use their test in the real world.

This one problem could be fixed by simply adding autocomplete as an officially supported feature – but that wouldn’t change the fundamental issue: somewhere, someone is deciding what these tests should cover, and you won’t always agree. Even when it’s us.

This problem is unavoidable. The test-designer can argue their case, use extensive research and a thorough testing methodology, but fundamentally you’re making a choice to accept their conclusions and ascribe them a certain level of worth. How much worth is up to you.

For our part, we’re trying to avoid the avoidable issues like W3C. That means regularly updating, being as open as possible, and involving our community in the refinement of our tests. We believe that by doing this consistently over a long period of time, we can earn the trust of users.

But remember, no test is perfect.