The limits of automated website testing

This is a challenge we face every day: what can we actually test with a computer?

Computers are really good at following precise instructions very quickly, which is pretty much the opposite of us. If you can explain the problem in simple steps, then a computer is your … erm … man:

1. Find all images on a page
2. Find out what size they claim to be
3. Find out their actual size
4. If 2 is different from 3, report an error

A rickety old PC could run a test like this with contemptuous ease, and it would always uncover images with the wrong sizes a billion times faster than you could. These tests are super easy to write, and reliable to run.

Now consider spell checking. This simple set of rules should do the trick:

1. Find all words on a page (split them by spaces and punctuation marks).
2. Look each word up in a dictionary.
3. If a word is not found, flag it as mis-spelt.

Except this is completely wrong. For starters, look at these ‘words’:

3.14157

30^th

BBC

£23,000

2/07/1979

bob@example.com

www.example.com

All of these won’t be found in a dictionary. There are certain rules which say that numbers are not words, and that uppercase words are acronyms (in English at least). Ordinals (like 1^st, 2^nd, 3^rd) go on forever, and you can’t add each one to the dictionary. Instead, you need a tonne of custom rules to recognise these exceptions, and ignore them correctly.

Don’t forget that spelling can vary depending on context (e.g. the first letter of a new sentence should be uppercase, but proper nouns like ‘Dave’ should always be uppercase). Oh, and you want to check spelling in languages other than English, right? Good luck with that.

Of course we have spell checkers in Word, and now in our web browsers, and they’re pretty good too. So although this problem is a total bitch, it is possible. And this gives us hope for the other bitchy problems we have our hearts set on solving.

Note however, that automated spell checkers are definitely not perfect. And unless they become actually as smart as human beings, they never will be. Try this in Word:

“This is what the student wrote: I dont no how to spel”.

Now as a human it’s obvious that the spelling ‘mistakes’ here were intentional, and in fact essential to the meaning. But without a near human level of understanding of this sentence, no computer is going to know that. Nor is it worth making our spell checkers 8 billion times more complicated to try and eliminate the 0.001% of fringe cases where they fail, if we even knew how to do that, which we don’t (yet).

And so it is with testing websites.

There are ‘soft’ tests, for easy problems – like counting broken links. These will have extremely high accuracy, but tend to be relatively superficial. They form the majority of automated tests available today.

And there are ‘hard’ tests, for problems like spell checking, or identifying different designs in your site, or telling you your graphic design makes my eyes bleed. These are vastly more complex, imperfect sets of rules which approximate things our human brains can often do easily. With work, they can be right over 99% of the time, but they never be perfect. Just like the spell checker in Word though, they can still be very useful.

We have a lot of interest in the hard problems, which is where we believe the most valuable feedback on a website can be identified. Wouldn’t it be great if SiteRay could tell you, for example, that the strategy for your website was wrong? Maybe, one day, it will…