Optical Character Recognition Still Struggling With Handwriting

Ian Lamont recently asked Google if they planned to extend their transcription of books and other printed media to include public records, many of which were handwritten before word processors became ubiquitous. Google wouldn't talk about any potential plans, but Lamont found out a bit more about the limits of optical character recognition in the process: "Even though some CAPTCHA schemes have been cracked in the past year, a far more difficult challenge lies in using software to recognize handwritten text. Optical character recognition has been used for years to convert printed documents into text data, but the enormous variation in handwriting styles has thwarted large-scale OCR imports of handwritten public documents and historical records. Ancestry.com took a surprising approach to digitizing and converting all publicly released US census records from 1790 to 1930: It contracted the job to Chinese firms whose staff manually transcribed the names and other information. The Chinese staff are specially trained to read the cursive and other handwriting styles from digitized paper records and microfilm. The task is ongoing with other handwritten records, at a cost of approximately $10 million per year, the company's CEO says."

Better approach?

[mandelbr0t]

It seems to me that it would be better to OCR everything and contract the proof-reading to the Chinese firm. The wide variation of writing styles and letter forms may make 100% accuracy of OCR impossible for this task, but starting from OCR should reduce the task, shouldn't it?

Half the time..

[Miststlkr]

I can't even read people's handwriting, I hardly expect a computer to.

general OCR harder than CAPTCHA OCR

[Anonymous Coward]

There is a simple reason that general OCR is much harder than cracking a CAPTCHA. General OCR has to recognize text *reliably*. CAPTCHA breakers are thrilled with a 10% success rate, because they use distributed systems created by worms to do the hard work a million times over. If you got 10% of the words right when scanning historical records you might as well not bother.

Re:Too variable, less reference

[Kickersny.com]

While handheld technology is indeed getting better, it's not directly applicable to the problem at hand. Real-time handwriting analysis uses stroke analysis as well as shape analysis to determine the letter(s). That is, the order in which you construct your letters matters very much. For example, if you crossed your T before drawing the vertical bar, the engine may have a difficult time figuring out what you intended.

When OCRing documents, all of that 'meta-information' is lost.

Now you have a training dataset.

[bigattichouse]

Now you take the human translated recognition, and use it to train your genetic algo or neural net against the original images.

Re:Half the time..

[glwtta]

Hell, I can't even read my own handwriting. Yeah, this is probably not going to happen.

Re:Better approach?

[mi]

I expect we'll have the capability in the next decade or so as processor core density, memory, and storage continues to increase at their current rate -- eventually, the machine will be able to "brute-force" through the docs just like the Chinese data entry folks in this article.

In the next decade or so we will have increased our processing power about 1000 times over. This work is scalable "sideways" — two pages can be processed by two computers independently. Which means, a thousand of today's computers could've done the work @home-style.

The problem is not with the processing power — it is the lack of algorithms. You and I reassemble the hand-written characters quite differently from how today's computers do it. The software will need to be created — and it is not the lack of CPU/memory/storage power, that's holding it.

One thing for sure is that the new algorithms will need to use the spell-checking engine(s) to better guess, what the next letter might be. On top of that, they would need to be equipped with grammar-checkers too, to be able to guess the next word, however illegible. Human speech (and thus writing) is quite redundant often — even if a misplaced coma can reverse the meaning on occasion.

Our brain certainly uses its knowledge of both the general rules of the language and that of the domain of what's written — this is why another doctor can decipher another doctor's handwriting, for example, that's infamously illegible to mere mortals. The software will have to do the same — and it can start doing it already.

Re:Better approach?

[fyoder]

Teaching someone English at that level would be more difficult that teaching them to recognize characters. In ancient Rome the people who engraved dies for coins weren't always literate, but they managed for the most part to get the inscriptions right. Barbarians who made copies had more trouble, but then perhaps they thought the inscription part was purely decorative allowing for artistic interpretation. Or perhaps they weren't flogged for making mistakes. Point is, you can copy without having the high level of literacy required for proof reading.

lecture notes

[emj]

Have you ever taken lecture notes?