Computers To Mark English Essays 243
digitig writes "According to The Guardian, computers are to be used in the UK to mark English examination essays. 'Pearson, the American-based parent company of Edexcel, is to use computers to "read" and assess essays for international English tests in a move that has fueled speculation that GCSEs and A-levels will be next. ... Pearson claims this will be more accurate than human marking.' Can computers now understand all the subtle nuances of language, or are people going to have to learn an especially bland form of English to pass exams?"
Context... (Score:3, Interesting)
"Time flies like the wind, fruit flies like a banana." -- Groucho Marx
This is a classic example of context which a machine would fail to get. :)
I would like to see an automated engine figure that one out.
GC
Re:Context... (Score:2, Interesting)
I am sure it was hilarious when Groucho delivered that line, spoken. As written? Meh.
Depressing (Score:4, Interesting)
Re:I doubt it! (Score:5, Interesting)
As an English prof myself, I'd like to confirm that we spend a lot of time on students' papers. Good papers are easy to breeze through, but the worse the paper, the more time it takes.
As for machine-grading goes, people have been working on that for 30 years. I have no doubt that, statistically, it can provide useful results.
The problem I'm seeing in these comments, however, is a common confusion of testing for assessment and standardized testing. I can't imagine using software to grade a student's paper in class. The student-teacher relationship is a personal one. That person is paying me to help them get better at writing, for example. It is my job to pore over that paper and show them where and how they can improve.
I am also a tester (I actually mostly work with multiple-choice data, but I've also worked on performance rating--speaking and writing). The relationship between a rater and an examinee is very different from that of a teacher and student. The examinee is paying the rater to put them on a scale with other people. This is not a fine-grained assessment; it is always done at extremely "low resolution." When rating a paper for something like the GRE or other standardized test, it is the rater's job to compare the paper to scoring rubrics and make a call on which box of text best describes the paper, and then make note of the number in that box. That's it. It can't really go any more in-depth than that.
For this reason, your comment about "five-paragraph themes" is an important one: Test task design always needs to be clear about what kind of performance is expected, because it is nigh impossible to write rubrics that can be applied to any performance (believe me on this, I beg of you). However, this is actually a question of test specification, not of the software or raters in question. Personally, as someone who works in EFL, I am actually in favor of retaining the "five-paragraph" formula, at least for timed essay tasks. That format is at the heart of all good rhetoric. Yes, it's stilted and silly, but if you can do it, it means that you know basically how information is expected to be organized in Western, especially Anglophone, societies. No good writer would actually use it, but any good writer could.
Again, this is about putting people in boxes, not reading their essays. I can rate a 1-page essay in about 2 minutes, with excellent model fit (I have always used many-facet Rasch modeling for my multi-rater performance testing). I have no doubt that software could be employed whose ratings would be highly predictive of those of human raters.
Re:Graduate Record Exam (Score:3, Interesting)
It's a stretch to say that thereby the computerised programme marks the essay, or even that it takes a direct part in the actual marking of the essay (that is, in creating the mark which is given to the examinee). The programme really marks the human marker in that scenario, if it marks anything at all.
Re:Judging from... (Score:3, Interesting)
Actually the last time I did any serious writing in a word processor (at least two years ago), I found that enabling inline grammar checking and setting it to the strictest mode did tend to improve my writing. There were a few exceptions (it can never seen to decide between affect and effect), and while the suggestions weren't always great, it seemed to catch errors in syntax and structure often enough that I could go back and overall improve the writing.
That being said, it's certainly not foolproof and absolutely not ready to replace a human - let alone a trained English teacher. I'm sure it could catch papers that ought to fail miserably with relative ease, but once you get into papers that would get probably a C or better, it's time for something with a brain to take over.
I do not have the same experience (Score:2, Interesting)
No and no (Score:5, Interesting)
It's worth pointing out that certain types of exams are designed to elicit extraordinary prose from respondents, that which yields a sense of competence or even brilliance, say. In these cases, the idea is not so much to detect the high end of the bell curve, but to identify the tiny pool of applicants who may be capable of Nobel Prize work in future realms of science or service. No 'bot can do that job, just as no 'bot except Deep Blue can beat Gary Kasparov, and no 'bot at all deserves the monicker Fujiwara no Sai (although Go-playing 'bots are approaching the mid-levels of highly ranked amateur players).
That's the objective part. My personal opinion is that using robots to sort the hopes and aspirations of college-bound men and women is just begging for lawsuits. It's an approach in which differences of opinion quickly escalate to class action against universities as well as test administrators, and would not be an approach I could comfortably recommend.
How will it mark this poem ? (Score:5, Interesting)
My New Spell Checker
Eye halve a spelling chequer
It came with my pea sea
It plainly Marx four my revue
Miss steaks eye kin knot sea
Eye strike a key and type a word
And weight four it two say
Weather eye am wrong oar write
It shows me strait a weigh
As soon as a mist ache is maid
It nose bee fore two long
And eye can put the error rite
Its rare lea ever wrong
Eye have run this poem threw it
I am shore your pleased two no
Its letter perfect awl the weigh
My chequer tolled me sew
(Sauce unknown)
Re:Graduate Record Exam (Score:4, Interesting)
It also scores great writing and even greater speaking very inconsistently.
When fed Kennedy's "I am a Berliner" speech these systems always scored it rather low. Repetitious. Gratuitous use of foreign words: Ich bin ein Berliner.
Re:Graduate Record Exam (Score:2, Interesting)
Feed it real Shakespeare and watch it grade him an imbecile with poor grammar.
Re:The beginnings of Newspeak (Score:3, Interesting)
So, no, this is not the beginning of real-world Newspeak. The beginning was over half a century ago.
Re:Don't they already do this? (Score:2, Interesting)
As to the argument that writing mistakes and errors correlate with poor quality writing, I can agree to a certain extent. If the examinee is a native English speaker, it may well hold true in the majority of cases. But if English is their second - or a foreign - language, there is a much weaker correlation.
Language register (degrees of formality) is important in these examinations, especially the IGCSE and AS level English. There is also an important differentiation between grammar (the basic rules of language) and structure (putting elements together using appropriate linking words and punctuation). Good structure possesses the quality that linguists call cohesion. Nowadays examinations tend to be less strict about grammar, and place more emphasis on the command of structure.
My own take on machine marking of English composition may be summed up in two words - utter bollocks.
Re:Graduate Record Exam (Score:4, Interesting)
Re:Especially bland form of English, a bad thing? (Score:2, Interesting)
Re:Don't they already do this? (Score:2, Interesting)
Formal written language is different from casual spoken language in terms of grammar, syntax, vocabulary. That's true in any literate language. So, for the purpose of writing an English paper (or some of the things it prepares you for: a newspaper article, a grant proposal, a cover letter, etc.) some of the things that you are saying "aren't actually mistakes" are actually mistakes.
Re:Especially bland form of English, a bad thing? (Score:2, Interesting)
Besides constructed languages, this is the case for practically every language there is. There are always irregularities; this is down to the inherently human nature of linguistic evolution. If you learn English without a single irregularity, what you have learned is not really English, but some other English-derived language which English speakers will be unlikely to understand at all - at which point, you may as well force everyone to learn Esperanto.
I also rather doubt that getting rid of odd past tense forms would really make learning English a great deal easier.
Re:Don't they already do this? (Score:2, Interesting)