|
morrowasted
Worldwide Stepper



Registered: 10/30/09
Posts: 31,378
Loc: House of Mirrors
Last seen: 17 days, 17 hours
|
Automated writing exam grading
#19333257 - 12/27/13 02:51 PM (10 years, 1 month ago) |
|
|
So my cousin's husband is doing a PhD in applied linguistics and his dissertation involves writing a computer program that will be used to automatically score the writing portion of the TOEFL exam. He claims that automated systems have actually been shown to have less scoring variation than systems that involve human graders. What do you think about this?
|
andrewmurray86
Θεολογος




Registered: 02/05/13
Posts: 1,120
Loc: Hunter Valley, NSW
Last seen: 6 years, 1 month
|
Re: Automated writing exam grading [Re: morrowasted]
#19333266 - 12/27/13 02:54 PM (10 years, 1 month ago) |
|
|
It would have be an amazingly complex system to understand any human language, especially English, probably one of the more complex languages out there.
|
andrewmurray86
Θεολογος




Registered: 02/05/13
Posts: 1,120
Loc: Hunter Valley, NSW
Last seen: 6 years, 1 month
|
|
I mean, the idiosyncrasies of English, the ability of English to have puns and double entendres... Especially things like poetry and other writing intended to evoke emotions.
I'd be fascinated to see the end product.
|
nooneman


Registered: 04/24/09
Posts: 14,608
Loc: Utah
|
Re: Automated writing exam grading [Re: morrowasted]
#19333300 - 12/27/13 02:59 PM (10 years, 1 month ago) |
|
|
Automated grading systems have been around for a while, and they have known problems. Researchers have made some really cool nonsense papers that have been rigged to cheat automated paper grading systems. They're pure nonsense, but get perfect scores. It's pretty classic.
|
Me_Roy
Stranger
Registered: 07/30/02
Posts: 3,230
|
Re: Automated writing exam grading [Re: nooneman]
#19333312 - 12/27/13 03:03 PM (10 years, 1 month ago) |
|
|
Quote:
nooneman said: Automated grading systems have been around for a while, and they have known problems. Researchers have made some really cool nonsense papers that have been rigged to cheat automated paper grading systems. They're pure nonsense, but get perfect scores. It's pretty classic.
witness drkenny.
|
badchad
Mad Scientist

Registered: 03/02/05
Posts: 13,373
|
Re: Automated writing exam grading [Re: nooneman]
#19333323 - 12/27/13 03:07 PM (10 years, 1 month ago) |
|
|
I'd be cautiously skeptical but think such a system has potential. I don't know anything about the content of the TOEFL, but would imagine an automated system would be most appropriate for an exam format with relatively short, concise, "closed" answers.
-------------------- ...the whole experience is (and is as) a profound piece of knowledge. It is an indellible experience; it is forever known. I have known myself in a way I doubt I would have ever occurred except as it did. Smith, P. Bull. Menninger Clinic (1959) 23:20-27; p. 27. ...most subjects find the experience valuable, some find it frightening, and many say that is it uniquely lovely. Osmond, H. Annals, NY Acad Science (1957) 66:418-434; p.436
|
morrowasted
Worldwide Stepper



Registered: 10/30/09
Posts: 31,378
Loc: House of Mirrors
Last seen: 17 days, 17 hours
|
Re: Automated writing exam grading [Re: badchad]
#19338372 - 12/28/13 07:24 PM (10 years, 1 month ago) |
|
|
Quote:
badchad said: I'd be cautiously skeptical but think such a system has potential. I don't know anything about the content of the TOEFL, but would imagine an automated system would be most appropriate for an exam format with relatively short, concise, "closed" answers.
The TOEFL is just a test for measuring one's ability to use English; it isn't interested in content. The program works, according to my understanding, by measuring the prescriptive grammatical accuracy and lexical complexity of the writing sample. The latter is accomplished by creating ratios; for example, writing sample A contains 5000 words. 500 of those words are "the"; thus 1 in 10 words is "the". 2000 of those words are nouns; 1000 are verbs, etc. 20 of those words contain 4 syllables. 5 words contain 5 syllables. These ratios generate a score of written complexity.
Don't get me wrong, I am also skeptical. I was told that by one of the philosophy professors at my school that the graduate department no longer considers the writing score from the GRE to be relevant because a portion of the grade is generated by an automated system.
|
|