Project Summary

Marking and grading ‘essay type’ questions take about a third of the time of teachers. The cost of this in economic terms is very large. This has provided the motivation for the development of an automated essay grading system. Whilst some automated essay grading systems have appeared both as commercial systems and research prototypes, they have had limited success and they mainly focus on mechanical and surface features. We propose a set of techniques using chunking of text to capture semantic features and sophisticated data mining techniques to help model the relationship between these features and the essay grade.

Summary of National/Community Benefit

Making and grading essays is estimated to cost around AUD$2 Billion in Australia each year. It takes up approximately a third of the teachers’ time. An Automated Essay Grading System would mean significant financial savings in the education sector both public and private. It would also free up teachers to concentrate on higher level pedagogical tasks resulting in a better quality of education. The potential for sales overseas for such a system are also great.

Curtin has developed an AEG system, currently known as MarkIT (essaygrading.com) - and has tested it with 390 Year 10 essays provided by WA-DET. The topic of these essays is “The School Leaving Age”. The inter-rater reliability for human scores on the test essays was about 0.80, and MarkIT matched this performance against the average scores for the three human graders who graded each test essay. This encouraging outcome has led the research team to take the position that the MarkIT system can be used as a suitable base system from which the WA-DET system can be developed.

It is recommended that the initial development of the system focus on building a system for the Year 9 argumentative essays. The original trial of the existing system generated a positive outcome for the Year 10 essays, and it is estimated that minimal changes to the system are needed to produce a satisfactory outcome for Year 9 essays. It should be kept in mind that the total score for the essays, calculated from the sum of the scores for individual traits, is the score predicted by the system. Prediction of the individual traits will be subject to trial and error, but it is expected that prediction of On-Balance-Judgement