New Delhi, May 30: Say “C” for examiner.

Researchers at the Indian Institute of Information Technology (IIIT), Hyderabad, have developed an automated evaluation system that makes it possible for computers to assess “long and subjective-type” answers.

Till now, automated evaluation has been limited to assessing only objective-type answers.

The technology, which uses the Natural Language Processing (NLP) method to understand if a sentence makes sense, is capable of assessing essay-type answers within a few seconds with up to 99 per cent accuracy.

“This new technology is unique as it will assess long and subjective answers. It has been put to trial test and found to be successful. The marks or grades granted under automated evaluation is 95 to 99 per cent close to the marks or grades granted by human evaluation,” Vasudeva Verma, a member of the computer science faculty at IIIT Hyderabad who developed the technology with 20 researchers, told The Telegraph.

Evaluating subjective-type questions through computers has always been a challenge. The reason is computers fail to read and comprehend texts, Verma said. With the new technology, however, computers will be able to read and understand typed texts.

After understanding the texts, the computers will compare their “essence” with certain model answers and award marks accordingly.

“Instead of just matching the answer by words, the technology will help computers get the essence of the essay-type answers. The computers will have certain model answers (programmed in). They will then award marks or grades on the basis of how the essence conveyed in the answers is close to the model answers,” Verma said.

But a key part of evaluating content is syntax. While computers do have tools to correct spelling and grammar, they cannot differentiate between a sentence that makes sense and one that doesn’t. The NLP method takes care of this problem.

“(Using this method), the computer can pinpoint wrong words and meaningless sentences. If a text has a sentence like ‘banana ate my room’, the computer will recognise the fault in the sentence,” Verma said.

Verma said the technology had been tested on a trial basis at the Enhanced Education Centre of IIIT Hyderabad and been found “95 to 99 per cent” accurate.

“The technology cannot give 100 per cent accuracy in evaluation. But there is a very high degree of correlation with human evaluation. Its agreement with human evaluation is 95 to 99 per cent,” he said.

A big plus-point of the technology is that if it fails to comprehend a sentence, it will flash a question mark, seeking human intervention.

Verma said the technology had been commercialised recently by Setu Software, an incubating company of IIIT Hyderabad, which has approached several universities and school boards to adopt the method.

“This technology will save time, cost and effort. If you use multiple computers, the answer papers of lakhs of students can be assessed in a day and the results can be published in a short time,” Verma said.

He said school boards spend between Rs 100 and Rs 500 to evaluate a subjective answer paper and this cost could be reduced by using the new technology. Universities offering distance learning can also use this method for timely publication of results, Verma added.

Verma said the researchers took seven years to develop the technology, which will be made available in other Indian languages too.

An HRD ministry official said the development was a “significant step forward” in India’s plans to launch a virtual university in the next two years. “For starting a virtual university, we need to develop a system under which we can hold examinations online, evaluate automatically and publish results quickly. This automated evaluation technology is a significant step forward in starting a virtual university,” the official said.

The researchers at IIIT Hyderabad are now working on how the automated evaluation technology can be used to assess handwritten, essay-type answers. “It is possible to translate handwritten text to digital format. Now we are exploring whether handwritten answer papers can be translated to digital format and evaluated by this technology,” Verma said.