— I am in charge of managing the overall AI infrastructure for the automatic grading. Hi guys, this is Brian from TOEFLBANK

Speaking of AI, I’m sure most of you who have used our service realized at one point that your writing responses are being graded by a machine. Now, some of you might have thought that’s a bit weird, and for some of you who are interested in the field of deep learning might not have found it so surprising.

But regardless of where you stand on the topic of AI, the truth of the matter is that ETS is also using some form of automated scoring. That’s right, when you walk into an ETS testing center today to take the TOEFL Exam, your writing and speaking responses are partly graded by a machine. So, for those of you who feel uneasy about this whole “AI is grading my essay” thing, we want to let you know that this is real.

However, the way we do it here at TOEFL BANK is relatively new and different — so I’m sure you have questions on how we grade your essays, so I wanted to take this time to answer some questions you sent us about our AI.

Question #1 — Why use AI for scoring in the first place?

Awesome question — now, the key to getting better in writing and speaking, not just in TOEFL, but in language acquisition in general, is repetition. However, if you simply repeat without any feedback or guidance, you are wasting your time. You need someone or, in this case, something, to tell you that you are going in the right direction. Conventionally, that someone is a teacher at your school, or a tutor at your prep school. But think about the last time you had to wait for their feedback. They don’t respond so well during weekends right? What we aim to do in TOEFL BANK is having an AI provide feedback for you so that you can request for grades without having to worry about time or location. I’m in South Korea, and based on our user-data, most of you are not here and we are not even in the same time-zone, but you still use our service right? It’s already working out!

Question #2 — How does this work and how is this different?

Alright — before I answer this question I’d like to give you a brief overview on how this whole automated scoring thing started. Basically, we’re not the first guys to try this out — BUT, we are the first guys to use deep learning algorithms for automated scoring service — then how is deep learning different? — Previously, everything was about rules. You have 2 spelling errors, 3 spelling errors, and those counts worked as a deduction against your grade. This already sounds like a painful process doesn’t it? Think about how many factors work together interactively to grade your essay. And plus, spelling and grammar are simple surface level features. How is the machine going to understand abstract concepts like logic or context? The answer is, you can’t, and that’s why these rule-based systems fail so miserably in natural language understanding.

At TOEFLBANK, we use ZERO rules. Everything is about representation. Your essay is broken down to text tokens. Text tokens are transformed to context based representations that a computer can understand. Next, the computer will look at your essay representation — and a score assigned by a professional human grader, and start associating your essay with the given score. With enough data and training, the machine will be able to grade your essay without the supervision of a human grader. Of course, since it is learning from a human pattern, it also picks up on biases.

For example, same grader, scoring on Monday, scoring on Friday, can you guess which day they give higher grades? That’s why while we emphasize adopting new models and algorithms, our main focus is data quality. Without clean data, algorithms are useless. Our human scorers are fully trained and experienced TOEFL professionals, and our AI is properly learning from their scoring pattern. Our AI is learning every single day — which means we are continuing to improve every day.

Question #3 — I made corrections on my essay but why has my score not changed?

Well, I can assure you if you have done the right corrections, your scores would have gotten better. However, there is a threshold for scores to change. The AI itself is spitting out scores in numbers with decimals. So, when we look at the decimal points, your score might have improved, but when the AI score is being converted to an ETS score, there are some rounding procedures involved — and that’s why you might end up with the same converted score.

Then you might ask “if AI scores are more precise than the converted score, why not give the AI score in the first place?” AI grading is fun and everything, but at the end of the day we want you to do well in TOEFL. Let’s not forget why we are doing this in the first place. All scores and grading layouts will follow what you would normally see at the actual TOEFL examination to make sure you are prepared for the real test.

Alright, so I hope this post properly answered some of the questions you had about our AI service. If you have any other questions — please feel free to let us know, and we will do our very best to get back to you as soon as possible. Also, as mentioned earlier, our speaking autograder is coming out soon. So once it is out, please try it out. Feedback and repetition are key in achieving your target score. So, good luck, and keep on practicing!

Source: https://www.toeflbank.com/ko/blog/TOEFLBANK-Writing-Test-AI-Grading

#engineering #ai #tech #toeflbank