As Remote Schooling Persists, Bots Are Grading Homework - And They're Often Wrong

A 'wrong' answer might be because a student wrote '3' instead of 'three.' 

Software providers are blaming teachers for the problems but the reality is that the product is not yet intelligent enough to discern the subtleties of student performance. It is yet another example of technology promoting itself as a time-saving answer which turns out to be counterproductive and destructive in the current situation. JL

Julie Jargon reports in the Wall Street Journal:

Lower-than-expected grades might be less an accurate reflection of  children’s work than a glitch in the automated systems many schools are using to check schoolwork. The root of the failure is in the simplistic design of many grading bots: They try to match a student’s response to a teacher’s answer key. If the two aren’t identical, the response is often marked wrong, even if a human could easily see it’s correct. It is yet another headache for teachers, parents and students brought on by the sudden shift to remote learning.

Some parents might be in for an unpleasant surprise when report cards come out this fall: lower-than-expected grades. But that might be less an accurate reflection of their children’s work than a glitch in the automated systems many schools are using to check schoolwork.

Unless parents and teachers review students’ tests, the problem can be easy to miss. I discovered the problem only when my fifth-grader bombed a science test for which I knew he was prepared. I opened the test online to see what he had missed and found some of his typed-in answers were correct but marked wrong. The “error”? They contained capital letters. I figured I’d better go through every test he had taken, and when I did I found several more examples.

Auto-grading promised to save teachers time but now has many of them spending hours poring over answers to their students’ every assessment and exam—if they’re aware of the grading problems at all.

In some instances, the Canvas learning platform has flagged responses as incorrect due to capitalization discrepancies.

PHOTO: JULIE JARGON/THE WALL STREET JOURNAL/INSTRUCTURE, INC.

The root of the failure is in the simplistic design of many grading bots: They try to match a student’s response to a teacher’s answer key. If the two aren’t identical, the response is often marked wrong, even if a human could easily see it’s correct. It is yet another headache for teachers, parents and students brought on by the sudden shift to remote learning.

The makers of such educational software acknowledge the problems with auto-grading and say many teachers didn’t have enough time to properly train on how to use the online platforms.

Cristina Perez, a single mother in Aurora, Colo., said her three children have become dispirited by the whole remote-school experience, including receiving numerous incorrectly graded tests in the Canvas Learning Management Platform, the same virtual classroom my children use in California.

“All of my kids were ‘B’ students, and now they’re failing in most classes. That’s how bad it is,” said Ms. Perez, whose children are in seventh, eighth and 10th grades. “What does that do to a kid’s mental health and their willingness to even try?”

During school hours, Ms. Perez works delivering car parts and said she finds it difficult to reach teachers. Each of her children has seven teachers. She said she has emailed some of them about the problems but the test scores haven’t been corrected.

“I don’t have time to be sitting there monitoring all of this,” she said. “I got to a point where I told my kids to just log in and do what they can. They’re stressed enough as it is.”

Some students’ responses to questions in Canvas are automatically marked incorrect because they don’t match a teacher’s exact wording. Teachers have manually adjusted grades in such instances.

PHOTO: JULIE JARGON/THE WALL STREET JOURNAL/INSTRUCTURE, INC.

Patrick Lytle, who runs a design firm in Highland Park, Ill., had better luck when his fifth-grade daughter told him she got an answer wrong on a math quiz in the online platform Otus. Her teacher had instructed the students to include units in their answers, and she correctly replied “3 grams” to one question. However, the auto-grader marked it wrong and showed the correct answer as simply “3.” Mr. Lytle emailed the teacher about it and she corrected the girl’s score.

Mitch Benson, chief product officer at Instructure Inc., the Utah-based educational software company that makes Canvas, said the company hears about problems with its auto-grading feature frequently. In the rush to virtual learning, he said, many school districts and universities didn’t have enough time to thoroughly train teachers on all aspects of Canvas, which bills itself as a one-stop shop for distance learning.

“Teachers are having to use a tool they’re not intimately familiar with, and it poses a bunch of new challenges,” Mr. Benson said.

Teachers set up their own virtual classrooms in Canvas, where students can log in to video meetings, access educational apps, receive and submit assignments and take tests. In an older version of the quiz feature—which Instructure said about 75% of its 6,000 district and university customers are still using—students must answer questions exactly as the teacher input them. About 18 months ago, Canvas began rolling out an updated quiz feature with an algorithm that can detect when an answer is close enough.

Mr. Benson said it has been hard to keep up with all the issues during the rapid growth in business. The peak number of concurrent Canvas users has risen from 2 million in April to 6.5 million in September. Customer-support calls have risen alongside usage, reaching more than 200,000 help requests last month.

Mr. Benson said the company aims to make teachers’ jobs easier. “I’m apologetic for the areas where we’ve fallen down a bit,” he said.

Keith Westman, chief operating officer of Illinois-based Otus, used in 170 K-12 school districts, also said auto-grading problems aren’t a tech issue but a teacher-training issue.

When teachers design tests in Otus, they can include several alternate answers. For example, if the correct answer to a math question is “3 grams,” a teacher can designate several answers that will be recognized as correct, such as “three grams,” or just “3.” The teacher can choose whether to give the student a full or half point for a given answer and can require the answer to be an exact match, or contain part of the correct text.

If teachers don’t like auto-grading, they can disable it altogether, Mr. Westman said. “Auto-grading is great when it works, but when it doesn’t work, it can be devastating to a student,” he said.

The platform Otus, used in 170 K-12 school districts, lets teachers include several alternate answers. The company has said auto-grading problems aren’t a tech issue but a teacher-training issue.

PHOTO: OTUS

Kristin Beamish-Brown, a senior teaching specialist at the University of Minnesota, Twin Cities, deals with devastation every day from remote students in her third-semester Spanish class. Even though she warns them that the automatic grades they see in Canvas might be incorrect, she often hears from them before she can double-check the auto scores.

“I have 71 students. It takes me hours to grade what was supposed to be an automatically graded quiz,” she said. “Not only do I have to go back in and check their grades, I have to respond to 50 students who are emailing me worrying about their grades.”

Ms. Beamish-Brown said she has started hiding the grades from students until she can review all the answers. When the auto-grader marks something wrong, she finds herself reading the answer three or four times to be sure.

“I’m second-guessing myself because the machine has said it’s wrong,” she said. “So much for technology making our lives easier.”