Grading Graders

Your local school district 🏫 needs your help. They issue a standardized test to compare students across all schools in the district. Most subjects (Math, English, Science, ...) are multiple choice, so it's easy to compare student performance. However, in the Writing portion of the test, each student writes an essay which is manually graded by a small group of graders.

Notes

• Essays are randomly partitioned and assigned to each grader to grade.

• Graders score each essay on the scale [0, 100].

• Scores are then mapped to letter grades (A, B, C, D, F) using the following percentiles

  Grade	Percentile
  A	≥ 90%
  B	[70%, 90%)
  C	[30%, 70%)
  D	[10%, 30%)
  F	< 10%

  This mapping is applied per grader. In other words, each grader assigns the same number of As, Bs, etc.

It's really important that essays are graded in a consistent manner. If grader 1 gives student 193828 an F, we expect that graders 2, 3, 4, and 5 would also give that student an F.

So, prior to the test, you dig up 50 essays from last year's test and tell the graders to score them. Their scores are given in this CSV file.

| grader_id | student_id | score |
|----------:|-----------:|------:|
|         1 |       6648 | 35.45 |
|         1 |      29337 |  5.45 |
|         1 |      38658 |  8.86 |
|         1 |      44058 |  4.03 |
|         1 |      54112 | 87.54 |
|         1 |      70645 | 82.98 |
|       ... |        ... |   ... |
|         5 |     890187 | 98.70 |
|         5 |     915496 | 40.68 |
|         5 |     946355 | 67.44 |
|         5 |     949135 | 87.37 |
|         5 |     951898 | 23.33 |
|         5 |     956616 | 66.87 |

Design a tool that evaluates how much each grader conforms to the group.

If you had to fire one grader, which one would you fire?

Loading the data¶

You can load the data directly from GitHub.

import pandas as pd
grades = pd.read_csv("https://raw.githubusercontent.com/practiceprobs/datasets/main/essay-grades/grades.csv")

library(data.table)
grades <- fread("https://raw.githubusercontent.com/practiceprobs/datasets/main/essay-grades/grades.csv")

Solution¶