Created
November 11, 2020 00:34
-
-
Save sko9370/e8cfd6edc88f9ad547b0b61170b69c59 to your computer and use it in GitHub Desktop.
Revisions
-
Sang Oh created this gist
Nov 11, 2020 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,140 @@ <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Welcome file</title> <link rel="stylesheet" href="https://stackedit.io/style.css" /> </head> <body class="stackedit"> <div class="stackedit__html"><h1 id="topic-bias-of-item-analysis-system-on-grades">Topic: Bias of item analysis system on grades</h1> <h2 id="situation">Situation</h2> <ul> <li>Bad questions ranging from typos to question-answer mismatch are picked</li> <li>Students with those bad questions are refunded points if they got the wrong answer</li> </ul> <h2 id="problem">Problem</h2> <ul> <li>This system favors students that receive a high number of bad questions</li> <li>This in turn hurts students that receive a low number of bad questions</li> </ul> <h2 id="explanation">Explanation</h2> <ul> <li>Optimistic assumption that a student would have gotten a good question right, even if they got a bad question wrong</li> <li>Example: 2 students take the exam, one exam contains 5 bad questions, the other contains none <ul> <li>Scores <ul> <li>Student 1 (5 bad questions): 45/50, only bad questions were incorrect <ul> <li>actual score after item analysis: 50/50</li> </ul> </li> <li>Student 2 (0 bad questions): 50/50 <ul> <li>actual score after item analysis: 50/50</li> </ul> </li> </ul> </li> <li>Because Student 1 had 5 bad questions, the two students end up with the same score. However, this assumes that Student 1 would have answered 5 additional (good) questions correctly. Student 1 gains points based on an unfounded assumption whereas Student 2 has clearly demonstrated correct answers to earn the points.</li> <li>This bias strongly favors Student 1 because if they had not been able to answer an additional 5 (good) questions correctly, they are getting more points than they deserve.</li> <li>In other words, Students 1 and 2 got the same score through item analysis even though Student 2 was able to answer 50 good questions and Student 1 was only able to answer 45 good questions.</li> </ul> </li> </ul> <h2 id="alternative-to-current-system">Alternative to Current System</h2> <ul> <li>Possible Solution 1: give every (5 bad questions) student that had a bad question an additional point, regardless of if they got it correct or not <ul> <li>This may unfairly reward students that may have randomly guessed the correct answer for a bad question</li> </ul> </li> <li>Possible Solution 2: subtract any bad questions from both the number of correct answers and total questions, regardless of if they got it correct or not <ul> <li>Although a reasonable solution, this is not fair because of the different levels of severity that a question could be “bad.” A question with a simple spelling error where the meaning could be derived by a reasonable person should be easier than not to answer correctly, given the student were able to answer it correctly without the spelling error. However, some questions were worse or ambiguous to the point where a reasonable person would not be able to derive the intended meaning of the question.</li> <li>This solution would be benefit those that understood the meaning of the question and got it wrong simply because they did not have the knowledge while penalizing those that understood the meaning of the question and got it right.</li> </ul> </li> <li>Possible Solution 3: <strong>Recommended</strong> <ul> <li>If a student get a bad question wrong, remove the question from the exam so that it is neither counted as wrong or correct.</li> <li>Example <ul> <li>A student gets a score of 45/50 with 5 bad questions wrong. <ul> <li>Actual score = 45/(50 - 5 bad questions wrong) = 45/45</li> </ul> </li> <li>A student gets a score of 48/50 with 2 out of 5 bad questions wrong. <ul> <li>Actual score = 48/(50 - 2 bad questions wrong) = 48/48</li> </ul> </li> </ul> </li> <li>This solution is a better alternative to the current system because of the wide range of bad questions on different students’ exams.</li> <li>Example: Current System, one exam has 10 bad questions, the other has 5 <ul> <li>Student 1 (10 bad questions): 43/50, 2 good and 5 bad questions wrong <ul> <li>Actual score = (43 + 5 bad questions wrong)/50 = 48/50 = <strong>96%</strong></li> <li>Score of right questions = (50 - 10 bad questions - 2 good questions)/(50 - 10 bad questions) = 38/40 = <strong>95%</strong></li> <li>Difference of raw vs actual score = <strong>+1%</strong></li> </ul> </li> <li>Student 2 (5 bad questions): 48/50, 2 good questions wrong <ul> <li>Actual score = 48/50 = <strong>96%</strong></li> <li>Score of right questions = (50 - 5 bad questions - 2 good questions)/(50 - 5 bad questions) = 43/45 = <strong>95.5%</strong></li> <li>Difference of raw vs actual score = <strong>+0.5%</strong></li> </ul> </li> <li>System Bias for student with more bad questions <ul> <li>1% - 0.5% = <strong>0.5%</strong></li> </ul> </li> </ul> </li> <li>Example: Recommended System, same exams <ul> <li>Student 1 (10 bad questions): 43/50, 2 good and 5 bad questions wrong <ul> <li>Actual score = 43/(50 - 5 bad questions wrong) = 43/45 = <strong>95.5%</strong></li> <li>Difference of raw vs actual score = <strong>0.5%</strong></li> </ul> </li> <li>Student 2 (5 bad questions): 48/50, 2 good questions wrong <ul> <li>Actual score = 48/50 = <strong>96%</strong></li> <li>Difference of raw vs actual score = <strong>0.5%</strong></li> </ul> </li> <li>System Bias for student with more bad questions <ul> <li><strong>0%</strong></li> </ul> </li> </ul> </li> </ul> </li> </ul> <h2 id="conclusion">Conclusion</h2> <ul> <li>Given… <ul> <li>range of severity of bad questions</li> <li>range of numbers of bad questions for different students</li> <li>current system bias for students with more bad questions</li> </ul> </li> <li>I recommend Solution 3 for use in applying item analysis for scores for the final exam to promote fairness and accuracy of assessment scores.</li> </ul> </div> </body> </html>
Sang Oh
created
this gist