1 of 11

Beyond Open Book Examination

​

The Pros and Cons of Open Internet Programming Examination

Steven Halim

Computer Science

School of Computing

2 of 11

Beyond Open Book Assessment

​

The Pros and Cons of Open Internet Programming Assessment

Steven Halim

Computer Science

School of Computing

3 of 11

Assessments in NUS

Closed Book

Can have questions involving memorizations

Lesser importance with prevalence of Internet

Update on 01 April 2020: Student has to be physically present

Open Book

More encouraged in NUS nowadays

“The answers are not in the book(s)”

Apply, analyze, synthesize, compare/contrast or evaluate-type Questions

Update on 01 April 2020: Student has to be physically present

4 of 11

Beyond Open Book… Open Internet?

“The answers are not (easily found) in the Internet”

What are allowed?

  • Internet access, full? restricted?
  • A computer (laptop), tablet, smartphone?
  • Restricted control software? Junglebyte monitoring, Examplify, etc?�Update on 01 April 2020: Considering Zoom, Proctoru, etc.
  • “Read-only” mode, i.e., can use the Internet to read existing information, but not ask for the answer of our *new* question (in StackOverflow, Quora, secret discussion forum, email each other, etc)

​

5 of 11

Open Internet - Pros

  • Closer to real life (for software engineers)
  • Minor stumbling blocks avoided�(students can Google for hints if their “initial resources” are not enough)
  • For the examiner: (Slightly) easier to setup (SoC is facing ‘lab space issue’)�Update on 25 March 2020: Solve the issue of no physical meeting due to COVID-19�Update on 01 April 2020: Maybe the solution for e-exam due to COVID-19?
  • Misc 1: Save paper (by not printing lecture/tutorial/lab notes etc)
  • Misc 2: Familiarity with own laptop (not always allowed)

Source: Summary of Week 07+Week 12 feedback out of ~120 students

6 of 11

Open Internet - Cons

  • Too hard to fully supervise 100%, a question of assessment fairness/integrity�Update on 01 April 2020: There are more such tools available worldwide due to COVID-19, e.g., https://www.proctoru.com/ (to be tested, not a free tool)
  • The questions become naturally harder (than an open book assessment)�Update on 01 April 2020: Some lecturers purposely increase number of questions or change the level of questions so that it is hard even for the potential A+ students to complete the paper in ≤ 2 hours, making it near impossible for them to “help” others, then use bell curve to rank students
  • For inexperienced ones, Googling (too much) wastes their (assessment) time
  • For the examiner: Has to use special software for invigilating the assessment
  • For the examiner: Need to be even more creative than in open book format
  • New in 2023: The presence of Chat-GPT (and its variants) makes this Open Internet format dangerous

Source: Summary of Week 07+Week 12 feedback out of ~120 students

7 of 11

My Experiments this Sem 1 AY2019/20 (1)

CS2040C Practical Exam (PE): PE1, re-PE1, PE2, re-PE2 (96 students, weightage 21% of grade)

Format: Open Internet, 2 hours, individual, easy but not trivially Google-able problems, in lab

Precautions: Question paper printed and given onsite (not public), Junglebyte monitoring tool used, (Kattis) Online Judge used to pairwise compare every student submissions�Update on 14 October 2020: Due to e-classes, question paper will be public from the start this time

8 of 11

My Experiments this Sem 1 AY2019/20 (2)

CS4234 Mini Project (30 students, weightage 15% of grade)

Format: Open Internet, 3 weeks, project group of 4 students/group, NP-hard problems

Precautions: “None”, the problems themselves are the safeguards, no one on earth knows how to solve them efficiently unless P == NP (an open problem that still bothering Computer Scientists)

9 of 11

My Experiments this Sem 2 AY2019/20 (2)

CS3233 Open Internet Final Team Contest (12 students, weightage 13.5% of grade)

Format: Open Internet, 5 hours, team of 3 students/team, (very) hard problems

Precautions: e-proctoring via Zoom and the inherent difficulty of the problem set itself

​

​

Slide added on 02 June 2020

​

Red highlight: The 4x3 = 12 official students

​

​

​

​

10 of 11

The Jury is Still Already Out

  • More experiments still needed
  • Both successful (and failed) experiments are reported/shared
  • Failed experiments (within acceptable bounds) should not attract (career) penalties…
  • The presence of Chat-GPT makes this format dangerous

11 of 11

Thanks

​

Any Questions?