Background
Over the past 15–20 years many leading Schools of Medicine, Healthcare Sciences, and Healthcare Recruitment Bodies have slowly completed the digital transformation of their traditional paper intensive OSCEs (Objective Structured Clinical Examinations) and MMIs (Multiple Mini Interviews). This brought some major benefits to exam coordinators including the reduction in human error associated with marking and scoring paper scoresheets, minimising administrative costs, real-time results, monitoring and psychometric analysis whilst the exam is ongoing.
In 2007, The College of Medicine, Nursing and Health Sciences at the National University of Ireland, Galway made the same transition when a web based solution, OMIS (OSCE Management Information System), was developed in-house at the department of Medical Information and Medical Education (MIME) lead by Dr. Thomas Kropmans senior lecturer in medical informatics. The school adopted the web application for the majority of their examinations. In 2008, during the financial crisis, Qpercom was born to commercialise the software now named Observe.
Pandemic
When COVID-19 hit, campuses closed, exams were postponed and students were sent home. Medical educators and exam coordinators had to find alternative solutions to carry out assessments remotely while continuing to use the existing assessment software package. There were (and still are) great video conferencing tools out there, Google Meet, MS Teams, just to name a few but there was little in the form of an integrated solution for the logistics, scheduling and video problem. We listened to our clients’ concerns, deviated from our product development roadmap and reverted back to design thinking with all hands on the pump to come up with a solution.
Solution
Our first port of call was to understand the What before getting too carried away with the How at an early stage from requirements elicitation to software design. It was a valuable lesson learned from our IT Consultant Stephen B. Morris (also independent writer) who has worked with us on previous projects and has stressed the importance of sticking to this basis rule.
At the design phase, we tried to replicate the physical OSCE setting as much as we could while minimising changes to user flow on the existing assessment component, particularly for the examiner role. The only enhancements we made here was to include embedded video calls, notifications and exam status messages. We developed a new student and actor dashboard to allow them to participate in the online exam which wasn’t previously possible. Actors would be allocated their stations and students sent to a waiting area ready to be invited into the station room by examiners for ID Confirmation.
We extended the existing real time monitoring feature used by exam administrators to include an exam circuit control panel where they can switch through the exam states, activate assessment groups and can optionally send prompts/notifications to specific roles and all users e.g. Students: “Please read your instructions”, Examiners: “Rendezvous with actors at your stations”, Actors: “Please read station scenario” etc. Behind the scenes, the system is auto-incrementing the circuit rotations number, assisting with moving students through the stations (virtual rooms) and delivering exam status and notifications to participants.
Technical Insights
The backend API was extended to manage the logistics based on the following state diagram that captures all the states of a typical MCQ and OSCE exam:
Here is a sample code of that “smart” or shall we say simple logic in Node.js, courtesy of Volodymyr Bortniak @Inventorsoft (outsourcing partners), to handle the state change (ASSESSMENT -> READING_TIME) and to automatically remove students from all virtual stations:
On the front-end we managed to embed video call capability by using programmable video technology delivered by Twilio Inc. Below is sample angular code courtesy of Diana Lazariuk @Inventorsoft, used to join a virtual station in a circuit:
Lessons learned
- We wish we had more time and resources to add extra functionality to the initial release but we had such a short turnaround to develop quite a complex system in order to meet the expectations of our clients.
- Communication at this point in time on the system is unidirectional from the administrators -> participants (SSE — Server Sent Events) carrying exam state updates and notifications/prompts. We plan to include duplex communication in the next release to allow participants to message the administrators in case of issues and participant status (online/offline, current position on circuit, network quality indicators etc).
- Since actors and now students are users of the platform and participants in the video integration, logging in with their own devices (more moving parts) means a lot more can go wrong.
- We’re currently improving the logistics somewhat to allow the administrators to keep rotating the circuit post exam for any students/candidates who may have had connection or technical issues and are missing assessments at certain stations. This will also include the ability to regroup affected students and schedule their assessment for another time.
- Participant screen sharing and invigilator access to virtual stations rooms is in the pipeline.
About the Author
David Cunningham is CTO/COO and co-founder of Qpercom and you can connect with him on linkedin or follow him here on medium
