Building an Intelligent Proctoring System with MediaPipe and YOLOv8

Introduction

Online examinations and remote hiring platforms have become essential in today’s distributed digital ecosystem. However, ensuring fairness, detecting suspicious behavior, and verifying candidate authenticity remain significant challenges. We developed an AI-driven proctoring system that leverages real-time computer vision using MediaPipe and YOLOv8 to address these concerns.

This intelligent system tracks eye movements and head orientation, detects multiple faces, and identifies the presence of mobile phones, all from a single video feed. It is optimized for hiring assessments and exams and performs efficiently without requiring cloud-based inference engines. The solution is open-source, scalable, and highly accurate across varied environments.

Overview of Components

Our system combines the strengths of two powerful models:

1. MediaPipe – For head pose estimation and eye tracking using facial landmarks.
2. YOLOv8 – This is for multi-face detection and mobile phone detection in video streams.

Each component identifies behaviour that might indicate cheating or violating guidelines during an exam or interview session.

Head and Eye Tracking using MediaPipe

MediaPipe Face Mesh offers 468 facial landmarks. These compute the head’s pitch, yaw, and roll, allowing us to determine how much the candidate moves or turns their head. Simultaneously, iris tracking provides insight into where the user is looking, which is useful for detecting eye movement patterns that may suggest distractions or off-screen focus.

For example, if the candidate consistently looks away from the screen or shows frequent head turns, these behaviors can be flagged for review.

Sample Code:

Face Detection using YOLOv8

The YOLOv8 model (we used yolov8n.pt) provides fast and accurate object detection. We used it to detect:

• Faces – To identify when more than one person appears in the frame.
• Mobile Phones – To flag if the candidate uses or has access to a phone during the test.

The model processes all video frames and saves annotated evidence whenever a person and a cell phone are detected simultaneously.

Sample Code:

This allows for powerful, lightweight on-device detection without relying on third-party APIs or cloud services. Only relevant frames with multiple faces or phone presence are stored as evidence, reducing storage overhead.

Technical Challenges and Optimizations

While developing this proctoring solution, a few key challenges had to be addressed for efficiency and accuracy:

1. Frame Sampling
   Analysing every frame in a long video is resource-heavy. To optimize, the system can sample fewer frames per second (e.g., 5 FPS instead of 30), reducing computation without missing critical evidence.
2. Memory Management
   Long videos risk memory overflow if frames aren’t released promptly. To counter this, processed frames are freed immediately, and lightweight buffers are used to keep resource usage low.
3. Threshold Calibration
   Movement thresholds were carefully tuned to reduce false positives:

• Head Yaw/Pitch: Movements beyond 40° are flagged.
• Eye Movement: Normalized iris displacement beyond 0.4 is flagged.

These optimizations ensured the system remained accurate, scalable, and efficient even for extended video sessions.

Conclusion

This AI-based proctoring system merges the efficiency of MediaPipe and the power of YOLOv8 to create a holistic solution for monitoring online exams and interviews.

Whether you’re building a next-gen interview bot, an online assessment tool, or an AI invigilator for remote exams, this system can be a plug-and-play proctoring module.

About CloudThat