EyeQ — Engagement Detection

Measuring genuine engagement (in online learning, assessment or research) is hard. Any single cue (just a smile, or just where someone is looking) is easily fooled. You need several complementary signals, scored continuously and captured per learner over time.

Score engagement from ordinary video by combining multiple behavioural signals (facial expression, head orientation and body posture) into one reliable read per frame, and log it per student and session so it can drive analytics and real research.

Three models score each frame in parallel. A Keras / TensorFlow CNN reads emotion / engagement from a 48×48 face crop (the face located via a Haar cascade). A scikit-learn classifier reads body language from MediaPipe Holistic pose-and-face landmarks. And head pose is solved from MediaPipe FaceMesh landmarks via OpenCV solvePnP: yaw / pitch / roll thresholded into engaged or disengaged. A FastAPI endpoint ingests frames per student and session and writes the three readings to a MySQL store.

1. 1A FastAPI endpoint receives a frame with student, organization, video and timestamp
2. 2A Haar cascade locates the face; a Keras CNN scores emotion / engagement on a 48×48 crop
3. 3MediaPipe Holistic landmarks feed a scikit-learn body-pose classifier
4. 4MediaPipe FaceMesh + OpenCV solvePnP compute head-pose angles → engaged / disengaged
5. 5All three readings are written to MySQL, keyed by student, session and second

• Emotion / engagement CNN (Keras / TensorFlow, 48×48 face input)
• Body-pose classifier (scikit-learn on MediaPipe Holistic landmarks)
• Head-pose estimation (FaceMesh + OpenCV solvePnP → yaw / pitch / roll)
• Haar-cascade face & eye detection (OpenCV)
• Custom models trained on Amazon SageMaker

• Three complementary signals fused: emotion + head pose + body pose
• FastAPI service scores frames per student, session and timestamp
• Per-learner engagement logged to MySQL for analytics
• Runs on ordinary video with no special hardware
• Custom-trained models (Amazon SageMaker)
• Used in peer-reviewed academic research

A Python FastAPI service loads three models at startup (a Keras CNN, a pickled scikit-learn body-language model, and OpenCV Haar cascades) plus MediaPipe Holistic and FaceMesh. Each uploaded frame is scored by all three (emotion, body pose, head pose) and the results are committed to a MySQL table (student, organization, video, second, emotion, head_pose, body_pose), ready for downstream analytics.

EyeQ is an API-first engagement service: it exposes a scoring endpoint and writes structured, per-second readings to MySQL, so engagement can be aggregated per student, session or cohort, and visualised live via the on-frame overlay of class, probability and head angles.

• Amazon SageMaker (custom model training)
• FastAPI (scoring API) + MySQL (per-student readings)
• MediaPipe (Holistic + FaceMesh) & OpenCV (Haar cascades, solvePnP)
• TensorFlow / Keras & scikit-learn (models)

Models are custom-trained on Amazon SageMaker and exported to run inside the FastAPI service, which scores frames and persists readings to MySQL, deployable on a plain server with no GPU required at run time.