FormCoach
Hands-Free AI Form Coach
FormCoach is an on-device AI coach that analyzes exercise form in real time using only the iPhone camera and delivers spoken feedback. It uses Apple Vision 3D body-pose estimation to measure joint angles and correct form across five lifts (squat, push-up, lunge, deadlift, overhead press), scoring each set on four axes — alignment, range of motion, symmetry, and stability. You advance sets and hear your score entirely hands-free via voice commands, and all video and audio processing stays on the device and never leaves the phone.
Working out alone makes it hard to notice when your form breaks down, and hiring a trainer for every session is expensive. FormCoach gives real-time form feedback by voice — even when you're several meters from the camera and can't touch the screen — so you can train safely, fully hands-free.
App preview
What it does
On-Device 3D Pose Estimation
Runs Apple Vision's VNDetectHumanBodyPose3DRequest asynchronously on a dedicated queue to extract 3D positions and 2D screen projections for 17 joints per camera frame. All inference runs entirely on-device, weighting core joints (hips, knees, spine) with higher confidence.
simd Joint-Angle Computation
Computes knee, elbow, hip, and shoulder flexion angles via simd 3D vector dot products and acos, plus spine vertical deviation, torso lean, and knee valgus (inward collapse) angles — and derives a left/right asymmetry index for each joint pair.
Multi-Dimensional Rule-Based Scoring
A per-exercise rule engine defines ideal angle ranges and thresholds, measures deviation per rep phase to produce prioritized (critical/high) feedback, and combines four axes — alignment, range of motion, symmetry, and stability — with per-exercise weights into a 0–100 form score. It is a deterministic rule engine over Vision estimates, not an ML model.
State-Machine Rep Counting
Tracks the tracking-joint angle through a five-phase state machine (idle, descending, bottom, ascending, top) to count reps automatically, distinguishing flexion lifts from the overhead-press extension direction and using cooldown logic to prevent double counts.
Hands-Free Voice Commands
Wraps SFSpeechRecognizer in an actor and runs it with on-device recognition, parsing commands like "next set", "pause", "stop", and "score" through Korean/English matchers. Debouncing filters duplicate utterances, and input is suppressed during spoken coaching to avoid the app mishearing its own voice.
Spoken Coaching Cues
An AVSpeechSynthesizer-based voice coach auto-detects the device language and speaks form corrections, rep counts, set-complete summaries, and encouragement, using per-joint-group cooldowns to avoid nagging and .duckOthers to lower background music while it talks.
Thermal & Battery Adaptation
Dynamically lowers the pose-estimation frame rate (15–30fps) based on device thermal state and battery level, and smooths joint jitter with an exponential-moving-average JointSmoother, containing heat and battery drain across long sessions.
History, Trends & Achievements
Persists set results, personal records, and streaks with SwiftData, visualizes trends with Swift Charts, and awards achievements. The free tier caps daily sets, history depth, and long-range trends, all unlocked by a Pro subscription.
How it's built
Language
UI
Vision & Geometry
Audio & Speech
Data & Commerce
Build & Tooling
Testing
A Clean Architecture + MVVM design that abstracts every device service (camera, pose, form analysis, rep counting, voice coaching, voice commands, storage, subscription, achievements, notifications) behind protocols via a ServiceContainer, pairing each on-device implementation with a 1:1 mock. The entire pipeline runs on-device with no video or audio leaving the phone.
- 1
A protocol-based ServiceContainer dependency-injects 10 services, swapping device (makeDevice) and mock (shared) implementations to simplify testing, previews, and simulator screenshots
- 2
On-device, privacy-first design — Vision pose estimation and SFSpeech recognition both run with on-device recognition, and unsupported locales return an empty stream rather than uploading audio to the cloud, so video and audio never leave the phone
- 3
A hands-free closed loop — camera frame → 3D pose estimation → simd angle computation → rule-based multi-dimensional analysis → spoken coaching and voice-command control cycle with no screen touches, suppressing voice-command input while coaching speaks to block self-recognition
- 4
Form analysis is a deterministic rule engine, not an ML model — it compares Vision 3D estimates against per-exercise ideal ranges and thresholds, then weights alignment, range of motion, symmetry, and stability per exercise into an explainable score
- 5
Subscription gating — observes tier via StoreKit 2 Transaction.updates/currentEntitlements streams, gates free-tier exercises, daily set count, history depth, and long-range trends, and checks intro-offer eligibility
- 6
A restrained paywall policy (PaywallTriggerPolicy) — surfaces the upsell only at the highest-intent moment right after a completed workout where the score just climbed, but enforces a three-workout cooldown after the first to avoid "too pushy" reviews; a pure value type that is trivially unit-testable
- 7
Full Swift 6 strict concurrency — actor isolation, Sendable, and AsyncStream handle async camera, voice, and subscription events with concurrency safety, with the project managed declaratively via SPM + XcodeGen
659
Unit tests
37
UI tests
104
Swift source files
5
Supported exercises
17
Tracked joints
4
Score dimensions