If you want to understand the future of a play, you first need to understand how the players are moving; not just where they are on the pitch, but how their bodies are positioned, loading, and about to change.
Most human motion prediction research is built on clean, lab-captured data with millimeter-accurate tracking. In sports analytics, that's a luxury we rarely have. Broadcast video is noisy, occluded, and compressed, but it's also the largest collection of authentic athletic performance. So the question I asked was: can we accurately predict motion from raw broadcast footage from a live World Cup match?
To find out, I applied SimpliHuMoN, my simple and general human motion prediction architecture, to footage from the 2022 FIFA World Cup using the WorldPose dataset. We predict a player's full 3D skeletal motion 1 second into the future, using only 1 second of past broadcast video.
The best way to understand what this looks like in practice is to watch it.
Here's what's happening:
The camera zooms into a player at the exact moment prediction begins. The observed skeleton (in black) freezes. Instantly, three of the proposed futures materialise for the player, each colour-coded by the model's confidence: blue (low), yellow (medium), and red (high). Along with the skeletons, the ball is tracked using SAM3, grounding each prediction visual in the actual game context.
As time moves forward and ground truth is progressively revealed frame-by-frame, the confidence labels update in real time, so you can watch, live, which prediction was closest to what actually happened.
This is beyond a dot moving across a 2D pitch; this is the full 24-joint, 3D biomechanical trajectory of a player. You can observe their hips loading, their arms counterbalancing, and their weight shifting, all predicted from nothing but a second of broadcast video.
For the full architecture and ablation studies, see the SimpliHuMoN paper. Here's the intuition behind what makes it work on dynamic, real-world sports data.
1. No encoder-decoder split. Just unified attention.
Most prediction models divide the task in two: an encoder reads the past and a decoder generates the future. SimpliHuMoN removes that divide entirely. 1 second of observed motion and 1 second of learnable future proposals are processed simultaneously through a unified attention mechanism where each informs the other in the same pass. The model dynamically adjusts its interpretation of what it just saw based on what it's trying to predict, and vice versa.
2. Ten futures from one forward pass.
Sports are inherently stochastic. A striker might cut left, check right, or brake completely. SimpliHuMoN predicts 10 distinct hypothesis trajectories to capture the full range of possible outcomes. This makes the method lightweight and efficient compared to diffusion-based approaches as we are able to generate multiple futures in a single forward pass.
3. Keeping the hypotheses genuinely different.
The hardest problem in multi-hypothesis prediction is mode collapse (converging to nearly identical guesses). I address this with two complementary training penalties:
• A balance penalty that stops any single prediction head from dominating during training, forcing all hypotheses to stay active and viable.
• A diversity hinge that applies an explicit spatial penalty when two predicted futures come too close to each other, ensuring predictions meaningfully span the space of outcomes.
I benchmarked SimpliHuMoN against leading multi-person prediction methods on WorldPose. Two numbers stand out for practical deployment:
Accuracy: SimpliHuMoN achieves sub-0.3 m full-body error, an 18% improvement in full-body error versus the best baseline. Moreover, we see that using skeletal information helps in predicting "player-as-a-point" future trajectories. The model isn't just tracking where players go; it's accurately forecasting how their bodies move to get there.
Speed: ~70 samples per second on a single GPU (>6× faster than diffusion-based approaches). For a 22-player scene, that's roughly 3 complete scene predictions per second. With standard engineering optimisations (batching, quantisation), real-time broadcast integration is within reach.
One thing worth noting: The model has no idea it's watching football. It sees 24 joints moving through 3D space. It learns motion patterns, not sport-specific priors, which makes it surprisingly adaptable.
Here's what I'm thinking about next:
1. Integrating pose with ball tracking and event data.
The current model predicts player motion independently of the ball. That's an obvious ceiling. The natural next step is fusing 3D pose with ball trajectory and event data (passes, pressures, shots) to move from motion prediction into decision evaluation. We could assess defensive shape, anticipate passing lanes, and quantify decision quality in real time. The main bottleneck right now is temporal synchronisation between pose data and event feeds. Has anyone worked on this pipeline? I'm keen to explore what's possible.
2. Modelling player interactions.
Right now each player is predicted independently. But in team sports like football, every movement is relational. A defender's body position anticipates the attacker's next touch; a midfielder's run constrains a teammate's options. Modelling the interaction graph between players feels like where a lot of interesting questions live, and a natural next layer to build on top of the current architecture.
3. Expanding to new sports and modalities.
The architecture should generalise. If you're working with multi-view or markerless mocap data in any sport and want to see what this model does with it, let's talk. I'm particularly curious about basketball, with its rapid, high-diversity body poses and tight spatial interactions, and American football, with its complex multi-agent coordination and clearly defined roles.
If you're working in sports analytics, computer vision, or biomechanics - or just find this kind of problem interesting - I'd love to hear your thoughts. What would you build with this?
The SimpliHuMoN paper is available on arXiv and code on GitHub. Happy to share more on the architecture, the WorldPose pipeline, or the demo code!