Estimating 3D Poses of Athletes at Live Sporting Events
Estimating 3D Poses of Athletes at Live Sporting Events
Advances in computer vision are giving us deeper insights into our surroundings than ever before. For journalists, they have the potential to unlock data that humans alone can't perceive and inform new types of reporting. In preparation for the 2020 Summer Olympics, last year we began exploring how we might use computer vision during live sporting events to better understand athletic performance. The goal was to help our reporters measure athlete movement in a way that typically requires sensors. Traditional motion capture techniques require an athlete to wear physical markers. But this isn’t possible during live sporting events. Instead, we built a solution that uses our photographers’ cameras, machine learning and computer vision to capture this data as an event unfolds."Simone Biles Takes Gymnastics to a New Level. Again." The New York Times, August 9, 2019.Read the articleThis work expands on previous Times sports coverage. For the 2019 United States Gymnastics Championships, The Times used burst photography to analyze Simone Biles’s floor routine, analyzing her historic triple-double (two flips with three twists) frame by frame. What else could we learn about Simone Biles’s triple-double if we could extract data from these photos that would let us quantify her split-second movements in 3D space?Our Process Our work thus far has focused on gymnastics since their poses — the locations of their joints in space — are such a vital part of the sport and its scoring. We also chose gymnastics because women’s gymnastics is consistently one of the most popular Summer Olympic events.Earlier this year, we conducted field tests at practice sessions of the Rutgers women’s gymnastics team, where we experimented with capturing the athletes with multiple cameras, running the photos and videos through computer-vision algorithms and turning the outputs into meaningful data. Through these field tests and a lot of iteration, we created a pipeline that enables us to extract 3D poses and performance metrics at live sporting events from burst photography captured with just three cameras. Here is an overview of how it works: Using burst photography and specialized hardware, we capture the athlete from multiple angles and synchronize the frames across the cameras. We estimate 2D poses of all the visible people in each frame using machine learning. With a combination of manual and automated tools, we determine which poses belong to the target gymnast. Drawing on regulation-size sports equipment and markings that are visible to the cameras, we determine camera properties such as focal length, position and rotation. We then triangulate the 2D poses into 3D using the synchronized frames and the known camera properties in each of those frames.With this sequence of 3D poses, we’re able to determine metrics such as peak height, rotation speed and more. Below, we’ve detailed some of our challenges and how we’ve addressed them.Synchronizing Burst Photography Across a Large Physical SpaceIf we can capture synchronized photos from multiple angles, we can transform poses extracted from all images into a 3D skeleton. Existing techniques to accomplish this require the cameras to be physically connected. This isn’t possible at sports events, as we usually have a few photographers spread out around an arena. To account for this, our team built hardware that synchronizes the shutters of multiple cameras using GPS clocks and a remote radio trigger — allowing photos from each camera to be taken at the same time from different vantages.Niko Koppel/The New York TimesCustom-built hardware uses radio frequency and GPS clocks to synchronize camera shutters across large sporting venues.Niko Koppel/The New York TimesCapturing action sequences of gymnasts at Rutgers University.Why would we use burst photography instead of video? First, our photographers are already using burst photography in their sports coverage. Additionally, each frame yields a sharper, higher-resolution image than video. Sharper images allow machine-learning models to perceive the details more clearly and provide more accurate poses. Comparison of Wrnch pose estimation results on a video frame, left, and burst photography frame, right. With burst photography we can have faster shutter speeds and sharper images, resulting in more accurate pose estimation results. Estimating 2D Poses in Photos of GymnastsOnce we have these images, we estimate the 2D poses of the athletes by feeding each frame through a machine-learning model. This gives us the location of each joint of the body on the screen. We evaluated many computer-vision models on photos of gymnasts, but most of them failed to identify their poses because most models are trained on images of everyday activities such as walking. We even started to train our own pose-estimation model on datasets we created that were over-fit towards gymnastics poses. But recently we discovered a model from Wrnch that provides exceptional results for images of gymnasts out of the box, so we decided to use it instead.Tracking the Pose of a GymnastPose-estimation models usually pick up the poses for all people in a given image. In order to know which pose to triangulate into 3D, we need to know which pose belongs to the gymnast in every frame of a sequence. To determine this, we implemented a solution in which a user selects which pose to track in the first frame, and then an algorithm finds the pose closest to the previous one for each subsequent frame.Determining Camera Properties from Sports Equipment and MarkersTo determine each camera’s location and orientation, existing systems require them to be placed in a fixed position with a physical calibration board visible to all cameras. But this type of calibration is not possible at live sports events, as we can’t place a calibration board in the middle of a court or field. Additionally, our photographers are likely to be in different locations throughout the course of an event. Instead, we rely on a technique detailed in a 2018 research paper jointly created by the University of Washington, Facebook and Google. The researchers found that one can use the known geometry from a regulation-size soccer field in place of a calibration board. To do this, the researchers marked the points of the soccer field visible on screen and used the screen-to-real-world correspondences to calibrate the camera. Using this method, we can similarly calibrate the position of a camera for a balance beam routine using the beam itself, the exact geometry of which is known based on regulation sizes for equipment.Calibrating a camera’s position and orientation using the visible points of a balance beam. The magenta lines that appear at the end are the top and bottom surfaces of the beam, drawn on the screen based on the determined camera pose. While we implemented this algorithm for gymnastics, we found that it extended well to other sports, such as tennis and track and field.Using the same camera-calibration method, the magenta lines on the tennis court above are drawn based on estimated camera position and orientation. 3D Pose TriangulationOnce we know where each camera is located and oriented and which poses belong to the gymnast, we can triangulate these 2D poses from all cameras into 3D using traditional computer-vision techniques. This gives us the x, y, and z coordinates of each joint of the athlete in real-world units such as feet or meters. To get the 3D pose of a gymnast throughout a move, we do this for all frames in the sequence.Results of triangulation of 2D poses from three cameras into 3D.Human-in-the-Loop for Data Accuracy  To ensure data accuracy, we also built a tool that allows anyone in the newsroom to correct inaccuracies at any step in the automated pipeline. This can include incorrectly estimated 2D points of a pose that didn’t track properly throughout a sequence. Using the editor tool, we can edit the points where the pose estimator confuses the left joints from the right.Extracting Real-World Metrics From 3D PosesOnce we have the 3D poses of a gymnast throughout a routine, we can extract meaningful metrics about the performance, such as jump height, body acceleration or rotation speed.With 3D pose, we have the position of each body joint in feet or meters, which can be turned into meaningful real-world metrics. Here we can compare the height of the pelvis, left, and right ankles during an athlete’s two jumps in a sequence. Looking AheadWe’re exploring how this capability might be useful in future coverage, including at next summer’s Tokyo Olympics. Beyond sports, we’re excited about the potential of computer vision to help journalists responsibly gather new types of data to inform their reporting. If you’re a researcher interested in our work, we’d love to hear from you. 