{ "id": "epcggr3hj7", "slug": "point-tracking", "title": "Point tracking", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.perception_language", "label": "Language and vision AI", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.perception_language" }, { "id": "technology.computing.ai.computer_vision", "label": "Computer vision", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.computer_vision" }, { "id": "technology.computing.ai.cv_methods", "label": "Vision methods and geometry", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.cv_methods" }, { "id": "technology.computing.ai.cv_methods.motion_analysis", "label": "Motion analysis and optical flow", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.cv_methods.motion_analysis" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.perception_language", "label": "Non-geographic · 2001 to 2020: Language and vision AI", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.perception_language", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.perception_language", "label": "Language and vision AI", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.perception_language" } ] } ], "excerpt": "Point tracking is a computer vision method that follows individually specified points across video frames, estimating each point's full trajectory and occlusion status, as in the tracking any point (TAP) formulation.", "snippet": "Point tracking is a computer vision method that follows individually specified points across video frames, estimating each point's full trajectory and occlusion status, as in the tracking any point (TAP) formulation.", "node": "technology.computing.ai.cv_methods.motion_analysis", "markdown": "# Point tracking\n\nPoint tracking is a computer vision method that follows the motion of individually specified points across the frames of a video, estimating each point's full trajectory. In the widely used \"tracking any point\" (TAP) formulation, the algorithm receives a video and a list of query points (x, y, t), where x and y are 2D position and t is time, and must output one position per frame plus a binary value indicating whether the point is occluded on each frame.[1](https://doi.org/10.48550/arxiv.2211.03726) Unlike optical flow, which gives a dense two-frame displacement field, a point tracker returns multi-frame trajectories that survive occlusion; PIPs, for example, takes an (x, y) query on the first frame and returns a \\( T \\times 2 \\) matrix of positions with a per-timestep visibility score.[2](https://doi.org/10.48550/arxiv.2204.04153) The task can be read as an extension of optical flow to longer timeframes with occlusion estimation, or of structure-from-motion keypoint matching to nonrigid, weakly textured surfaces.[1](https://doi.org/10.48550/arxiv.2211.03726)\n\n| Key fact | Detail |\n|---|---|\n| Input and output | Query points (x, y, t); per-frame positions \\( (x_{t}, y_{t}) \\) and a binary occlusion flag \\( o_{t} \\) for each query[1](https://doi.org/10.48550/arxiv.2211.03726) |\n| Trajectory format | PIPs returns \\( T \\times 2 \\) positions plus visibility \\( v_{t} \\in [0, 1] \\); its code returns tensors shaped B, S, N, 2 (batch, sequence length, points, x/y)[2](https://doi.org/10.48550/arxiv.2204.04153) • [3](https://github.com/aharley/pips/) |\n| Standard metrics | TAP-Vid scores occlusion accuracy (OA), \\( \\delta^{x}_{\\mathrm{avg}} \\) averaged over 1, 2, 4, 8, and 16 pixel thresholds, and Average Jaccard (AJ)[1](https://doi.org/10.48550/arxiv.2211.03726) |\n| Scale | CoTracker jointly tracks up to 70,000 points on a single GPU using proxy tokens[4](https://doi.org/10.48550/arxiv.2307.07635) • [5](https://co-tracker.github.io/) |\n| Speed | With JIT-compiled JAX on one GPU, TAPIR tracks 50 query points in 0.3 s (about 150 frames per second); PIPs takes 34.5 s on the same video[6](https://doi.org/10.48550/arxiv.2306.08637) |\n| Benchmark | TAP-Vid: 30 DAVIS val videos, 1000 Kinetics val videos, 50 RGB-Stacking videos for evaluation, and near-infinite synthetic Kubric ground truth for training[1](https://doi.org/10.48550/arxiv.2211.03726) |\n\n## How it works\n\nModern point trackers estimate a trajectory by matching appearance features across frames and refining positions iteratively. PIPs initializes a zero-velocity estimate by copying the query coordinate to every timestep, extracts CNN features for all frames, and then repeatedly updates positions and features using multi-scale cross-correlation similarity maps together with the current trajectory as a temporal prior; a 12-block MLP-Mixer produces updates applied as \\( F^{k+1} = F^{k} + \\Delta F \\) and \\( X^{k+1} = X^{k} + \\Delta X \\), after which a linear layer and sigmoid give visibility scores.[2](https://doi.org/10.48550/arxiv.2204.04153) • [7](https://blog.ml.cmu.edu/2022/09/09/tracking-any-pixel-in-a-video/)\n\nTAP-Net instead builds cost volumes, inspired by optical flow, and is trained with a [Huber loss](https://www.edgepedia.org/huber-loss) on position and cross-entropy on occlusion.[1](https://doi.org/10.48550/arxiv.2211.03726) TAPIR combines the two ideas in two stages: a matching stage that independently locates a candidate match for the query on every frame, and a refinement stage that updates the trajectory and query features from local correlations.[6](https://doi.org/10.48550/arxiv.2306.08637) CoTracker formulates tracking as joint inference over many points with attention between tracks; ablations show joint tracking beats independent tracking by 3.9 AJ at equal parameters.[4](https://doi.org/10.48550/arxiv.2307.07635) DOT, an occlusion-aware flow estimator that interpolates sparse point tracks, is trained with an L1 flow reconstruction objective plus a binary cross-entropy visibility objective on Kubric.[8](https://doi.org/10.48550/arxiv.2312.00786)\n\n## How it is done\n\nRunning a pretrained tracker involves four steps. First, load the video as an RGB tensor; PIPs processes 8-frame subsequences.[3](https://github.com/aharley/pips/) Second, specify query points, either manually or on a uniform grid: CoTracker tracks points on a grid from the initial frame, and TAP-Vid evaluation supports \"strided\" and \"first\" query modes at 256 × 256 resolution.[5](https://co-tracker.github.io/) • [9](https://github.com/google-deepmind/tapnet/) Third, run inference and read out trajectories; PIPs returns estimated trajectories shaped B, S, N, 2, where S is sequence length, N the number of particles, and 2 the x and y coordinates, together with visibility, and CoTracker3 additionally estimates a confidence value in [0, 1] per point.[3](https://github.com/aharley/pips/) • [10](https://doi.org/10.48550/arxiv.2410.11831) Fourth, for videos longer than the model's window, chain segments: PIPs' chain_demo.py re-initializes from a late timestep with high visibility (threshold started at 0.99 and decreased in 0.01 increments) while reusing the original query feature to avoid identity switches.[2](https://doi.org/10.48550/arxiv.2204.04153) • [3](https://github.com/aharley/pips/) TAP-Vid requires that outputs for a query not depend on other queries in the batch, so results are the same whether queries are passed one at a time or all at once.[1](https://doi.org/10.48550/arxiv.2211.03726) CoTracker's causal window design makes it suitable for online tasks, where windows are unrolled across the stream.[5](https://co-tracker.github.io/) Open implementations include the tapnet repository (TAP-Net, TAPIR, BootsTAPIR, TAPNext), the PIPs repository, and CoTracker.[9](https://github.com/google-deepmind/tapnet/)\n\n## Origin\n\nAn earlier \"particle video\" representation positioned itself as a middle ground between feature tracking and optical flow, describing pixels with trajectories that locate them in multiple future frames. The PIPs paper, Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories by Harley, Fang, and Fragkiadaki (2022, arXiv), rebuilt this idea with dense cost maps, iterative optimization, and learned appearance updates.[2](https://doi.org/10.48550/arxiv.2204.04153) In parallel, the TAP-Vid paper by Doersch and colleagues (2022, arXiv) states that they \"first formalize the problem, naming it tracking any point (TAP)\", and released the TAP-Vid benchmark with human-annotated and synthetic videos.[1](https://doi.org/10.48550/arxiv.2211.03726) The two accounts of the task's origin differ: the CoTracker authors credit particle video with introducing the TAP problem and PIPs with occlusion-aware sliding-window tracking, while the TAP-Vid authors present their paper as the formalization.[4](https://doi.org/10.48550/arxiv.2307.07635) • [1](https://doi.org/10.48550/arxiv.2211.03726) TAPIR (Doersch and colleagues, 2023, arXiv) added the two-stage matching-plus-refinement design,[6](https://doi.org/10.48550/arxiv.2306.08637) and CoTracker (Karaev and colleagues, 2023, arXiv) introduced transformer-based joint tracking.[4](https://doi.org/10.48550/arxiv.2307.07635)\n\n## Variants\n\n**TAP-Net** is a cost-volume baseline trained on Kubric; **PIPs** is trained entirely on synthetic FlyingThings++ data with multi-frame amodal trajectories.[1](https://doi.org/10.48550/arxiv.2211.03726) • [2](https://doi.org/10.48550/arxiv.2204.04153) PIPs++, released with the PointOdyssey paper by Zheng and colleagues (2023), correlates feature crops with the initial point feature and recently tracked features to handle appearance change, with a 1D ResNet predicting position updates.[11](https://doi.org/10.48550/arxiv.2307.15055) **DOT** obtains dense long-range motion by interpolating a sparse set of point tracks, roughly 100× faster than applying point trackers densely.[8](https://doi.org/10.48550/arxiv.2312.00786) **TrackOn** tracks online frame by frame with spatial and context memory, formulating tracking as patch classification followed by offset refinement, and flags predictions as uncertain when error exceeds 8 pixels.[12](https://arxiv.org/pdf/2501.18487) **TAPNext** formulates TAP as next token prediction; TAPNext++ extends stable tracking 40× longer with occlusion tracking and re-detection.[9](https://github.com/google-deepmind/tapnet/) **BootsTAP** (Doersch and colleagues, 2024) trains by consistency on unlabeled real video across spatial transformations, corruptions, and query choices, yielding BootsTAPIR.[13](https://doi.org/10.48550/arxiv.2402.00847) **GenPT** (Tesfaldet and colleagues, 2025) is a 12M-parameter generative tracker trained with flow matching that captures multi-modality under occlusions.[14](https://doi.org/10.48550/arxiv.2510.20951) CoTracker3 pseudo-labels real videos with off-the-shelf trackers and fine-tunes a model that outperforms all its teachers, beating BootsTAPIR using 15k real videos against BootsTAPIR's 15M training clips, and ships offline (bidirectional, better on occluded points but memory-bound in length) and online (indefinite) versions.[10](https://doi.org/10.48550/arxiv.2410.11831) On full-resolution 1080p DAVIS, PIPs raises AJ from 42.0% to 48.6% relative to a flow-chaining baseline.[1](https://doi.org/10.48550/arxiv.2211.03726)\n\n## Applications\n\n**Robotics** uses point tracking for imitation learning: RoboTAP (Vecerik and colleagues, 2023) provides 265 real, manually annotated robotics videos for evaluating trackers on manipulation scenes.[15](https://doi.org/10.48550/arxiv.2308.15975) **3D scene understanding** is served by TAPVid-3D (Koppula and colleagues, 2024), a benchmark of 4,000+ real-world videos with metric (x, y, z) ground-truth trajectories; it extends the metrics to 3D with a depth-relative threshold, and a jointly trained TAPIR-3D reaches only 9.4 3D-AJ, showing 3D tracking lags 2D.[16](https://doi.org/10.48550/arxiv.2407.05921) **Dense motion estimation** is the target of DOT, which turns sparse tracks into a dense occlusion-aware flow field.[8](https://doi.org/10.48550/arxiv.2312.00786)\n\n## Limitations and alternatives\n\nPIPs performs inference on the eight frames from \\( N \\) to \\( N + 7 \\) (with zero-based indexing) assuming the N-th frame result is approximately correct; if the track is lost for more than 8 frames, such as during a long occlusion, or the sequence has a discontinuity like a video cut, it is likely to fail catastrophically.[2](https://doi.org/10.48550/arxiv.2204.04153) A common failure mode across trackers is drift toward different objects or the background, which objectness regularization mitigates.[17](https://arxiv.org/pdf/2409.05786) Textureless objects remain hard: DOT gains over 13% relative AJ on RGB-Stacking by handling them better, and point trackers are too slow and memory-intensive to track every pixel, with CoTracker's accuracy dropping when simultaneous tracks increase too much.[8](https://doi.org/10.48550/arxiv.2312.00786)\n\nAgainst alternatives: optical flow gives accurate two-frame estimates, but as soon as the target is occluded it is no longer represented in the flow field and tracking fails; chaining flow vectors accumulates error.[2](https://doi.org/10.48550/arxiv.2204.04153) On FlyingThings++ 8-frame videos, RAFT's errors increase drastically for heavily occluded trajectories, often drifting to follow the occluder, and DINO features cannot match during occlusions.[2](https://doi.org/10.48550/arxiv.2204.04153) Classical object-tracking point correspondence, historically solved with qualitative motion heuristics such as proximity, maximum velocity, and common motion, struggles with occlusions, misdetections, and entries and exits; graph-theoretic multi-frame methods handle occlusions shorter than their matching window.[18](https://web.engr.oregonstate.edu/~sinisa/courses/OSU/CS556/literature/ObjTracking.pdf) In bioimaging particle tracking, no universally best method exists, and users should be cautious when image SNR is below about 4.[19](https://www.nature.com/articles/nmeth.2808) Discriminative trackers regress to a single mean or mode and miss multi-modality under uncertainty, which GenPT addresses.[14](https://doi.org/10.48550/arxiv.2510.20951)\n\n## References\n\n1. [Doersch, Carl and colleagues (2022). TAP-Vid: A Benchmark for Tracking Any Point in a Video. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2211.03726)\n2. [Harley, Adam W., Fang, Zhaoyuan, Fragkiadaki, Katerina (2022). Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2204.04153)\n3. [aharley/pips, official PIPs code release](https://github.com/aharley/pips/)\n4. [Karaev, Nikita and colleagues (2023). CoTracker: It is Better to Track Together. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2307.07635)\n5. [CoTracker project page](https://co-tracker.github.io/)\n6. [Doersch, Carl and colleagues (2023). TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2306.08637)\n7. [Tracking Any Pixel in a Video (CMU ML blog, by Adam Harley)](https://blog.ml.cmu.edu/2022/09/09/tracking-any-pixel-in-a-video/)\n8. [Moing, Guillaume Le, Ponce, Jean, Schmid, Cordelia (2023). Dense Optical Tracking: Connecting the Dots. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2312.00786)\n9. [google-deepmind/tapnet official repository (incl. TAP-Vid benchmark README)](https://github.com/google-deepmind/tapnet/)\n10. [Karaev, Nikita and colleagues (2024). CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2410.11831)\n11. [Zheng, Yang and colleagues (2023). PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2307.15055)\n12. [TrackOn: Online Long-term Point Tracking with Memory (arXiv preprint)](https://arxiv.org/pdf/2501.18487)\n13. [Doersch, Carl and colleagues (2024). BootsTAP: Bootstrapped Training for Tracking-Any-Point. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2402.00847)\n14. [Tesfaldet, Mattie and colleagues (2025). Generative Point Tracking with Flow Matching. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2510.20951)\n15. [Vecerik, Mel and colleagues (2023). RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2308.15975)\n16. [Koppula, Skanda and colleagues (2024). TAPVid-3D: A Benchmark for Tracking Any Point in 3D. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2407.05921)\n17. [Objectness-aware point tracking building on PIPs++ (arXiv 2409.05786)](https://arxiv.org/pdf/2409.05786)\n18. [Object Tracking: A Survey (Yilmaz, Javed, Shah, ACM Computing Surveys 2006)](https://web.engr.oregonstate.edu/~sinisa/courses/OSU/CS556/literature/ObjTracking.pdf)\n19. [Objective comparison of particle tracking methods | Nature Methods](https://www.nature.com/articles/nmeth.2808)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Language and vision AI › Computer vision › Vision methods and geometry › Motion analysis and optical flow*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://web.engr.oregonstate.edu/~sinisa/courses/OSU/CS556/literature/ObjTracking.pdf" ], "url": "https://www.edgepedia.org/point-tracking", "markdown_url": "https://www.edgepedia.org/point-tracking.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Point tracking\", Edgepedia (EdgeChat), https://www.edgepedia.org/point-tracking. Edgepedia Community License 1.0.", "credit_md": "\"[Point tracking](https://www.edgepedia.org/point-tracking)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/point-tracking](https://www.edgepedia.org/point-tracking). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Point tracking\", Edgepedia (EdgeChat), https://www.edgepedia.org/point-tracking. Edgepedia Community License 1.0.", "speakable": "Point tracking is a computer vision method that follows individually specified points across video frames, estimating each point's full trajectory and occlusion status, as in the tracking any point formulation." }