WACV 2027  ·  workshop version at CV4Ecology, ECCV 2026

Automated Temporal Localization of Behavior in Cichlid Fish with Few‑Shot Trajectory Tokens

Bhargav Srinivasan*, William Lamousin*, Jason Liu†, Charles Phan†, Ofure Osunbor, Shubh Sharma, Cade McGeehan, Tahir Haroon, Vijay Jayasuriya, Coltan Parker, Scott Juntti, Pulkit Kumar, Abhinav Shrivastava

University of Maryland, College Park · Departments of Computer Science and Biology · * † equal contribution

The problem

Behavioral experiments are limited by how fast experts can watch video.

Experts annotate at real-time playback, hour by hour, in BORIS.
Four observers per recording, reconciled by inter-rater reliability.
Each label is a single time point. No start, no end.
Raw recording · 25 fps
0
hour-long recordings
0
expert point annotations
0.0%
fall within 2 s of another annotation
Expert point annotations a timestamp and a class 0 s 60 s FSH-TAL output start, end, and class, including overlaps Bite Chase Lead Peck Quiver Tilt
The dataset  ·  17 hour-long recordings of Astatotilapia burtoni pairs, side view, 25 fps

Six short, visually similar, often overlapping behaviors.

Bite3,516 events
Chase846 events
Lead1,455 events
Peck529 events
Quiver1,262 events
Tilt594 events
Stage A  ·  object-grounded trajectories

Segmentation based keypoint tracking.

Raw clip
SAM3 segmentation
Point tracking
Lead
Chase

No skeleton or keypoint labels provided. Tracking each fish is worth 13.1 mAP over a uniform patch grid, and a single point per fish recovers 74% of that gain.

The model  ·  FSH-TAL

A few-shot classifier feeds a point-supervised localizer.

FSH-TAL architecture: SAM3 masks and point tracking, a multiple-instance few-shot classifier on trajectory-aligned tokens, and a temporal localizer with dilated convolutions, transformer blocks, and three decoder heads
Few-shot MIL classifier

Trajectory-aligned tokens at the tracked points fused with motion features, trained in few-shot episodes.

Point-supervised localizer

Behavior, timespan, and centerness heads, trained on ±2 s pseudo-segments around each point annotation.

Results  ·  5-fold cross-validation over all 17 held-out recordings

State of the art at every IoU threshold over varied supervision.

Average mAP, IoU 0.1–0.7
0.0%
+17.4 points over ActionFormer, fully supervised on the same pseudo-segments, and ahead of every baseline on all 17 recordings.
ActionFormer23.6
MIL + post-processing16.0
HR-Pro14.1
MIL classifier12.2
ASM-Loc8.3
Per-behavior average precision  ·  IoU 0.1–0.7, recordings containing the behavior

Well-defined behaviors localize best, even when they are rare.

0 20 40 60 AP (%) 49.4 48.3 43.4 42.6 35.9 35.2 Tilt 594 · 11 Peck 529 · 11 Lead 1,455 · 16 Quiver 1,262 · 17 Bite 3,516 · 15 Chase 846 · 12 annotated events · recordings containing the behavior
Precisely defined behaviors beat loosely defined ones: Tilt and Peck are rare, yet among the best localized.
Results vary most for the three behaviors present in 12 or fewer recordings; Lead, present in 16, varies least.