Stereo video analysis of drift feeding behavior

My research is heavy on visuals, because much of it involves using VidSync to collect 3-D measurements from stereo (side-by-side) video footage. This page is not intended to give a complete overview of any part of this research, but just to show the role that video technology and 3-D visualization plays in bringing some new kinds of data to bear on ecological questions.

Below, a dolly varden makes several prey capture attempts. Pac-Man icons indicate the positions from which the fish begins each attempt and the position at which it either captures the item or abandons the pursuit.

By combining the types of points measured above with water velocity data (also measured from video by calculating the trajectory of natural or artificial tracers), we obtain a 3-D estimates of the positions at which the fish detected potential prey. Below is one such dataset for a 19″ Arctic grayling. The view rotates around it on a circular path to help visualize the 3-D structure of the detection locations.

I am currently working on a model to predict the trajectory, duration, and energy cost of the optimal prey capture maneuver for the fish to capture an item detected in a particular location. The model will help us better understand the energetic value of different habitats to the fish. Below is part of a test of this model, in which yellow dots indicate the model-predicted positions at which a dolly varden captured prey, whereas green dots indicate the actual capture position. The predicted path of the fish’s head is shown by a looped line with direction arrows and colors indicating swimming speed throughout the maneuver.

It is also possible to digitize real maneuvers for comparison to model predictions of the full maneuver path. Below is a juvenile Chinook salmon’s prey capture attempt digitized in VidSync and recreated in Mathematica. (It is not compared to model predictions here.)

Territoriality and social behavior

One advantage of using video to study behavior is that it’s possible to analyze many interacting animals in detail over the same time period, as opposed to live observation in which it’s hard to watch more than one at a time. One of my Ph.D. chapters examined territoriality in juvenile Chinook salmon by gathering 3-D measurements of all the relevant actions by every fish in a group for twenty minutes. Below is a brief clip showing what a video looks like in VidSync after taking this many measurements on it.

Although juvenile salmonids are widely recognized to establish 2-D mosaics of territories in small streams, it was unclear whether and how this behavior would translate to a large river where the fish live in tight groups to stay safe from predators. Are the fish in the group individually staying in consistent areas, or roaming around within the group at random?

My first look at this question involved measuring prey capture attempt positions for a subset of the above fish (colored spheres below, with a different color for each fish) and wrapping them (minus a few outliers) with 3-D “convex hulls” that roughly visualize each fish’s territory. This showed that the fish were using largely distinct spaces despite being in a tight group in a 3-D environment with some fish directly above or below others.

However, these convex hulls raise some questions they cannot answer. Are the fish with large convex hulls really feeding and repelling competitors within a large amount of space at any given time, compared to the other fish? Or are they using a smaller space that slowly moves over time? I developed a new technique to track the center of the space the fish was using over time and look at how much it deviates from that center as an indicator of the amount of space each fish really requires within the group. Below, each bold squiggly line represents the center of a fish’s activity over a 60-second period. Longer lines indicate the fish moved a lot during those 60 seconds; short lines show that it was fairly stationary. Thin offshoots to spheres mark prey capture attempts. Each color represents a different fish, except for white, which was used for all the fish that only briefly passed through the field of view. The 60-second window rolls forward at a rate of one second of animation per minute of video footage, giving a compete picture of space use within this group over a 20-minute period. It reveals a variety if individual space use strategies, including clear territoriality by some individuals.

Invertebrate drift and debris ampling

A project in Alaska in 2021 and 2022 included fieldwork using two new devices I designed and built. The first was a system to sample drifting invertebrates by pumping water out of the river, dissipating the violent force of the water pump in a conical chamber, and draining the water through a fine-mesh net suspended in the air. A paper in Hydrobiologia describes this system, which found that the amount of prey available (per unit volume of water) for juvenile Chinook Salmon was eleven times higher than the amount estimated using a conventional drift net.

The other device in the water, the black box, is a computer vision system I built to count and measure drifting particles, including debris which distract fish from prey. Inside the box are a camera and six super-bright Cree XHP70.2 LED lights, controlled together by a Raspberry Pi microcomputer. I couldn’t use a regular camera flash, because it would overheat when taking 1800 pictures an hour. The bright LEDs, however, were able to dissipate heat through thermally conductive epoxy into the metal frame of the device and the cold river water. The photos taken by this system are analyzed by a Python program that counts and measures debris particles and distinguishes them from other objects, such as bubbles and scratches on the glass. We found that debris outnumber prey by a factor ranging from hundreds to thousands.

Just for fun

In summer 2015 and 2016 I led a three-person field team collecting video footage, drift samples, and diet samples from drift-feeding salmonids in three Alaskan streams. Here’s a compilation of footage from our first month in the field.