Projects

Hard numerical problems, solved on a phone.

The two featured apps have the same constraint: the analysis has to finish fast enough to give the user feedback while they are still playing or throwing. Below them are open-source imaging tools from my research career.

Flight Plate Monitor app icon

Flight Plate Monitor & Flight Plate Control

Disc golf launch analysis from a phone lying on the ground · Android (Java), iOS (C++/Swift), Python prototypes

A printed 3 × 3 dot pattern is stuck to the underside of a disc, and a phone on the ground films the throw from below at 120 fps. Within a couple of seconds the Flight Plate Monitor app announces the disc's speed, spin, launch angle, nose angle, hyzer angle and wobble. A second phone runs the Flight Plate Control app, which shows the live preview, guides the user through camera calibration and receives the results over local Wi-Fi.

The project was inspired by commercial launch monitors such as TechDisc (a sensor disc) and RipReader (a phone app). It explores the measurement problem rather than competing with them.

Why it is hard

The disc passes 1–1.7 m above the lens and moves at up to 20 m/s, spinning at around 900 rpm. It is visible for only 4–18 frames. The phone's rolling shutter reads the sensor row by row over about 5 ms, and in that time the disc moves around 10 cm and turns up to 27°. Ignoring this shears the image of the disc and produces false tilt angles.

Pipeline

  1. Cascaded trigger. The camera records continuously into a 3 s ring buffer. A cheap 160 × 90 frame-difference motion gate runs on every frame, and the neural detector runs only when something moves.
  2. Retrained disc detector (ML). I fine-tuned a YOLOX-nano detector from pretrained weights on a data set I labelled myself (frames from my own throw videos, labelled with LabelImg and converted to COCO format). It was exported to ONNX and runs on the phone through OpenCV DNN at 320 × 180 input. A lattice check with normalised cross-correlation then rejects people, pets and clothing that the detector lets through.
  3. Sub-pixel pattern nodes. A robust Gauss–Newton (Lucas–Kanade family) alignment of a blurred 300 dpi template, using an 8-dof homography and Huber-weighted IRLS over a three-level pyramid, locates the nine nodes far more accurately than the 0.5–0.9 px of blob detection.
  4. Rolling-shutter physics fit. A 17-parameter model, combining constant-acceleration translation with torque-free symmetric-top rotation (spin plus wobble), is fitted with Levenberg–Marquardt to every node in every frame at once. Each node is evaluated at the moment its sensor row was read out.
  5. Metrics with uncertainty. All metrics come from the fitted parameters, not from frame-to-frame differences. The accelerometer gives the direction of gravity, and the parameter covariance is propagated to a ± value for every metric.

Validation

Three phones (two Samsung A16s and an iPhone 12 Pro) lay side by side and recorded the same throws independently. The pooled between-phone SDs were:

SpeedSpinLaunchNoseHyzerWobble
0.31 km/h3.0 rpm1.7°2.8°1.4°0.5°

iOS does not report its rolling-shutter readout time. I estimated it from the throws themselves by fitting it as an additional free parameter. On the Android phones the method recovered the readout time the phone reports to within about 0.2 ms. For the iPhone it gave 2.4 ms rather than the 5 ms assumed until then, and the correction cut the iPhone's nose-angle disagreement with the Android phones from 5.5° to 1.6° RMS.

Engineering

The algorithm was prototyped in Python, implemented in Java for Android, and ported line for line to portable C++ for iOS. Calibration uses Zhang's method with a guided set of poses. On the iPhone it can instead use Apple's factory lens data, for which I measured the undocumented mapping to the binned 120 fps video mode (0.12 px residual).

  • Java
  • C++
  • Swift
  • Python
  • OpenCV
  • YOLOX
  • ONNX
  • Levenberg–Marquardt
  • rolling shutter
  • Camera2
  • Wi-Fi / NSD
Audio FFT Visualise app icon

Audio FFT Visualise

Real-time spectrum and multiple-pitch estimation · Android (Java, C++/NDK, Vulkan compute)

A free Android app that shows the live spectrum of what the microphone hears and estimates the fundamental frequencies of the notes being played. It is designed for string and pipe instruments, where several notes and their harmonics overlap.

Why it is hard

Identifying a single pitch is easy. Identifying several simultaneous pitches is not, because the harmonics of one note fall on top of the fundamentals and harmonics of the others. The app implements Klapuri's (2006) multiple-F0 estimation, which whitens the spectrum, computes a harmonic-summation salience for every F0 candidate, then iteratively picks the strongest candidate and cancels its harmonics. All of this has to be repeated for every audio frame fast enough for the display to keep up with the music.

How it runs in real time

  1. Low-latency capture. Audio is recorded with Google's Oboe library in native C++ in low-latency performance mode, and handed to Java through JNI.
  2. GPU FFT. The FFT runs as a Vulkan compute shader on the phone's GPU (the successor to an earlier RenderScript implementation).
  3. GPU salience. The harmonic-summation salience for all F0 candidates is computed in parallel on the GPU, using precomputed index and weight tables for the harmonics.
  4. Custom rendering. The spectrum and the detected notes are drawn on a dedicated surface view.
  • Java
  • C++
  • Android NDK
  • Oboe
  • Vulkan compute
  • FFT
  • DSP

Open-source imaging tools

Tools from my research career that have been used by other research groups.

pQCT analysis plugin for ImageJ / BoneJ

ImageJ / Fiji tools for Stratec pQCT scans. They cover conventional bone analyses (cortical area, volumetric BMD), regional density distribution around the cortex, bone marrow density, and soft-tissue analyses (muscle and subcutaneous fat cross-sectional area). The density distribution method was integrated into BoneJ, the standard open-source toolkit for bone image analysis, and is used by other research groups.

  • Java
  • ImageJ / Fiji
  • image analysis

MRI slice segmentation: RoiAnalyzer

An ImageJ plugin for computer-assisted segmentation of regions of interest in MRI slices, built for the MRI segmentation in Daniel Belavý's Spine Activity project. Companion plugins add LiveWire boundary tracing and threshold-guided region selection.

  • Java
  • ImageJ
  • MRI
  • segmentation