Back to selected work

Master's research · 2026

Digitising
Sand Stories.

Multimodal Capture of Sand Drawing

A drawing in sand changes while the story is told. I built capture tools to record the surface, the drawing actions, and the narration, then examined what could be followed in the saved recordings.

My role
Capture software, replay workflow, and evaluation
Tools
Swift / ARKit, Python, Azure Kinect, Godot
Context
Master of Computer Science, University of Melbourne
Study
36 paired performances, controlled indoors
Annotated setup showing the articulated support arms, Azure Kinect DK, iPhone 15 Pro, support stand, sand tray, host computer, and power strip
The annotated indoor capture setup. Open full-size image ↗

The final drawing is only part of the record

A photograph of the finished surface cannot show a line that was drawn and then erased. It also cannot preserve how narration and pointing related to those changes.

My research compared two capture configurations: an iPhone 15 Pro using ARKit and LiDAR, and an Azure Kinect DK connected to a Windows computer. The question was what each recording preserved, and what evidence was available for reconstruction.

My contribution

I developed the capture applications and worked on the offline processing and replay workflow. I then recorded and evaluated 36 paired performances under combinations of device height, lighting, and sand preparation.

  • iPhone capture. An application to retain RGB video, depth, audio, timing metadata, and estimated stroke points in an exportable session package.
  • Azure Kinect capture. A desktop recording workflow for RGB, depth, audio, and gesture-control events.
  • Offline inspection. Package checks, recording analysis, and review of reconstructed 3D surfaces in the replayer.

The work was supervised by Dr Adelaide Genay and Prof Martin Tomko.

Keep the recorded evidence inspectable

Retain separate streams and their metadata

Video, depth, and audio were saved with metadata so they could be examined after recording. File-presence, readability, and timestamp checks helped establish whether the recorded material could be processed.

Keep pairing separate from synchronisation

The applications started and stopped independently. Matching labels identified recordings of the same performance, but did not give the devices a shared time origin or establish synchronisation accuracy.

Distinguish an estimate from a measurement

The iPhone's stroke points were estimated by intersecting image rays with detected planes. Those estimates were not a continuous, measured 3D fingertip trajectory. The native Kinect recordings retained gesture-control events rather than a continuous hand-landmark stream.

What the recordings show

All 72 recording packages contained the required files. Recording checks described saved frame rates, returned depth values, and stream continuity. These checks established access to the stored material; they did not establish geometric accuracy.

Thesis comparison of RGB and depth samples from performance S01. The iPhone depth shows broad surface variation; the Azure Kinect depth shows drawing outlines.
One sampled performance at 60 cm with dry sand and room lighting. The crops show different physical areas and sampled moments. This is a visual diagnostic, not an accuracy test. Open full-size figure ↗

In the sampled depth images, Azure Kinect retained circular and linear drawing outlines, with irregular or incomplete boundaries in places. The iPhone depth samples showed broader surface variation, even where the drawing remained visible in RGB.

I reviewed all 36 Azure Kinect recordings as reconstructed 3D surfaces with RGB colour mapped onto them. In that review, I could follow the drawing and erasing sequence, understand the narration, and identify the locations indicated by pointing.

What this finding supports

These are the researcher's qualitative observations under the tested indoor conditions. They do not establish that every event was preserved or that other viewers would make the same judgements.

Limits and next steps

The study used researcher-performed, non-cultural sequences. It did not assess community use, outdoor recording, participant experience, or geometric accuracy against a measured reference.

A timestamped, event-level assessment remains proposed work. It would check individual drawing, erasing, pointing, and narration events against the recordings. It should not be presented as a completed result.