Matthew Wnuk
← All work

Symbotic · Senior Software Engineer, ML and Perception · 2023 to 2026

CaseID: SKU Identification for Warehouse Automation

Identifying the SKU of every case a warehouse robot handles, with the whole pipeline running onboard the robot.

  • Vision Transformers
  • Metric learning
  • ONNX
  • NVIDIA Jetson
  • ROS
Onboard the mobile robot: NVIDIA Jetson, ROS nodesRobot camera frameDetect andcrop the caseViT encoder(ONNX)EmbeddingSearch: nearest SKUs in the reference gallery123Verify: does it match the SKU the system expects?expected SKUsimilarityover threshold?acceptor flag

Overview

Organization
Symbotic
Role
Tech lead
Period
2023 to 2026

Symbotic builds end-to-end warehouse automation for some of the largest retailers and wholesalers in the world. Robotic cells break inbound pallets down into individual cases, a fleet of autonomous mobile robots stores and retrieves those cases in a dense storage structure, and outbound cells build mixed pallets ready for the store.

Because the system handles goods as individual cases in their native packaging, everything downstream depends on knowing exactly what each case is. CaseID is the system I led for that problem: identifying the SKU of a case from images taken by cameras on the mobile robots.

The range of products is effectively unbounded: packaging varies widely, new SKUs arrive constantly, and many cases look alike. As tech lead I took CaseID from ideation and literature review, through model development and experimentation, to production deployment and monitoring.

SKU ASKU BSKU C
Three SKUs of one product family. The cases share a design and differ mainly in variety, which is the kind of distinction the embedding has to capture.
anchorsame SKUdifferent SKUPretrainedViT,shared weightspullpushembedding space

01

Metric Learning on a Foundation Model

The encoder starts from a pretrained Vision Transformer foundation model and is fine-tuned on case images with metric learning: the loss pulls embeddings of the same SKU together and pushes different SKUs apart. Distance in the resulting space measures how likely two images are the same product.

Searchquery1: SKU A2: SKU B3: SKU CVerifyqueryvsexpected: SKU Blow similarity:flag a mismatch

02

Search and Verification

The embedding is used in two ways. In search, it is compared against a gallery of reference embeddings and the nearest SKUs are returned. In verification, the system already expects a particular SKU, and the embedding is checked against that SKU to accept the case or flag a mismatch.

Fine-tuneAzureExportto ONNXContainerizefor the robotROS nodes on the Jetsondetectembedmatchlimited, shared memory

03

The Full Pipeline on the Robot

Detection and cropping of the case, embedding, and matching all run onboard the robot on an NVIDIA Jetson, as ROS nodes. Models are fine-tuned in the cloud on Azure, exported to ONNX, and containerized. Memory on the robot is limited and shared, so memory management matters as much as model accuracy.

Photos from Wikimedia Commons: Cases at a warehouse store by BrokenSphere (CC BY-SA 3.0); Jetson developer kit by Ubahnverleih (CC0).