Personal robotics project

Omnirobot

From a Wi-Fi-controlled chassis to a prompt-driven robot: custom mechanics, embedded sensing, and three generations of control software, now brought together in OmniController MK3.

WhenJuly–October 2026 · Ongoing
My roleMechanical design · System design & integration
StatusBasic autonomy demonstrated · MK3 v0.2.0
The assembled Omnirobot mobile robot with its electronics and mechanical structure visible
My Fusion designs and assembled prototype

The goal

Turn a typed task into useful physical motion.

My work

Chassis and brackets, hardware integration, system requirements, and AI-assisted controller development.

Hardest problem

Keep movement useful while images, model decisions, and wireless commands arrive at different times.

Result so far

Basic object approach and person following; MK3 v0.2.0 with simulation and managed updates.

The project.

Omni began with a simple goal: type a task on my laptop and have a physical robot carry it out. I designed the chassis and motor-controller brackets in Fusion, assembled the hardware, and directed an AI-assisted software development process. The project grew from browser-based driving into person following, several MK2 navigation experiments, and a rebuilt MK3 controller. By October 28, I had demonstrated basic object approach and person following through GPT-6 mode. The work now also includes a standalone Unity simulator and a versioned desktop application with managed updates.

The wider project

A personal four-motor omnidirectional robot linking custom mechanics, ESP32 firmware, computer vision, and a Windows control application.

My contribution

I designed the chassis and brackets, assembled and integrated the hardware, set requirements, tested behavior, and directed the controller’s successive redesigns. AI coding tools assisted with software implementation and testing.

Prompt-based tasks

I demonstrated object approach and person following in GPT-6 mode; repeatability testing continues.

MK3 · v0.2.0

Standalone Unity simulation, managed updates, preserved user data, and a retained previous release.

Obstacle avoidance

General navigation reliability and measured stopping behavior remain open physical-testing questions.

Development · July–October 2026

One robot.
A changing control system.

The hardware provided continuity while the software changed. Each stage exposed a different integration problem and shaped the next design.

  1. July 2026

    Manual control

    Start with the motors and a browser

    The first layer was direct control: map four motors to forward, sideways, and rotational movement, then send WASD/QE commands over Wi-Fi. Early work involved pin assignments, wheel direction, flashing problems, and a robot-hosted control page.

    Engineering decision

    I worked through the physical wiring and direction mapping before adding autonomy. The earliest recovered browser firmware includes a command timeout and Stop, but no camera or IMU.

    What changed: Establish a dependable path from an input to the correct motor outputs before adding perception.

  2. August 2026

    Camera + IMU

    Add a view of the world

    Camera streaming and MPU-6050 orientation feedback turned the robot into a remotely observable platform. Firmware experiments added speed ramps, combined movement, sensor resets, and separate tasks for sensing and control.

    Engineering decision

    When camera and sensor changes degraded WASD response, I asked to preserve the working controls and add diagnostics. Later dual-core work separated attitude processing, while I kept pushing for compact, useful telemetry.

    What changed: Image capture, image delivery, sensor sampling, and control response have different timing constraints.

  3. Late August 2026

    Omni-Controller

    Narrow the task to following a person

    The early Omni-Controller desktop focused on one behavior: select a person, turn to center them, then move toward an estimated standoff distance. It combined person detection, identity tracking, learned depth, and IMU-guided turns.

    Engineering decision

    I deliberately narrowed the scope to understand one useful behavior. The preserved implementation uses YOLO11s and BoT-SORT in a native PySide6 application, with handling for target loss and reacquisition.

    What changed: A specialized follower made perception-to-motion integration more concrete; it was not a general navigator.

  4. September–early October 2026

    OmniControlMk2

    Explore a broader navigation stack

    MK2 explored measured navigation with ROS 2, OpenVINS, and Nav2, followed by a ROS-free ER2/NoMaD approach. I specified a slower visual reasoning layer paired with faster local navigation, then pushed toward a simpler camera-relative operating mode.

    Engineering decision

    Connecting models required a custom bridge: a semantic instruction is not automatically a navigation policy’s visual goal. MK2 also strengthened command authentication, finite motion leases, timing checks, and the separation between manual and autonomous control.

    What changed: Loading the models and passing software tests did not establish calibrated physical navigation.

  5. Mid-October 2026

    OmniController MK3

    Rebuild the controller around clear responsibilities

    I requested a fresh MK3 implementation around the existing hardware and GPT-6.1 Sol. The new structure separated planning, local checks, embedded motor control, and recorded feedback, with a native Windows interface and later optional depth and D1 correction.

    Engineering decision

    The rewrite carried forward hardware knowledge and lessons from MK2. Later, specific camera and IMU implementations were ported by request. Local corrections were bounded so they could adjust an authorized maneuver without taking over the mission.

    What changed: A planner’s intent, the host’s permission to move, and the firmware’s motor output need distinct boundaries.

  6. October 23–28, 2026

    Physical milestones

    Bring the software back to physical results

    Unity simulation, camera-freshness repairs, and a JEV visual-waypoint engine expanded the test environment. A short October 25 floor test recorded forward travel and a Stop. On October 28, I reported basic prompt-based object approach and person following in GPT-6 mode.

    Engineering decision

    The short JEV test ended with a low-confidence hold and a time-limit Stop. I kept that narrow result separate from the later basic-autonomy demonstrations and from broader claims about navigation.

    What changed: Obstacle avoidance, measured approach accuracy, and repeatability still require their own physical tests.

  7. October 29, 2026

    MK3 · Release 0.2.0

    Deliver a maintained desktop application

    MK3 v0.2.0 combined the revised dashboard, a bundled Unity Player, a richer workshop scene, and managed updates. Recorded acceptance confirmed the installed app staged and activated the new release while preserving settings and retaining the previous version.

    Engineering decision

    My delivery requirement included checking the actual installation. The report records 369 passing package tests, two Linux-only skips, matching installed-file hashes, and unchanged shared data after the update.

    What changed: Release verification establishes the delivery path; simulator results remain separate from real-world robot performance.

Inside OmniController MK3

From a goal
to bounded motion.

The laptop handles visual reasoning and supervision. The robot handles sensing and motor output, with its own command deadlines.

Operator inputA typed goal or manual controls
01 / Laptop + model services

Interpret & check

The selected engine interprets images and the task. The host checks frame freshness, command ownership, operator limits, and whether motion is still allowed.

Python · FastAPI · Windows WebView2
02 / Embedded control

Accept & supervise

The ESP32-S3 accepts authenticated commands with finite lifetimes. A separate motor-supervisor task enforces deadlines and updates output independently of camera work.

XIAO ESP32-S3 Sense · C++ firmware
03 / Physical platform

Move & observe

Two dual motor drivers control four motors for forward motion, strafe, and rotation. The front camera and MPU-6050 return images and orientation feedback.

Custom Fusion chassis & brackets

Feedback returns to the laptop. Images, relative orientation, and command acknowledgments inform the next decision. Learned depth is associated with the source frame; it is not calibrated wheel odometry.

Alternate test target

Unity Player supplies a rendered camera and simulated robot behind the same controller interface. It runs without the Unity Editor; its geometry and motion remain approximations of the physical robot.

Three selectable autonomy approaches

These are alternative operating modes within MK3.

General visual planning

SOL

GPT-6.1 Sol uses the goal and visual evidence to choose observations, movement, or Stop. The host validates each proposed action.

Planning + local correction

SOL + D1

A local GPU model makes limited adjustments between Sol decisions. It can correct or stop an authorized maneuver without changing the mission.

Visual waypoint steering

JEV

A remote model selects a short-range waypoint in the image. A local policy aligns, advances, or holds according to visibility, confidence, and current evidence.

Engineering decisions

How it came together.

Keep motion continuous, but bounded

Repeated short commands could restart acceleration ramps, leaving the robot barely moving even when the planner kept asking it to advance. I pushed for holding useful motion while the next decision was computed. MK3 separates that intent from short commands that must be renewed; fresh sensor evidence and independent firmware deadlines still determine whether movement can continue.

Use the newest image, not a growing queue

More camera resolution did not automatically make control better. Camera transport, model processing, and the display could each add delay. The later system prioritizes the latest frame, checks its identity and age, and associates depth with that exact image. Reducing physical capture to 320 × 240 also required adapting the upload image to the waypoint model’s input requirements.

Separate what was requested from what happened

A model can ask for movement while another part of the system withholds it. I requested clearer logs and feedback so a stopped robot would not simply be classified as stuck. The controller distinguishes proposed actions, accepted commands, electrical motor output, and visual feedback. The MPU-6050 supplies relative orientation; it does not establish absolute position or measured wheel speed.

Treat installation as part of the project

I wanted OmniController to run as an application with a bundled simulator, preserved settings and recordings, and updates from tested GitHub releases. The release process now uses versioned packages, checksums, a stable launcher, health checks, and a retained previous version for rollback. This extended the engineering work from robot behavior to maintaining the software used to operate it.

Results & next questions

What the evidence shows.

Physical demonstrations, software tests, and simulation answer different questions.

Physical robot

Basic autonomy

My October 28 update reported object approach and person following in GPT-6 mode. A separate October 25 JEV record documents short forward travel, a low-confidence hold, and acknowledged zero motor output at Stop.

These are initial demonstrations, not a measured success rate or proof of obstacle avoidance.

Software delivery

Verified release path

The v0.2.0 report records 369 passing package tests and two Linux-only skips. Installed package hashes matched the archive, shared data remained unchanged, and the previous release was retained after activation.

Recorded acceptance on my laptop; broad fresh-machine compatibility remains separate.

Simulation

A repeatable test target

The bundled Unity Player delivered rendered 320 × 240 camera frames. Recorded checks covered translation, strafe, rotation, Stop, source loss, and restarting disarmed.

Simulation supports development; it does not establish physical accuracy or collision avoidance.

Project record through October 29, 2026. Based on the recovered development history and release acceptance in Omni: comprehensive engineering and development report, second iteration. Early experiments and reported physical results retain their original scope.

Project photographs

A look at the build.

Explore the hardware, design views, and development stages.

Inside the software

Decisions become
motor commands.

Three earlier OmniControlMk2 source excerpts, developed with AI coding assistance. These document an earlier implementation, not the current GPT-6 mode.

Limits forward and turn effort, and rejects camera evidence whose timing or identity is stale. This is a source excerpt from the development code; it does not establish physical test results.

omnicontrolmk2/visual_control.py · Lines 113–133
def bounded_effort(values,limits=DEFAULT_LIMITS):
    if not isinstance(values,(list,tuple)) or len(values)!=3 or not all(math.isfinite(v) for v in values):
        raise ValueError('Invalid visual effort')
    forward,lateral,turn=values
    if forward<0 or lateral!=0:raise ValueError('Visual control supports forward and turn only')
    forward=min(forward,MAX_FORWARD_EFFORT*limits[0]/DEFAULT_LIMITS[0])
    turn=max(-MAX_TURN_EFFORT*limits[2]/DEFAULT_LIMITS[2],min(MAX_TURN_EFFORT*limits[2]/DEFAULT_LIMITS[2],turn))
    scale=max(1.,(forward+abs(turn))/MAX_COMBINED_EFFORT)
    return float(forward/scale),0.,float(turn/scale)


def visual_frame_time(frame,evidence,now):
    if evidence.get('qualified_frame') is not True or evidence.get('source')!=CONTROL_MODE:
        raise ValueError('; '.join(evidence.get('blockers',[])) or 'Waiting for camera/IMU timing')
    if (frame.boot,frame.session,frame.profile)!=(evidence.get('boot'),evidence.get('session'),evidence.get('profile')):
        raise ValueError('Visual frame identity changed')
    delta=(frame.captured_us-evidence['captured_us'])/1e6
    stamp=evidence['captured_monotonic']+delta
    if not -.1<=delta<=.25 or not 0<=now-stamp<=.25:
        raise ValueError('Visual capture expired')
    return stamp
Source code explains the implementation. Demonstrations establish physical behavior.Download excerpts

Currently in development

The next milestone.

Measure repeatability, approach accuracy, and obstacle handling on the physical robot.

Evidence to look forRepeated trials across scenes, target loss and reacquisition, loaded stopping tests, and Wi-Fi performance while the motors run.

Skills developed

FusionMechanical designHardware integrationESP32-S3PythonC++ firmwareControl architectureComputer vision integrationSimulation & testingAI-assisted development

Project demonstration

Original project recordingOpen original video ↗