The goal is simple to say: put a camera on a small RC car, drive it around for a while, and have it learn to drive the same way on its own. Getting there has taken a few detours.
Starting in simulation
I started in Webots, a robot simulator, with a car in a little city world. The first controller did nothing smarter than steer along a cosine wave, but it got me comfortable with the steering/throttle loop before any hardware could break. A couple more simulated worlds later, I wanted something real.
Picking a brain
I prototyped an iOS app to use an iPhone as the onboard computer (great camera, great chip, already in my pocket), then settled on a Raspberry Pi with a camera module, which is much easier to bolt onto a car and hack on.
The real car
camera --> [ Raspberry Pi ] --> web dashboard
|
WiFi / HTTP
v
RC receiver --> [ ESP32 ] --> steering servo
--> motor (ESC)
- A 1/16 RC car as the chassis.
- An ESP32 drives the steering servo and motor controller. It also reads the normal RC receiver, so I can always grab the transmitter and override the car.
- A Raspberry Pi handles the camera and serves a small dashboard: live video, drive controls, telemetry and a record button. It talks to the ESP32 over the ESP32's own WiFi network.
Learning by copying
The plan is behavioural cloning, the same idea as NVIDIA's PilotNet: a small CNN that takes a camera frame and outputs a steering angle. To train it, I drive the car by hand and record what I did.
- Frames are recorded at 20 Hz and downscaled to 320x240 (about one frame every 10-15 cm of travel).
- Each frame is labelled with steering and throttle normalised to -1..1, taken from what the servos were actually doing, not what was requested. That matters, because manual driving is exactly when the two differ.
- A small inspection script checks each recorded session (dropped frames, dead stretches, label ranges) before it's allowed anywhere near training.
- Autonomous throttle is deliberately capped for now. Crashes are cheap at a walking pace.
Where it's at
The hardware works and I'm in the collecting-laps phase. Next up: train the first model, put it on the Pi, and see how many laps it lasts before it finds a wall. I'll update this post as it goes.