Running AWS IoT Core at Scale: How We Rolled Out OTA Firmware to 300 EV Chargers

When we first imagined a network of smart EV chargers, the idea of updating their firmware remotely felt like a sci‑fi plot twist. Yet, as the pilot grew, we quickly realized that manual flashing wasn’t just inconvenient—it was impossible at scale. That’s when AWS IoT Core entered the picture, promising a managed way to talk to devices, handle certificates, and keep an eye on telemetry.

Our first real‑world test involved a handful of chargers installed at a corporate campus. The goal? Push a minor bug‑fix without sending a technician on site. We wrote a tiny script, paired it with AWS IoT’s Device Shadow, and—voilà—everything updated in under five minutes. The thrill was real, but it also raised a dozen new questions: How do we verify the update didn’t brick a charger? What happens if the network drops halfway through? And how do we keep a tidy audit trail for compliance?

Those “what‑ifs” drove the architecture we eventually settled on. At the core sits an AWS IoT Core thing registry, one entry per charger, each equipped with X.509 certificates for mutual TLS authentication. We leaned on the Device Defender service to flag any anomalous behavior—like a charger trying to download a firmware image that didn’t match its model number.

The OTA pipeline itself is a three‑stage dance. First, a CI/CD job (built on CodePipeline and CodeBuild) compiles the firmware, signs it with our internal PKI, and publishes the binary to an S3 bucket that is locked down with bucket policies. Second, a Lambda function watches the bucket, creates a signed URL that’s valid for just a few minutes, and writes a new “desired” state into the charger’s Device Shadow. Finally, the charger, which runs a lightweight MQTT client, sees the shadow change, fetches the firmware via HTTPS, verifies the signature, and reboots into the new image.

We deliberately added a safety net: a dual‑bank flash layout. The charger writes the new image to a secondary bank while the current one stays untouched. After a successful checksum, a tiny bootloader swaps the active bank and reports success back to the shadow. If anything goes sideways—checksum failure, power loss, or a bad signature—the bootloader simply boots the old bank, and the shadow is updated with a failure status. This “fail‑fast, fail‑safe” pattern saved us from a handful of near‑catastrophes during early roll‑outs.

Scalability turned out to be a pleasant surprise. AWS IoT Core can comfortably handle thousands of concurrent MQTT connections, but we still hit throttling on the S3 signed‑URL generation when we tried to push updates to all 300 chargers at once. The fix? Introduce a simple back‑off loop in the Lambda, and throttle the Lambda invocation itself using a reserved concurrency limit. The result was a smooth, staggered rollout that never overwhelmed any single service.

One of the most human moments of the project happened when a field engineer reported a charger that refused to reboot after an update. Our logs showed the device had downloaded the binary, but the checksum failed. Turns out a rogue developer had accidentally pushed a debug build that left out a required library. Because our bootloader kept the old firmware intact, the charger stayed online, and we could roll back the change with a single shadow update. No hardware had to be replaced, and the engineer got to sip coffee while we fixed the code.

Compliance was another beast we wrestled with. The energy sector demands a clear audit trail of every firmware version that ever ran on a charger. To satisfy auditors, we ship every signed firmware hash to an immutable ledger in AWS QLDB. The ledger entry includes the S3 object version, the signing certificate fingerprint, and the timestamp of the deployment. When a regulator asks, we can produce a tamper‑proof report in seconds.

Looking back, three takeaways stand out:

• Automation over manual. Even a “simple” firmware bump becomes a massive coordination problem without a pipeline.
• Design for rollback. A robust bootloader that can revert to a known‑good image is priceless.
• Observability is non‑negotiable. Device Shadows, CloudWatch metrics, and Device Defender alerts together gave us the confidence to push updates at scale.

Today, with more than 300 chargers spread across three states, the OTA system runs quietly in the background. Updates are scheduled during low‑usage windows, and most of the time, you’ll never notice them. If you do, you’ll probably just see a blinking LED as the charger finishes its reboot—exactly the kind of unobtrusive experience we aimed for.

We’re still iterating. Next on the roadmap are delta updates to shrink download size, and a more granular role‑based access model so partners can push feature flags without touching the core firmware. But for now, AWS IoT Core has proved itself as a reliable backbone for a production‑grade OTA solution, and we’re grateful for the peace of mind it brings to both engineers and EV drivers.