Terminal firmware upgrades: the field replacement problem

Firmware upgrades for a terminal fleet are one of those operations that looks trivial in a lab and turns into a nightmare in the field. The lab has good power, good network, and no customers. The field has none of those.

Based on fleet operations reviews 2023; written April 2025.

What goes wrong in the field that doesn't in the lab

Four failure classes I've seen repeatedly:

• Interrupted downloads. The terminal has enough battery for the download but not for the flash cycle; a customer transaction lands mid-upgrade and the firmware image is partially written.
• Network windows too narrow. The terminal only has connectivity during business hours; the upgrade window falls outside that.
• Version-skew issues with the acquirer. The new firmware handles some field correctly; the acquirer's front-end doesn't know the field yet.
• Merchant-specific configuration lost. The upgrade overwrites merchant-specific settings that were manually adjusted in the field.

What actually works

• Staged rollouts with observability at each stage. Deploy to a small fleet first; watch failure rates; scale up. This is boring and necessary.
• Resumable, idempotent update flow. The terminal must be able to resume an interrupted download without corrupting state, and the update process itself must be safe to retry.
• A rollback that works from the terminal side. If the new firmware doesn't complete first-transaction validation, roll back automatically without operator intervention.
• Merchant configuration held server-side, applied post-update. Never overwrite it and expect the merchant to re-enter it.

The firmware payload is a small fraction of the operational effort. The rollout mechanism is where the actual engineering lives.