Technical Plan - Compare with Repository and Import Firmware from Connected ESP32
Date: 2026-07-14
Status: proposed, not implemented
Scope
This plan covers two approved future capabilities:
1. Compare with repository.
2. Import firmware from connected ESP32.
Explicitly out of scope:
- Restore backup package.
- OTA.
- Provisioning.
- Fleet Management.
- Edge OS.
Existing Model
The current implementation already provides the required foundation:
- "firmware": firmware catalog entry.
- "firmware_segment": ordered segment relation for multipart firmware.
- "artifact": binary artifact library identified by SHA-256.
- "device_backup": backup entity separated from firmware.
- "backup_region": detected/read flash regions associated with a backup.
- "backup_artifact": full dump and ZIP artifacts for backups.
Important rule: backup and firmware remain different concepts. A backup is a restorable device-state copy and may contain sensitive or device-specific state. Imported firmware is a reusable catalog item and excludes sensitive/device-specific regions by default.
Part A - Compare with Repository
Goal
Compare a captured backup, local partition read, or connected-device layout against the firmware repository using reusable region hashes, not only full dump hash.
The comparison must detect exact reusable-firmware matches even when NVS, OTA data, coredump or other mutable state differs.
Matching Levels
Implement explicit match levels:
- "FULL_DUMP_MATCH": complete backup SHA-256 matches a stored artifact or backup.
- "FIRMWARE_FINGERPRINT_MATCH": reusable-region fingerprint matches a known firmware or backup fingerprint.
- "REGION_EXACT_MATCH": one region SHA-256 matches an artifact.
- "SEGMENT_LAYOUT_MATCH": offsets, sizes and segment names match a firmware layout, even if one or more hashes differ.
- "NO_MATCH": no useful match found.
Data Model Additions
Add a small persistent comparison model only if needed for audit/history. Minimum proposal:
- "repository_compare_result"
   - "id"
   - "source_type": "BACKUP", "CONNECTED_DEVICE", "LOCAL_REGIONS"
   - "source_id": nullable for browser-only previews
   - "firmware_fingerprint_sha256"
   - "match_level"
   - "best_firmware_id"
   - "best_backup_id"
   - "matched_region_count"
   - "total_reusable_region_count"
   - "created_at"
   - "result_json"
For MVP, comparison can be computed on demand and returned by API without persistence. Persist only if it helps audit and user workflow.
Backend API
Add read-only API endpoints:
- "GET /api/v1/backups/{id}/compare-repository"
   - Computes comparison for a saved backup.
   - Uses existing "backup_region.sha256" and "firmware_fingerprint_sha256".
- "POST /api/v1/repository/compare-regions"
   - Accepts a list of regions from the browser or an import preview.
   - Fields: name, type, subtype, offset, size, classification, sha256.
   - Returns match summary and per-region matches.
Optional later endpoint:
- "GET /api/v1/firmware/{id}/fingerprint"
   - Returns firmware fingerprint computed from its reusable segments.
Backend Logic
Create a service such as "RepositoryCompareService".
Steps:
1. Normalize region names and subtypes.
2. Exclude non-reusable regions by default:
   - NVS
   - OTA data
   - coredump
   - NVS keys
   - Sensitive
   - Device state
   - Diagnostic
3. Sort reusable regions by "offset_address".
4. Compare each region SHA-256 against "artifact.checksum_sha256".
5. Join artifact matches back to "firmware_segment" and "firmware".
6. Count matches per firmware.
7. Compute or reuse firmware fingerprint for known firmware entries.
8. Return:
   - best candidate firmware;
   - matched regions;
   - missing regions;
   - differing regions;
   - sensitive regions ignored;
   - confidence label.
Firmware Fingerprints for Repository Entries
Current firmware catalog does not yet store a normalized reusable fingerprint. Add one of these approaches:
Preferred MVP:
- Add nullable columns to "firmware":
   - "firmware_fingerprint_sha256"
   - "firmware_fingerprint_size_bytes"
   - "firmware_fingerprint_regions_json"
- Compute it for multipart firmware by concatenating included segment artifacts in offset order.
- For simple/merged firmware, use the whole artifact as a single reusable region at its flash address unless metadata states otherwise.
Alternative:
- Keep computed values transient and avoid migration until Import proves the format. This is simpler but slower and less inspectable.
Recommended: persist fingerprint columns because backups already persist their fingerprint and UI comparison becomes clearer.
UI
Add Compare with repository controls in Backups:
- On each saved backup row: "Compare with repository".
- In flash layout/import preview: "Compare selected regions".
Result panel:
- Best match.
- Match level badge.
- Firmware name/version/application/hardware.
- Reusable regions matched versus total.
- Ignored sensitive regions.
- Table:
   - Region
   - Offset
   - Size
   - SHA-256
   - Repository match
   - Firmware candidates
   - Classification
Validation
1. Compare existing backup #2.
2. Confirm "app0" matches the known artifact SHA-256.
3. Confirm firmware fingerprint remains stable when full dump SHA changes due to NVS.
4. Confirm NVS is reported as ignored/sensitive, not as a firmware mismatch.
5. Confirm at least one repository match is displayed.
6. Confirm no write operation is performed on ESP32.
Part B - Import Firmware from Connected ESP32
Goal
Read a connected ESP32 and create a reusable firmware catalog entry from selected non-sensitive regions.
Default behavior must avoid importing device-specific or private data.
Import Flow
1. User opens Backups > Import firmware from connected ESP32.
2. Connect ESP32 via Web Serial over HTTPS.
3. Detect chip, revision, MAC, flash ID and flash size.
4. Analyze flash layout.
5. Display regions with classifications.
6. Select safe defaults:
   - bootloader: included.
   - partition table: included.
   - active application: included.
   - reusable filesystem: included with warning.
   - NVS: excluded.
   - OTA data: excluded.
   - coredump: excluded.
7. Optional advanced selection with explicit warning for sensitive regions.
8. Read selected regions in browser using existing chunked reader and dual progress model.
9. Upload selected regions to backend as a temporary import session.
10. Backend computes SHA-256 for every uploaded region.
11. Backend compares with artifact library and reuses existing artifacts by SHA-256.
12. UI shows import preview.
13. User enters metadata:
   - name;
   - version;
   - application;
   - hardware compatible;
   - description;
   - origin;
   - tags;
   - notes.
14. User confirms "Create firmware".
15. Backend creates "firmware" and "firmware_segment" rows linked to "artifact" rows.
16. Firmware is marked:
   - source: "CONNECTED_DEVICE".
   - status/validation: "UNVERIFIED".
17. User can edit, download and flash it like other multipart firmware.
Import Session Model
Add temporary model independent from saved backups:
- "firmware_import_session"
   - "id"
   - "status": "CREATED", "UPLOADING", "READY_FOR_PREVIEW", "COMPLETED", "DISCARDED", "FAILED"
   - "chip"
   - "chip_revision"
   - "source_mac"
   - "flash_id"
   - "flash_size_bytes"
   - "created_at"
   - "expires_at"
   - "metadata_json"
   - "error_message"
- "firmware_import_region"
   - "id"
   - "session_id"
   - "name"
   - "type"
   - "subtype"
   - "offset_address"
   - "size_bytes"
   - "flags"
   - "classification"
   - "included"
   - "sha256"
   - "temporary_path"
   - "artifact_id", nullable until finalized
Use filesystem temporary storage outside public web paths, for example:
- "/srv/esp-platform/imports/tmp/<session-id>/"
Final artifacts stay in the existing artifact library location managed by "FirmwareService"/artifact storage rules.
Backend API
Add endpoints under "/api/v1/firmware-imports":
- "POST /api/v1/firmware-imports/sessions"
   - Creates a temporary import session.
- "PUT /api/v1/firmware-imports/sessions/{id}/regions/{regionIndex}"
   - Uploads one selected region.
   - Includes offset, size, name, subtype, classification and checksum if browser already calculated it.
- "POST /api/v1/firmware-imports/sessions/{id}/verify"
   - Verifies file sizes, SHA-256 and region metadata.
   - Checks overlaps and flash bounds.
   - Reuses artifacts by SHA-256 when possible.
- "GET /api/v1/firmware-imports/sessions/{id}/preview"
   - Returns import preview and repository comparison.
- "POST /api/v1/firmware-imports/sessions/{id}/complete"
   - Creates firmware and firmware_segment rows.
   - Moves or links artifacts.
- "POST /api/v1/firmware-imports/sessions/{id}/discard"
   - Removes temporary files.
Artifact Rules
Follow the already approved artifact-library rules:
1. Every imported region is hashed with SHA-256.
2. If SHA-256 exists in "artifact", reuse it.
3. If not, store a new artifact.
4. A "firmware_segment" references an "artifact".
5. Deleting imported firmware must not delete shared artifacts.
6. Artifact physical cleanup remains an orphan-cleanup operation, not part of firmware deletion.
Segment Naming
Recommended names:
- "bootloader"
- "partition-table"
- "app0"
- "app1"
- "spiffs"
- "littlefs"
- "filesystem"
Preserve original partition names where possible. Store offsets exactly as detected.
Active App Selection
For MVP, include app partitions selected by the user. Do not attempt to infer active OTA boot state from "otadata" unless explicitly needed.
Future enhancement:
- Read and interpret "otadata" to mark active app, but keep it excluded from imported reusable firmware unless user intentionally includes it.
Security and Privacy
Before importing filesystem or sensitive partitions, show warnings:
- NVS may contain SSIDs, passwords, tokens, certificates and device identity.
- Filesystem may contain credentials or customer data.
- Coredump is diagnostic only.
Defaults:
- Exclude NVS.
- Exclude OTA data.
- Exclude coredump.
- Include filesystem only with visible warning and clear classification.
Backend logs must not print credentials or file contents.
UI
Within Backups > Import firmware from connected ESP32:
Sections:
1. Device detection.
2. Flash layout.
3. Region selection.
4. Repository comparison preview.
5. Firmware metadata form.
6. Import confirmation.
7. Result with link to firmware catalog entry.
Use existing dual progress:
- Global progress: total bytes across all selected regions and uploads.
- Partial progress: current region read or current upload.
Validation
1. Connect known ESP32.
2. Analyze flash layout.
3. Confirm safe defaults exclude NVS, OTA data and coredump.
4. Import bootloader, partition table, app0/app1 and filesystem as selected.
5. Confirm artifacts are created or reused by SHA-256.
6. Confirm firmware row has source "CONNECTED_DEVICE" and validation "UNVERIFIED".
7. Confirm firmware segments keep offsets and sizes.
8. Confirm imported firmware appears in Firmware list.
9. Confirm it can be selected in Flasher as multipart firmware.
10. Flash imported firmware to another compatible ESP32 only after explicit approval for that physical validation.
11. Confirm deleting imported firmware does not remove shared artifacts.
12. Confirm temporary import session cleanup leaves no orphan files.
Recommended Implementation Order
Step 1 - Repository Comparison Backend
- Add comparison DTOs.
- Add "RepositoryCompareService".
- Add backup comparison endpoint.
- Add tests with existing backup regions and artifact matches.
Step 2 - Repository Comparison UI
- Add "Compare with repository" button on saved backup rows.
- Add result panel with best match and per-region table.
- Validate against backup #2.
Step 3 - Firmware Fingerprints for Catalog Entries
- Add firmware fingerprint columns if approved.
- Compute fingerprints for existing firmware records.
- Display fingerprint in firmware detail only where useful.
Step 4 - Import Session Backend
- Add import session and import region migrations/entities.
- Add temp upload endpoints.
- Add verify/preview/discard endpoints.
- Add cleanup for expired sessions.
Step 5 - Import UI Read and Preview
- Add Import firmware screen inside Backups.
- Reuse connection/layout/read code from Backups Stage 1.
- Upload selected regions to import session.
- Show repository comparison preview.
Step 6 - Complete Import
- Create firmware metadata form.
- Finalize artifacts and segments.
- Create firmware entry with source "CONNECTED_DEVICE" and validation "UNVERIFIED".
- Link to Firmware detail page.
Step 7 - Validation
- Run non-destructive import from the known ESP32.
- Verify database rows and filesystem artifacts.
- Confirm no sensitive regions are imported by default.
- Do not flash to another ESP32 until explicit physical validation approval.
Open Decisions Before Implementation
1. Should firmware catalog entries get persisted fingerprint columns now, or should comparison compute them on demand for MVP?
2. Should "firmware" get explicit "source", "tags" and "validation_status" columns before Import, or encode them temporarily in description/metadata? Recommended: add columns.
3. Should imported filesystem regions be selected by default? Recommended: yes for known reusable firmware, but with a visible warning; no for unknown/custom devices.
4. Should Import support a complete raw full-flash artifact as advanced option in this phase? Recommended: not in first import MVP; keep full dumps under Backups.