# Sovereign Hybrid Compute, Microgrid Resilience & Thermal Cogeneration — Blueprint v2

> Design concept for discussion. Not stamped engineering, financial advice, or a certified election system.
> All electrical, plumbing, fire and structural work requires a licensed PE, permits and AHJ inspection.
> Vendor figures marked "verify" must be confirmed against current datasheets.

## 1. Summary

Three electrical tiers and two heat paths:

- **T1 Essential IT (1,880 W):** A/B double-conversion UPS. Either UPS carries 100% (70% of 2,700 W).
- **T2 Deferrable AI batch (3,000 W):** parks gracefully below 50% SOC and resumes at 60%.
- **T3 Interruptible hydro mining (11,000 W budget):** ramps with SOC from 80% (full) to 65% (zero). Hard trip below 40% through a hardwired coil chain.
- **Facility peak:** 15.9 kW. Shedding T3 removes 69%; shedding T2+T3 removes 88%.
- **Recovered heat (hydro loop, 95% capture):** 35,657 BTU/h ≈ 10.45 kW thermal, needing 3.71 GPM at ΔT 20 °F with 30% PG.
- **Efficiency:** reported as metered ERF (ISO/IEC 30134-6), not an assumed percentage. A rear-door HX captures the air-side IT heat.

## 2. Gap audit (30 findings)

### [CRITICAL] Thermal trip could never fire (Automation)
- **v1:** ESPHome DS18B20 sensors report °C, but the cutoff was "above: 155.0 # Fahrenheit". 155 °C is never reached, so the protection silently does nothing.
- **v2:** Thresholds are now in °C (150 °F = 65.6 °C) and checked on the device, so the trip works even when Home Assistant or MQTT is down.

### [CRITICAL] A/B UPS pair could not carry the load (Electrical)
- **v1:** The essential bus carried 4,030 W (including the GPU server), but each SRT3000 is rated 2,700 W. In an A/B design each UPS must carry 100% of the load alone.
- **v2:** The GPU server moves to a Tier-2 deferrable circuit. Tier-1 is 1,880 W, about 70% of one UPS, so either UPS can carry everything.

### [CRITICAL] Load shed was not fail-safe (Electrical)
- **v1:** A shunt-trip coil must be energized to open. If control power, the ESP32 or the relay fails, the mining bus stays energized, which defeats the protection.
- **v2:** An energize-to-run contactor. Its coil runs through a hardwired series chain: BESS SOC dry contact, the thermal interlock and the EMS permissive. Losing any link sheds the load in under 50 ms with no software involved.

### [CRITICAL] Three-phase calculation was wrong (Electrical)
- **v1:** "60A × 0.80 × √3 ≈ 66.5A (17,280 W)" mixes current and power: 66.5 A is not a meaningful figure here.
- **v2:** Continuous current = 60 A × 0.80 = 48 A per phase. Power = √3 × 208 V × 48 A ≈ 17.3 kW.

### [CRITICAL] Mining fleet numbers did not agree (Hardware)
- **v1:** "4× Whatsminer M53S+ … 120–140 TH/s per unit … 11,000 W". Hydro M53S-class units are multi-kilowatt each, so four would draw well over 11 kW. The hash rate given is also far below that class.
- **v2:** The fleet is defined by an 11 kW power budget: for example, 2 hydro units with a firmware power limit. Unit count and TH/s must come from the current MicroBT datasheet.

### [HIGH] GPIO12 is an ESP32 strapping pin (Automation)
- **v1:** GPIO12 sets the flash voltage at boot. A relay pulling it high can stop the board booting, or brown out the flash.
- **v2:** The relay moves to GPIO26. 1-Wire is on GPIO4, flow on GPIO27, and the I²C DAC on GPIO21/22.

### [HIGH] The ESP32 cannot output 0–10 V (Automation)
- **v1:** The valve output was listed as "0–10V Analog Output", but the ESP32 DAC is 0–3.3 V and too weak to drive a Belimo actuator input.
- **v2:** A GP8403 I²C 0–10 V DAC module (ESPHome gp8403 platform). The valve fails toward the dry cooler.

### [HIGH] Invalid YAML: two systems mixed together (Automation)
- **v1:** One file mixed ESPHome device config with a Home Assistant automation: block. Neither tool will accept it.
- **v2:** Split into two files: an ESPHome node with on-device interlocks, and Home Assistant supervisory automations (2024.10+ syntax).

### [HIGH] Load-shed thresholds contradicted each other (Electrical)
- **v1:** The text said shed below 65% SOC; the code said below 40%.
- **v2:** Tiered response: below 80%, curtail mining in proportion via the power-limit API. At 65%, mining is at zero. Below 40%, hardwired hard trip, latched until 60%. Re-enable only above 75% with solar surplus.

### [HIGH] "Air-gapped" zone sat on a shared cluster (Security)
- **v1:** VLAN 20 was called air-gapped while sharing the hypervisor cluster, the core switch and the trunk with tenants and mining. A VLAN is not an air gap.
- **v2:** A physically separate R650 enclave with its own switch. Export goes out only through a hardware one-way data diode (signed hashes and audit evidence).

### [HIGH] Election role had no scope limit (Governance)
- **v1:** The "election verification engine" had no limits on what it does or what data it may hold.
- **v2:** Audit-evidence support only. Certified voting systems remain the official record (EAC VVSG and state law). Voter identity, biometric data and ballot selections are refused. A named human approves every placement.

### [HIGH] Legionella risk in DHW preheat (Thermal)
- **v1:** Preheating potable water to 77–120 °F puts storage in the Legionella growth range.
- **v2:** Double-wall vented heat exchanger for potable water. The downstream heater stores at ≥140 °F (60 °C), with thermostatic mixing at the fixtures and scheduled thermal disinfection.

### [HIGH] BESS and interconnection codes missing (Safety)
- **v1:** The battery, solar and generator design cited no fire or interconnection standard.
- **v2:** NFPA 855 with UL 9540/9540A-listed BESS, IEEE 1547 / UL 1741 inverters, NEC 690/705/706/710, a utility interconnection agreement and review by a licensed PE.

### [HIGH] Glycol manifold mounted above the UPS units (Safety)
- **v1:** The rack put the hydronic manifold (U7–10) directly above both UPS units (U2–5), with no leak detection. A leak would drip onto the power supply for every load.
- **v2:** The manifold moves out of the IT rack into the hydro enclosure. The rack keeps only bottom-fed rear-door HX lines with a drip tray, leak rope and auto-closing isolation valves. EPO per NEC 645.

### [HIGH] "Automated multisig revenue routing" contradicts 3-of-5 (Governance)
- **v1:** Phase 3 called for automated routing of funds, but a 3-of-5 multisig only works if three humans sign. Automation that can sign removes the protection.
- **v2:** Automation only prepares the transactions (PSBTs) to allowlisted addresses. Three keyholders review and sign on hardware devices. The coordinator holds no keys.

### [MEDIUM] Mixed service voltages (Electrical)
- **v1:** A 240 V single-phase essential circuit and a 208 V three-phase mining panel normally cannot both come from one service.
- **v2:** One 208Y/120 V three-phase service. Feed A and Feed B are separate 208 V/20 A two-pole circuits (3.3 kW continuous each).

### [MEDIUM] Shed percentage overstated (Electrical)
- **v1:** Shedding was claimed to drop "80%+ of the facility load".
- **v2:** 11.0 of 15.9 kW ≈ 69%. Adding the Tier-2 GPU shutdown brings it to about 88%.

### [MEDIUM] GPU server power understated (Hardware)
- **v1:** The GPU server was listed at 2,200 W, but four A100 SXM4 GPUs alone are 4 × 400 W, before CPUs, memory and fans.
- **v2:** Planned at 3.0 kW on its own 208 V/20 A circuit (3.3 kW continuous).

### [MEDIUM] Two-node Proxmox cluster had no quorum (Compute)
- **v1:** Two hypervisor nodes cannot keep quorum when one fails or sheds.
- **v2:** A QDevice provides the third vote. The GPU node can power down on SOC thresholds without stopping the cluster.

### [MEDIUM] One VLAN does not isolate tenants from each other (Security)
- **v1:** Every tenant shared VLAN 30, so tenant-to-tenant traffic was possible at layer 2.
- **v2:** Proxmox SDN gives each tenant a VNet, firewall group and MAC/IP filtering, with no shared broadcast domain.

### [MEDIUM] "Authenticated Modbus TCP" is not standard (Security)
- **v1:** Plain Modbus TCP has no authentication.
- **v2:** Use Modbus/TCP Security (TLS, port 802), or keep Modbus behind a protocol gateway. MQTT runs on 8883 with per-device certificates and topic ACLs.

### [MEDIUM] Fully blocked mining VLAN could not be curtailed (Security)
- **v1:** The EMS must reach the miners to change their power limit, but VLAN 50 was blocked from everything.
- **v2:** One pinned allow rule from the EMS host to the miner API port. Pool traffic uses Stratum V2 (encrypted and authenticated).

### [MEDIUM] Storage platform described as two different things (Storage)
- **v1:** "TrueNAS Enterprise / R740xd2, Dual Controller": the R740xd2 is a single-controller Dell server. The R650 also ran hardware RAID1 under ZFS.
- **v2:** TrueNAS on R740xd2 (single controller) or an iX M-series for dual-controller high availability. All ZFS pools run on HBAs. Use 2× 6-wide RAID-Z2 and 3-2-1 replication.

### [MEDIUM] Freeze point overstated (Thermal)
- **v1:** The doc said 30% propylene glycol protects to 2 °F. Approximate freeze point is about 8 °F (−13 °C).
- **v2:** 35% propylene glycol (≈0 °F) for northern sites. Check fluid concentration with a refractometer every year.

### [MEDIUM] Heat exchanger and pump specs were muddled (Thermal)
- **v1:** "Titanium brazed PHE" is rarely available. "0–10V Modbus signal" mixes two different control methods.
- **v2:** Gasketed titanium plate-and-frame for marine sites, or 316L brazed inland. MAGNA3 pumps use a CIM 200 Modbus RTU module.

### [MEDIUM] ">85% efficiency" claim was unmeasured (Thermal)
- **v1:** Air-cooled server heat was never captured, and heat delivered off-season is zero.
- **v2:** Report ERF (ISO/IEC 30134-6) from metered heat. A rear-door heat exchanger captures the air-side heat.

### [MEDIUM] Trip setpoint above typical hydro limits (Thermal)
- **v1:** 155 °F (68 °C) is above the outlet limit many hydro miners allow.
- **v2:** Warn at 145 °F, trip at 150 °F, and lower both to the vendor maximum if it is lower.

### [MEDIUM] "Category 5 rated bunker" is not a standard (Resilience)
- **v1:** The Saffir-Simpson scale rates storms, not buildings. ">3 ft AFF" is arbitrary.
- **v2:** Design to ICC 500 / FEMA P-361. Elevate above the base flood elevation plus freeboard per ASCE 24.

### [MEDIUM] "Smart contract" multisig on Bitcoin (Governance)
- **v1:** Bitcoin multisig is native script, not a smart contract. Holding public funds is restricted by law in many places.
- **v2:** 3-of-5 P2WSH/Taproot multisig on geographically split hardware signers, with a time-locked recovery path. Revenue converts to fiat daily, subject to public-funds counsel review.

### [LOW] Rack and roadmap labels (Documentation)
- **v1:** The rack was labelled "24U / 42U". Phase 1 was marked complete with no evidence.
- **v2:** Labelled 42U. Phase status now shows "reported, needs verification" until site evidence is attached.

## 3. Network segmentation

| VLAN | Zone | Policy |
| --- | --- | --- |
| 10 | Management / IPMI / iDRAC | Physically separate out-of-band switch. Reached only through a bastion host with FIDO2 hardware keys. No route to tenants or mining. |
| 20 | Sovereign enclave & election audit support | Not a VLAN on the shared cluster. A dedicated R650, a dedicated switch and a one-way data diode that exports signed hashes only. No inbound path from any network. |
| 30 | Multi-tenant hosting | Proxmox SDN gives each tenant its own VNet and firewall group, so tenants cannot reach each other at layer 2. Each tenant has CPU, RAM, IOPS and bandwidth quotas. |
| 40 | Smart farm & IoT telemetry | MQTT over TLS on port 8883 with a certificate for each device. Modbus has no authentication, so it stays behind a protocol gateway or uses Modbus/TCP Security (TLS, port 802). |
| 50 | ASIC mining & Stratum V2 | Default deny toward every internal subnet. Egress only to pinned pool endpoints, encrypted with Stratum V2. One allow rule from the EMS host to the miner power-limit API. |

Inter-VLAN default is **deny**. VLAN 20 is a physical enclave, not a logical VLAN on the shared cluster. Its only egress is a hardware one-way data diode.

**Election scope:** audit-evidence support only (artifact hash verification, risk-limiting-audit math). It never tabulates. Certified voting systems remain the official record (EAC VVSG, state law). Voter identity, biometric data and ballot selections are refused. A named human approves every placement.

## 4. Bill of materials

| Device | Role | Configuration | Form | Power | Tier |
| --- | --- | --- | --- | --- | --- |
| Dell PowerEdge R650 | Sovereign enclave (stand-alone, not clustered) | 2× Xeon Silver 4314, 256 GB ECC, 2× 960 GB NVMe as a ZFS mirror (HBA mode, no HW RAID) | 1U | 450 W | T1 |
| Dell PowerEdge R750 | Proxmox node A, multi-tenant host | 2× Xeon Gold 6330, 512 GB ECC, 4× 1.92 TB NVMe (ZFS mirrored vdevs) | 2U | 750 W | T1 |
| Supermicro 4U GPU | Proxmox node B, deferrable AI batch | 2× EPYC 7543, 512 GB, 4× A100 80 GB SXM4. The GPUs alone draw 1.6 kW | 4U | 3,000 W (plan) | T2 |
| TrueNAS on Dell R740xd2 | Storage (single controller) | 12× 18 TB SAS as 2× 6-wide RAID-Z2 (about 144 TB raw data space), plus offsite replication | 2U | 500 W | T1 |
| UniFi EnterpriseXG 24 | Core L3 switch | 10G access, 25G SFP28 uplinks. Check port media against the optics you buy | 1U | 80 W | T1 |
| Netgate 8200 | Edge firewall | pfSense Plus. Inter-VLAN default deny, Stratum V2 egress pinning | 1U | 50 W | T1 |
| OOB switch + bastion + QDevice | Management plane and cluster quorum | Small fanless node. Provides the third Proxmox quorum vote | 1U | 30 W | T1 |
| Enclave switch + data diode | One-way export from the sovereign enclave | Hardware-enforced unidirectional link | 1U | 20 W | T1 |
| Hydro ASIC fleet | Interruptible, flexible load | Sized by power budget, not unit count: e.g. 2 hydro units power-limited to about 5.5 kW each. Verify against the MicroBT datasheet | Hydro enclosure | 11,000 W | T3 |
| APC SRT3000 ×2 (208/230 V) | A/B double-conversion UPS | 2,700 W each. Each one carries the full T1 load (1,880 W ≈ 70%) | 2U ×2 | — | UPS |
| Rear-door heat exchanger | Captures the air-side IT heat | Tied into the low-temperature preheat stage | Rack door | — | Thermal |
| Leak-detection rope + EPO | Rack safety | Rope sensors under the manifold. Emergency power off per NEC 645 | — | — | Safety |

## 5. Electrical (208Y/120 V three-phase service)

| Circuit | Breaker | Continuous (80%) | Load |
| --- | --- | --- | --- |
| T1 Feed A → UPS A | 208 V 20 A 2P | 16 A · 3.3 kW | 1.88 kW |
| T1 Feed B → UPS B | 208 V 20 A 2P | 16 A · 3.3 kW | 1.88 kW (redundant) |
| T2 GPU | 208 V 20 A 2P | 16 A · 3.3 kW | 3.0 kW |
| T3 Mining sub-panel | 208 V 60 A 3P | 48 A · √3 × 208 × 48 ≈ 17.3 kW | 11.0 kW |

**Fail-safe shed:** the mining contactor is energize-to-run. Its coil is fed through a series chain: BESS SOC dry contact, thermal switch, then EMS/ESPHome permissive. Opening any link drops T3 in about 20–50 ms with no software in the path.

**Standards:** NEC 645/690/705/706/710, NFPA 855, UL 9540/9540A, IEEE 1547, UL 1741, utility interconnection agreement.

## 6. Hydronics

- Q [BTU/h] = P [W] × 3.412142
- GPM = Q ÷ (k × ΔT °F), with k ≈ 500 for water and ≈ 480 for 30% PG (verify at operating temperature)
- Primary: glycol 135 → 115 °F. Secondary: water 105 → 125 °F. Approach about 10 °F.
- 30% PG freezes at about 8 °F. Use 35% (≈0 °F) in northern sites.
- Plate HX: gasketed titanium plate-and-frame for marine sites, 316L brazed inland.
- Potable DHW: double-wall vented HX, downstream storage at ≥140 °F, thermostatic mixing at fixtures.
- Belimo 3-way valve fails toward the dry cooler. The hydronic manifold stays outside the IT rack.

## 7. Automation (three layers)

1. **Hardwired** coil chain (<50 ms).
2. **ESPHome on-device** interlocks: over-temperature (65.6 °C / 150 °F), low flow, sensor fault, supervisor loss. Mining is off at boot. See `config/thermal-node-01.yaml` (validated with `esphome config`).
3. **Home Assistant** supervisory layer: SOC-proportional power target, GPU parking, heat routing, leak isolation. See `config/ha-automations.yaml`.

## 8. Risk register

| Risk | Severity | Impact | Mitigation |
| --- | --- | --- | --- |
| Crypto price volatility | High | Mining revenue swings hurt cash flow | Mined coin converts to fiat daily. Base economics are modeled on heat sales only, so mining is upside. |
| Thermal saturation | High | Heat users stop drawing heat and compute overheats | Belimo valve fails toward the dry cooler. The device trips at 150 °F independently of HA, and low flow also trips. |
| BESS thermal runaway | Critical | Lithium battery fire | NFPA 855 spacing, UL 9540A-tested system, off-gas detection, fire-department pre-plan. |
| Glycol leak in IT rack | High | Short circuit or hardware loss | Manifold kept out of the IT rack. Bottom-fed rear-door HX, drip tray, leak rope, auto-closing valves. |
| Marine corrosion | Critical | Coastal electronics and piping degrade quickly | NEMA 4X enclosures, titanium heat exchanger plates, sacrificial anodes, conformal-coated PCBs. |
| Hurricane / flood | High | Grid loss, wind and flood damage | ICC 500 / FEMA P-361 structure, ASCE 24 elevation, islanding via ATS. |
| OT/IT cyber intrusion | High | Attacker pivots from IoT or mining into the core | Default-deny segmentation, Stratum V2, mTLS MQTT, data-diode enclave, hardware-key bastion. |
| Legionella in DHW | High | Public-health risk | Double-wall heat exchanger, ≥140 °F storage, thermostatic mixing valves, disinfection log. |
| Regulatory / interconnection | Medium | Utility, noise or zoning blocks operation | IEEE 1547 agreement, dry-cooler acoustic study, early review by the local authority. |

## 9. Governance

3-of-5 native Bitcoin multisig (P2WSH/Taproot) on hardware signers in different locations, with a time-locked recovery path. Keyholders: Municipal Infrastructure Director, Lead System Architect, Co-op Board Member, Independent Auditor and Microgrid Engineering Lead. Mined revenue converts to fiat daily, subject to public-funds counsel review. Base economics are modeled on heat sales alone.

## 10. Roadmap

See `phases/playbook.md` for the full test-gated execution plan.
- **Phase 1: Foundation** (Months 1–3): Site, permits & interconnection; Microgrid: PV, BESS, ATS; Essential IT (T1) & enclave; Network & zero-trust
- **Phase 2: Thermal integration** (Months 4–6): Mining bus & hydro fleet; Primary & secondary plumbing; Controls & sensing; Commissioning
- **Phase 3: Scaling & cooperative** (Months 7–12): Municipal hydronic connection; Community compute onboarding; Treasury & revenue routing; Reporting & audit