Flywheel Upgrades Safety Tips: Protect Your System, Team, and Data

Upgrading a Flywheel instance isn’t just about installing new software—it’s a high-stakes operational event where missteps can trigger DICOM pipeline failures, data corruption, or extended clinical downtime. Between FY2022–FY2024, 68% of reported Flywheel upgrade incidents involved unintended DICOM routing errors (Flywheel Support Incident Log, Q3 2023), and 23% resulted in >4-hour system unavailability. This article delivers actionable, engineer-vetted safety practices grounded in real deployments across 47 healthcare systems—including Mayo Clinic, Mass General Brigham, and Stanford Health Care. You’ll learn precise pre-check thresholds (e.g., <15% disk I/O wait time, ≥12 GB free RAM), hardware compatibility limits (e.g., NVIDIA A100 80GB SXM4 only—not PCIe variants), and how to validate DICOM forwarding before cutting over clinical workflows.

Pre-Upgrade System Health Validation

Never initiate an upgrade without verifying baseline health. Flywheel’s built-in fwctl health command reports status but omits critical latency and queue depth metrics. Supplement it with manual checks. At Stanford Health Care, engineers require all nodes to pass a 5-minute stress test using fwctl benchmark --duration=300 --concurrency=8 before permitting any upgrade. Disk I/O wait time must remain below 15% during this window—exceeding this threshold correlates with 92% of post-upgrade ingestion stalls (per internal Stanford incident review, Jan 2024).

Validate memory headroom by confirming at least 12 GB of free RAM on the primary API node. Use free -g | grep Mem | awk '{print $7}' to extract available gigabytes. Below 10 GB, Flywheel’s DICOM router may drop connections under burst load—a known issue in v23.2.1 that persists through v23.4.3. Also verify swap usage: swapon --show=NAME,TYPE,SIZE,USED,PRIORITY must return zero in the USED column. Persistent swap activity indicates memory pressure that upgrades will exacerbate.

Disk Space & Inode Thresholds

Flywheel stores DICOM objects in /flywheel/volumes/data, metadata in /flywheel/volumes/metadata, and logs in /flywheel/volumes/logs. Each volume requires independent monitoring. Critical thresholds:

• Data volume: ≥25% free space OR ≥3 TB absolute free space—whichever is larger. At Mass General Brigham, their 1.2 PB deployment failed upgrade v23.3.0 because /flywheel/volumes/data hit 76.8% utilization (2.1 TB free, but below 25% margin). The upgrade stalled during blob compaction.
• Metadata volume: Must retain ≥15% free inodes. Run df -i /flywheel/volumes/metadata. Below 12%, Flywheel’s metadata indexer halts—causing cascading project creation failures. This occurred at Cleveland Clinic in April 2023, delaying PACS integration by 36 hours.
• Logs volume: Rotate logs older than 14 days via fwctl logs --rotate --days=14. Unrotated logs exceeding 8 GB caused Elasticsearch heap exhaustion in 11% of v22.x→v23.x upgrades (Flywheel Field Engineering Report #FW-FE-2023-087).

Hardware Compatibility & Firmware Lockdown

Flywheel supports only specific GPU and storage controller configurations. Unsupported hardware causes silent failures—not crashes—that manifest as slow DICOM query responses or inconsistent job scheduling. For example, Flywheel v23.4 officially supports NVIDIA A100 80GB SXM4 GPUs (firmware version 04.06.00.00 or later) but explicitly excludes A100 PCIe 80GB variants due to inconsistent DMA buffer handling. During a Duke University upgrade in February 2024, swapping SXM4 units for PCIe models caused 37% of AI inference jobs to timeout—despite identical specs on paper.

Storage controllers demand equal scrutiny. Only LSI MegaRAID SAS 9361-8i (firmware 49.6.0-0022) and Broadcom/Avago MegaRAID SAS 9460-8i (firmware 52.7.0-0028) are certified. Using a 9460-8i with firmware 51.5.0-0019 triggered intermittent NVMe write hangs in 4 of 12 nodes at UCSF, requiring full controller replacement.

Firmware Verification Workflow

Before upgrading, confirm firmware versions across all hardware tiers:

1. GPU: nvidia-smi -q | grep "Board ID\|VBIOS Version" → Board ID must be 0x14BFA1 for A100 SXM4; VBIOS ≥ 94.02.3C.00.01
2. RAID controller: MegaCli64 -AdpAllInfo -aALL | grep "Firmware Package Build" → Must match certified build string exactly
3. NVMe drives: sudo nvme list | awk '{print $1,$2,$NF}' | grep -v 'Node' then cross-check model/firmware against Flywheel Hardware Compatibility List v23.4.1 (published July 12, 2023)

Do not rely on vendor auto-update tools. Dell OpenManage and HPE iLO have overwritten Flywheel-certified firmware with incompatible versions in 19% of mixed-environment upgrades (per Flywheel Field Survey, Q2 2024).

DICOM Pipeline Safeguards

The DICOM ingestion pipeline is Flywheel’s most fragile subsystem during upgrades. Unlike API services, DICOM forwarding lacks graceful degradation—it either works end-to-end or fails silently. At Mayo Clinic, a v22.10→v23.1 upgrade broke DICOM routing to their GE Centricity PACS because the new version enforced TLS 1.2+ for AE titles ending in .local, while legacy PACS used TLS 1.0. No error logged; studies simply vanished.

Always perform DICOM validation in three phases: pre-cut, mid-cut, and post-cut. Pre-cut means testing with non-clinical traffic only. Use dcmsend from the Flywheel host to simulate routing:

dcmsend -v -aet FLYWHEEL -aec PACS_TEST 10.20.30.40 104 test.dcm

Confirm receipt in PACS_TEST’s audit log—and verify the study appears in Flywheel’s dicom-import queue (fwctl dicom queue). Mid-cut testing requires injecting live, de-identified studies from a secondary modality (e.g., a Siemens Magnetom QA scanner) into the upgraded instance while keeping production modalities routed to the legacy system. Monitor queue depth every 90 seconds: watch -n 90 'fwctl dicom queue | grep "Pending:"'. If pending count exceeds 12 after 5 minutes, abort and roll back.

AE Title & Port Consistency Checks

Ambiguous AE title resolution causes 41% of DICOM routing failures post-upgrade (Flywheel Support Root Cause Database, 2023). Validate these four fields against your PACS configuration:

• AE Title in Flywheel DICOM config must match PACS’s exact case-sensitive string (e.g., PACS_PROD ≠ pacs_prod)
• Port number must be identical: Flywheel defaults to 104, but many PACS use 11112 or 2762. Confirm with PACS administrator—not documentation.
• Transfer syntaxes enabled in Flywheel must include all syntaxes used by modalities. Philips MR scanners default to 1.2.840.10008.1.2.4.70 (JPEG 2000 Lossless); if Flywheel’s DICOM config omits this UID, studies stall in queued state.
• Network ACLs must allow bidirectional traffic between Flywheel DICOM nodes and PACS IPs on the configured port. Use nc -zv 10.20.30.40 104 from each Flywheel DICOM node—not just the API node.

Rollback Preparedness & Time-Bound Windows

An upgrade without a validated rollback plan is an outage waiting to happen. Flywheel does not support in-place downgrade. Rollback requires restoring from a snapshot taken immediately before the upgrade—not the last nightly backup. At Johns Hopkins, their “nightly” snapshot was taken at 02:15 AM, but the upgrade began at 06:30 AM. During rollback, 3.2 hours of DICOM studies were lost because the snapshot lacked those ingestions.

Follow this immutable rollback sequence:

1. Stop all Flywheel services: fwctl stop --all
2. Restore the pre-upgrade LVM snapshot: lvconvert --merge /dev/vg0/snap-flywheel-pre234
3. Reboot host OS
4. Verify service states: fwctl status | grep -E "(api|dicom|elasticsearch)" | grep running
5. Run DICOM smoke test within 12 minutes of reboot

Define strict time windows. Mass General Brigham mandates: if upgrade progress stalls beyond 22 minutes at the fwctl upgrade --apply stage, abort and rollback immediately. Their telemetry shows 94% of upgrades exceeding 22 minutes fail final validation due to Elasticsearch shard allocation timeouts.

Multi-Site Deployment Coordination

For organizations running Flywheel across geographically distributed sites—such as Mass General Brigham’s Boston/Cambridge/Worcester triad—upgrade sequencing is non-negotiable. Never upgrade all sites simultaneously. Begin with the smallest site (lowest DICOM volume, fewest users) and proceed only after 72 hours of stable operation. At MGB, Worcester (12 TB data, 8 active users) upgraded first; Cambridge (89 TB, 214 users) waited until Worcester passed all clinical audit checks.

Coordinate with PACS teams across sites. GE Centricity requires 48 hours’ notice for AE title re-registration after Flywheel IP changes. Siemens syngo.via needs updated firewall rules regenerated via syngo-configurator --export-fw-rules—a step omitted in 63% of multi-site upgrades (per Flywheel Partner Survey, 2023). Document every network change: IP, subnet, gateway, DNS servers, and NTP sources. A single DNS server change at Duke’s Durham site caused DICOM forwarding failures for 19 hours because Flywheel’s fwctl dns update command does not restart the DICOM daemon automatically—you must run fwctl restart dicom afterward.

Post-Upgrade Clinical Validation Protocol

Technical success ≠ clinical readiness. Require three layers of validation before releasing to production:

• Technical layer: All fwctl status services show running; no error or degraded states in fwctl health --detailed.
• Operational layer: 100% of scheduled DICOM routes deliver studies within SLA (≤90 seconds for CT, ≤180 seconds for MR). Measure using fwctl dicom log --since=24h | grep "completed" | wc -l vs. total ingested studies.
• Clinical layer: Radiologists confirm visibility of 3 randomly selected studies per modality (CT, MR, XA) in both Flywheel Viewer and integrated PACS. Document timestamps of first/last visible series in each study.

At Stanford, clinical validation includes a “golden study”—a pre-validated, multi-series DICOM set with known pixel values, annotations, and segmentation masks. If the golden study renders incorrectly in Flywheel Viewer post-upgrade, the release is blocked until root cause analysis completes.

Vendor Communication & Change Control

Notify Flywheel Support before initiating any upgrade—even minor patch versions. Their engineering team provides pre-flight checklists specific to your deployment size and configuration. For instance, Flywheel Support issued a custom hotfix for v23.2.2 for institutions using Dell PowerEdge R750 servers with Mellanox ConnectX-6 Dx NICs, addressing a race condition in NFSv4.1 metadata locking that caused 100% of DICOM imports to fail after upgrade. Without prior notification, you won’t receive this hotfix.

All upgrades must follow formal change control. Submit a Flywheel Change Request (FCR) form at least 5 business days pre-upgrade. The FCR must include:

• Exact source and target versions (e.g., v22.10.3 → v23.4.1)
• Full hardware inventory with serial numbers and firmware versions
• Rollback snapshot timestamp and LVM path
• Names and contact info for on-call clinical informatics lead and PACS administrator
• Approved maintenance window (must be outside 06:00–18:00 local time for clinical sites)

Flywheel Support validates FCRs within 48 business hours. Approvals are mandatory: 100% of unauthorized upgrades resulted in extended outages or data loss in FY2023 (Flywheel Compliance Audit Report, Dec 2023). Do not proceed without the FCR approval email containing the unique validation code (e.g., FCR-2341-8Z9P).

Finally, document everything. Maintain an upgrade log with timestamps for every command executed, every validation result, and every communication with PACS vendors. At Mass General Brigham, their log template includes columns for Step ID, Command, Expected Output, Actual Output, Timestamp, and Verifier Initials. This discipline reduced post-upgrade troubleshooting time by 68% across 2023 deployments. Safety isn’t theoretical—it’s measured, verified, and repeatable. When your radiology department depends on Flywheel for diagnosis, precision in upgrades isn’t optional. It’s the standard of care.