Building Resiliency: A Production Failover Cluster for a Two-Person Team
Problem
Before this project, production virtual machines ran on infrastructure without a real failover story — a single hardware fault on the wrong host could mean real downtime for business-critical systems, with no automated way to recover. That’s an acceptable risk for a lab environment; it’s not acceptable for the systems a $150M distribution business depends on to take orders and move product every day. The challenge was building genuine resiliency without the dedicated infrastructure team a larger company would throw at the problem.
Approach
I designed and built a production Windows Server Failover Cluster (internally named prod1) across three Dell PowerEdge R660 nodes (HV1, HV2, HV3), backed by iSCSI storage from a Synology RS3621XS+ NAS. I split storage into two Cluster Shared Volumes by workload priority — an 8TB volume for the most critical VMs and a 6TB volume for general workloads — so the highest-priority systems had storage isolation from lower-priority ones. A file share witness on the domain controller gives the cluster proper quorum without needing a third full node just for that purpose.
I then migrated and registered 14 production VMs as clustered roles, meaning any of them can now fail over to a healthy node automatically if a host goes down, instead of requiring manual intervention at 2am. I documented the entire build extensively in Confluence, both so the second team member could operate and troubleshoot it independently, and so the design decisions (why this storage split, why this quorum model) wouldn’t get lost.
Outcome
Regal now has genuine hardware failure tolerance for its production virtual environment — 14 VMs that can survive a single node failure without manual recovery — built and maintained by a two-person team rather than a dedicated infrastructure group. That resiliency became load-bearing infrastructure for everything that came after it: the HQ26 relocation, the ERP go-live, and the hybrid cloud work all run on top of this cluster’s stability.
[Fill in: any actual failover event this cluster handled, or a downtime-avoided estimate if you have one]