BHZ2A-03-BLOCK I-(1) · Systems data · Internal, for Nicholai and Davané

Agent-first VFX infrastructure.
The agent never runs on the machine doing the work.

A specification for scaling Biohazard's compute, storage and AI workloads past one workstation: an always-on control plane for Antimony and Signet, workstations and render nodes that are allowed to crash, a storage tier that can feed them, and a cost model built from what SXRS actually spent on the farm.

Prices observed 4 Nov 2026, USD, pre-tax Farm data SXRS, Oct 2026 Status Proposed, nothing purchased
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
SUM
SECTION 2  ·  SUBSECTION 2.0
SUMMARY AND DECISIONS
2.0.1

INTRODUCTION.

During SXRS the agent, Houdini, Nuke, Blender, ComfyUI and every render shared one Ryzen 9 7900X with 93 GB of RAM and a 24 GB RTX 3090. When a cook filled memory, the kernel's OOM killer took whatever it picked: once another session's Blender render, and in one overloaded hour the agent's own gateway went down. This handbook splits that machine into planes that fail independently. The control plane hosts the agent and its memory and does no DCC work. Execution machines run the DCCs and may crash without stopping anyone else. A storage tier feeds them, and the farm absorbs bursts.

2.0.2

FINDINGS THAT SET THE PLAN.

  • SXRS billed $1,234.69 on Fox for 1,098 node-hours: GPU 384 h at $1.87/h, CPU 714 h at $0.72/h (section 2.8).
  • The farm's real value was width. On 18 Oct it ran 326 GPU node-hours in one day, and on 20 Oct 504 CPU node-hours. Matching that in 24 h would take about 20 owned GPU nodes or 21 CPU nodes.
  • On render cost alone, owned GPU nodes pay back only near full utilisation: about 14,900 busy hours per node, or 1.7 years running 24/7. Four nodes cost what roughly 99 SXRS-sized GPU farm bills cost.
  • Owned hardware is therefore justified by what the farm cannot do: hosting the agent, iteration without upload and per-test billing, local AI models (SXRS spent $1,045.53 on hosted generation), sims and lookdev.
  • The TrueNAS server is degraded and its active link negotiates 1 GbE. That has to be fixed before anything else on this page can use it (section 2.6).
Table 2.0-1. Recommended Build Sequence
PhaseScopeParts, USD
0Repair NAS: 3 × 18 TB disks, mirrored boot pair, restore 10 GbE link1,807.32
1Control plane (3 × MS-01, Proxmox HA) and redundant 25/100 GbE fabric21,196.88
2NVMe render storage tier, 15.36 TB mirrored, 100 GbE23,201.02
3Agent workstation, MID tier (TR 9970X, 256 GB, RTX PRO 5000 72 GB)26,753.75
4Two GPU render nodes (RTX PRO 5000 72 GB, 4U)35,688.42
Starter build, phases 0–4108,647.39
5After pilot: GPU nodes 3–4; one refurb CPU node (quote: £12,895 + $1,079.96)35,688.42 +

Excluded: tax, shipping, assembly, UPS, electrical work, software, Proxmox subscription. Fabric is counted once (phase 1). Several prices are spot retail during a DDR5 shortage and should be re-quoted (2.3.4).

WARNING

Data pool DEGRADED: a second 12 TB member is FAULTED with 62 read errors. Boot SSD shows 295 write errors. Replace disks one resilver at a time before adding any load.

DECISIONS NEEDEDNV / DV
  1. 1NAS REPAIR - approve now
  2. 2SITE - studio room or colo
  3. 3POWER - two 240 V/20 A runs
  4. 4HOUDINI - Indie cap vs FX
  5. 5FINANCE - buy, lease, 179
  6. 6PHASES - approve 0-2 first
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
ARC
SECTION 2  ·  SUBSECTION 2.1
SYSTEM ARCHITECTURE
2.1.1

INTRODUCTION.

The system has four planes joined by one data fabric (figure 2.1-1). The control plane holds the agent gateway, Signet memory and the render manager on three small hosts that fail over between each other. The execution plane holds every machine that runs a DCC. The data plane holds the storage the work is read from and written to. The burst plane is Fox Renderfarm, reached over the internet and released only through a price gate.

OPERATORS NICHOLAI DAVANE CREW, VENDORS DISCORD, CHAT, LOCAL CLIENTS CONTROL PLANE · PROXMOX VE HA · 3 × MS-01 CP-1 CP-2 CP-3 AGENT GATEWAY SIGNET MEMORY RENDER MANAGER REPLICA REPLICA REPLICA REPLICA+ QUORUM ZFS REPL. EVERY 1 MINHA RESTART 1–3 MIN BURST PLANE FOX RENDERFARM UP TO 200 NODES PER JOB TEST FRAMES, THEN PRICE GATE WAN DATA PLANE NVME TIER 2 × 2 MIRROR 15.36 TB NFS / SMB3 HOURLY SNAPSHOTS HDD NAS RAIDZ2 × 10 34.7 TIB FREE ARCHIVE NEXTCLOUD SYNC, DELIVERY NOT RENDER STORAGE DATA FABRIC · 25 GBE TO NODES · 100 GBE TO STORAGE SWITCH A · CRS518 SWITCH B · CRS518 2 × 100G 10G 100G EXECUTION PLANE · DCC WORKERS 25 GBE WS-1 AGENT WS TR 9970X 256 GB 72 GB GPU WS-2 NICHOLAI 7900X 93 GB 3090 24 GB GPU-1 RENDER PRO 5000 72 GB 4U GPU-2 RENDER PRO 5000 72 GB 4U GPU-3 PHASE 5 GPU-4 PHASE 5 CPU-1 SIM, NUKE 2 × EPYC 512 GB PHASE 5 EACH WORKER: DCC SESSIONS + TOOL ENDPOINT + LOCAL NVME SCENE CACHE + GPU LEASE IN STARTER BUILD (PHASES 0–4) AFTER PILOT (PHASE 5) ARC-501A
Figure 2.1-1.Biohazard Compute, Functional Block Diagram
2.1.2

FUNCTIONAL DESCRIPTION.

A request reaches the agent gateway on CP-1. The agent recalls the shot's rules from Signet, publishes an immutable scene version to the NVMe tier, and submits a job to the render manager, which leases a GPU on a worker. The worker stages the scene to local NVMe, runs the DCC and writes frames back to the tier. When the queue exceeds owned capacity, the agent prices test frames on Fox and asks before releasing the range. If CP-1 dies, its VMs restart on CP-2 from the last one-minute replica, and running frames on the workers carry on.

2.1.2.1

Fault Isolation Rule.

A DCC process may run on any machine except a control host. A crash takes down at most one worker, and the manager requeues that worker's frames.

FRONT ELEVATION · 27U RACK SIDE ELEVATION · TOWERS CRS326 A CRS326 B CRS518 A CRS518 B CP-1 CP-2 CP-3 CPU-1 GPU-1 GPU-2 GPU-3 GPU-4 UPS 47.25 [1200] MGMT, COROSYNC SWITCH (2) DATA SWITCH 25/100 GBE (2) CONTROL HOSTS (3), 2U SHELF CPU NODE, 1U (PHASE 5) GPU NODE, 4U, 600 W (2) GPU NODES 3–4 (PHASE 5) UPS, A/B PDU (NOT PRICED) WS-1 AGENT WORKSTATION MESHIFY 2 XL NVME STORAGE TIER DEFINE 7 XL GPU 72 GB TR 9970X U.2 NVME (4) 21.3 [542] ARC-502A
Figure 2.1-2.Physical Arrangement, Starter Build and Phase 5 Positions
Table 2.1-1. Plane Responsibilities
PlaneHoldsOn failure
ControlAgent gateway, Signet, render manager, chat bridgesVM restarts on a peer, 1–3 min
ExecutionHoudini, Nuke, Blender, ComfyUI, LLM inferenceOne machine lost, frames requeued
DataPublished scenes, caches, renders, archiveRenders pause, last hourly snapshot on NAS
BurstFox Renderfarm, priced per jobWork stays on owned nodes
NOTE

Starter build draw is about 3 kW: two GPU nodes 1.2 kW, workstation 0.95 kW, control 0.24 kW, switches 0.24 kW, storage. All figures are planning estimates, not meter readings.

BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
CTL
SECTION 2  ·  SUBSECTION 2.2
CONTROL PLANE
2.2.1

INTRODUCTION.

The control plane is three identical Minisforum MS-01 hosts in a Proxmox VE cluster. Each host stores its VMs on a local ZFS mirror, and each VM is replicated to both peers every minute. If a host fails, Proxmox fences it and restarts its VMs on a survivor. The documented typical failover time is about two minutes. Planned maintenance uses live migration, which moves a running VM with no interruption.

2.2.2

WHY NOT KUBERNETES.

Kubernetes fails over seamlessly when it runs several identical stateless copies of a service. The agent gateway and Signet are not that. Signet writes one SQLite database, which allows a single writer, so only one copy can run. For a single-copy service on a lost node, Kubernetes waits for the node to go NotReady, then waits out the default 300 s eviction toleration, and then may wait up to 6 min to force-detach the volume before the pod can start elsewhere (figure 2.2-1). It also needs the same replicated storage layer Proxmox needs. The result is slower recovery with more to operate.

Kubernetes earns its place later in the execution plane, as a pool of stateless GPU services such as ComfyUI workers or model servers, where many identical copies are the point.

02 46 810 12 MINUTES AFTER HOST LOSS PROXMOX HA VM ON ZFS REPLICA FENCE + RECOVER BOOT, SERVICES UP ≈ 2.5 MIN KUBERNETES ONE POD, RWO VOLUME NOTREADY ≈ 40–50 S EVICTED AT 300 S VOLUME FORCE-DETACH, UP TO 6 MIN DETECT, WAIT START SERVICE CTL-501A
Figure 2.2-1.Recovery Timeline After a Host Loss, Default Settings (Illustrative)
2.2.3

FUNCTIONAL DESCRIPTION.

CP-1 runs one 8 vCPU / 16 GiB VM with the agent gateway, the Signet daemon and a CPU embedding worker, plus a 4 GiB VM for the render manager. Signet's SQLite file, WAL and SHM stay together on the VM's own disk. They are never placed on NFS or SMB. Both onboard SFP+ ports run at 10 GbE, one to each data switch. Both 2.5 GbE ports carry corosync, one to each management switch, so a single cable or switch never breaks quorum. Heavy AI inference stays on the GPU fleet. The control hosts carry no GPUs and no passthrough, which keeps live migration possible.

2.2.3.1

Recovery Point.

Replication is asynchronous. A crash loses at most the writes since the last successful replica, nominally one minute. Independent SQLite online backups go to the NAS and offsite, because replicas are not backups.

Table 2.2-1. Control Plane Bill of Materials
ItemQtyUSD
Minisforum MS-01, i9-13900H, 32 GB, 1 TB boot33,885.00
Samsung PM9A3 960 GB M.2 22110, PLP (ZFS mirror)611,633.94
Control plane total15,518.94
Estimated draw240 W
Proxmox subscriptionnot priced

The PLP drives are 75% of this cost at CDW's $1,938.99 list price. Re-quote them, or use another PLP M.2 22110 model, before ordering.

CAUTION

Never disable fencing to make a failover test pass. Two hosts running the same SQLite writer will corrupt Signet.

HOST LOSS DRILLANT
  1. 1SIGNET - write test memory
  2. 2CP-1 PWR - pull cord
  3. √ CP-2 - VMs running < 3 min
  4. 3SQLITE - integrity_check
  5. √ LAST RECORD - present
  6. 4CP-1 PWR - restore, migrate back
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
WKS
SECTION 2  ·  SUBSECTION 2.3
WORKSTATIONS
2.3.1

INTRODUCTION.

A workstation in this system is an execution machine that the agent drives with several DCC sessions open at once, and that an artist can also sit at. Three tiers were priced from live listings on 4 November 2026 (table 2.3-1). The hard requirement comes from SXRS: a city frame needs 28–33 GB, which does not fit a 24 GB 3090 and leaves no margin on a 32 GB RTX 5090. Every tier therefore carries at least 48 GB of VRAM.

A. STAR-KILLER 9995WX, 96 C PRO 6000 MAX-Q96 GB (3) 1 TB DDR5 ECC 12 TB NVME SCRATCH 100 GBE, 2050 W, 240 V $123,859.84 B. MID (RECOMMENDED) TR 9970X, 32 C PRO 5000 72 GB SLOT FOR 2ND GPU 256 GB DDR5 ECC 4 TB NVME SCRATCH 10 GBE, 1500 W $26,753.75 C. BUDGET 5900XT, 16 C A6000 48 GBREFURB 128 GB DDR4 850 W, 120 V $6,827.84 WKS-501A
Figure 2.3-1.Workstation Tiers, Front Elevation (Towers to Common Scale)
Table 2.3-1. Workstation Tier Data
ParameterStar-killerMidBudget
CPUTR PRO 9995WX, 96 CTR 9970X, 32 CRyzen 9 5900XT, 16 C
Memory1 TB DDR5-6400 ECC ($53,382.99)256 GB DDR5-5600 ECC ($11,790.99)128 GB DDR4-3200 ($916.99)
GPU3 × RTX PRO 6000 Max-Q 96 GBRTX PRO 5000 72 GBRTX A6000 48 GB, refurbished
GPU, Blender 5.2.0 OptiX median15,505 each10,9564,641
CPU, Blender 5.2.0 median2,6401,377412
Wall draw, estimated1,650–1,850 W, 240 V800–950 W550–650 W
UnlocksThree isolated 96 GB jobs; full sim, comp and model stack at onceThe city scene with room to spare; parallel DCCsThe city scene; little else at the same time
Main limit96 GB per device, no NVLink poolingOne GPU shared by render and AIOlder CPU, non-ECC, below a 3090 in speed
Parts total$123,859.84$26,753.75$6,827.84
2.3.2

SELECTION.

MID is the agent workstation (WS-1). One 72 GB card holds the city scene with margin, 32 cores and 256 GB carry several DCC sessions, and the board keeps a slot and the power supply sized for a second GPU. Reference: the current 3090 scores 5,087 on the same benchmark, so the MID card renders about 2.15 times faster before counting the end of out-of-core spill.

2.3.3

BUDGET ALTERNATIVE.

The cheapest fix for the scene-size problem alone is the same $4,799.99 refurbished A6000 installed in the existing 7900X workstation. It renders slightly slower than the 3090 but fits the scene. It does not separate the agent from the work.

2.3.4

MEMORY PRICING.

DDR5 is in shortage. Memory is 43% of the star-killer and 44% of MID. Exxact configured a comparable star-killer (9995WX, 1 TB, 3 × PRO 6000 Max-Q, rack chassis, three-year support) at $101,068, which is less than the retail parts. Get integrator quotes (Exxact, Puget, BOXX) before buying above the MID tier.

BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
GPU
SECTION 2  ·  SUBSECTION 2.4
GPU RENDER NODES
2.4.1

INTRODUCTION.

A render node is a 4U rack server with one RTX PRO 5000 Blackwell 72 GB, a Ryzen 9 9950X on a Supermicro server board with ECC, 128 GB of RAM, two mirrored NVMe drives for the scene cache and a dual 25 GbE NIC (figure 2.4-1). One GPU per node keeps a failure to one card's worth of work and fits a single 1000 W supply. Buy two, pilot them on the heaviest real frame, then add two more.

PLAN VIEW, LID REMOVED FRONT REAR I/O PSU 1000 W ATX 3.1 NVME 2 TB (2), MIRROR ECC UDIMM 32 GB (4) RTX PRO 5000 72 GB300 W, PCIE GEN5 CONNECTX-4 LX 2 × 25G RYZEN 9 9950X, NH-D12L ARCTIC P12 (3) SUPERMICRO H13SAE-MF REV 2.0 GPU-501A
Figure 2.4-1.GPU Render Node, General Arrangement (Not to Scale)
2.4.2

FUNCTIONAL DESCRIPTION.

The manager hands the node a chunk of frames. The node copies the published scene to its local NVMe mirror, keeps it resident across the chunk, renders, and writes frames back over 25 GbE. Loading the scene once per chunk instead of once per frame is the largest throughput lever (table 2.4-1). A node runs one GPU job at a time. ComfyUI and model jobs take the whole card through the same lease, never alongside a render.

CITY FRAME 28–33 GB RTX 3090, CURRENT24 RTX 5090, REJECTED32 RTX A6000, BUDGET WS48 RTX PRO 5000, NODE + MID WS72 RTX PRO 6000, STAR WS96 020406080100 VRAM PER DEVICE, GB GPU-502A
Figure 2.4-2.VRAM per Device Against the SXRS City Frame
Table 2.4-1. City Frame Throughput, Frames per Hour (Estimate)
Dispatch1 node2 nodes4 nodes
Scene resident, render only155310620
10-frame chunks, one 60 s load each123246493
Reload every frame4387173

Basis: 50 s/frame + 60 s load on a 3090, scaled by the Open Data ratio 10,956 / 5,087 = 2.15. Unverified until the pilot renders a real city frame.

Table 2.4-2. Node Bill of Materials
RTX PRO 5000 Blackwell 72 GB8,999.99
4 × 32 GB DDR5 ECC UDIMM5,847.96
9950X, H13SAE-MF, cooler1,303.31
2 × 990 PRO 2 TB, CX-4 Lx, RM1000x1,289.95
Sliger CX4170a 4U, fans, rails403.00
Per node / two / four17,844.21 / 35,688.42 / 71,376.84
CAUTION

Two 32 GB cards are not a 64 GB pool. Each device must hold the whole scene. Blender can spill to host RAM, at a large speed cost.

NODE ACCEPTANCEANT
  1. 1BIOS - rev 2.0, ECC on
  2. 2CITY 034 - render heaviest fr
  3. √ LOG - OPTIX, no CPU fallback
  4. √ PIXELS - match farm frame
  5. 3CHUNK - 10 fr, cold and warm
  6. 4PDU - record watts at load
  7. 5ORDER - nodes 3, 4 if pass
NOTE

Rejected: RTX 5090 (32 GB, no margin) and PRO 5500 84 GB (no firm price or lead time yet). Alternative if a job needs more than 72 GB: PRO 6000 96 GB, $16,513.99, 600 W.

BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
CPU
SECTION 2  ·  SUBSECTION 2.5
CPU NODES AND FARM SCALE
2.5.1

INTRODUCTION.

CPU nodes are for work GPUs do not do well: Houdini sims, Mantra and Karma CPU, Nuke renders, and scenes larger than any card. They are a poor way to buy Cycles throughput. On Blender 5.2.0 medians, matching one RTX PRO 5000 takes about ten new single-socket EPYC 9355P nodes, six dual EPYC 7713 or three dual EPYC 9555 (figure 2.5-1). A 16- or 64-node CPU render array is therefore the wrong first purchase.

A. CPU NODES PER RTX PRO 5000 (CYCLES) EPYC 9355P, 1 SOCKET9.5 2 × EPYC 77135.7 2 × EPYC 95552.9 2 × EPYC 7763 (QUOTED)NO 5.2 DATA 0510 NODES B. FARM DRAW, KW 4 GPU NODES2.4 16 REFURB CPU13.6 · 57 A 64 REFURB CPU54.4 · 227 A 64 REFURB, PUE 1.476.2 50 KW SITE LIMIT, RATE C 0204060 KW AT 100% LOAD, AMPS AT 240 V CPU-501A
Figure 2.5-1.Cycles Equivalence and Facility Load of a CPU Farm
2.5.2

WHERE SCALE BECOMES USEFUL.

Size owned compute to daily iteration and the farm to the deadline peak. SXRS's peak CPU day (20 Oct) was 504 node-hours on Fox's 48-thread nodes. Finishing that in 24 hours takes about 21 such nodes. The whole project's CPU work, 714 node-hours, billed $516.25. At $0.72 per node-hour on Fox against about $0.11 per hour in electricity alone for an 850 W refurb node, an owned CPU farm cannot pay for itself at this volume. Sixteen refurb nodes also need a 57 A feed and dedicated cooling, and sixty-four exceed the 50 kW site limit of Xcel's Schedule C.

2.5.3

RECOMMENDATION.

Buy one CPU node, after the GPU pilot, for sims and Nuke that are slow to farm out. The quote target is CTO Servers' 1U dual EPYC 7763 with 512 GB DDR4, listed at £12,895. That price is advisory, not a confirmed quote: the seller is in the UK and requires a quotation. Add two NVMe drives and a 25 GbE NIC for $1,079.96. The alternative is a new single EPYC 9355P with 512 GB DDR5, $39,664.39 in parts, of which $31,392 is RAM at spot retail.

Table 2.5-1. Monthly Electricity, 720 h at Full Load (Xcel CO, eff. 19 Oct 2026)
FleetkWRate C, winter energyRate SG, all-in
1 refurb CPU node0.85$77.80$154.35
16 refurb CPU nodes13.6$1,244.86$883.15
64 refurb CPU nodes54.4not eligible$3,215.31
64 new 9355P nodes35.2$3,221.98$2,117.82

Rate C winter $0.12713/kWh, summer $0.16878. Cooling excluded; at PUE 1.4, 64 refurb nodes reach about $4,459/month on SG.

CAUTION

Cycles scores do not predict Houdini sims, Mantra or Nuke. Benchmark the studio's own sim and comp script on the pilot node before buying a second.

CPU NODE QUOTEDV
  1. 1OEM MODEL - exact chassis
  2. 2RAM - real DRAM, not PMem
  3. 3PSU - 2 x 1200 W redundant
  4. 4NVME - 2 x4 paths confirmed
  5. 5PRICE - USD delivered, duty
  6. 6WARRANTY - US RTB terms
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
STO
SECTION 2  ·  SUBSECTION 2.6
STORAGE
2.6.1

INTRODUCTION.

The current TrueNAS server was inspected read-only on 4 November 2026. It is a Ryzen 5 3600 with 64 GB of non-ECC RAM, one 10-wide RAIDZ2 vdev of 12 TB disks, TrueNAS SCALE 24.04.0, and SMB only. Its ARC hit rate is 99.76%, mostly metadata. It cannot feed render nodes: the active port on its 10GBASE-T card negotiates 1 GbE, and the card sits in a PCIe 2.0 x4 slot. At 1 GbE, one node needs four minutes to pull a 30 GB scene, and eight nodes need 32 minutes.

A. AS FOUND RAIDZ210 × 12 TB DEGRADED 1 FAILED (SPARE IN USE) 1 FAULTED, 62 READ ERR 2 MORE > 600 REALLOCATED BOOT SSD: 295 WRITE ERR SLOG: SINGLE SATA SSD X540 10GBASE-T 1 GBE LINK PCIE 2.0 X4 ALL CLIENTS, SMB B. PROPOSED NVME TIER2 × 2 MIRROR SOLIDIGM D7-P5520 7.68 TB (4) EPYC 7302P, 256 GB ECC ARC CAP 192 GIB PLAN: 8 GB/S AGGREGATE SWITCH A/B 100 GBE 25 GBE EACH NODES, NFS 4.1 ARTISTS, SMB3 HDD NAS: SNAPSHOTS HOURLY: 48 H / 30 D / 8 W STO-501A
Figure 2.6-1.Storage As Found and Proposed Render Tier
2.6.2

FUNCTIONAL DESCRIPTION.

A separate NVMe server becomes the render tier. It holds active projects only, at most about 12.3 TB of live data and snapshots. Linux nodes mount it read-only over NFSv4.1, artists use SMB3, and render outputs land in separate datasets. The repaired HDD NAS becomes the archive and the hourly snapshot target. Nextcloud stays the sync and client-delivery layer and is never mounted as render storage. Option 1, upgrading the existing box to the same platform, costs $28,276.60 in total but keeps Nextcloud, apps and render serving in one failure domain.

2.6.2.1

No Cache Bolt-Ons.

A special vdev or L2ARC on the HDD pool does not speed up the first read of a freshly published scene, and losing a special vdev can lose the pool. Hot data goes on its own mirrored flash pool.

Table 2.6-1. Time to Stage a 30 GB Scene, Two Artists Streaming
Nodes loading togetherToday, 1 GbEOption 1Option 2
14 min12 s12 s
28 min20 s12 s
416 min40 s24 s
832 min80 s48 s

Planning arithmetic at 2.5 GB/s per client; today's column assumes the whole 0.125 GB/s link and no artists. Not measured.

Table 2.6-2. Render Tier Bill of Materials
4 × Solidigm D7-P5520 7.68 TB U.215,800.00
EPYC 7302P, H12SSL-i, 256 GB DDR4 ECC5,089.48
ConnectX-5 100 GbE, U.2 carriers, cooling1,045.80
Define 7 XL, RM850e, boot pair, 25G NIC for NAS, DACs1,265.74
Render tier, fabric excluded23,201.02
NAS repair: 3 × IronWolf Pro 18 TB, 2 × boot SSD1,807.32
WARNING

Replace one disk at a time and let each resilver finish. A second failure during resilver on this pool leaves no parity margin.

NAS REPAIRNV
  1. 1BACKUP - verify offsite copy
  2. 2FAULTED DISK - replace 18 TB
  3. √ RESILVER - complete, 0 err
  4. 3HIGH-REALLOC (2) - repeat
  5. 4BOOT - mirror 2 x 500 GB
  6. 5X540 - cable/port to 10 GbE
  7. √ LINK - 10000 Mb/s
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
NET
SECTION 2  ·  SUBSECTION 2.7
NETWORK FABRIC
2.7.1

INTRODUCTION.

Two MikroTik CRS518-16XS-2XQ switches carry data. Each has 16 SFP28 ports at 25 GbE and two QSFP28 ports at 100 GbE. The 100 GbE pair joins A to B. Two CRS326 switches carry management and corosync. Every node has two links, one to each switch, bonded active-backup, so either switch can be powered off without stopping work. The SFP28 cages accept 10 GbE with the same DACs, so no second 10 GbE switch is needed.

Table 2.7-1. Fabric Bill of Materials
ItemQtyUSD
MikroTik CRS518-16XS-2XQ-RM23,190.00
MikroTik CRS326-24G-2S+RM2418.00
XQ+DA0001 100G DAC, 1 m278.00
XS+DA0003 25G DAC, 3 m281,092.00
ConnectX-4 Lx 2 × 25G (workstation, storage)3899.94
Fabric total5,677.94
Table 2.7-2. Ports per Data Switch
GPU nodes, 25G4
CPU nodes, 25G2
Workstation, 25G1
Storage, 25G2
Control hosts, 10G3
Mgmt uplinks, 10G2
Used / spare14 / 2
NOTE

3 m DACs reach only within the rack. Workstations in another room need optics and fibre, not priced.

BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
CST
SECTION 2  ·  SUBSECTION 2.8
SXRS COST MODEL: FARM VERSUS OWNED
2.8.1

INTRODUCTION.

Every SXRS task on the Fox account was read through the free API: 99 billable layers under 73 root jobs, including eight jobs in the recycle bin that ordinary listings omit. SXRS billed $1,234.69 for 14,687 completed frame records and 1,098.37 observed node-hours. Nearly all of it (99.98%) fell between 17 and 23 October (figure 2.8-1).

0200400 NODE-HOURS 16 OCT0.3 1753 18326 GPU 1950 20504 CPU 21124 2236 GPU, BLENDER CPU, HOUDINI/MANTRA LOCAL DATE, OCTOBER 2026 · PEAK 327 FRAMES RENDERING AT ONCE CST-501A
Figure 2.8-1.SXRS Farm Work by Day
2.8.2

BREAK-EVEN.

A Fox GPU node-hour cost $1.87. Fox's GPU nodes are mostly RTX 5090 class (Open Data 16,072), so one hour of an owned RTX PRO 5000 (10,956) replaces 0.68 Fox hours, worth $1.27. Electricity for a 600 W node at Rate C winter is $0.08 per hour, leaving $1.20 per busy hour. A $17,844.21 node breaks even after about 14,900 busy hours: 1.7 years at 24/7, 3.4 years at half utilisation, 6.8 years at a quarter (figure 2.8-2). SXRS's GPU work would have kept four owned nodes busy for 141 hours each.

05,00010,00015,00020,000 BUSY HOURS PER NODE $0$10K$20K$30K OWNED NODE: $17,844 + $0.08/H POWER FOX EQUIVALENT: $1.27/H BREAK-EVEN 14,889 H 1 YEAR AT 24/7 SXRS PER NODE, 4 NODES: 141 H CST-502A
Figure 2.8-2.Cumulative Cost of One GPU Node Against Renting the Same Work
Table 2.8-1. SXRS Farm Totals (Fox API, 4 Nov 2026)
MeasureGPU (75)CPU (64)
Billed$718.44$516.25
Observed node-hours384.3714.1
Effective $/node-hour1.870.72
Completed frames4,17310,514
Peak day, node-hours326 (18 Oct)504 (20 Oct)
Stopped for low balance5 layers, $199.49
Hosted AI generation (fal)$1,045.53

Four GPU jobs ran on 2-GPU nodes and most report no model, so the 5090 equivalence is an assumption. The fal figure is usage from the 23 Oct advance reconciliation.

NOTE

On render cost alone, four owned nodes ($71,377) equal about 99 SXRS-sized GPU farm bills. The case for owning rests on 2.8.3.

CAUTION

Five SXRS layers stopped mid-range when the Fox balance went negative. Keep the balance above the projection plus 30% before any release.

2.8.3

WHAT OWNED HARDWARE BUYS.

  • A home for the agent that a DCC cannot take down. This one has no farm equivalent.
  • Iteration with no upload, no 15-minute cloud analysis and no per-test billing. Look rounds, flipbooks and test frames run at the speed of the network.
  • Local AI. SXRS spent more on hosted generation ($1,045.53) than on GPU rendering ($718.44). Open-weight video, roto and LLM workloads run on owned 72 GB cards for the cost of power. Closed models stay hosted.
  • Sims and lookdev that are impractical to farm, and client material that should not leave the building.
  • The farm stays as the burst plane. Owned nodes cover the daily baseline, and Fox covers the 18 Oct and 20 Oct peaks.
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
LIC
SECTION 2  ·  SUBSECTION 2.9
SOFTWARE LICENSING PER WORKER
2.9.1

INTRODUCTION.

Hardware scales freely, and licences do not. Blender has no per-node cost. Houdini Indie allows three Indie seats and three free Engine Indie machines per entity, and only for entities under $100K in revenue. A four-node Indie farm already exceeds that. Nuke charges per render worker. The render manager starts as Flamenco, which is free, because Deadline 10 has been in maintenance mode since 7 November 2025.

Table 2.9-1. Licence Consequences of Adding Workers (Prices Seen 4 Nov 2026)
SoftwareTermsEffect at 4 GPU + 2 CPU workers
Blender / CyclesGPL, no per-node fee$0
Houdini Indie$299/yr; entity revenue ≤ $100K; 3 Indie + 3 Engine IndieExceeds 3 farm machines
Houdini FX (commercial)$3,505/yr rental or $4,495 + $2,740/yr AUP; bundles 5 KarmaArtist seats only
Houdini Engine (commercial)$525/yr workstation; floating $795/yr, 25-pack $16,500/yrNeeded to cook .hip on workers
Karma (husk on exported USD)5 bundled with FX; extra $195/yr each6 workers: 1 extra, $195/yr
Nuke render (nuke_r)$462/yr per render worker2 CPU workers: $924/yr
FlamencoGPL, free; Blender native, custom job types for others$0, integration work
OpenCueApache-2.0, freeLater, at larger scale
Deadline 10Free, maintenance mode since Nov 2025Not selected
CAUTION

Husk jobs with Python callbacks that load Houdini libraries can check out Engine or FX licences even from plain USD. Qualify each job type before budgeting it as Karma-only.

BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
AGT
SECTION 2  ·  SUBSECTION 2.10
AGENT-FIRST OPERATIONS
2.10.1

INTRODUCTION.

Antimony becomes the studio's operating layer. Antimony is the agent, Signet is its memory, and the render manager is its hands on the farm. Artists and producers ask in chat. The agent turns each request into versioned scenes, leased GPU time, farm jobs with price gates, and delivered files. On SXRS the agent did this from inside the same machine it was loading. Here, every step crosses a network boundary on purpose (figure 2.10-1).

OPERATOR ANTIMONY SIGNET MANAGER GPU-1 NVME TIER FOX 1 RENDER 034 LIGHT PASS 2 RECALL SHOT RULES CAPS, LAST VERSION 3 PUBLISH SCENE V005, IMMUTABLE, CHECKSUMMED 4 SUBMIT, LEASE 1 GPU, 72 GB 5 CHUNK 1001-1010 6 STAGE TO LOCAL 7 FRAMES, LOGS 8 QUEUE > OWNED CAPACITY 9 OVERFLOW RANGE: 3 TEST FRAMES, READ PRICE PER LAYER 10 STATUS, PRICE, ASK GO AGT-501A
Figure 2.10-1.Request Sequence, Owned Capacity With Farm Overflow
2.10.2

FUNCTIONAL DESCRIPTION.

Each worker runs a small agent-facing service that opens DCC sessions (Houdini, Nuke, Blender, ComfyUI), runs scripts in them and reports back. The agent reaches workers only through the manager's lease, so two sessions never land on the same GPU. GPU, RAM and disk are booked through the lease system. On SXRS that booking was done with ad-hoc claims and memory caps, and the caps still lost jobs to the OOM killer. The agent keeps its existing farm discipline: price every pass from test frames, keep a cost guard running, never overwrite a pass someone is comping, and never let a full-range render run on an artist's machine.

2.10.2.1

Many Sessions, One Memory.

Any number of agent sessions can run on the control plane at once. They share Signet, coordinate through leases, and survive a worker crash because none of them lives on a worker.

Table 2.10-1. Failure Effects
EventEffect
Houdini takes down WS-1WS-1 reboots; agent and other workers unaffected
GPU node dies mid-chunkManager requeues the chunk on another node
CP-1 power lossAgent back on CP-2 in 1–3 min; renders continue
Data switch A failsBonds fail over to B; no interruption
NVMe tier offlineNew chunks wait; staged chunks finish; archive intact
Fox balance lowAgent asks for a top-up before release
NEW SHOT ON THE GRIDANT
  1. 1SIGNET - recall shot rules
  2. 2SCENE - publish vNNN, sums
  3. 3LEASE - GPU, RAM per job
  4. 4TEST - 1 frame local, check
  5. 5RANGE - owned nodes first
  6. 6OVERFLOW - Fox test, price
  7. √ GO - from Nicholai
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
PLN
SECTION 2  ·  SUBSECTION 2.11
BUILD PLAN AND OPEN QUESTIONS
2.11.1

SEQUENCE.

  1. Phase 0, now. Repair the NAS and restore its 10 GbE link. This needs no other decision.
  2. Phase 1. Control plane and fabric. Move the agent gateway and Signet off the workstation, then run the host-loss drill (2.2).
  3. Phase 2. NVMe render tier. Set up publish-and-stage, and switch Linux workers to NFS.
  4. Phase 3. MID agent workstation. The agent's DCC sessions move to WS-1, and Nicholai's machine is his own again.
  5. Phase 4. Two GPU nodes, a Flamenco pilot on the city scene, and node acceptance (2.4).
  6. Phase 5. Based on pilot data: GPU nodes 3–4, one CPU node, and a second GPU in WS-1. The star-killer only against a firm integrator quote.
2.11.2

QUESTIONS FOR NICHOLAI AND DAVANE.

  1. Site. Studio room or colocation? The starter build draws about 3 kW continuously, and is loud. A full 4 + 2 build reaches about 4.6 kW and needs two dedicated 240 V/20 A circuits, an electrician and cooling.
  2. Houdini entitlement. Biohazard's revenue against the Indie $100K cap decides whether the farm needs commercial Engine licences (2.9).
  3. Finance. Buy, lease or finance. Ask the accountant how Section 179 or bonus depreciation applies to equipment placed in service this year. This handbook does not give tax advice.
  4. Timing. DDR5 is at shortage prices. Buy the MID tier now, and hold larger memory configurations until prices ease or an integrator quote beats retail.
  5. Utilisation target. Owned GPUs pay off on render cost only near continuous use (2.8). Agree what share of GPU hours should go to local AI and lookdev to justify nodes 3–4.
  6. Backups. No independent offsite copy was verified during this inspection. Choose an offsite target before the NVMe tier goes into production.
Table 2.11-1. Cumulative Spend by Phase
Through phaseCumulative USD
0 · NAS repair1,807.32
1 · control + fabric23,004.20
2 · NVMe tier46,205.22
3 · MID workstation72,958.97
4 · two GPU nodes108,647.39
5 · GPU nodes 3–4144,335.81
NOTE

Every throughput, wattage and fit statement here is a planning estimate until the acceptance checklists in 2.2, 2.4 and 2.6 have run on delivered hardware.

APPROVALNV / DV
  1. 1PHASE 0 - approve, order disks
  2. 2QUOTES - integrator, CPU node
  3. 3SITE - room, circuits, cooling
  4. 4PHASES 1-2 - approve
  5. 5PHASES 3-4 - after 1-2 pass
BHZ2A-03-BLOCK I-(1)
BIOHAZARD VFX COMPUTE OPERATIONS HANDBOOK
SYSTEMS DATA
APPENDIX A
SOURCES AND RESEARCH RECORDS
A.1

RESEARCH RECORDS.

Full bills of materials, with every row's retailer, availability and observation date, are in the research files. Each file lists its own numbered sources.

  • workstations.md · .json: three workstation tiers, Open Data medians, Exxact comparison
  • nodes.md · .json: GPU and CPU nodes, control plane, fabric, Xcel rates, licensing
  • storage.md · .json: read-only NAS inspection and upgrade options
  • sxrs_fox.md: SXRS farm ledger, per task, per shot and per day
A.2

PRINCIPAL SOURCES.

  1. Blender Open Data, Blender 5.2.0, OptiX and CPU medians by device: https://opendata.blender.org/benchmarks/query/?compute_type=OPTIX&blender_version=5.2.0&group_by=device_name
  2. NVIDIA RTX PRO desktop GPUs (PRO 5000 72 GB, PRO 6000 Max-Q): https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/
  3. AMD Threadripper PRO 9995WX: https://www.amd.com/en/products/processors/workstations/ryzen-threadripper/9000-wx-series/amd-ryzen-threadripper-pro-9995wx.html
  4. Exxact TensorEX TWS-193767794 configuration: https://configurator.exxactcorp.com/configure/TWS-193767794
  5. CTO Servers dual EPYC 7763 1U, 512 GB: https://www.ctoservers.com/dual-socket-amd-epyc--1u-server--2x--epyc--7763--512gb-ram--nvme-4955-p.asp
  6. Proxmox VE HA manager, failover behaviour: https://pve.proxmox.com/pve-docs/chapter-ha-manager.html
  7. Proxmox VE storage replication: https://pve.proxmox.com/pve-docs/chapter-pvesr.html
  8. Kubernetes taint-based evictions, default 300 s toleration: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions
  9. SQLite WAL, same-host requirement: https://sqlite.org/wal.html
  10. MikroTik CRS518-16XS-2XQ-RM: https://mikrotik.com/product/crs518_16xs_2xq
  11. Minisforum MS-01: https://store.minisforum.com/products/minisforum-ms-01
  12. Xcel Energy Colorado rate books (Schedule C, SG, eff. 19 Oct 2026): https://www.xcelenergy.com/company/rates_and_regulations/rates/rate_books
  13. SideFX product comparison and pricing: https://www.sidefx.com/products/compare
  14. SideFX Indie render farm limits: https://www.sidefx.com/faq/question/indie-renderfarm-setup
  15. SideFX, licence use by husk: https://www.sidefx.com/faq/question/how-are-licenses-used-when-issuing-the-husk-command
  16. Foundry Nuke render licences: https://support.foundry.com/hc/en-us/articles/207823889
  17. AWS Thinkbox Deadline, free and maintenance status: https://awsthinkbox.zendesk.com/hc/en-us/articles/7865096598551
  18. Flamenco: https://flamenco.blender.org
  19. OpenZFS special vdev and caching: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Pool%20Structure/Special%20vdev.html
  20. Solidigm D7-P5520 product brief: https://solidigmtechnology.jp/content/dam/solidigm/en/site/products/data-center/d7/p5520/documents/Solidigm_D7_P5520_P5620.pdf
  21. Blender Cycles GPU rendering, out-of-core behaviour: https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html
  22. Fox Renderfarm account API (task list, recycle list, per-frame feeAmount and frameExecuteTime), read 4 Nov 2026; internal.