The 8-weekend AI factory plan
Twenty-three modules in four parts and ten hands-on HPC cluster labs, from AI workloads and GPU silicon to fabrics, facilities, orchestration and runbooks.
01Weekend 01: AI Workloads, Parallelism and Collectives
Module 1: AI Fundamentals for Infrastructure Engineers
- AI vs ML vs Deep Learning for Systems Engineers
- Why GPUs and Accelerators Beat CPUs for AI
- Model Parameter Scaling
- Compute Bounds vs Memory Bandwidth Limits
- Why the Fabric Becomes Mission-Critical at Scale
- MFU: Measuring Real Compute Efficiency
Module 2: AI Workloads and Network Traffic
- North-South vs East-West Traffic
- Elephant Flows and Synchronized Phases
- Microburst Dynamics
- Incast: Many-to-One and Buffer Exhaustion
- Forward Pass, Backward Pass and All-Reduce
- Why AI Overturns Enterprise Network Assumptions
Module 3: AI Parallelism Strategies
- Data Parallelism (DP) and FSDP
- Tensor Parallelism (TP) Inside the Node
- Pipeline Parallelism (PP) and Pipeline Bubbles
- 3D and Hybrid Parallelism
- Aligning Models with Network Topology
- Profiling PyTorch and TensorFlow Jobs
Module 4: Collective Communications and NCCL
- Broadcast, Gather, Scatter, Reduce
- All-Reduce, All-Gather, Reduce-Scatter
- Ring vs Tree vs Double-Binary Tree
- NCCL Environment Tuning
- Synchronization Barriers and Tail Latency
- How Collectives Drive Fabric Design
Profile a distributed PyTorch job
Lab 2: Collectives in Action (NCCL Ring vs Tree)
02Weekend 02: GPU Systems and Rack-Scale Architecture
Module 5: GPU and Compute System Architecture
- CPU, GPU, TPU, DPU and xPU Roles
- Hopper (H100/H200) vs Blackwell (B200/GB200/GB300)
- Tensor Cores, Transformer Engine, FP4/FP8
- HBM3e vs GDDR vs SRAM and NUMA Locality
- PCIe Gen5/Gen6, Root Complexes and Switches
- DGX and HGX Baseboards and SXM Modules
- ConnectX SuperNICs and BlueField DPUs
- Scale-Up vs Scale-Out Systems
Module 6: Scale-Up Networking
- NVLink 5 Generations and Signaling
- NVSwitch 5 Fabric ASICs
- Connecting 8, 16 and 72 GPUs
- Blind-Mate Copper Backplane vs Optics
- UALink and SUE-T Open Standards
- Scale-Up Design Principles
AI Factory deep dive: Rack Facilities and Bill of Materials
- 120 to 132 kW Power Shelves
- 415V/480V 3-Phase AC to 48V DC Busbars
- Direct-to-Chip Liquid Cooling Loops
- CDUs and Leak Detection
- Exploded BOM: Compute and Switch Trays
- OOB Switches and Cabling for Proposals
Lab 1: Meet the GPU Node (nvidia-smi, lspci, topology)
Size an exploded BOM for a GB300 NVL72 rack
03Weekend 03: Scale-Out AI Fabrics and Topologies
Module 7: Scale-Out AI Networks
- Quantum-2 and Quantum-X800 InfiniBand
- Subnet Manager (OpenSM)
- Spectrum-X Ethernet and SuperNICs
- Credit-Based Flow Control vs PFC/ECN
- Fat-Tree and Non-Blocking Clos
- Oversubscription and Bisection Bandwidth
- In-Network Computing with NVIDIA SHARP
- UFM Telemetry, BER Audits, Cable Health
Module 8: AI Network Topologies
- Multi-Tier Clos and High-Radix Switching
- Rail-Optimized 2-Tier vs 3-Tier Fat-Tree
- Managing the 3-Hop to 5-Hop Transition
- Dragonfly and Dragonfly+
- Slimfly and Diameter Reduction
- 1D, 2D and 3D Torus Interconnects
- Cost, Cabling and Resilience Trade-Offs
- OOB Management and North-South Ingress
Module 13: Load Balancing and Dynamic Routing
- Why ECMP Fails with Elephant Flows
- Flowlet Switching
- Packet Spraying
- Adaptive Routing and Queue Inspection
- RDMA Packet Reordering and Reassembly
- Maximizing GPU Utilization
Compare topologies on cost, hops and cabling
Read a UFM fabric telemetry and BER report
04Weekend 04: RDMA and Lossless Networking
Module 9: RDMA Fundamentals and Zero-Copy Architecture
- Kernel Bypass and Zero-Copy Transfers
- Queue Pairs, Work and Completion Queues
- Memory Registration and Protection Domains
- RDMA Verbs: Read, Write, Atomic
- GPUDirect RDMA Across Nodes
Module 10: RDMA Transport Technologies
- Native InfiniBand vs RoCEv1 vs RoCEv2
- iWARP over TCP
- Packet Formats and Encapsulation
- Latency and Throughput Benchmarks
- Choosing a Production Transport
Module 11: Lossless Ethernet for AI
- The Cost of Drops in RDMA
- PFC (802.1Qbb) and Headroom Buffers
- Head-of-Line Blocking and PFC Deadlocks
- DCB and DCBX
- ETS (802.1Qaz) Bandwidth Allocation
Module 12: Congestion Control Mechanisms
- ECN Marking and RED Thresholds
- DCQCN: CNP and Rate Control
- TIMELY, HPCC and Swift
- In-band Network Telemetry (INT)
- Production Tuning Recipes
Lab 3: RDMA Basics and Validation (ib_write_bw)
Lab 4: Lossless Ethernet Tuning (PFC, ECN, DCQCN)
05Weekend 05: AI Cluster and Data Centre Design
Module 14: Training vs Inference Networks
- Training Traffic: Bursty All-Reduce
- Inference Traffic: Low-Latency Streaming
- Frontend vs Backend vs OOB Networks
- Dedicated Low-Jitter Inference Design
Module 15: AI Data Centre and GPU Cluster Design
- Rail-Optimized vs Rail-Unified Designs
- Single-Rail vs Multi-Rail NIC Mapping
- GPU Pods and Scalable Units (SUs)
- NVIDIA SuperPOD Reference Architectures
- Spine-Leaf Cabling and Optics
- Failure Domains and Blast Radius
- Digital Twin Design with NVIDIA DSX Air
- Simulate AI factory topologies, BCM provisioning scripts, cabling maps, and failure injection scenarios in software using NVIDIA DSX Air before physical cluster deployment.
Module 16: Ultra Ethernet
- Ultra Ethernet Consortium (UEC) Vision
- Limits of Classic 802.3 Ethernet
- Multi-Pathing and Packet Spraying
- Out-of-Order Delivery, Selective Retransmit
- UEC vs RoCEv2 vs InfiniBand
- Ethernet Roadmap in HPC
Map NICs to rails for an 8-GPU node
Lab 5: Fabric Design Studio (32-node rail-optimized pod)
06Weekend 06: Power, Cooling and AI Storage
Module 17: Power Infrastructure for AI Data Centres
- 8 to 15 kW Racks vs 40 to 132 kW Racks
- 3-Phase 415V/480V and Overhead Busways
- Smart PDUs and Power Capping
- N+1, 2N and UPS Battery Sizing
- PUE and Carbon Efficiency
Module 18: Advanced Thermal Management and Cooling
- Air Cooling Limits
- Rear-Door Heat Exchangers (RDHx)
- Direct-to-Chip (DTC) Cold Plates
- Single-Phase vs Two-Phase Immersion
- CDUs, Manifolds and Facility Loops
- Leak Detection and Fluid Chemistry
Module 19: Storage and Checkpointing for AI
- Storage Across the AI Lifecycle
- HBM vs Local NVMe vs Shared Storage
- Lustre, GPFS (Storage Scale), WEKA, VAST
- Checkpoint Bursts and Recovery Time
- GPUDirect Storage (GDS) and NVMe-oF
- Dedicated Storage Rails vs HoL Blocking
Pre-deployment site survey: power and CDU audit
Lab 6: Storage Benchmarking with FIO and gdscheck
07Weekend 07: Provisioning, Software Stack and Orchestration
Module 20: NVIDIA Software Stack and AI Ecosystem
- GPU Drivers, Kernel Modules, CUDA
- Container Toolkit and NGC Containers
- MIG Partitioning vs vGPU and vWS
- NVIDIA AI Enterprise: TensorRT, RAPIDS, NeMo
- Serving with Triton and NVIDIA NIM
Module 21: Cluster Management and Workload Scheduling
- Base Command Manager: Head Nodes, PXE Boot
- Golden Images, OFED, CUDA, Zero Drift
- Slurm Partitions, QoS and GPU GRES
- GPU Operator and Network Operator
- Multus CNI for RDMA in Pods
- Run:ai Fractional GPUs and KAI Scheduler
- Fair-Share Queues and Preemption
- Slurm vs Kubernetes and Gang Scheduling
Lab 7: Build the Software Stack and slice a GPU with MIG
Lab 8: Schedule GPU Jobs on Slurm and Kubernetes
08Weekend 08: Observability, Resiliency and Acceptance
Module 22: GPU Monitoring and Observability
- nvidia-smi Query Loops and Telemetry
- DCGM Health Checks and Diagnostic Levels
- DCGM Exporter, Prometheus and Grafana
- InfiniBand Symbol Errors (OpenSM/UFM)
- Ethernet Pause Frame Counters
- BMC and Redfish Telemetry
Module 23: Troubleshooting, Validation and Acceptance
- NCCL Tests and the perftest Suite
- XID Errors (31, 45, 62, 79) and ECC Faults
- Thermal Throttling and PCIe Dropouts
- MTBF and Stragglers at 1,000+ GPUs
- AHR Node Cordoning, AJR Job Recovery
- 3-Phase Client Acceptance Protocol
- 24-Hour HPL Burn-In and DCGM Level 3
- Runbooks for Cordon, Drain and RMA
Lab 9: Monitor the Cluster with DCGM and Prometheus
Lab 10: Break-Fix Scenarios on injected faults
Ten labs on a live GPU cluster
Every lab runs on your own SSH seat on a multi-GPU HPC cluster with NVIDIA accelerators, ConnectX SuperNICs and high-speed fabrics.
| Lab | Title | Exercises | Modules |
|---|---|---|---|
| Lab 1 | Meet the GPU Node | Inspect GPUs, memory hierarchy, PCIe root complexes and NVLink topology using nvidia-smi, lspci -tv and topology matrix commands. | 1, 5, 6 |
| Lab 2 | Collectives in Action | Run NCCL benchmarks (all_reduce_perf), compare Ring vs Tree efficiency, and launch a distributed PyTorch benchmark across GPUs. | 3, 4 |
| Lab 3 | RDMA Basics and Validation | Query RDMA devices with ibv_devinfo, configure Queue Pairs, and run ib_write_bw and ib_read_lat between nodes. | 9, 10 |
| Lab 4 | Lossless Ethernet Tuning | Configure PFC, ECN and DCQCN on switches and SuperNICs, and observe flow control under simulated congestion. | 11, 12 |
| Lab 5 | Fabric Design Studio | Design a rail-optimized fabric for a 32-node GPU pod: switch ports, spine-leaf oversubscription and cable tables. | 7, 8, 13, 15 |
| Lab 6 | Storage Benchmarking | Benchmark parallel file system throughput with FIO, time checkpoint reads and writes, and configure GPUDirect Storage (gdscheck). | 19 |
| Lab 7 | Build the Software Stack | Install drivers, CUDA and the NVIDIA Container Toolkit, partition a GPU into MIG slices, and run jobs in NGC containers. | 20 |
| Lab 8 | Schedule GPU Jobs | Write sbatch scripts requesting multi-GPU resources, deploy the GPU Operator on Kubernetes, and schedule inference pods. | 21 |
| Lab 9 | Monitor the Cluster | Deploy DCGM Exporter, configure Prometheus collection, and build Grafana monitoring views for GPU utilization, power and temperature. | 22 |
| Lab 10 | Break-Fix Scenarios | Diagnose live injected faults: failed links, thermal and power throttling, driver mismatches and real XID errors. | 23 |
Prerequisites: the Linux command line, file systems and bash, plus networking fundamentals (IP, subnets, VLANs, routing, OSI). Docker and Kubernetes awareness helps. No prior InfiniBand, AI, deep learning or Python required.
Preparing for NVIDIA certifications? Practise with the NCA-AIIO, NCP-AIN and NCP-AII practice exams. CareerByteCode is an independent training program, not affiliated with or endorsed by NVIDIA Corporation.
What you walk away with, and where you grow
Eight weekends in, you can size, build, run and troubleshoot a GPU cluster, with a design portfolio and runbooks to prove it. Then you keep growing through our community and freelancing network.
Real cluster experience
Hours of keyboard-level work on a live multi-GPU HPC cluster: RDMA benchmarks, NCCL sweeps, Slurm and Kubernetes scheduling, and injected-fault drills.
A design portfolio
An NVL72 rack bill of materials, a 32-node rail-optimized fabric design, a UFM telemetry review and a site survey you can walk an interviewer through.
Runbooks you keep
Session recordings and notes for 9 weeks from the course start date, plus a Git repository of runbooks for cordon, drain, RMA and acceptance testing, ready for your first production cluster.
Grow through our community
Join a global community of engineers. Showcase your work, stay visible to recruiters, and speed up the opportunities that come your way.
Earn through freelancing
Turn your AI infrastructure skills into income. Strong candidates are invited into our freelancing programs to get paid for real work.
Ready to start Weekend 01?
Join the next weekend cohort, get your own seat on a live GPU cluster, and run your first NCCL benchmark in week one.