Overview
MILABENCH_BASE=workdir/results
workdir
├── milabench
│ ├── benchmarks # Each benchmark is inside a folder
│ │ ├── torchvision
| | | ├── benchfile.py # Benchmark configuration (source to checkout, script to runs etc...)
| | | ├── voirfile.py # Instrumentation to insert timers
| | | ├── prepare.py # Prepare script executed to fetch datasets, download pretrained models
| | | ├── main.py # benchmark script to be ran
| | | ├── requirements.in # base requirements
| | | ├── requirements.cuda.txt # pinned requirements for cuda
| | | ├── requirements.rocm.txt # pinned requirements for rocm
| | | └── requirements.xpu.txt # pinned requirements for xpu
| | └── timm
| | ├── benchfile.py # Benchmark configuration (source to checkout, script to runs etc...)
| | ├── voirfile.py # Instrumentation to insert timers
| | ├── prepare.py # Prepare script executed to fetch datasets, download pretrained models
| | └── main.py # benchmark script to be ran
│ ├── benchmate # benchmate module
│ ├── milabench # milabench module
│ ├── constraints # pip constraints for different vendors
│ └── config # benchmark suite configurations
│ └── standard.yaml # <= MILABENCH_CONFIG
├── env # virtual environment where milabench is installed
└── results # <= MILABENCH_BASE
├── data # Datasets, pre-trained models
├── extra
├── venv # Benchmark virtual environments
│ └── torch # each benchmark can have their own environments
└── runs # Raw metrics gathered by milabench
└── {runname}.{time}
└── {benchname}.D{device_id}.data # Stream of metrics for a benchmark
Benchmark configuration
milabench was created first as a procurement tool. It was design to measure performance of a given system in order to measure its suitabilty given a specific purpose (suite of benchmark).
As such milabench benchmark suite is fully configurable. Users can create their own benchmark suite from it in order to ensure the systems are tested for their real use case.
Milabench is configured using a yaml file that specify where are the benchmark and how to install them.
# you can include a previous configuration
# and override its values
include:
- base.yaml
_defaults:
max_duration: 600 # Bench time out
voir:
options:
stop: 60 # Bench stops after gathering 60 observations
interval: "1s" # Gathering interval
validation: # Validation (disabled by default)
usage:
gpu_load_threshold: 0.5 # ensure GPU load is higher than 50%
gpu_mem_threshold: 0.5 # ensure GPU memory is higher than 50%
_torchvision:
inherits: _defaults # base configuration
definition: ../benchmarks/torchvision # benchmark definition location
group: torchvision
install_group: torch # venv name to use for this benchmark
plan: # Specify how the benchmark is scheduled
method: per_gpu # `per_gpu` means it will spawn one bench per GPU
argv: # arguments to forward
--precision: 'tf32-fp16'
--lr: 0.01
--no-stdout: true
--epochs: 50
--num-workers: 8
resnet50: # benchmark name "_" are "private" and never run
inherits: _torchvision
tags: # selection tags
- vision
- classification
- convnet
- resnet
argv:
--model: resnet50
--batch-size: 64
--num-workers: "{cpu_per_gpu}" # Placeholder variable to be resolved
# milabench can also define matrix job
resnet-matrix-noio:
matrix:
batch-size: [32, 64, 128, 256, 512, 1024]
job:
name: 'resnet50-noio-bs{batch-size}'
inherits: _resnet50
argv:
--batch-size: '{batch-size}'
--synthetic-data: true
--fixed-batch: true
System Configuration
milabench can run benchmarks across multiple nodes, to do so a system configuration needs to be provided. This file will define all the nodes accessible to milabench.
system:
arch: cuda # Default arch
sshkey: ~/.ssh/id_ed25519 # sshkey used in remote milabench operations
# Docker image to use
docker_image: ghcr.io/mila-iqia/milabench:${system.arch}-nightly
# Nodes list
nodes:
# Alias used to reference the node
- name: manager
ip: 192.168.11.11
port: 5000
main: true # Use this node as the rank=0 node or not
user: manager # User to use in remote milabench operations
- name: node1
ip: 192.168.11.12
main: false
user: username
Multinode
Milabench takes care of sending the commands to all the nodes when appropriate.
Methodology
for i in range(epoch):
events = []
# Creation of the iterator from the dataloader is time consuming
# it would get amortized across many batch during real training
# but we want benchmarking to be fast so it is something we cannot afford
batch_iter = iter(loader)
total_obs = 0
# Avoid sync in the batch loop
start = Event()
start.record()
for batch in batch_iter:
pred = model(batch)
loss = fn(pred, target)
end = Event() # +->
end.record() # |
events.append((start, end, len(batch), loss.detach())) # | Limited overhead
if len(events) + total_obs >= 60: # |
break # |
start = end # +->
# Force sync at the end of the epoch # +->
for start, end, bs, loss in events: # | Timer is off does not impact perf measures
end.wait() # |
log(loss=loss.item()) # |
log(rate=bs / (end - start)) # |
# |
total_obs += len(events) # |
if total_obs >= 60: # |
raise StopProgram() # +->
Instrumentations
To minimize code change, milabench use ptera to modify the code that will be run and insert the necessary hooks to measure performance.
The hooks are defined inside the voirfile.py.
The example below override the return value of the dataloader() function which is defined in the __main__ module.
It wraps the original object with a custom wrapper that will time the time between __next__ calls.
This allows milabench to integrate benchmarks from code coming from third parties without modifying the code directly.
def wrapper(loader):
print("received loader obj")
return Wrapper(loader)
probe = ov.probe("//dataloader() as loader", overridable=True)
probe['loader'].override(wrapper)
Execution Flow
milabench installCreates virtual env for benchmarks and install their dependencies
Modify:
$MILABENCH_BASE/venv/{bench}
milabench prepareCall the prepare script for each benchmarks to download/generate dataset and download pretrained models
Modify:
$MILABENCH_BASE/data/{dataset}
milabench runExecute each benchmark
Modify:
$MILABENCH_BASE/runs/{runame}.{time}- Steps
init: Voir has initialized itself. You can add command-line arguments here.
parse_args: The command-line arguments have been parsed.
load_script: The script has been loaded: its imports have been done, its functions defined, but the top level statements have not been executed. You can perform some manipulations prior to the script running.
run_script: the script will start to run now
finalize: tearing down
Execution Plan
milabench main process * gather metrics from benchmark processes, save them to file * manages the benchmarks (timeout etc…)
if
per_gpuis used, milabench will launch one process per GPU (setsCUDA_VISIBLE_DEVCES) * each processes log their GPU data * might spawn a monitor processwill init pynvml
dataloader will also spawn process workers * usually not using GPU
if
njobsis used, milabench will launch a single process (torchrun) * torchrun in turn will spawn one process per GPURANK 0 is used for logging
RANK 0 might spawn a monitor process * will init pynvml
dataloader will also spawn process workers * usually not using GPU
per_gpu: used for mono gpu benchmarks, spawn one process per gpu and run the same benchmark
_torchvision:
inherits: _defaults
definition: ../benchmarks/torchvision
group: torchvision
install_group: torch
plan:
method: per_gpu
Milabench will essentially execute something akin to below.
echo "---"
echo "fp16"
echo "===="
time (
CUDA_VISIBLE_DEVICES=0 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
CUDA_VISIBLE_DEVICES=1 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
CUDA_VISIBLE_DEVICES=2 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
CUDA_VISIBLE_DEVICES=3 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
CUDA_VISIBLE_DEVICES=4 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
CUDA_VISIBLE_DEVICES=5 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
CUDA_VISIBLE_DEVICES=6 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
CUDA_VISIBLE_DEVICES=7 $SRC/milabench/benchmarks/flops/activator $BASE/venv/torch $SRC/milabench/benchmarks/flops/main.py --number 30 --repeat 90 --m 8192 --n 8192 --dtype fp16 &
wait
)
njobs used to launch a single jobs that can see all the gpus.
_torchvision_ddp:
inherits: _defaults
definition: ../benchmarks/torchvision_ddp
group: torchvision
install_group: torch
plan:
method: njobs
n: 1
Milabench will essentially execute something akin to below.
echo "---"
echo "lightning-gpus"
echo "=============="
time (
$BASE/venv/torch/bin/benchrun --nnodes=1 --rdzv-backend=c10d --rdzv-endpoint=127.0.0.1:29400 --master-addr=127.0.0.1 --master-port=29400 --nproc-per-node=8 --no-python -- python $SRC/milabench/benchmarks/lightning/main.py --epochs 10 --num-workers 8 --loader pytorch --data $BASE/data/FakeImageNet --model resnet152 --batch-size 16 &
wait
)