Overview

The OpenTelemetry GPU Collector is a lightweight, single-binary metrics collector written in Go. It exports GPU hardware telemetry, host system metrics, and process metrics via OpenTelemetry (OTLP) - with no Python dependencies, no DCGM daemon, and no vendor-specific agents.

It is fully configured via standard OpenTelemetry environment variables and follows the OTel semantic conventions for hardware metrics.

Goals

• OpenTelemetry-native - uses standard OTEL_* env vars, exports via OTLP gRPC or HTTP to any OTel-compatible backend
• Cross-vendor GPU support - NVIDIA (NVML), AMD, and Intel on Linux and Windows from a single binary
• OTel semantic conventions - hw.gpu.* metric names, hw.id / hw.name / hw.vendor attributes per spec
• Zero dependencies - no DCGM, no Python, no CUDA toolkit needed at runtime for hardware metrics
• Resilient - continues exporting host metrics even when no GPUs are present; retries GPU discovery every 30s

What it collects

• GPU Hardware Telemetry — Utilization, memory, temperature, power, energy, clocks, PCIe/interconnect throughput, throttle, ECC/XID/RAS — NVIDIA, AMD, and Intel on Linux and Windows
• Process + workload attribution — Per-process VRAM/util plus cmdline, OS state (zombie), owner, uptime, and LLM framework labels (vllm, ollama, …). Kubernetes pod name/namespace via PodResources
• eBPF CUDA Tracing — Kernel launch counts, grid/block sizes, memory allocations, and memory copies via uprobes on libcudart (on by default on Linux; discovers libs from /proc with host PID — no CUDA mount)

GPU vendor support

Vendor	Linux	Windows
NVIDIA	NVML (+ MIG, NVLink, XID events)	NVML (nvml.dll, NVLink) + PDH process attribution (MIG not available on Windows)
AMD	sysfs/hwmon/DRM + libamd_smi (PCIe/XGMI/throttle/RAS when present)	DXGI + PDH (incl. encode/decode) + ADL (util/temp/clocks/fan)
Intel	DRM/hwmon + Level Zero Sysman (util/VRAM/power/media)	DXGI + PDH (incl. encode/decode) + Level Zero Sysman

Platform support

Feature	Linux	macOS	Windows
System metrics (CPU, memory, disk, network)	Yes	Yes	Yes
Process metrics (CPU, memory, threads, FDs)	Yes	Yes	Yes
GPU device + process metrics	Yes	-	Yes
eBPF CUDA tracing / occupancy	Yes	-	-

How it works

Host Metrics (all platforms via gopsutil)
    +-- CPU utilization, memory, disk I/O, filesystem, network
    +-- Process: self CPU, memory, threads, FDs, Go runtime

GPU Metrics (Linux + Windows)
    +-- Linux: PCI scan → NVML / sysfs / DRM (+ eBPF CUDA, default on)
    +-- Windows: NVML.dll / DXGI + PDH (no eBPF)

Export
    +-- OTel SDK --> OTLP gRPC/HTTP --> your OTel collector / backend

Discovers GPUs

On Linux, the collector scans /sys/bus/pci/devices/ for GPU class codes. On Windows, it loads nvml.dll when present and enumerates adapters via DXGI.

Initialises vendor backend

Each detected GPU is handed to its vendor-specific backend. NVIDIA uses NVML. Linux AMD/Intel read sysfs/hwmon; Windows AMD/Intel use DXGI memory totals and PDH for utilization and per-process attribution.

Registers OTel instruments

Observable gauge and counter instruments are registered with the OTel SDK meter. On each collection tick, the SDK calls back into the collector to read fresh values from each GPU.

Exports via OTLP

Metrics are exported via OTLP to any compatible backend - ARMS, Grafana, Datadog, New Relic, or a standard OTel Collector.