Skip to content

Add optional Loki + Alloy log search for Slurm and Kubernetes - #1447

Open
dholt wants to merge 2 commits into
masterfrom
dholt/loki-alloy-logs
Open

dholt wants to merge 2 commits into
masterfrom
dholt/loki-alloy-logs

Conversation

@dholt

@dholt dholt commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Adds optional, off-by-default log search to DeepOps, replacing the archived Elastic logging guide.

  • Slurm: new loki and alloy roles, Grafana Loki datasource, a Cluster logs dashboard, 7-day retention. Enable with slurm_enable_log_search: true.
  • Kubernetes: scripts/k8s/deploy_logging.sh with pinned Loki and Alloy charts, example Helm values, updated logging docs.
  • Molecule scenarios for both roles (service start skipped in containers without systemd).
  • Render tests in tests/logging/.

Tested locally with render tests and a full Slurm install on Ubuntu 24.04.

Off by default (slurm_enable_log_search: false). Slurm: loki and alloy
roles, Grafana Loki datasource and Cluster logs dashboard, 7-day
retention. Kubernetes: scripts/k8s/deploy_logging.sh with pinned Loki
and Alloy charts. Replaces the archived Elastic logging guide.

Signed-off-by: Doug Holt <dholt@nvidia.com>
Docker molecule containers have no systemd as PID 1, so the service
task and restart handler fail converge. Tag the service task
molecule-notest (same as nvidia_dcgm) and guard the handlers, which
ignore tags, on ansible_skip_tags.

Signed-off-by: Doug Holt <dholt@nvidia.com>
Comment thread tests/logging/test_logging_render.py Dismissed
@dholt
dholt requested a review from michael-balint November 17, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants