Operations

Observability (OpenTelemetry)

Argus exports metrics, logs, and traces for every run over OTLP, enabled purely by environment variables, k6-style, and shipped to your own OpenTelemetry collector. A versioned Grafana dashboard suite turns it into a fleet-wide view: run health, where the time goes, what was found and fixed, and the exact failure output without downloading job artifacts.

What you get

Metrics: 18 instruments covering runs, phases, vulnerabilities found/fixed/residual, strategy mix, qualification verdicts, grand-slam hit rate, MRs, and upstream latency.
Traces: one trace per run, argus.run → a span per pipeline phase → per-artifact qualification and per-MR publication spans. The “why was this run slow” tool.
Logs: every run-log line mirrored (honouring ARGUS_LOG_LEVEL), the per-CVE decision records, the anchor.log diagnostics, and structured failure events carrying the actual verify-output tail.

Telemetry never changes a run’s behaviour: export failures log a warning, the end-of-run flush is bounded (default 5s), and disabled telemetry costs nothing. Console and artifact output are byte-identical either way.

Enabling it

.gitlab-ci.yml

# .gitlab-ci.yml: via the Argus CI/CD component you publish in your own catalog
include:
  - component: $CI_SERVER_FQDN/your-group/ci-components/argus@4.2.1
    inputs:
      otel: "on"                        # default off; explicit opt-in
      otel_exporter: "grpc"             # or "http"
      otel_org: "acme-bank"             # optional multi-tenant org id
      otel_attributes: "team=payments"  # embedded in every signal

# Project/group CI/CD variables (masked; never component inputs):
#   OTEL_EXPORTER_OTLP_ENDPOINT = https://otel-collector.internal:4317
#   OTEL_EXPORTER_OTLP_HEADERS  = authorization=Bearer ****

# .gitlab-ci.yml: via the Argus CI/CD component you publish in your own catalog
include:
  - component: $CI_SERVER_FQDN/your-group/ci-components/argus@4.2.1
    inputs:
      otel: "on"                        # default off; explicit opt-in
      otel_exporter: "grpc"             # or "http"
      otel_org: "acme-bank"             # optional multi-tenant org id
      otel_attributes: "team=payments"  # embedded in every signal

# Project/group CI/CD variables (masked; never component inputs):
#   OTEL_EXPORTER_OTLP_ENDPOINT = https://otel-collector.internal:4317
#   OTEL_EXPORTER_OTLP_HEADERS  = authorization=Bearer ****

Environment variables

ARGUS-prefixed variables win over their standard SDK equivalents (the k6 convention):

Property	Default	Description
`ARGUS_OTEL`	`off`	The master switch. Telemetry is explicit opt-in: org-wide `OTEL_*` CI variables never silently enable export from a security tool.
`ARGUS_OTEL_EXPORTER`	`grpc`	`grpc` (collector :4317) or `http` (OTLP/HTTP protobuf, :4318). Wins over `OTEL_EXPORTER_OTLP_PROTOCOL`.
`OTEL_EXPORTER_OTLP_ENDPOINT`	`localhost:4317/4318`	Standard SDK variable (with `_HEADERS`, `_CERTIFICATE`, `_TIMEOUT`, and per-signal forms). Deliberately not mirrored as an ARGUS variable: headers carry credentials and belong in masked CI/CD variables.
`ARGUS_OTEL_SERVICE_NAME`	`argus`	Overrides `service.name`. Wins over `OTEL_SERVICE_NAME`.
`ARGUS_OTEL_ATTRIBUTES`	`unset`	Extra resource attributes embedded in all metrics, logs, and traces, as `k1=v1,k2=v2` (e.g. `team=payments,env=prod`). Wins per key over `OTEL_RESOURCE_ATTRIBUTES`.
`ARGUS_OTEL_ORG`	`unset`	Explicit multi-tenant `org` attribute. Never derived from the project path; unset omits the attribute entirely.
`ARGUS_OTEL_METRICS / _LOGS / _TRACES`	`on`	Per-signal toggles.
`ARGUS_OTEL_METRICS_TEMPORALITY`	`delta`	`delta` (recommended; pair with the collector’s `deltatocumulative` processor) or `cumulative` for pipelines without it.
`ARGUS_OTEL_FLUSH_TIMEOUT`	`5s`	Hard budget for the end-of-run flush. A hung collector costs at most this long and a warning, never a failed run.

Identity attributes

Every metric, log, and trace carries the drill-down hierarchy the dashboards chain on:

org: only from ARGUS_OTEL_ORG (explicit multi-tenant identity; never derived, omitted when unset);
subgroup: the project’s immediate parent group, derived from CI_PROJECT_PATH;
project: the project name from CI_PROJECT_NAME (never a subgroup path; falls back to the path slug when unset);
service.name / service.version: argus and the release;
your ARGUS_OTEL_ATTRIBUTES (e.g. team=payments).

No per-run identifiers (pipeline id, job id, ref) ever appear on metrics: that would mint a new Prometheus series set per run. Individual runs are found in Tempo via the argus.run.id span attribute; per-CVE detail lives in the logs, never in metric labels.

Metric reference

Attribute values are strictly low-cardinality. After the collector’s Prometheus export, argus.run.duration becomes argus_run_duration_seconds, counters gain _total, and so on.

Property	Default	Description
`argus.run.count / argus.run.duration`	`Counter / Histogram`	Runs by ecosystem and outcome; wall-clock duration. Fleet health and success rate.
`argus.phase.duration`	`Histogram`	Seconds per pipeline phase: parse, resolve_graph, hydrate_advisories, plan, curate, qualify, publish, summarize. Where the time goes.
`argus.dependencies.scanned / argus.components.scanned`	`Gauge`	Manifest size and resolved-graph size for the run.
`argus.vulnerabilities.found / .remediated / .residual`	`Gauge`	Per-run absolutes by `severity`; residuals also carry `kind` (`residual` or `delegated`). Read with `max_over_time`.
`argus.remediations.planned`	`Counter`	Plans by `strategy`: the ladder mix (BUMP, OVERRIDE, EXCLUDE_ADD, ANCHOR_UPGRADE, PARENT_UPGRADE, LOCK_REFRESH).
`argus.qualification.count / .duration`	`Counter / Histogram`	Verdicts by `status`, `validation`, and `mode` (`grand_slam`, `per_artifact`, `final_pass`); mvn/npm verify cost.
`argus.grand_slam.count`	`Counter`	Grand-slam-first outcomes: `qualified` up front, `fallback` to per-artifact, or `skipped` (single fix). The ADR-039 hit rate.
`argus.final_pass.fixes`	`Gauge`	Extra fixes the final remediation pass accepted this run.
`argus.merge_requests.published / .errors`	`Counter`	MRs by `kind` (single, aggregate, cleanup); failures by `stage` (lookup, commit, labels, mr).
`argus.publication.deferred / argus.stale_overrides.removed`	`Gauge`	Budget-deferred remediations (max_mrs) and curation activity.
`argus.client.request.duration`	`Histogram`	Upstream HTTP latency by `target` (osv, gitlab, registry) and `status_class`.

Exact failures in Grafana, no artifact downloads

Beyond mirrored log lines, Argus emits structured ERROR events at each failure seam:

argus.qualification.failed: the artifact, its strategy, and the last ~100 lines of the actual mvn/npm verify output as the log body;
argus.publish.failed: the publication stage (lookup, commit, labels, mr) and the GitLab API error;
argus.run.failed: which phase the run died in, and why.

The Investigation dashboard surfaces these directly, so “why did this team’s run fail” is answered from Grafana. The full output always remains in the job artifacts for deep dives.

Collector configuration

Two collector settings make the shipped dashboards work as-is:

otel-collector.yml

processors:
  deltatocumulative:        # Argus exports delta counters (short-lived CLI)

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.internal/api/v1/push
    resource_to_telemetry_conversion:
      enabled: true         # promote org/subgroup/project to metric labels

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [deltatocumulative, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

processors:
  deltatocumulative:        # Argus exports delta counters (short-lived CLI)

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.internal/api/v1/push
    resource_to_telemetry_conversion:
      enabled: true         # promote org/subgroup/project to metric labels

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [deltatocumulative, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Grafana dashboards

Each release ships argus-grafana-dashboards-<version>.zip alongside the binaries: Fleet Overview (org-wide health), Run Performance (phases, qualification cost, upstream latency, traces), Security Outcomes (found vs fixed vs residual, strategy mix, grand-slam hit rate), and Investigation (drill into one project: run history, exact failures, traces). All four share chained $org → $subgroup → $project variables and every panel has a tooltip. Import the JSONs, pick your Prometheus/Loki/Tempo datasources, done. The set is versioned with the release because the metric taxonomy is. See grafana/README.md in the zip for the full walkthrough.

Monitoring playbook: using the data without drowning

The dashboard pack ships a runbook (MONITORING.md in the zip) so teams consume the signal deliberately. The short version:

Daily (5 minutes, Fleet Overview): success-rate gauge green? Runs-per-day comb intact (missing teeth mean pipelines not running: a scheduler problem, not Argus)? Any hot cells in the latest-run table? Click through to Investigation.
Weekly (Security Outcomes): remediation-rate gauge trending up; the critical/high residual table trending empty (each row needs a human decision: raise a ceiling, accept, or chase the delegated owner); strategy mix not drifting toward overrides.
Alert on four things only: repeated failed runs per project, any critical residual, any MR publication error (token expiry!), and fleet p95 duration blowing out. Paste-ready PromQL for each lives in the runbook. Resist alerting on more.
Triage failed runs in Investigation: outcome timeline (streak vs one-off) → the failure log panels (exact verify-output tail / GitLab API error, no artifact downloads) → the run trace for phase-level timing. Escalate to job artifacts + the diagnostic skill only after that.

Decision record: ADR-042. The environment variables page lists the OTEL variables alongside everything else.