8000 Added tenant_id config to send X-Scope-OrgID for Mimir by abd-ulbasit · Pull Request #21 · dominikhei/cardamon · GitHub
[go: up one dir, main page]

Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
</a>
</p>

**Cardamon** is a metric auditor for Prometheus. It identifies metrics that exist in your TSDB but are never actually queried by dashboards, alerting rules, recording rules, or any other consumer. The idea is to generate insight into that not to blindly drop all metrics as they might be valuable during an incident.
**Cardamon** is a metric auditor for Prometheus. It identifies metrics that exist in your TSDB but are never actually queried by dashboards, alerting rules, recording rules, or any other consumer. The idea is to generate insight into that not to blindly drop all metrics as they might be valuable during an incident.

## Demo
![](./media/cardamon_demo.gif)
Expand Down Expand Up @@ -58,6 +58,7 @@ Cardamon is configured via a YAML file. Pass the path with `--config`.
prometheus:
address: "http://localhost:9090"
path_prefix: "" # Set if Prometheus is served under a subpath, e.g. /prometheus
tenant_id: "" # Optional, sent as the X-Scope-OrgID header, e.g. for Mimir
# If no auth is configured leave this blank
token: "" # Token based auth, takes precedence over password auth if both are set
username: "" # Basic auth / LDAP fallback
Expand Down Expand Up @@ -97,6 +98,24 @@ They [need to be enabled](https://prometheus.io/docs/guides/query-log/), e.g via

> **Note:** Cardamon handles both active `.log` files and rotated `.log.gz` files. It reads log entries in order and stops early once it reaches entries older than `lookback_days`, so scanning is efficient even on large logs.

### Using cardamon with Grafana Mimir

Mimir serves the Prometheus HTTP API under a prefix (`/prometheus` by default) and reads the tenant from the `X-Scope-OrgID` header. Set `path_prefix` to that prefix and `tenant_id` to the tenant you want to audit:

```yaml
prometheus:
address: "http://mimir.example.com" # Mimir gateway, or Mimir in monolithic mode
path_prefix: "/prometheus"
tenant_id: "team-a"
```

- One run audits one tenant. To audit several tenants, run cardamon once per tenant.
- Federated tenant IDs such as `team-a|team-b` are not supported. Mimir's rules endpoint accepts a single tenant only, so the rules call would fail.
- If multi-tenancy is disabled in Mimir, leave `tenant_id` blank.
- Rules are read from `<path_prefix>/api/v1/rules`. Mimir serves that path from the ruler, not the query-frontend, so `address` has to route it to the ruler. If the rules call fails, cardamon only prints a warning, and metrics used only in rules will show up as unused.
- The query log source reads the Prometheus query log format and has not been tested with Mimir. Without it, only dashboards and rules count as usage, so more metrics may be flagged as unused.
- Grafana dashboards are not matched to tenants yet, see [Limitations](#limitations).

---

## Usage
Expand Down Expand Up @@ -183,7 +202,7 @@ No write access is required.

## Limitations

Cardamon currently assumes a 1:1 relationship between a single Prometheus and Grafana instance. Multi-tenancy support (e.g. Mimir or Thanos with multiple tenants) is planned for a future release. All metrics found across all dashboards are treated as used, regardless of which datasource the panel queries. This means that in setups with multiple datasources, metrics from one datasource may prevent metrics in another from being flagged as unused even if nothing in that datasource actually queries them.
Cardamon currently assumes a 1:1 relationship between a single Prometheus and Grafana instance. For Mimir, one run audits a single tenant (see [Using cardamon with Grafana Mimir](#using-cardamon-with-grafana-mimir)). Matching dashboards to tenants and Thanos multi-tenancy are not supported yet. All metrics found across all dashboards are treated as used, regardless of which datasource the panel queries. This means that in setups with multiple datasources, metrics from one datasource may prevent metrics in another from being flagged as unused even if nothing in that datasource actually queries them.

---

Expand Down
2 changes: 1 addition & 1 deletion cmd/cardamon/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ func main() {
os.Exit(1)
}

promClient, err := prom.NewClient(cfg.Prometheus.Address+cfg.Prometheus.PathPrefix, cfg.Prometheus.Token, cfg.Prometheus.Username, cfg.Prometheus.Password)
promClient, err := prom.NewClient(cfg.Prometheus.Address+cfg.Prometheus.PathPrefix, cfg.Prometheus.Token, cfg.Prometheus.Username, cfg.Prometheus.Password, cfg.Prometheus.TenantID)
if err != nil {
log.Fatalf("Failed to initialize Prometheus client: %v", err)
}
Expand Down
1 change: 1 addition & 0 deletions pkg/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ type Config struct {
Token string `yaml:"token"`
Username string `yaml:"username"`
Password string `yaml:"password"`
TenantID string `yaml:"tenant_id"`
} `yaml:"prometheus"`

Storage struct {
Expand Down
8 changes: 7 additions & 1 deletion pkg/prom/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ type roundTripper struct {
token string
username string
password string
tenantID string
base http.RoundTripper
}

Expand All @@ -39,16 +40,21 @@ func (rt *roundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
} else if rt.username != "" {
r.SetBasicAuth(rt.username, rt.password)
}
// Multi-tenant backends such as Grafana Mimir read the tenant from this header.
if rt.tenantID != "" {
r.Header.Set("X-Scope-OrgID", rt.tenantID)
}
return rt.base.RoundTrip(r)
}

func NewClient(address, token, username, password string) (*Client, error) {
func NewClient(address, token, username, password, tenantID string) (*Client, error) {
c, err := api.NewClient(api.Config{
Address: address,
RoundTripper: &roundTripper{
token: token,
username: username,
password: password,
tenantID: tenantID,
base: api.DefaultRoundTripper,
},
})
Expand Down
100 changes: 100 additions & 0 deletions pkg/prom/client_test.go
9D19
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
// Copyright 2026 dominikhei
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package prom

import (
"context"
"net/http"
"net/http/httptest"
"slices"
"sync"
"testing"
"time"
)

// seenRequest is what the fake Prometheus API received.
type seenRequest struct {
path string
tenant []string
auth string
}

func TestNewClient_TenantHeader(t *testing.T) {
tests := []struct {
name string
token string
tenantID string
wantTenant []string
wantAuth string
}{
{name: "no tenant sends no header"},
{name: "tenant is sent", tenantID: "team-a", wantTenant: []string{"team-a"}},
{name: "value is sent unchanged", tenantID: "team-a|team-b", wantTenant: []string{"team-a|team-b"}},
{name: "tenant is sent alongside auth", token: "secret", tenantID: "team-a", wantTenant: []string{"team-a"}, wantAuth: "Bearer secret"},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var (
mu sync.Mutex
seen []seenRequest
)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
seen = append(seen, seenRequest{
path: r.URL.Path,
tenant: r.Header.Values("X-Scope-OrgID"),
auth: r.Header.Get("Authorization"),
})
mu.Unlock()
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"status":"success","data":["up","http_requests_total"]}`))
}))
defer srv.Close()

// Mimir serves the Prometheus API under /prometheus, which cardamon gets via path_prefix.
c, err := NewClient(srv.URL+"/prometheus", tt.token, "", "", tt.tenantID)
if err != nil {
t.Fatalf("NewClient: %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

names, err := NewAnalyzer(c).GetAllMetricNames(ctx)
if err != nil {
t.Fatalf("GetAllMetricNames: %v", err)
}
if want := []string{"up", "http_requests_total"}; !slices.Equal(names, want) {
t.Errorf("metric names = %v, want %v", names, want)
}

mu.Lock()
defer mu.Unlock()
if len(seen) != 1 {
t.Fatalf("got %d requests, want 1", len(seen))
}
got := seen[0]
if want := "/prometheus/api/v1/label/__name__/values"; got.path != want {
t.Errorf("path = %q, want %q", got.path, want)
}
if !slices.Equal(got.tenant, tt.wantTenant) {
t.Errorf("X-Scope-OrgID = %q, want %q", got.tenant, tt.wantTenant)
}
if got.auth != tt.wantAuth {
t.Errorf("Authorization = %q, want %q", got.auth, tt.wantAuth)
}
})
}
}
0