8000 Credentials via egress injection: keep real keys out of tasks (rotation + multi-tenant) · Issue #475 · google/ax · GitHub
[go: up one dir, main page]

Skip to content

Credentials via egress injection: keep real keys out of tasks (rotation + multi-tenant) #475

Open

Description

@MushuEE

Summary

I think we're going to need a proper story for credentials in tasks. Ideally the agent never holds the real key at all. Substrate already has the pieces for this with egress credential injection: the agent sends a placeholder header, and the egress gateway swaps in the real credential on the way out. AX should lean on that.

This builds on #449, since AX needs to be creating egress policies first.

What it could look like

Tasks, Models and Workspaces reference secrets instead of carrying values:

kind: Model
spec:
  provider: gemini
  model: gemini-2.5-pro
  secretKey: { name: gemini-credentials, key: api-key }
---
kind: Workspace
spec:
  git:
    - repo: https://github.com/acme/private-repo
      credentialRef: { name: github-acme, key: token }
---
kind: Task
spec:
  env:
    - name: STRIPE_API_KEY
      secretRef:
        name: stripe
        key: api-key
        inject: { host: api.stripe.com, header: Authorization, prefix: "Bearer " }

AX turns those into rules on the actor's egress policy:

rules:
- https:
    hostnames: ["generativelanguage.googleapis.com"]
    effects:
      replaceHeaders:
      - header: x-goog-api-key
        credentialUri: ate-secret://k8s.io/default/team-a/gemini-credentials/api-key

Inside the sandbox the agent just sees GEMINI_API_KEY=ax-placeholder, so SDKs still work. Git over HTTPS works the same way with a placeholder Authorization header.

Key rotation

Since everything is a reference, rotating a key should just be updating the secret. No template rebuilds, nothing baked into snapshots. Substrate's Kubernetes provider looks like it reads the Secret per request, so a rotation should take effect pretty quickly. Worth confirming whether the gateway caches anything.

Multi-tenant / many keys to many tasks

  • Tenants define their Models and secrets in their own atespace. The same Model can be shared by lots of tasks, and one task can pull in several keys.
  • Substrate's provider already scopes atespace → secret namespace, so team-a can't resolve team-b's secrets even if it names them.
  • AX should check that mapping up front when a Task is created, rather than letting it fail on the first request.

A few things we'd probably want to raise with Substrate:

  • The namespace policy only reloads on a provider restart, which gets painful with lots of tenants.
  • One task can't use two different credentials for the same host (say two GitHub tokens for different orgs).

Limits

Rough phases

  1. Tasks have no network access on current Agent Substrate because ax never creates an egress policy #449: AX creates the egress policy
  2. Model credentials through the gateway
  3. Workspace git credentials
  4. env[].secretRef for arbitrary APIs
  5. Rotation and multi-tenant polish

Open questions

  • Does AX own the egress policy outright, or merge its rules into one the user provides?
  • Default to the Kubernetes provider, or make it a per-cluster choice (k8s, Secret Manager, Vault)?
  • A built-in table of header/prefix for common providers (Gemini, OpenAI, Anthropic, GitHub), with an override?

Related: #449, #427, #467, #454

Activity

  1. added
    devxDeveloper experience: onboarding, defaults, and gaps found building on AX
    on Oct 15, 2026
  2. MushuEE commented on Oct 18, 2026

    @MushuEE
    CollaboratorAuthor

    Quick update from starting on this. We're going to scope secret providers per atespace: a task just says which secret it needs (secretRef: {name, key, inject}), and the atespace decides where that secret lives through a SecretStore.

    One catch on the Substrate side: the egress gateway only serves a single credential provider today. To actually let different atespaces use different providers (k8s Secrets for one, Secret Manager or Vault for another), we'd need either:

    • multi-provider support in the gateway, routing on the ate-secret://<provider>/... host it already parses, or
    • per-atespace gateways (the per-atespace gateway routing in the roadmap).

    So for v1 I'm going with a SecretStore resource backed by k8s Secrets only, and a task that references a secret in an atespace without a store gets rejected rather than falling back to anything. The task-facing API shouldn't need to change when more backends show up.

    Also worth raising with Substrate alongside multi-provider: the k8s provider looks like it resolves the Secret on every request, so caching will matter at scale.

  3. MushuEE commented on Oct 19, 2026

    @MushuEE
    CollaboratorAuthor

    Breaking the implementation into a stack of smaller PRs so each one is reviewable on its own and nothing turns on before it's ready:

    • 1. API (feat(api): add egress, secret-backed env vars, and SecretStore to the API #499): egress.hosts, env[].secretRef (with inject mirroring Substrate's CredentialHeader), and the SecretStore resource. No behavior yet: CreateTask returns Unimplemented for the new fields.
    • 2. SecretStore: storage (Redis + memory), server CRUD, ax apply/get/delete secretstore, and docs for the Substrate credential provider + namespace policy setup. Nothing reads stores yet.
    • 3. Egress policies (Tasks have no network access on current Agent Substrate because ax never creates an egress policy #449): the reconciler creates the actor's egress policy from egress.hosts before the actor can run, and mounts the egress trust bundle (SSL_CERT_FILE etc.). Lifts the egress check.
    • 4. Credential injection: secretRef hosts get https rules that replace the header with an ate-secret:// reference through the atespace's store; the task only sees a placeholder. CreateTask requires the store to exist. Lifts the last check. Will be tested on a kind cluster with the k8s.io provider before it merges.

    2-4 are ready locally and will go up as each one before them lands. Follow-ups after that: a server-wide default egress policy (#449), moving GEMINI_API_KEY and git credentials to injection, and the multi-provider / caching asks for Substrate.

  4. self-assigned this
    on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

controllerdevxDeveloper experience: onboarding, defaults, and gaps found building on AXprotosubstrate

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    0