[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apiext: restrict which crds are watched and patched. #5540

Merged
merged 2 commits into from
Feb 5, 2024

Conversation

LanceEa
Copy link
Contributor
@LanceEa LanceEa commented Feb 2, 2024

Description

The apiext server watches for CRD's and will ensure they are patched properly (unless disabled). Previously,
we were restricting our resources by the default provided label selectors of app.kubernetes.io/part-of: emissary-apiext when setting up our watch of the CRD resources in a cluster. Unfortunately, K8s doesn't support sets in the field selectors so we couldn't watch for specifically named CRD's per this:

https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/#supported-operators

So, removing this restriction had a few downside affects:

  • If a user customizes the Labels so they no longer match the default provided manifest then it would cause
    the apiext server to never become ready.
  • It caches all CRD's in the cluster and for large clusters we observed the apiext deployment memory going
    from about 20mb to 80mb.
  • Without the label selector restriction we attempt to patch all resources in the getambassador.io group
    which means other resources not related to the APIGateway are then patched when they should not be.

The fix does the following:

  1. Adds a flag for setting the crd-label-selectors
    1. Defaults to --crd-label-selectors="app.kubernetes.io/part-of=emissary-apiext"
  2. crd-label-selectors configured via the new flags are passed along to the CRD List Watch options to limit
    what is cached.
  3. We added an additional check when patching the CRD to ignore CRD's explicitly set with Spec.Conversion.Strategy == "None".

These three changes will ensure:

  • we can limit memory usage of apiext with clusters with lots of CRD's installed
  • we only patch the required CRD's
  • accommodate custom installs for labels and namespaces

Related Issues

n/a

Testing

Manually tested in a cluster and added two new e2e tests for running apiext in a different namespace and modifying CRD labels.

Checklist

  • Does my change need to be backported to a previous release?
  • I made sure to update CHANGELOG.md.
  • This is unlikely to impact how Ambassador performs at scale.
  • My change is adequately tested.
  • I updated DEVELOPING.md with any any special dev tricks I had to use to work on this code efficiently.
  • The changes in this PR have been reviewed for security concerns and adherence to security best practices.

When a getambassador.io CRD explicitly sets webhook strategy
to None then the following should occur:

- CRD Patching ignores CRD and doesn't reconcile it
- CRD readiness checks ignore crd when checking for readiness

Signed-off-by: Lance Austin <laustin@datawire.io>
@LanceEa LanceEa changed the title apiext: debug ready check behavior apiext: restrict which crds are watched and patched. Feb 3, 2024
This returns the default restrictions on CRD labels for what
gets watched but makes it configurable for custom installs.

Signed-off-by: Lance Austin <laustin@datawire.io>
@LanceEa LanceEa marked this pull request as ready for review February 3, 2024 22:26
@LanceEa LanceEa merged commit 2bf8880 into master Feb 5, 2024
39 checks passed
@LanceEa LanceEa deleted the laustin/apiext-debug branch February 5, 2024 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants