Apigee known issues

Following the upgrade to 1-16-0-apigee-3, some Apigee organizations may encounter HTTP 500 errors stemming from a java.lang.NoClassDefFoundError within a Java Callout policy. The error appears similar to the following:

{"fault":{"faultstring":"Failed to execute JavaCallout. org/apache/commons/lang/StringUtils","detail":{"errorcode":"steps.javacallout.ExecutionError"}}}

This error occurs when a Java Callout policy incorrectly depends on libraries intended for Apigee's internal use instead of providing its own dependencies.

Action required: To prevent this issue, developers must ensure that all Java Callouts are self-contained and utilize their own dedicated libraries, avoiding any reliance on Apigee's internal class path.

In the API proxy that utilizes the Apache Commons JAR library, import the Apache Commons JAR file to include as a resource for the proxy. For more information, see Java resource guidelines.

Upgrading the apigee-org chart can fail if the mintTaskScheduler.serviceAccountPath property is not set, even if Monetization is not enabled:

Error: UPGRADE FAILED: execution error at (apigee-org/templates/mint-task-scheduler-gsa-secret.yaml:12:63): mintTaskScheduler.serviceAccountPath is required!

Workaround: Remove the apigee-org/templates/mint-task-scheduler-gsa-secret.yaml and apitee-org/templates/mint-task-scheduler-sa.yaml files.

You can optionally move the files to a separate location outside of the Helm chart directory. Alternatively you can download them again if you want to enable Monetization in the future, following the instructions in Step 2: Download the Apigee Helm charts.

For example, from the helm-charts/ directory:

1. ls apigee-org/templates/

   Output:

   apigee-org-guardrails.yaml              mart-sa.yaml
   apigee-proxy-chaining-certificate.yaml  mint-task-scheduler-gsa-secret.yaml
   apigee-proxy-chaining-route.yaml        mint-task-scheduler-sa.yaml
   ax-hash-salt-secret.yaml                NOTES.txt
   connect-agent-gsa-secret.yaml           organization.yaml
   connect-agent-sa.yaml                   udca-gsa-secret.yaml
   data-encryption-secret.yaml             udca-sa.yaml
   encryption-keys-secret.yaml             watcher-gsa-secret.yaml
   _helpers.tpl                            watcher-sa.yaml
   mart-gsa-secret.yaml
   

2. (Optional:)

   cp apigee-org/templates/mint-task-scheduler-gsa-secret.yaml /tmp/
   cp apigee-org/templates/mint-task-scheduler-sa.yaml /tmp/

3. rm apigee-org/templates/mint-task-scheduler-gsa-secret.yaml

4. rm apigee-org/templates/mint-task-scheduler-sa.yaml

5. ls apigee-org/templates/

   Output:

   apigee-org-guardrails.yaml              mart-gsa-secret.yaml
   apigee-proxy-chaining-certificate.yaml  mart-sa.yaml
   apigee-proxy-chaining-route.yaml        NOTES.txt
   ax-hash-salt-secret.yaml                organization.yaml
   connect-agent-gsa-secret.yaml           udca-gsa-secret.yaml
   connect-agent-sa.yaml                   udca-sa.yaml
   data-encryption-secret.yaml             watcher-gsa-secret.yaml
   encryption-keys-secret.yaml             watcher-sa.yaml
   _helpers.tpl
   

The apigee-pull-push.sh script can return a No such image error message, for example:

Error response from daemon: No such image: gcr.io/apigee-release/hybrid/apigee-stackdriver-logging-agent:latest

Workaround: Edit the HELM_CHARTS_DIR/apigee-operator/etc/tools/apigee-pull-push.sh script to change line 114 in the docker_tag()function from:

  docker tag "${source}/$i" "${dest}/$i:${TAG}"

To:

  docker tag "${source}/$i:${TAG}" "${dest}/$i:${TAG}"

When metrics.serviceAccountRef or metrics.serviceAccountSecretProviderClass is specified in the overrides.yaml file, the telemetry role will target the wrong service account.

Workaround: Patch _helper.tpl in the apigee-operator/ chart and reapply it.

1. Edit apigee-operator/templates/_helpers.tpl and remove the following bold lines:

   {{- define "metricsSA" -}}
     {{- $metricsName := "apigee-metrics" }}
     {{- $telemetryName := "apigee-telemetry" -}}
     {{- $generatedName := include "orgScopeEncodedName" (dict "name" .Values.org) -}}
     {{- if .Values.gcp.workloadIdentity.enabled -}}
     {{- printf "%s-sa" $metricsName -}}
     {{- else if .Values.serviceAccountSecretProviderClass -}}  <-- DELETE
     {{- .Values.serviceAccountSecretProviderClass -}}          <-- DELETE
     {{- else if .Values.metrics.serviceAccountRef -}}          <-- DELETE
     {{- .Values.metrics.serviceAccountRef -}}                  <-- DELETE
     {{- else if .Values.multiOrgCluster -}}
     {{- printf "%s-%s" $metricsName $generatedName -}}
     {{- else -}}
     {{- printf "%s-%s" $metricsName $telemetryName -}}
     {{- end -}}
   {{- end -}}

   The resulting section should look like the following:

   {{- define "metricsSA" -}}
     {{- $metricsName := "apigee-metrics" }}
     {{- $telemetryName := "apigee-telemetry" -}}
     {{- $generatedName := include "orgScopeEncodedName" (dict "name" .Values.org) -}}
     {{- if .Values.gcp.workloadIdentity.enabled -}}
     {{- printf "%s-sa" $metricsName -}}
     {{- else if .Values.multiOrgCluster -}}
     {{- printf "%s-%s" $metricsName $generatedName -}}
     {{- else -}}
     {{- printf "%s-%s" $metricsName $telemetryName -}}
     {{- end -}}
   {{- end -}}

2. Reapply the apigee-operator chart.

   helm upgrade operator apigee-operator/ \
       --namespace APIGEE_NAMESPACE \
       --atomic \
       -f overrides.yaml

For a workaround, follow the steps in Known issue 416634326.

The use of wildcards (*) in Apigee proxy basepaths may conflict with other explicit basepaths, resulting in a 404 error. For example, use of the following basepaths for two proxies deployed in the same environment may result in a 404 error when calling Proxy-2:

Proxy-1: /a/v1/b
Proxy-2: /a/*/c

In this case, calls to the explicit base path will resolve successfully, but calls to the Proxy-2 may return a 404, if the wildcard path is evaluated as /a/v1/c.

Fixed: This issue is fixed in Apigee and in hybrid 1.14.3 and later. However, the fix is not enabled by default. If you wish to enable the use of wildcards in basepaths:

• Customers with Apigee orgs should reach out to Apigee support to enable the fix if needed.
• Customers with Apigee hybrid orgs should use the following procedure to enable the fix if needed.

Procedure: To enable use of wildcards (*) in Apigee proxy basepaths in Apigee hybrid:

1. Add the following stanza to your overrides.yaml file before upgrading to hybrid 1.14.3 or later:

   runtime:
     cwcAppend:
       conf_message-processor-communication_classificationV2.enabled: "true"
   

2. Apply the changes to the apigee-env chart. Repeat the change for each environment in your installation.

   helm upgrade ENV_RELEASE_NAME apigee-env/ \
     --install \
     --namespace APIGEE_NAMESPACE \
     --set env=ENV_NAME \
     -f OVERRIDES_FILE
   

   • ENV_RELEASE_NAME is a name used to keep track of installation and upgrades of the apigee-env chart. This name must be unique from the other Helm release names in your installation. Usually this is the same as ENV_NAME. However, if your environment has the same name as your environment group, you must use different release names for the environment and environment group, for example dev-env-release and dev-envgroup-release. For more information on releases in Helm, see Three big concepts in the Helm documentation.
   • ENV_NAME is the name of the environment you are upgrading.
   • OVERRIDES_FILE is your overrides file.
3. Upgrade to hybrid 1.14.3 or later.

LogTimer usage in SecurityPolicy can cause a memory leak.

In some circumstances, Apigee hybrid's logger threads could consume all available memory. For example, frequent log entries documenting permissions errors associated with the Javacallout could cause OOM.

The Apigee Extension Processor does not support more than 100kb of data processing as part of request and response body events.

When updating a keystore or truststore without creating a new keystore or truststore, runtime updates may fail and cause the following intermittent error:

  {"fault":{"faultstring":"SSL Handshake failed sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","detail":{"errorcode":"messaging.adaptors.http.flow.SslHandshakeFailed"}}}

The error is intermittent because the keystore or truststore update may fail on one runtime pod but succeed on other pods. To avoid this issue, update the keystore or truststore by creating a new keystore or truststore. Repoint your reference to the new keystore or truststore, as described in When a cert expires.

Starting with Apigee hybrid version 1.14.0, automatic addition of Zipkin trace headers (x-b3-*) has been removed.

N/A

If you only specify httpProxy, you must also ensure that *.googleapis.com is allowlisted for your Apigee Runtime pods to enable connectivity to these services.

See Configure forward proxying for API proxies.

Presence of istio.io Custom Resource Definitions (CRDs) in an Apigee hybrid cluster may cause failure in apigee-ingressgateway-manager pods.

During Apigee Hybrid upgrade from older versions to 1.14.2 or later, presence of existing istio.io CRDs may cause failed readiness probes in the discovery containers of the apigee-ingressgateway-manager pods.

Workaround: The istio.io CRDs are not required by Apigee hybrid v1.14.2 or newer. There are two options to fix this issue:

• Delete the istio.io CRDs if you are not using Istio for any purpose other than Apigee in your cluster.
• Update the apigee-ingressgateway-manager clusterrole to add permissions for istio.io.

Afer each of the above options, you will need to restart your apigee-ingressgateway-manager pods.

After you have completed the above options, you will need to restart your apigee-ingressgateway-manager pods.

1. List the ingress-manager pods to reinstall or recreate:

   kubectl get deployments -n APIGEE_NAMESPACE

   Example output:

   NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
   apigee-controller-manager       1/1     1            1           32d
   apigee-ingressgateway-manager   2/2     2            2           32d
   

2. Restart the ingress-manager pods:

   kubectl rollout restart deployment -n APIGEE_NAMESPACE apigee-ingressgateway-manager

3. After a few minutes, monitor the apigee-ingressgateway-manager pods:

   watch -n 10 kubectl -n APIGEE_NAMESPACE get pods -l app=apigee-ingressgateway-manager

   Example output:

   NAME                                             READY   STATUS    RESTARTS   AGE
   apigee-ingressgateway-manager-12345abcde-678wx   3/3     Running   0          10m
   apigee-ingressgateway-manager-12345abcde-901yz   3/3     Running   0          10m
   

ApigeeTelemetry can become stuck in creating state.

This issue has been seen on OpenShift installations.

Workaround: Modify the apigee-operator chart template to create the correct clusterrole access.

1. Edit the helm-charts/apigee-operator/templates/apigee-operators.yaml template file, and locate the definition of the -apigee-manager-role- clusterrole. It starts with:

   kind: ClusterRole
   metadata:
     name: apigee-manager-role-{{ include 'namespace' }}
   rules:
     ...
                 

2. Find the - apiGroups: apiregistration.k8s.io block, and add the apiservices/finalizers resource to the list of resources:

   - apiGroups:
     - apiregistration.k8s.io
     resources:
     - apiservices
     - apiservices/finalizers
     verbs:
     - create
     - delete
     - get
     - patch
     - update
     

3. Find the - apiGroups: authorization.k8s.io block, and add the - apiGroups: apigee.cloud.google.com block after the end of the block with the following text:

   - apiGroups:
     - apigee.cloud.google.com
     resources:
     - apigeetelemetries/finalizers
     verbs:
     - get
     - patch
     - update
   

   For example:

   - apiGroups:
     - authorization.k8s.io
     resources:
     - subjectaccessreviews
     verbs:
     - create
     - get
     - list
   - apiGroups:
     - apigee.cloud.google.com
     resources:
     - apigeetelemetries/finalizers
     verbs:
     - get
     - patch
     - update
   

4. Apply the changes to the apigee-operator chart:

   Dry run:

   helm upgrade operator apigee-operator/ \
     --install \
     --namespace APIGEE_NAMESPACE \
     --atomic \
     -f OVERRIDES_FILE \
     --dry-run=server
   

   upgrade the chart:

   helm upgrade operator apigee-operator/ \
     --install \
     --namespace APIGEE_NAMESPACE \
     --atomic \
     -f OVERRIDES_FILE \
   

APIproducts are limited to 50 paths. If you add additional paths, an error message is displayed: Operation group limit of 51 exceeded in Operation Config

Workaround: Use wildcard patterns to combine resource paths and operations as described in Configuring resource paths.

Workaround: Create multiple API Products. This workaround has been verified and is viable for those who add a lot of API Proxies to an API Product. The App that is associated with your API Products would need to be updated to include the new one, but credentials and client-side request details have been proven to not need changing.

Error using Custom Reports and Stats APIs when selecting by fee type for installations using Monetization.

Workaround: Fetch all fee types on the client side before filtering.

Scaling of Istio ingress pods can cause a 503 error.

Scaling of the Istio ingress pods can occasionally cause a 503 error. If a 503 error occurs, logs for the load balancer display the following message: statusDetails: backend_connection_closed_before_data_sent_to_client.

Workaround: Manually scale the Istio ingress pods.

ESS and non-ESS Cassandra credential rotation does not work in organizations with Enhanced Proxy Limits.

Runtime traffic is not affected.

Backups taken from the multi-region setup using GCP and HYBRID Cloud Providers contain information about all the Hybrid regions that existed when the snapshot was taken. And, since those regions do not exist anymore, the restore job will conflict with the current state of Cassandra and fail with the following error: Unrecognized strategy option passed to NetworkTopologyStrategy.

The Nimbus JOSE + JWT library may cause a java.lang.ClassCircularityError when using a JavaCallout policy .

If you have a Apigee hybrid-enabled organization with a JavaCallout policy that uses the Nimbus JOSE + JWT library, do not upgrade to hybrid 1.12.4, hybrid 1.13.3, hybrid 1.14.1.

ESS and non-ESS Multi-region Cassandra credential rotation will fail in every region except the first.

Runtime traffic is not affected.

Follow the provided workaround to resolve this issue.

Cassandra pods will be restarted after applying Cassandra overrides changes, such when re-enabling backup, which will trigger this issue. The logs of a Cassandra pod in the CrashLoopBackoff state will show the following error: Cannot change the number of tokens from 512 to 256.

Follow the provided workaround to resolve this issue.

In some rare cases, the Apigee backup job doesn't clean up the Cassandra intermediate snapshots it creates while taking backups using HYBRID or GCP Cloud Providers. This only happens if an underlying issue prevents the backup process from successfully connecting to the remote server or Cloud Storage. If the connection issue persists, those leftover Cassandra snapshots can accumulate over time, utilizing storage on Cassandra's disks. If you're affected, please fix the underlying connection issue and then follow the steps provided in the Cassandra troubleshooting guide to clear Cassandra snapshots manually.

When a KeyValueMapOperations policy is used with apiproxy scope, and the policy's <Put> operation is called in a shared flow via a flow hook, the KVM entry is created under the shared flow name. It is expected to be created under the API proxy name.

384937220

When there are multiple virtual hosts, the Helm release creation may fail due to conflicting ApigeeRoute names. The workaround is to run the following commands for every virtual host when creating:

kubectl annotate ar apigee-ingressgateway-internal-chaining-PROJECT_ID_SUFFIX -n APIGEE_NAMESPACE meta.helm.sh/release-name=NEW_ENV_GROUP_NAME --overwrite
kubectl annotate cert apigee-ingressgateway-internal-chaining-PROJECT_ID_SUFFIX -n APIGEE_NAMESPACE meta.helm.sh/release-name=NEW_ENV_GROUP_NAME --overwrite

where:

• PROJECT_ID_SUFFIX is a unique suffix for internal chaining for your project in Kubernetes. You can find this suffix with the following command:

  kubectl get svc -n apigee -l app=apigee-ingressgateway | grep internal-chaining

  Your output will look something like:

  kubectl get svc -n apigee -l app=apigee-ingressgateway | grep internal-chaining
  apigee-ingressgateway-internal-chaining-my-project--1234567    ClusterIP  34.118.226.140  <none>    15021/TCP,443/TCP    5d6h

  In the example output, my-project--1234567 is the PROJECT_ID_SUFFIX.

• APIGEE_NAMESPACE is your Apigee namespace.
• NEW_ENV_GROUP_NAME is the name the additional environment group. Update this value for each virtual host.

N/A

If a user provides an invalid service account to the UpdateControlPlaneAccess api, the operation goes on a retry loop effectively locking the organization from invoking the API until the operation times out.

When using the GCP Cloud Provider, the Apigee backup job is unable to upload to Cloud Storage buckets with retention policies. Backup files may be left in the Cloud Storage bucket with 0 byte file size.

Workaround: Disable retention policies on the Cloud Storage bucket.

341099433

apigee-logger utilizes Google IAM service accounts for shipping logs to Cloud Logging. This is due to FluentBit's lack of support for Workload Identity Federation, which prevents the apigee-logger from utilizing this feature.

N/A

N/A

270574696

268104619

364872027

For Apigee and Apigee hybrid versions 1.13 and higher, any deviations in the required PEM format of keys used in Apigee JWS or JWT policies may result in a parsing error. For example, placing any character other an a newline (/n) immediately before the "-----END" line (post-encapsulation boundary) is not allowed and will result in an error.

To prevent this error, make sure that no characters other than a newline, such as trailing spaces or slashes, immediately precede the post-encapsulation boundary.

For more information about the encoding used for public or private keys, see IETF RFC 7468.

310191899

The following endpoints may experience timeouts when used with a high volume of queries per second (QPS):

• organizations.environments.apis.revisions.
  deployments.deploy
• organizations.environments.apis.revisions.
  deployments.undeploy
• organizations.environments.sharedflows.revisions.
  deployments.deploy
• organizations.environments.sharedflows.revisions.
  deployments.undeploy

To reduce the likelihood of timeouts, we recommend setting a target of 1 QPS when using these endpoints or checking the status of a deployment before attempting another deployment.

329304975

Apigee is enforcing a temporary limit of 1000 basepaths per environment to avoid potential failures when deploying API proxy revisions.

While this limit is in place, you can deploy up to 1000 API proxy revisions (each containing a single basepath) per environment. If your API proxies or revisions contain more than one basepath, the total number of basepaths per environment must not exceed 1000.

333791378

For the steps required to install a patch for the workaround, see Troubleshooting.

310384001

289583112

If the OASValidation policy specifies an <OASResource> with security requirements set at a global level, the security requirements are not enforced.

Workaround: To ensure enforcement, all security requirements must be set at the operation level in the OpenAPI specification passed in the <OASResource> element of the OASValidation policy.

205666368

See About setting TLS options in a target endpoint or target server.

295929616

Installing or upgrading to Apigee hybrid 1.10.0 through 1.10.2 could fail on OSE due to out-of-memory issues. Fixed in Apigee hybrid version 1.10.3.

292118812

If the firewall forces all traffic through a forward-proxy, apigee-udca may go into a crash-loop backoff state.

292558790

• The OASValidation Policy fails when the JSON content does not match the anticipated pattern. For example, if a header is expecting a value in the format <text>@<text> and is populated with text missing the @ symbol, the policy will fail with an Unable to parse JSON error.
• If the OASValidation policy specifies an <OASResource> containing a path parameter that utilizes a $ref schema, the policy will fail with an Unable to parse JSON - Unrecognized token error.

  Workaround: Do not use $ref in the path parameters of the OpenAPI spec specified in the <OASResource> element.

297012500

• Apigee deployment will fail for OAS Validation policy when using circular references for OpenAPI 3.0.0 specification as it gets into an infinite loop.

Workaround: Use an OpenAPI specification yaml without circular references.

289254725

Proxy deployments that include the OASValidation policy may fail if:

• The OpenAPI specification used for validation in the OASValidation policy is in YAML format, and
• The YAML-formatted OpenAPI specification contains a floating number. For example:

  schema:
  type: number
  example: 2.345

284500460

To avoid increasing latency in responses to the client, the Message Logging policy should be attached to the PostClientFlow. For more information on using policies in PostClientFlows, see Controlling API proxies with flows.

282997216

Use only alphanumeric characters for the Cassandra Jolokia password. Using special characters (including but not limited to "!", "@", "#", "$", "%", "^", "&", & "*") can cause Cassandra startup to fail.

270371160

Apigee Ingress gateway only supports TLS1.2+, and not earlier versions of TLS.

269139342

Apigee organization validation does not follow HTTP Forward proxy rules set in overrides.yaml. Set validateOrg: false to skip this validation.

266452840

In certain circumstances, web sockets are not working for Apigee X and Apigee Hybrid when using Anthos Service Mesh 1.15.3-asm.6.

242213234

This error might be returned when attempting to load API products: "Products were not loaded successfully. Error: no connections available from the Apigee connect agent(s)."

The problem occurs after enabling VPC service control in the Google Cloud project and adding iamcredentials.googleapis.com as one of the restricted services in the service perimeter.

Workaround: Manually create an egress rule, such as the following:

-egressTo:
    operations:
    -serviceName: "iamcredentials.googleapis.com"
        methodSelectors:
        -method:
    resources:
    -projects/608305225983
  egressFrom:
    identityType: ANY_IDENTITY

247540503

In certain circumstances at very high throughput a race condition with encryption key lookup can cause KVM lookup failures.

258699204

If you see problems with the apigee-telemetry-app or apigee-telemetry-proxy pods not running, change the metrics resources requests and resources limits properties to match the following defaults in Configuration property reference: metrics.

Apply the changes with apigeectl apply with the ‑‑telemetry flag:

apigeectl apply --telemetry -f overrides.yaml

260324159

API proxies and shared flows could take around 20 to 30 minutes to deploy in the runtime plane in certain circumstances due to a 'socket closed' error in synchronizer.

214447386

This is expected to occur every minute and does not affect your billing cost.

260772383

If installing hybrid on AKS, you may see this error:

envoy config listener '0.0.0.0_443' failed to bind or apply socket options: cannot bind '0.0.0.0:443': Permission denied

Workaround: Add the following svcAnnotations stanza to the overrides file:

ingressGateways:
- name: INGRESS_NAME
  ...
  svcAnnotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

See Configure the hybrid runtime. See also Use an internal load balancer with AKS.

241786534

When using Org-scoped UDCA, MART is sometimes unable to connect to FluentD. Org-scoped UDCA is the default in Apigee hybrid version 1.8. See orgScopedUDCA in the Configuration property reference.

After migration of apigee-logger from fluend to fluent-bit in Apigee hybrid version 1.6.6, the logger stopped working on Anthos BareMetal with CentOS or RHEL.

Current apigee-logger configurations, including liveness probes, are incompatible with the Kubernetes runtime, resulting in logs not being shipped to Cloud Logging as expected. This problem also causes the apigee-logger pods to crash regularly. This issue affects Apigee hybrid installations on AKS, Anthos Bare Metal, and other platforms. Note that in some cases this issue results in excessive log volume.

In the Apigee UI, you cannot view, confirm deployment status, or manage your archive deployments, as described Deploying an API proxy, or use the Debug UI as described in Using Debug. As a workaround, you can use gcloud or the API to List all archive deployments in an environment and use the Debug API.

Rolling back an archive deployment is not currently supported. To remove a version of an archive deployment you need to either redeploy a previous version of an archive or delete the environment.

421402073

Google authentication in ServiceCallout and ExternalCallout policies, as described in Using Google Authentication, is not supported in Apigee in VS Code.

x-b3 headers not available when DistributedTrace is disabled

In the Apigee 1-15-0-apigee-5 and hybrid 1.15.0 releases, Apigee stopped sending x-b3 headers to target endpoints when Distributed Trace is disabled. However, some Apigee installations were relying on x-b3 headers being sent regardless of whether Distributed Trace was enabled or disabled. Therefore this change has been reverted starting in Apigee 1-15-0-apigee-7.

Invalid HTTP Header error: The Istio ingress switches all incoming target responses to the HTTP2 protocol. Because the hybrid message processor only supports HTTP1, you may see the following error when an API proxy is called:

http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1,

name: [:authority], value: [domain_name]

If you see this error, you can take either of the following actions to correct the problem:

• Modify the target service to omit the Host header in the response.
• Remove the Host header using the AssignMessage policy in your API proxy if necessary.

420985360

• Apigee supports OpenAPI Specification 3.0 when you publish your APIs using SmartDocs on your portal, though a subset of features are not yet supported. For example, allOf properties for combining and extending schemas.

  If an unsupported feature is referenced in your OpenAPI Specification, in some cases the tools will ignore the feature but still render the API reference documentation. In other cases, an unsupported feature will cause errors that prevent the successful rendering of the API reference documentation. In either case, you will need to modify your OpenAPI Specification to avoid use of the unsupported feature until it is supported in a future release.

• Try this API has the following limitations:
  • The Accept header is set to application/json regardless of the value set for consumes in the OpenAPI Specification.
  • Request headers of type other than string are not supported.

• Simultaneous portal updates (such as page, theme, CSS, or script edits) by multiple users is not supported at this time.
• If you delete an API reference documentation page from the portal, there is no way to recreate it; you'll need to delete and re-add the API product, and regenerate the API reference documentation.
• When customizing your portal theme, it may take up to 5 minutes for changes to fully apply.

Search will be integrated into the integrated portal in a future release.

Single logout (SLO) with the SAML identity provider is not supported for custom domains. To enable a custom domain with a SAML identity provider, leave the Sign-out URL field blank when you configure SAML settings.

• API Proxy request and response counts (for proxy and targets) show abnormal spikes

  Here is a sample showing such a spike:

  (view larger image)

  
• Due to a bug, the system registers the count incorrectly for a brief period and the count is corrected. This happens where there is a reduction in API traffic (which results in a scale down of API gateways).
• To distinguish actual spikes in requests vs. this issue, please consult the API Analytics page (specifically the Proxy Performance and Target Performance pages)

Affected Metrics:

• apigee.googleapis.com/proxyv2/request_count
• apigee.googleapis.com/proxyv2/response_count
• apigee.googleapis.com/targetv2/request_count
• apigee.googleapis.com/targetv2/response_count

New metrics

You can use the new metrics to avoid this issue.

For Apigee hybrid, see: Metrics collection overview and View metrics.

Workaround: Disable the logging agent on hybrid.

Workaround: To maintain the fire and forget behavior:

1. Add <Response>calloutResponse</Response> to the ServiceCallout.
2. Set continueOnError to true.

Workaround: Avoid using more than one SpikeArrest policy in the proxy to prevent the issue.

For a particular key, if there is continuous traffic, the key may not be rate limited at the updated rate. If there is five minutes of no traffic for a particular key, the rate will be reflected.

Workaround: Redeploy the proxy with a new reference variable if the rate has to take effect immediately. Or use two conditional spike arrests with different flow variables to adjust the rate.

API Monitoring of Configurable API Proxies may display a fault code of '(not set)' for responses with a non-2xx status from the target.

If a proxy sets two or more io.timeout.millis values in two or more flows using the same target host, only one io.timeout.millis value is honored.

During upgrade to Apigee hybrid 1.8.x, after running apigeectl init and confirming that check-ready succeeded, you may notice, if you view the pods, that the Cassandra schema validation job is in an error state. This is a harmless condition, and you can safely move to the next step in the upgrade procedure.

Deploying proxies with the same path to multiple environments that are attached to the same instance and environment group is not allowed and should return a warning message about a base path conflict. Instead, no error is shown and the deployments appear to succeed.

Workaround: When deploying and after deployment, verify that there are not base path conflicts with deployed proxies and correct as needed.

When attempting to save a previously deployed proxy, the deployment might fail with an error stating that the revision is immutable.

Workaround: Click the dropdown arrow next to the Save button and select Save as new revision. Then reattempt the deployment.

When a call is made to a gRPC Target Server, the only trailer that's returned is the "grpc-status" trailer. All other trailers are removed from the response.

An infinite recursion occurs within the OASValidation policy when a backend response contains a discriminator type that is the same as the parent schema. This problem can lead to a StackOverflowError.

Use separate target endpoint definitions for SSE (server sent-event) targets. Mixing SSE and non-SSE target endpoints together might result in inconsistent behavior such as empty response.content flow variables.

Previously undetected DNS configuration problems might now cause DNS errors. Apigee removed the automatic DNS fallback functionality that was in 1-16-0-apigee-2. This removal eliminates the platform's resiliency to DNS misconfigurations and might now result in DNS errors.

Find related errors by checking the runtime logs for DNS resolution errors.