GitHub - atr93/karapace: Karapace - Your Apache Kafka® essentials in one tool

Karapace

karapace. Your Apache Kafka® essentials in one tool.

An open-source implementation of Kafka REST and Schema Registry.

Overview

Karapace supports the storing of schemas in a central repository, which clients can access to serialize and deserialize messages. The schemas also maintain their own version histories and can be checked for compatibility between their different respective versions.

Karapace rest provides a RESTful interface to your Apache Kafka cluster, allowing you to perform tasks such as producing and consuming messages and perform administrative cluster work, all the while using the language of the WEB.

Features

Drop-in replacement both on pre-existing Schema Registry / Kafka Rest Proxy client and server-sides
Moderate memory consumption
Asynchronous architecture based on aiohttp
Supports Avro, JSON Schema, and Protobuf
Leader/Replica architecture for HA and load balancing
Observability with Metrics and OpenTelemetry
Based on aiokafka (rdkafka) kafka library mostly
Schema registry based on FastApi

Compatibility details

Karapace is compatible with Schema Registry 6.1.1 on API level. When a new version of SR is released, the goal is to support it in a reasonable time. Karapace supports all operations in the API.

There are some caveats regarding the schema normalization, and the error messages being the same as in Schema Registry, which cannot be always fully guaranteed.

Setup

Using Docker

To get you up and running with the latest build of Karapace, a docker image is available:

# Fetch the latest build from main branch
docker pull ghcr.io/aiven-open/karapace:develop

# Fetch the latest release
docker pull ghcr.io/aiven-open/karapace:latest

Versions 3.7.1 and earlier are available from the ghcr.io/aiven registry:

docker pull ghcr.io/aiven/karapace:3.7.1

An example setup including configuration and Kafka connection is available as compose example:

docker compose -f ./container/compose.yml up -d

Then you should be able to reach two sets of endpoints:

Karapace schema registry on http://localhost:8081
Karapace REST on http://localhost:8082

For local development with Docker, use the convenient make commands (see Development section below).

Configuration

Each configuration key can be overridden with an environment variable prefixed with KARAPACE_, exception being configuration keys that actually start with the karapace string. For example, to override the bootstrap_uri config value, one would use the environment variable KARAPACE_BOOTSTRAP_URI. Here you can find an example configuration file to give you an idea what you need to change.

Using Sources

Install

You can do a source install using:

pip install .

Troubleshooting notes : - An updated version of wheel (https://pypi.org/project/wheel/) is required. - An updated version of 'go' and 'rust' is required - Create and activate virtual environment (venv) to manage dependencies

Run

Make sure kafka is running.

Start Karapace. This shout start karapace on http://localhost:8081

$ karapace karapace.config.json

Verify in browser http://localhost:8081/subjects should return an array of subjects if exist or an empty array. or with curl

$ curl -X GET http://localhost:8081/subjects

To enable oidc authentication on the karapace, configure oidc jwks url config details

sasl_oauthbearer_jwks_endpoint_url = "",
sasl_oauthbearer_expected_issuer = "",
sasl_oauthbearer_expected_audience = "",
sasl_oauthbearer_sub_claim_name = "sub",

There is a detailed section about OAuth2 authentication for karapace below.

To enable oidc authorization on karapace, configure the below params together with the above

sasl_oauthbearer_authorization_enabled: bool = False
sasl_oauthbearer_client_id: str | None = None
sasl_oauthbearer_roles_claim_path: str | None = None
sasl_oauthbearer_method_roles: dict[str, list[str]] = {"GET": [], "POST": [], "PUT": [], "DELETE": []}

There is a detailed section about OAuth2 authorization for karapace below.

Start Karapace rest proxy. This shout start karapace on http://localhost:8082

karapace rest-proxy-karapace.config.json

To enable authorization & authentication on the rest proxy, configure 'sasl_mechanism' in the config with values like PLAIN/OAUTHBEARER

sasl_mechanism = "OAUTHBEARER",
sasl_oauth_token_provider = token_provider,
security_protocol="SASL_SSL",
ssl_cafile="ca.pem",

If 'sasl_mechanism' is configured to PLAIN:

sasl_mechanism = "PLAIN",
security_protocol = "SASL_PLAINTEXT",
sasl_plain_username = "your_username",
sasl_plain_password = "your_password"

There is a detailed section about OAuth2 authentication for rest proxy below.

Verify with list topics:

$ curl "http://localhost:8082/topics"

Schema Registry Api reference

To register the first version of a schema under the subject "test" using Avro schema:

$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\": \"record\", \"name\": \"Obj\", \"fields\":[{\"name\": \"age\", \"type\": \"int\"}]}"}' \
  http://localhost:8081/subjects/test-key/versions
{"id":1}

To register a version of a schema using JSON Schema, one needs to use schemaType property:

$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schemaType": "JSON", "schema": "{\"type\": \"object\",\"properties\":{\"age\":{\"type\": \"number\"}},\"additionalProperties\":true}"}' \
  http://localhost:8081/subjects/test-key-json-schema/versions
{"id":2}

To list all subjects (including the one created just above):

$ curl -X GET http://localhost:8081/subjects
["test-key"]

To list all the versions of a given schema (including the one just created above):

$ curl -X GET http://localhost:8081/subjects/test-key/versions
[1]

To fetch back the schema whose global id is 1 (i.e. the one registered above):

$ curl -X GET http://localhost:8081/schemas/ids/1
{"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}

To get the specific version 1 of the schema just registered run:

$ curl -X GET http://localhost:8081/subjects/test-key/versions/1
{"subject":"test-key","version":1,"id":1,"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}

To get the latest version of the schema under subject test-key run:

$ curl -X GET http://localhost:8081/subjects/test-key/versions/latest
{"subject":"test-key","version":1,"id":1,"schema":"{\"fields\":[{\"name\":\"age\",\"type\":\"int\"}],\"name\":\"Obj\",\"type\":\"record\"}"}

In order to delete version 10 of the schema registered under subject "test-key" (if it exists):

$ curl -X DELETE http://localhost:8081/subjects/test-key/versions/10
 10

To Delete all versions of the schema registered under subject "test-key":

$ curl -X DELETE http://localhost:8081/subjects/test-key
[1]

Test the compatibility of a schema with the latest schema under subject "test-key":

$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\": \"int\"}"}' \
  http://localhost:8081/compatibility/subjects/test-key/versions/latest
{"is_compatible":true}

NOTE: if the subject's compatibility mode is transitive (BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE or FULL_TRANSITIVE) then the compatibility is checked not only against the latest schema, but also against all previous schemas, as it would be done when trying to register the new schema through the subjects/<subject-key>/versions endpoint.

Get current global backwards compatibility setting value:

$ curl -X GET http://localhost:8081/config
{"compatibilityLevel":"BACKWARD"}

Change compatibility requirements for all subjects where it's not specifically defined otherwise:

$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "NONE"}' http://localhost:8081/config
{"compatibility":"NONE"}

Change compatibility requirement to FULL for the test-key subject:

$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL"}' http://localhost:8081/config/test-key
{"compatibility":"FULL"}

< 10BC8 div class="markdown-heading" dir="auto">

Name		Name	Last commit message	Last commit date
.github		.github
benchmarks		benchmarks
bin		bin
container		container
examples		examples
go/protopace		go/protopace
performance-test		performance-test
requirements		requirements
runtime		runtime
src/karapace		src/karapace
stubs/isodate		stubs/isodate
tests		tests
website		website
.coveragerc		.coveragerc
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.env.example		.env.example
.git-blame-ignore-revs		.git-blame-ignore-revs
.gitignore		.gitignore
.isort.cfg		.isort.cfg
.pre-commit-config.yaml		.pre-commit-config.yaml
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
GNUmakefile		GNUmakefile
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
NEWS		NEWS
README.rst		README.rst
SECURITY.md		SECURITY.md
copyright.sh		copyright.sh
karapace.config.json		karapace.config.json
karapace.unit		karapace.unit
mypy.ini		mypy.ini
pyproject.toml		pyproject.toml
pytest.ini		pytest.ini
rest-proxy-karapace.config.json		rest-proxy-karapace.config.json
runtime.txt		runtime.txt
setup.py		setup.py

Format	Karapace	Confluent
Avro	80.95	7.22
Binary	66.32	46.99
Json	60.36	53.7

Format	Karapace	Confluent
Avro	25.05	18.14
Binary	21.35	15.85
Json	21.38	14.83

Format	Karapace	Confluent
Avro	6.54	5.67
Binary	6.51	4.56
Json	6.86	5.32

Consumers	Karapace combined	Confluent rest
1	47	200
10	55	400
20	83	530

Parameter	Default Value	Description
`http_request_max_size`	`1048576`	The maximum client HTTP request size. This value controls how large (POST) payloads are allowed. When configuration of `karapace_rest` is set to true and `http_request_max_size` is not set, Karapace configuration adapts the allowed client max size from the `producer_max_request_size`. In cases where automatically selected size is not enough the configuration can be overridden by setting a value in configuration. For schema registry operation set the client max size according to expected size of schema payloads if default size is not enough.
`advertised_protocol`	`http`	The protocol being advertised to other instances of Karapace that are attached to the same Kafka group.
`advertised_hostname`	`socket.gethostname()`	The hostname being advertised to other instances of Karapace that are attached to the same Kafka group. All nodes within the cluster need to have their `advertised_hostname`'s set so that they can all reach each other.
`advertised_port`	`None`	The port being advertised to other instances of Karapace that are attached to the same Kafka group. Fallbacks to `port` if not set.
`bootstrap_uri`	`localhost:9092`	The URI to the Kafka service where to store the schemas and to run coordination among the Karapace instances.
`sasl_bootstrap_uri`	`None`	The URI to the Kafka service to use with the Kafka REST API when SASL authorization with REST is used.
`client_id`	`sr-1`	The `client_id` Karapace will use when coordinating with other Karapace instances. The instance with the ID that sorts first alphabetically is chosen as master from the services with master_eligibility set to true.
`consumer_enable_autocommit`	`True`	Enable auto commit on rest proxy consumers
`consumer_request_timeout_ms`	`11000`	Rest proxy consumers timeout for reads that do not limit the max bytes or provide their own timeout
`consumer_request_max_bytes`	`67108864`	Rest proxy consumers maximum bytes to be fetched per request
`consumer_idle_disconnect_timeout`	`0`	Disconnect idle consumers after timeout seconds if not used. Inactivity leads to consumer leaving consumer group and consumer state. 0 (default) means no auto-disconnect.
`fetch_min_bytes`	`1`	Rest proxy consumers minimum bytes to be fetched per request.
`group_id`	`schema-registry`	The Kafka group name used for selecting a master service to coordinate the storing of Schemas.
`master_eligibility`	`true`	Should the service instance be considered for promotion to the master service. One reason to turn this off would be to have an instance of Karapace running somewhere else for HA purposes but which you wouldn't want to automatically promote to master if the primary instances become unavailable.
`producer_compression_type`	`None`	Type of compression to be used by rest proxy producers
`producer_acks`	`1`	Level of consistency desired by each producer message sent on the rest proxy. More on Kafka Producer
`producer_linger_ms`	`0`	Time to wait for grouping together requests. More on Kafka Producer
`producer_max_request_size`	`1048576`	The maximum size of a request in bytes. More on Kafka Producer configs
`security_protocol`	`PLAINTEXT`	Default Kafka security protocol needed to communicate with the Kafka cluster. Other options is to use SSL for SSL client certificate authentication, or SASL_PLAINTEXT / SASL_SSL for simple authentication.
`sentry`	`None`	Used to configure parameters for sentry integration (dsn, tags, ...). Setting the environment variable `SENTRY_DSN` will also enable sentry integration.
`ssl_cafile`	`/path/to/cafile`	Used when `security_protocol` is set to SSL, the path to the SSL CA certificate.
`ssl_certfile`	`/path/to/certfile`	Used when `security_protocol` is set to SSL, the path to the SSL certfile.
`ssl_keyfile`	`/path/to/keyfile`	Used when `security_protocol` is set to SSL, the path to the SSL keyfile.
`topic_name`	`_schemas`	The name of the Kafka topic where to store the schemas.
`replication_factor`	`1`	The replication factor to be used with the schema topic.
`host`	`127.0.0.1`	Listening host for the Karapace server. Use an empty string to listen to all available networks.
`port`	`8081`	Listening port for the Karapace server.
`server_tls_certfile`	`/path/to/certfile`	Filename to a certificate chain for the Karapace server in HTTPS mode.
`server_tls_keyfile`	`/path/to/keyfile`	Filename to a private key for the Karapace server in HTTPS mode.
`server_tls_cafile`	`/path/to/cafile`	Filename to the SSL CA certificate.
`server_tls_client_auth`	`none`	Client certificate requirement: `none`, `optional`, or `required`.
`registry_scheme`	`http`	Schema Registry scheme to use for rest-proxy, http \| https (if certificates are provided).
`registry_host`	`127.0.0.1`	Schema Registry host, used by Kafka Rest for schema related requests. If running both in the same process, it should be left to its default value
`registry_port`	`8081`	Schema Registry port, used by Kafka Rest for schema related requests. If running both in the same process, it should be left to its default value
`registry_user`	`None`	Schema Registry user for authentication, used by Kafka Rest for schema related requests.
`registry_password`	`None`	Schema Registry password for authentication, used by Kafka Rest for schema related requests.
`registry_ca`	`/path/to/cafile`	Kafka Registry CA certificate, used by Kafka Rest for Avro related requests. If this is set, Kafka Rest will use HTTPS to connect to the registry. If running both in the same process, it should be left to its default value
`registry_authfile`	`/path/to/authfile.json`	Filename to specify users and access control rules for Karapace Schema Registry. If this is set, Schema Segistry requires authentication for most of the endpoints and applies per endpoint authorization rules.
`rest_authorization`	`false`	Use REST API's calling authorization credentials to invoke Kafka operations over SASL authentication of `sasl_bootstrap_uri` to delegate REST proxy authorization to Kafka. If false, then use configured common credentials for all Kafka connections of REST proxy operations.
`rest_base_uri`	`None`	Publicly available URI of this instance advertised to the clients using stateful operations such as creating consumers. If not set, then construct URI using `advertised_protocol`, `advertised_hostname`, and `advertised_port`.
`metadata_max_age_ms`	`60000`	Period of time in milliseconds after Kafka metadata is force refreshed.
`karapace_rest`	`true`	If the rest part of the app should be included in the starting process At least one of this and `karapace_registry` options need to be enabled in order for the service to start
`karapace_registry`	`true`	If the registry part of the app should be included in the starting process At least one of this and `karapace_rest` options need to be enabled in order for the service to start
`protobuf_runtime_directory`	`runtime`	Runtime directory for the `protoc` protobuf schema parser and code generator
`name_strategy`	`topic_name`	Name strategy to use when storing schemas from the kafka rest proxy service. You can opt between `topic_name` , `record_name` and `topic_record_name`
`name_strategy_validation`	`true`	If enabled, validate that given schema is registered under used name strategy when producing messages from Kafka Rest
`master_election_strategy`	`lowest`	Decides on what basis the Karapace cluster master is chosen (only relevant in a multi node setup)
`kafka_schema_reader_strict_mode`	`false`	If enabled, causes the Karapace schema-registry service to shutdown when there are invalid schema records in the _schemas topic
`kafka_retriable_errors_silenced`	`true`	If enabled, kafka errors which can be retried or custom errors specififed for the service will not be raised, instead, a warning log is emitted. This will denoise issue tracking systems, i.e. sentry
`use_protobuf_formatter`	`false`	If protobuf formatter should be used on protobuf schemas in order to normalize schemas. The formatter is used on top and independent of regular normalization and schemas will be persisted in a formatted state.
`log_handler`	`stdout`	Select the log handler. Default is standard output. Alternative log handler is `systemd`.
`log_level`	`DEBUG`	Logging level. Default level is debug.
`log_format`	`%(name)-20s\t%(threadName)s\t%(levelname)-8s\t%(message)s`	Log format
`waiting_time_before_acting_as_master_ms`	`5000`	The time that a master wait before becoming an active master if at the previous round of election wasn't the master (in that case the waiting time its skipped). Should be an upper bound of the time required for a master to write a message in the kafka topic + the time required from a node in the cluster to consume the Log of messages. If the value its too low there is the risk under high load of producing different schemas with the ID.

Parameter	Description
`username`	A string
`algorithm`	One of supported hashing algorithms, `scrypt`, `sha1`, `sha256`, or `sha512`
`salt`	Salt used for hashing the password
`password_hash`	Hash string of the password calculated using given algorithm and salt.

Parameter	Description
`username`	A string to match against authenticated user
`operation`	Exact value of `Read` or `Write`. Write implies also read permissions. Write includes all mutable operations, e.g. deleting schema versions
`resource`	A regular expression used to match against accessed resource.

Resource	Description
`Config:`	Controls authorization to global schema registry configuration.
`Subject:<subject_name>`	Controls authorization to subject. The `<subject_name>` is a regular expression to match against the accessed subject.

Latest commit

History

Folders and files

Repository files navigation

Karapace

Overview

Features

Compatibility details

Setup

Using Docker

Configuration

Using Sources

Install

Run

Schema Registry Api reference

Schema Registry Rest proxy Api reference

Backing up your Karapace

Performance comparison to Confluent stack

Latency

Consuming RAM

Commands

Configuration keys

Authentication and authorization of Karapace Schema Registry REST API

Example of complete authorization file

Karapace Schema Registry access to the _schemas topic

OAuth2 authentication and authorization of Karapace

Testing the authentication and authorization with docker

Get token

Access schema registry

OAuth2 authentication and authorization of Karapace REST proxy

Token expiry

Schema Normalization

Uninstall

Installed via Docker

Installed from Sources

Development

Quick Start

Available Commands

License

Contact

Trademark

Credits

About

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages