hodgesrm · Dec 5, 2023 · Dec 1, 2023
diff --git a/README.md b/README.md
diff --git a/doc/OLD_README.md b/doc/OLD_README.md
diff --git a/doc/architecture.md b/doc/architecture.md
@@ -1,13 +1,62 @@
-# Kafka Sink Connector Architecture
+# Sink Connector Architecture Overview
 
-## Exactly Once Semantics
+The Sink Connector extracts transactions from upstream databases 
+using [Debezium](debezium) into a common log format and then applies those 
+transactions to tables in ClickHouse. 
 
-Exactly once semantics are guaranteed by storing offsets(by topic, partition) in a separate 
-clickhouse table. The table will be ordered by time and a materialized view will be created as a 
-cache to avoid expensive table scans.
+There are two modes of operation. 
+* Lightweight Sink Connector - Combines extract and apply operations 
+  into a single process. 
+* Kafka Sink Connector - Separates extract and apply operations into separate
+  processes, using a Kafka-compatible event stream for transport between them. 
 
-Topic management will be disabled in Kafka connect and the `precommit()` function in 
-SinkTask would return the offsets stored in clickhouse.
+Debezium offers change data capture on a number of database types. The 
+Sink Connector has been tested extensively on MySQL and PostgreSQL. 
+
+## Lightweight Sink Connector
+
+![](img/kafka_replication_tool.jpg)
+
+New tool to replicate data from MySQL, PostgreSQL, MariaDB and Mongo
+without additional dependencies.  Single executable and lightweight.
+
+The Lightweight Sink Connector translates MySQL DDL to ClickHouse 
+schema. This includes picking up and applying schema changes. 
+
+## Kafka Sink Connector 
+
+Kafka Sink Connector transfers data from Kafka to Clickhouse using
+the Kafka Connect framework.
+
+The connector is tested with the following converters
+- JsonConverter
+- AvroConverter (Using [Apicurio Schema Registry](https://www.apicur.io/registry/) or Confluent Schema Registry)
+
+![](img/sink_connector_mysql_architecture.jpg)
+
+The Kafka Sink Connector has a complete set of features for transactional 
+replication. 
+
+- Inserts, Updates and Deletes using ReplacingMergeTree - [Updates/Deletes](doc/mutable_data.md)
+- Auto create tables in ClickHouse
+- Exactly once semantics 
+- Bulk insert to Clickhouse 
+- Store [Kafka metadata](Kafka_metadata.md)
+- Kafka topic to ClickHouse table mapping, use case where MySQL table can be mapped to a different CH table name.
+- Store raw data in JSON(For Auditing purposes)
+- Monitoring(Using Grafana/Prometheus) Dashboard to monitor lag
+- Kafka Offset management in ClickHouse
+- Increased Parallelism(Customize thread pool for JDBC connections)
+
+### Exactly Once Semantics
+
+Exactly once semantics are guaranteed by storing offsets(by topic,
+partition) in a separate clickhouse table. The table will be ordered
+by time and a materialized view will be created as a cache to avoid
+expensive table scans.
+
+Topic management will be disabled in Kafka connect and the `precommit()`
+function in SinkTask would return the offsets stored in clickhouse.
 
 References:
-[1] https://stackoverflow.com/questions/55608325/clickhouse-select-last-record-without-max-on-all-table
+[1] https://stackoverflow.com/questions/55608325/clickhouse-select-last-record-without-max-on-all-table
diff --git a/doc/configuration.md b/doc/configuration.md
diff --git a/doc/mutable_data.md b/doc/mutable_data.md
@@ -1,3 +1,5 @@
+# Mutable Data Handling in Sink Connector
+
 ## How Updates/Deletes are Handled - ReplacingMergeTree
 Sink Connector will attempt to read the `engine_full` column from system.tables for the corresponding table and will 
 identify the `engine` and the `ver` column.

diff --git a/doc/quickstart.md b/doc/quickstart.md
@@ -0,0 +1,81 @@
+# Sink Connector QuickStart Guide
+
+Use [Docker Compose](https://docs.docker.com/compose/) to bring 
+up a complete configuration that illustrates operation of 
+Altinity Sink Connector.
+
+# QuickStart Installation for Lightweight Sink Connector
+
+This is the recommended path for initial use. It uses MySQL as the
+source database and brings up a full stack with MySQL, ClickHouse, 
+Lightweight Sink Connector, and Grafana. Example shown here is for
+Ubuntu but will work for any Linux or MacOS provided prerequisites
+are installed. 
+
+## Prerequisites
+
+Install Docker and Docker Compose.
+
+* [Docker Installation](https://docs.docker.com/engine/install/) 
+* [Docker Compose Installation](https://docs.docker.com/compose/) 
+
+Install MySQL client and ClickHouse client. 
+```
+sudo apt update
+sudo apt install mysql-client
+sudo apt install clickhouse-client
+```
+
+## Start the stack 
+
+Use Docker Compose to start containers. 
+```
+cd sink-connector-lightweight/docker
+export SINK_LIGHTWEIGHT_VERSION=latest
+docker compose -f docker-compose-mysql.yml up --renew-anon-volumes
+```
+
+## Test replication 
+
+The installed configuration will replicate any table added to MySQL
+database sysbench to ClickHouse table test.
+
+Login to MySQL, substituting your host name. 
+```
+mysql --user=root --password=root --host=<your host>
+```
+
+Enter SQL commands to Create a table in database sysbench and add data. 
+```
+mysql> use sbtest
+mysql> CREATE TABLE foo (id int PRIMARY KEY, value VARCHAR(100));
+Query OK, 0 rows affected (0.12 sec)
+mysql> INSERT INTO foo VALUES (1, '25');
+Query OK, 1 row affected (0.02 sec)
+```
+
+Login to ClickHouse, substituting your host name. 
+```
+clickhouse-client --user=root --password=root
+```
+
+Confirm that ClickHouse has the table in database test and that it 
+contains the data you entered. 
+```
+d69c9d8993cb :) use test
+Ok.
+d69c9d8993cb :) select * from foo
+SELECT *
+FROM foo
+
+┌─id─┬─value─┬─_sign─┬────────────_version─┐
+│  1 │ 25    │     1 │ 1731383644255158333 │
+└────┴───────┴───────┴─────────────────────┘
+```
+
+## Stop the stack
+
+Stop all components. 
+```
+docker compose -f docker-compose-mysql.yml down
+```
diff --git a/doc/setup.md b/doc/setup.md
@@ -1,6 +1,6 @@
 This doc describes how to set up CDC pipeline
 
-# Setup Kafka Sink Connector Pipeline
+# Set up Kafka Sink Connector Pipeline
 
 ![pipeline](img/pipeline.png)
 

diff --git a/doc/sink_connector_cli.md b/doc/sink_connector_cli.md
@@ -1,4 +1,6 @@
-### Sink Connector Lightweight (CLI)
+# Sink Connector Lightweight (CLI)
+
+## Design Overview 
 
 ![arch](img/sink_connector_cli.drawio.png)
 
@@ -18,5 +20,3 @@ This will give users option to set the binlog status/position, gtid
  Server will update the table with this information will restart the debezium event loop.
 5. **show replica status** Return the information from the **replica_status** table.
 
-## API
-