Kafka Connect for Integrations

What Connect Does

Kafka Connect moves data in and out of Kafka without writing custom clients. Connectors run as source (produce) or sink (consume) in worker processes.

Typical Connectors

• JDBC source - Stream DB tables/changes to Kafka.
• JDBC/Postgres sink - Persist topics into a database.
• Elasticsearch sink - Index topics for search.
• Debezium CDC - Stream Postgres/MySQL changes.
• File / S3 - Move logs and objects.

Configuration Shape

{
  "name": "jdbc-source",
  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
  "connection.url": "jdbc:postgresql://localhost:5432/app",
  "table.whitelist": "orders",
  "topics": "orders",
  "mode": "incrementing"
}

Submit it via the Connect REST API at /connectors.

Single vs Distributed Mode

Standalone mode runs one worker for simple local jobs; distributed mode runs a cluster of workers with automatic task allocation and the REST API for management.

Key Points

• Connect is config, not code, for data movement.
• Source and sink connectors pair with any store.
• CDC tools like Debezium stream DB changes.
• Distributed workers scale and self-heal tasks.