About the course
Apache Kafka is a powerful distributed event streaming platform used for building real-time data pipelines and streaming applications. 1 Known for its high throughput, low latency, and durability, Kafka has become a foundational technology for modern data architectures and microservices. This comprehensive training course provides participants with the knowledge and practical skills needed to design, build, and operate reliable and scalable systems using Apache Kafka.
You will dive deep into the core concepts of Kafka, including messages, topics, partitions, and brokers, and understand its architecture and key configuration aspects. The course provides in-depth coverage of the Producer and Consumer APIs, explaining different delivery semantics and how to build robust data pipelines. You will also explore essential ecosystem components like Schema Registry for data governance, Kafka Streams for real-time data processing, and ksqlDB for interacting with Kafka using SQL, understanding how to leverage these tools to build event-driven applications.
The workshop covers crucial aspects of Kafka security (Authentication, Authorisation, Encryption), important design and production considerations (monitoring, capacity planning, deployment strategies), and common design patterns. Additionally, the course includes a section comparing Open Source Apache Kafka with the Confluent Platform to highlight the differences and features offered by enterprise distributions. Through hands-on labs, you will gain practical experience working with Kafka components and building streaming applications.
Online and in-house face-to-face options are available - as part of a wider customised training programme, or as a standalone workshop, on-site at your offices or at one of many flexible meeting spaces in the UK and around the World.
-
- Explain the core concepts and architecture of Apache Kafka.
- Design and build robust Kafka Producers with appropriate delivery semantics.
- Develop efficient Kafka Consumers and manage offsets.
- Configure Kafka brokers and manage topics effectively.
- Understand replication, leader election, log retention, and log compaction.
- Utilise Schema Registry for data governance and schema evolution.
- Develop streaming applications using Kafka Streams (stateless and stateful transformations, time concepts, windowing, joins, KTable).
- Interact with Kafka data and build stream processing applications using ksqlDB.
- Implement Kafka security features (Authentication, Authorisation, Encryption).
- Understand the differences between Open Source Apache Kafka and the Confluent Platform.
- Identify key design and production considerations for deploying Kafka.
- Recognise common Kafka design patterns.
- Troubleshoot common issues in Kafka deployments.
-
This course is designed for technical professionals who need to work with Apache Kafka for building, designing, and operating scalable data streaming systems and microservices. It is particularly relevant for:
Software Developers (especially backend developers and those working with distributed systems)
Data Engineers
Data Architects
DevOps Engineers
Site Reliability Engineers (SREs)
Backend Engineers
System Architects
Anyone involved in designing or implementing event-driven architectures.
-
This course is designed for coming people from a technical background who are new to Kafka. Participants should ideally have:
Previous programming experience in any programming language.
Basic familiarity with command line or terminal interface.
A basic understanding of distributed systems concepts is helpful but not strictly required.
We can customise the training to match your team's experience and needs - with more time and coverage of fundamentals for people who are early in their journey (for instance as part of a graduate in-take training scheme).
-
This Kafka course is available for private / custom delivery for your team - as an in-house face-to-face workshop at your location of choice, or as online instructor-led training via MS Teams (or your own preferred platform).
Get in touch to find out how we can deliver tailored training which focuses on your project requirements and learning goals.
-
Kafka Overview
Use Cases
Apache Kafka Ecosystem
Kafka Message
What does a Kafka message look like?
Kafka Key
Body
Headers
Producer API
Sending messages to Kafka
High Level overview
Using the API
Buffer Pool
Discussion on thread safety
Consumer API
High Level
Using the API
Polling for messages
Discussion on why it is not thread safe
Consumer groups
Discussions of Kafka Offset
How do we store Offsets?
Multiple instances of the same service working together
Starting again from where we finished
Auto reset strategy
Configuration
Broker
Consumer
Producer
Zookeeper
What is Zookeeper
What does it contain?
Schema registry and Avro
Idempotent producer and Transactions
What does it mean to be idempotent?
How does it work?
Transactions in Kafka
Zombie fencing
Kafka Streams (Java only)
Kafka SQL and KSQL
Using Kafka with SQL syntax
How it uses Kafka streams underneath the hood
KSQL Server and CLI
Creating Tables vs Streams
Persisted vs non-persisted queries
Authentication and authorisation
How to connect to Kafka
ACLs for authorisation
Using ACLs with Kafka streams
Administering ACLs through CLI
Design Points
Persistence on disk
Producer design considerations
Consumer Design consideration
Message delivery
At least once, at most once and exactly once semantics
Replication
Log Compaction
Hardware and Cloud (AWS, Azure...) deployment considerations
Installation and deployment recommendations
Design patterns in a Kafka System
-
Official Apache Kafka Documentation - https://kafka.apache.org/documentation/
Apache Kafka Website - https://kafka.apache.org/
Confluent Website (Information on Confluent Platform and Cloud) - https://www.confluent.io/
Overview of Apache Kafka vs Confluent Platform (Search online for recent comparison guides, e.g., articles on Confluent's blog or independent reviews)
Kafka Streams Documentation - https://kafka.apache.org/documentation/streams/
ksqlDB Documentation - https://docs.ksqldb.io/
Schema Registry Documentation - https://docs.confluent.io/platform/current/schema-registry/index.html
Apache Avro Documentation - https://avro.apache.org/docs/current/
Trusted by
