Apache Kafka is a distributed event streaming platform used to handle large amounts of realtime data. Learn the basics of Kafka in this quickstart tutorial. 

#programming #datascience #100SecondsOfCode

💬 Chat with Me on Discord

https://discord.gg/fireship

🔗 Resources

- Kafka Homepage https://kafka.apache.org/
- Kafka Github https://github.com/apache/kafka
- RabbitMQ in 100 Seconds https://youtu.be/NQ3fZtyXji0


🔥 Get More Content - Upgrade to PRO

Upgrade at https://fireship.io/pro
Use code YT25 for 25% off PRO access 

🎨 My Editor Settings

- Atom One Dark 
- vscode-icons
- Fira Code Font

🔖 Topics Covered

- What is Apache Kafka?
- Who created Apache Kafka?
- What is Kafka used for?
- How do large apps handle streaming data?
- Apache Kafka basic examples
2024-02-04T05:14:50.885Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
a68f5e74-a810-4f10-9a92-b87e913c1347
9105a8a8f6b04c21ecc59f1f6309029f18d655de19d3ed8923d93f411d872865
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
Apache Kafka, a distributed event streaming platform that can scale massive pipelines of real time data.
It was created in 2011 at LinkedIn, written in Java and Scala, and named Kafka because it's a system optimized for writing.
Let's imagine using it to build a dashboard like Google Analytics.
When an event occurs, like a website visit, the producer API creates a new record.
These records are stored to disk in an ordered immutable log called a topic, which can persist forever, or disappear when no longer needed.
Topics are distributed and replicated in a cluster, which contains multiple servers called brokers.
Jers.
This makes Kafka fault tolerant and able to scale to any workload.
On the other side, multiple consumers can subscribe to this data.
They can read the most recent message like a queue, or read the entire topic log and listen to updates in real time.
In addition, it provides a very powerful streams API that can transform and aggregate these topics before they ever reach the consumer.
This all may sound similar to message brokers like RabbitMQ, but Kafka can handle more throughput and is ideal jail for streaming data applications.
For example, it's used today by companies like Lyft to collect and process geolocation data, Spotify and for log processing and Cloudflare for real time analytics.
To get started, download it and use a tool like ZooKeeper or KRaft to manage your cluster.
Now in one terminal start zookeeper, then in the other start the Kafka server.
With the environment up and running, we can now create our first topic.
Remember, a topic is just a log of events kept in order.
An event will have a key value and time stamp and may also contain optional metadata and headers.
Now let's use this command to publish an event to the topic, where every line represents a different event.
And now these events are written to a topic, which is stored durably and partitioned in the cluster.
Kafka guarantees that any consumer a given topic will always read the events in the exact same order.
Now with this command, we're able to consume the topic.
By default, it will give us the latest event, Ghent, although we can use the from beginning flag to read the entire log.
It's also possible to provide an offset to read a subset of records.
At this point, we've achieved basic event streaming, but the Kafka Streams API can take things to another level.
It's most well supported with Java and can do things like stateless transformation, like filtering out a a subset of events, or state full transformation, like an aggregation that combines multiple events into a single value over a certain window of time.
And at that point, you're able to manage real time streams of data at virtually any scale.
This has been Apache Kafka in 100 seconds.
Hit the like button and subscribe for more short videos like JS.
Thanks for watching and I will see you in the next one.

Speaker 0: Apache Kafka, a distributed event streaming platform that can scale massive pipelines of real time data. It was created in 2011 at LinkedIn, written in Java and Scala, and named Kafka because it's a system optimized for writing. Let's imagine using it to build a dashboard like Google Analytics. When an event occurs, like a website visit, the producer API creates a new record. These records are stored to disk in an ordered immutable log called a topic, which can persist forever, or disappear when no longer needed.

Topics are distributed and replicated in a cluster, which contains multiple servers called brokers. Jers. This makes Kafka fault tolerant and able to scale to any workload. On the other side, multiple consumers can subscribe to this data. They can read the most recent message like a queue, or read the entire topic log and listen to updates in real time.

In addition, it provides a very powerful streams API that can transform and aggregate these topics before they ever reach the consumer. This all may sound similar to message brokers like RabbitMQ, but Kafka can handle more throughput and is ideal jail for streaming data applications. For example, it's used today by companies like Lyft to collect and process geolocation data, Spotify and for log processing and Cloudflare for real time analytics. To get started, download it and use a tool like ZooKeeper or KRaft to manage your cluster. Now in one terminal start zookeeper, then in the other start the Kafka server.

With the environment up and running, we can now create our first topic. Remember, a topic is just a log of events kept in order. An event will have a key value and time stamp and may also contain optional metadata and headers. Now let's use this command to publish an event to the topic, where every line represents a different event. And now these events are written to a topic, which is stored durably and partitioned in the cluster.

Kafka guarantees that any consumer a given topic will always read the events in the exact same order. Now with this command, we're able to consume the topic. By default, it will give us the latest event, Ghent, although we can use the from beginning flag to read the entire log. It's also possible to provide an offset to read a subset of records. At this point, we've achieved basic event streaming, but the Kafka Streams API can take things to another level.

It's most well supported with Java and can do things like stateless transformation, like filtering out a a subset of events, or state full transformation, like an aggregation that combines multiple events into a single value over a certain window of time. And at that point, you're able to manage real time streams of data at virtually any scale. This has been Apache Kafka in 100 seconds. Hit the like button and subscribe for more short videos like JS. Thanks for watching and I will see you in the next one.
Apache Kafka, a distributed event streaming platform that can scale massive pipelines of real time data. It was created in 2011 at LinkedIn, written in Java and Scala, and named Kafka because it's a system optimized for writing.
Let's imagine using it to build a dashboard like Google Analytics. When an event occurs, like a website visit, the producer API creates a new record. These records are stored to disk in an ordered immutable log called a topic, which can persist forever, or disappear when no longer needed. Topics are distributed and replicated in a cluster, which contains multiple servers called brokers. Jers. This makes Kafka fault tolerant and able to scale to any workload.
data management
data
databases
publishing
On the other side, multiple consumers can subscribe to this data. They can read the most recent message like a queue, or read the entire topic log and listen to updates in real time.
In addition, it provides a very powerful streams API that can transform and aggregate these topics before they ever reach the consumer. This all may sound similar to message brokers like RabbitMQ, but Kafka can handle more throughput and is ideal jail for streaming data applications. For example, it's used today by companies like Lyft to collect and process geolocation data, Spotify and for log processing and Cloudflare for real time analytics. To get started, download it and use a tool like ZooKeeper or KRaft to manage your cluster. Now in one terminal start zookeeper, then in the other start the Kafka server.
streaming
databases
data management
google
With the environment up and running, we can now create our first topic. Remember, a topic is just a log of events kept in order. An event will have a key value and time stamp and may also contain optional metadata and headers. Now let's use this command to publish an event to the topic, where every line represents a different event. And now these events are written to a topic, which is stored durably and partitioned in the cluster. Kafka guarantees that any consumer a given topic will always read the events in the exact same order.
Now with this command, we're able to consume the topic. By default, it will give us the latest event, Ghent, although we can use the from beginning flag to read the entire log. It's also possible to provide an offset to read a subset of records. At this point, we've achieved basic event streaming, but the Kafka Streams API can take things to another level. It's most well supported with Java and can do things like stateless transformation, like filtering out a a subset of events, or state full transformation, like an aggregation that combines multiple events into a single value over a certain window of time.
streaming
And at that point, you're able to manage real time streams of data at virtually any scale. This has been Apache Kafka in 100 seconds. Hit the like button and subscribe for more short videos like JS. Thanks for watching and I will see you in the next one.
streaming
youtube
Apache Kafka, a distributed event streaming platform that can scale massive pipelines of real time data. It was created in 2011 at LinkedIn, written in Java and Scala, and named Kafka because it's a system optimized for writing. Let's imagine using it to build a dashboard like Google Analytics. When an event occurs, like a website visit, the producer API creates a new record. These records are stored to disk in an ordered immutable log called a topic, which can persist forever, or disappear when no longer needed. Topics are distributed and replicated in a cluster, which contains multiple servers called brokers. Jers. This makes Kafka fault tolerant and able to scale to any workload. On the other side, multiple consumers can subscribe to this data. They can read the most recent message like a queue, or read the entire topic log and listen to updates in real time. In addition, it provides a very powerful streams API that can transform and aggregate these topics before they ever reach the consumer. This all may sound similar to message brokers like RabbitMQ, but Kafka can handle more throughput and is ideal jail for streaming data applications. For example, it's used today by companies like Lyft to collect and process geolocation data, Spotify and for log processing and Cloudflare for real time analytics. To get started, download it and use a tool like ZooKeeper or KRaft to manage your cluster. Now in one terminal start zookeeper, then in the other start the Kafka server. With the environment up and running, we can now create our first topic. Remember, a topic is just a log of events kept in order. An event will have a key value and time stamp and may also contain optional metadata and headers. Now let's use this command to publish an event to the topic, where every line represents a different event. And now these events are written to a topic, which is stored durably and partitioned in the cluster. Kafka guarantees that any consumer a given topic will always read the events in the exact same order. Now with this command, we're able to consume the topic. By default, it will give us the latest event, Ghent, although we can use the from beginning flag to read the entire log. It's also possible to provide an offset to read a subset of records. At this point, we've achieved basic event streaming, but the Kafka Streams API can take things to another level. It's most well supported with Java and can do things like stateless transformation, like filtering out a a subset of events, or state full transformation, like an aggregation that combines multiple events into a single value over a certain window of time. And at that point, you're able to manage real time streams of data at virtually any scale. This has been Apache Kafka in 100 seconds. Hit the like button and subscribe for more short videos like JS. Thanks for watching and I will see you in the next one.
Apache
apache
Kafka,
kafka
a
a
distributed
distributed
event
event
streaming
streaming
platform
platform
that
that
can
can
scale
scale
massive
massive
pipelines
pipelines
of
of
real
real
time
time
data.
data
It
it
was
was
created
created
in
in
2011
2011
at
at
LinkedIn,
linkedin
written
written
in
in
Java
java
and
and
Scala,
scala
and
and
named
named
Kafka
kafka
because
because
it's
it's
a
a
system
system
optimized
optimized
for
for
writing.
writing
Let's
let's
imagine
imagine
using
using
it
it
to
to
build
build
a
a
dashboard
dashboard
like
like
Google
google
Analytics.
analytics
When
when
an
an
event
event
occurs,
occurs
like
like
a
a
website
website
visit,
visit
the
the
producer
producer
API
api
creates
creates
a
a
new
new
record.
record
These
these
records
records
are
are
stored
stored
to
to
disk
disk
in
in
an
an
ordered
ordered
immutable
immutable
log
log
called
called
a
a
topic,
topic
which
which
can
can
persist
persist
forever,
forever
or
or
disappear
disappear
when
when
no
no
longer
longer
needed.
needed
Topics
topics
are
are
distributed
distributed
and
and
replicated
replicated
in
in
a
a
cluster,
cluster
which
which
contains
contains
multiple
multiple
servers
servers
called
called
brokers.
brokers
Jers.
jers
This
this
makes
makes
Kafka
kafka
fault
fault
tolerant
tolerant
and
and
able
able
to
to
scale
scale
to
to
any
any
workload.
workload
On
on
the
the
other
other
side,
side
multiple
multiple
consumers
consumers
can
can
subscribe
subscribe
to
to
this
this
data.
data
They
they
can
can
read
read
the
the
most
most
recent
recent
message
message
like
like
a
a
queue,
queue
or
or
read
read
the
the
entire
entire
topic
topic
log
log
and
and
listen
listen
to
to
updates
updates
in
in
real
real
time.
time
In
in
addition,
addition
it
it
provides
provides
a
a
very
very
powerful
powerful
streams
streams
API
api
that
that
can
can
transform
transform
and
and
aggregate
aggregate
these
these
topics
topics
before
before
they
they
ever
ever
reach
reach
the
the
consumer.
consumer
This
this
all
all
may
may
sound
sound
similar
similar
to
to
message
message
brokers
brokers
like
like
RabbitMQ,
rabbitmq
but
but
Kafka
kafka
can
can
handle
handle
more
more
throughput
throughput
and
and
is
is
ideal
ideal
jail
jail
for
for
streaming
streaming
data
data
applications.
applications
For
for
example,
example
it's
it's
used
used
today
today
by
by
companies
companies
like
like
Lyft
lyft
to
to
collect
collect
and
and
process
process
geolocation
geolocation
data,
data
Spotify
spotify
and
and
for
for
log
log
processing
processing
and
and
Cloudflare
cloudflare
for
for
real
real
time
time
analytics.
analytics
To
to
get
get
started,
started
download
download
it
it
and
and
use
use
a
a
tool
tool
like
like
ZooKeeper
zookeeper
or
or
KRaft
kraft
to
to
manage
manage
your
your
cluster.
cluster
Now
now
in
in
one
one
terminal
terminal
start
start
zookeeper,
zookeeper
then
then
in
in
the
the
other
other
start
start
the
the
Kafka
kafka
server.
server
With
with
the
the
environment
environment
up
up
and
and
running,
running
we
we
can
can
now
now
create
create
our
our
first
first
topic.
topic
Remember,
remember
a
a
topic
topic
is
is
just
just
a
a
log
log
of
of
events
events
kept
kept
in
in
order.
order
An
an
event
event
will
will
have
have
a
a
key
key
value
value
and
and
time
time
stamp
stamp
and
and
may
may
also
also
contain
contain
optional
optional
metadata
metadata
and
and
headers.
headers
Now
now
let's
let's
use
use
this
this
command
command
to
to
publish
publish
an
an
event
event
to
to
the
the
topic,
topic
where
where
every
every
line
line
represents
represents
a
a
different
different
event.
event
And
and
now
now
these
these
events
events
are
are
written
written
to
to
a
a
topic,
topic
which
which
is
is
stored
stored
durably
durably
and
and
partitioned
partitioned
in
in
the
the
cluster.
cluster
Kafka
kafka
guarantees
guarantees
that
that
any
any
consumer
consumer
a
a
given
given
topic
topic
will
will
always
always
read
read
the
the
events
events
in
in
the
the
exact
exact
same
same
order.
order
Now
now
with
with
this
this
command,
command
we're
we're
able
able
to
to
consume
consume
the
the
topic.
topic
By
by
default,
default
it
it
will
will
give
give
us
us
the
the
latest
latest
event,
event
Ghent,
ghent
although
although
we
we
can
can
use
use
the
the
from
from
beginning
beginning
flag
flag
to
to
read
read
the
the
entire
entire
log.
log
It's
it's
also
also
possible
possible
to
to
provide
provide
an
an
offset
offset
to
to
read
read
a
a
subset
subset
of
of
records.
records
At
at
this
this
point,
point
we've
we've
achieved
achieved
basic
basic
event
event
streaming,
streaming
but
but
the
the
Kafka
kafka
Streams
streams
API
api
can
can
take
take
things
things
to
to
another
another
level.
level
It's
it's
most
most
well
well
supported
supported
with
with
Java
java
and
and
can
can
do
do
things
things
like
like
stateless
stateless
transformation,
transformation
like
like
filtering
filtering
out
out
a
a
a
a
subset
subset
of
of
events,
events
or
or
state
state
full
full
transformation,
transformation
like
like
an
an
aggregation
aggregation
that
that
combines
combines
multiple
multiple
events
events
into
into
a
a
single
single
value
value
over
over
a
a
certain
certain
window
window
of
of
time.
time
And
and
at
at
that
that
point,
point
you're
you're
able
able
to
to
manage
manage
real
real
time
time
streams
streams
of
of
data
data
at
at
virtually
virtually
any
any
scale.
scale
This
this
has
has
been
been
Apache
apache
Kafka
kafka
in
in
100
100
seconds.
seconds
Hit
hit
the
the
like
like
button
button
and
and
subscribe
subscribe
for
for
more
more
short
short
videos
videos
like
like
JS.
js
Thanks
thanks
for
for
watching
watching
and
and
I
i
will
will
see
see
you
you
in
in
the
the
next
next
one.
one
success
In this video, Speaker 0 discusses Apache Kafka, a distributed event streaming platform that allows massive pipelines of real time data. The system is optimized for writing and allows multiple consumers to subscribe to the event and read the most recent message. The system is used to handle more throughput and is ideal for streaming data applications like geolocation and real time analytics. The command is used to publish an event to a topic and create a new record, with the ability to manage real time streams of data at any scale.
Kafka in 100 Seconds
Apache Kafka, a distributed event streaming platform that can scale massive pipelines of real time data. It was created in 2011 at LinkedIn, written in Java and Scala, and named Kafka because it's a system optimized for writing. Let's imagine using it to build a dashboard like Google Analytics. When an event occurs, like a website visit, the producer API creates a new record. These records are stored to disk in an ordered immutable log called a topic, which can persist forever, or disappear when no longer needed. Topics are distributed and replicated in a cluster, which contains multiple servers called brokers. Jers. This makes Kafka fault tolerant and able to scale to any workload. On the other side, multiple consumers can subscribe to this data. They can read the most recent message like a queue, or read the entire topic log and listen to updates in real time. In addition, it provides a very powerful streams API that can transform and aggregate these topics before they ever reach the consumer. This all may sound similar to message brokers like RabbitMQ, but Kafka can handle more throughput and is ideal jail for streaming data applications. For example, it's used today by companies like Lyft to collect and process geolocation data, Spotify and for log processing and Cloudflare for real time analytics. To get started, download it and use a tool like ZooKeeper or KRaft to manage your cluster. Now in one terminal start zookeeper, then in the other start the Kafka server. With the environment up and running, we can now create our first topic. Remember, a topic is just a log of events kept in order. An event will have a key value and time stamp and may also contain optional metadata and headers. Now let's use this command to publish an event to the topic, where every line represents a different event. And now these events are written to a topic, which is stored durably and partitioned in the cluster. Kafka guarantees that any consumer a given topic will always read the events in the exact same order. Now with this command, we're able to consume the topic. By default, it will give us the latest event, Ghent, although we can use the from beginning flag to read the entire log. It's also possible to provide an offset to read a subset of records. At this point, we've achieved basic event streaming, but the Kafka Streams API can take things to another level. It's most well supported with Java and can do things like stateless transformation, like filtering out a a subset of events, or state full transformation, like an aggregation that combines multiple events into a single value over a certain window of time. And at that point, you're able to manage real time streams of data at virtually any scale. This has been Apache Kafka in 100 seconds. Hit the like button and subscribe for more short videos like JS. Thanks for watching and I will see you in the next one.
In this video, Speaker 0 discusses Apache Kafka, a distributed event streaming platform that allows massive pipelines of real time data. The system is optimized for writing and allows multiple consumers to subscribe to the event and read the most recent message. The system is used to handle more throughput and is ideal for streaming data applications like geolocation and real time analytics. The command is used to publish an event to a topic and create a new record, with the ability to manage real time streams of data at any scale.