Connect to Kafka running in Docker

In today's software development field, containerization technology has become an increasingly popular deployment method. As one of the most popular containerization solutions, Docker provides developers with convenient environment isolation and deployment methods. For developers who use Kafka as a message queue system, combining Kafka with Docker can make development and deployment more flexible and efficient. In this article, PHP editor Xigua will introduce how to run Kafka in Docker, so that you can easily enjoy the convenience brought by containerization.

Question content

I have set up a single node kafka docker container on my local machine as described in the confluence documentation (steps 2-3).

Additionally, I exposed zookeeper's port 2181 and kafka's port 9092 so that I can connect to them from a client running on my local machine:

$ docker run -d \
    -p 2181:2181 \
    --net=confluent \
    --name=zookeeper \
    -e zookeeper_client_port=2181 \
    confluentinc/cp-zookeeper:4.1.0

$ docker run -d \
    --net=confluent \
    --name=kafka \
    -p 9092:9092 \
    -e kafka_zookeeper_connect=zookeeper:2181 \
    -e kafka_advertised_listeners=plaintext://kafka:9092 \
    -e kafka_offsets_topic_replication_factor=1 \
    confluentinc/cp-kafka:4.1.0

Problem: When I try to connect to kafka from the host, the connection fails because the address: kafka:9092 cannot be resolved.

This is my java code:

properties props = new properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("client.id", "kafkaexampleproducer");
props.put("key.serializer", longserializer.class.getname());
props.put("value.serializer", stringserializer.class.getname());
kafkaproducer<long, string> producer = new kafkaproducer<>(props);
producerrecord<long, string> record = new producerrecord<>("foo", 1l, "test 1");
producer.send(record).get();
producer.flush();

exception:

java.io.IOException: Can't resolve address: kafka:9092
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.common.network.Selector.connect(Selector.java:214) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:265) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:266) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:176) [kafka-clients-2.0.0.jar:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.nio.channels.UnresolvedAddressException: null
    at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_144]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_144]
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:na]
    ... 7 common frames omitted

Question: How to connect to kafka running in docker? My code is running from the host, not docker.

NOTE: I know in theory I could try the dns settings and /etc/hosts but this is a workaround - it shouldn't be like this.

There is a similar question here, but it is based on the ches/kafka image. I use images based on confluenceinc which is different.

Workaround

tl;dr - Simple port forwarding from container to host will not work ... hosts file (e.g. * /etc/hosts on NIX systems) should not be modified to resolve Kafka network issues as this solution is not portable.

1) Which exact IP/hostname port do you want to connect to? Make sure the value is set on the proxy to advertished.listeners (not advertished.host.name and advertished.port as these are deprecated). If you see an error like Connection to node -1 (localhost/127.0.0.1:9092) it means your application container is trying to connect to itself. Are your application containers also running Kafka broker processes? maybe not.

2) Make sure the servers listed as part of bootstrap.servers are actually resolvable. E.g. ping IP/hostname, use netcat to check the port... If your client is in a container, you need to do this from the container, not (only) from the host Do this. If the container does not crash immediately to access its shell, use docker exec.

3) If running the process from the host instead of another container, to verify that the port is correctly mapped on the host, make sure docker ps shows that the kafka container is mapped from 0.0.0.0: 50c5a3c9bd6141bda139d1e2f6e9d778 -> cd3356af2f6e059e7bd54cd1ababa980 /tcp. If trying to run the client from outside the Docker network, the ports must match. No port forwarding is required between the two containers; use link/docker-network

The answer below uses the confluenceinc docker image to solve the problem raised, not wurstmeister/kafka. If you set the KAFKA_ADVERTISED_HOST_NAME variable, remove it (it is a deprecated property)

The following sections attempt to summarize all the details needed to use other images. For other commonly used Kafka images, are all run in containers Apache Kafka.
You only rely on how it is configured. And which variables are causing this.

wurstmeister/kafka

As of October 2023, this content no longer exists in DockerHub. In any case, it will not be maintained after 2022.

See the readme section on a> listener configuration, and also read their Connectivity wiki.

bitnami/kafka

If you want a small container, try these. The images are much smaller than Confluence and better maintained than wurstmeister. Refer to the readme file for their listener configuration.

debezium/kafka

Relevant documentation is mentioned herehere.

NOTE: Advertised host and port settings have been deprecated. Advertisinglistener covers both. Similar to the Confluence container, Debezium can update its properties using proxy settings prefixed with KAFKA_.

other

• ubuntu/kafka Requires you to add --overrideadvertising.listeners=kafka:9092 via Docker image parameters...I find it less portable than environment variables and therefore not recommended
• spotify/kafka Deprecated and obsolete.
• fast-data-dev or lensesio/box are great for an all-in-one solution with schema registry, Kafka Connect, etc., but if you only If you want Kafka, it seems bloated. Also, this is a Docker anti-pattern for running multiple services in a container
• Your own Dockerfile - Why? Are these other things incomplete? Start with a pull request instead of starting from scratch.

For additional reading, Full-featured docker-compose and network diagrams, please see This blog is written by: @rmoff

answer

Confluence Quick Start (Docker) Documentation It is assumed that all production and consumption requests will occur within the Docker network.

You can work around the issue of connecting to kafka:9092 by running the Kafka client code in its own container (using a Docker bridge), but otherwise you will need to add more environment variables to the container Exposed to the outside world while still having it work within a Docker network.

First add the protocol mapping of PLAINTEXT_HOST:PLAINTEXT, which maps the listener protocol to the Kafka protocol

Key: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
Value: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

Then set up two advertised listeners on different ports. (Here kafka refers to the docker container name; it may also be named broker, so double-check your service hostname).

Key:KAFKA_ADVERTISED_LISTENERS
Value: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

Please note that the protocol here matches the value on the left of the protocol mapping setting above

When running the container, add -p 29092:29092 for host port mapping and notify the PLAINTEXT_HOST listener.

So...(Use the above settings)

If still doesn't work, you can set KAFKA_LISTENERS to include 9625b882931466931ca10eb1a60367b3://0.0.0.0:a3b0c87895079be75e30be94102cc20b, where Both options match advertising settings and Docker forwarded ports

The client is on the same machine, not in the container

Advertising the localhost and associated port will allow you to connect outside the container, as you would expect.

In other words, when running any Kafka client outside of a Docker network (including CLI tools you may have installed locally), use localhost:29092 as the bootstrap server, using localhost:2181 as Zookeeper (requires Docker) port forwarding)

Client on another machine

If trying to connect from an external server, you need to advertise the host's external hostname/IP (e.g. 192.168.x.y) and / in place of localhost .
Simply advertising localhost via port forwarding will not work because the Kafka protocol will still continue to advertise your configured listeners.

This setup requires Docker port forwarding and Router port forwarding (and firewall/security group changes) if not in the same local network, e.g. your containers are running in the cloud and you want Interact with it from your local computer.

Client (or another proxy) in a container on the same host

This is the least error-prone configuration; you can use the DNS service name directly.

When running the application in a Docker network , use kafka:9092 (see the PLAINTEXT listener configuration advertised above) as the bootstrap server, Use zookeeper:2181 as Zookeeper, just like any other Docker service communication (no port forwarding required)

If you use a separate docker run command or Compose file, you will need to manually define the share using the compose networks section or docker network --create network

See the example Compose file for the full Confluence stack or the simpler for a single broker.

If multiple proxies are used, then they need to use unique hostnames for advertising listeners. View example一个>

Related questions

Connecting to Kafka on the host from Docker (ksqlDB)

appendix

For anyone interested in Kubernetes deployment:

• Access Kafka
• Operator (recommended): https://www.php.cn/link/61b07251e835d37322b7460d2b88c05b
• Helm Artifact Hub: https://artifacthub.io/packages/search ?ts_query_web=kafka&sort=stars&page=1

The above is the detailed content of Connect to Kafka running in Docker. For more information, please follow other related articles on the PHP Chinese website!