Is Kafka suitable for use in Docker? Does a stand alone cluster make sense

One of the core features of Docker is the ability to package any application, including Hadoop, into a Docker image. This tutorial describes the detailed steps to quickly set up a multi-node Hadoop cluster on a single machine using Docker. After discovering the problems of the current Hadoop on Docker project, the author developed a Hadoop image that is close to minimum, and supports the rapid establishment of a Hadoop cluster with any number of nodes.

github: kiwanlau hadoop-cluster-docker Setting up a Hadoop cluster directly from a machine can be a painful process, especially for beginners. They haven't even started to run wordcounts, and they may be tormented by this question.

And not everyone has several machines, right? You can try to build with multiple virtual machines, as long as you have a machine with good performance. My goal is to run Hadoop clusters in Docker containers, making it quick and easy for Hadoop developers to set up multi-node Hadoop clusters natively.

In fact, there have been many implementations of this idea, but none of them are ideal, they either have too large images, or they are too slow to use, or they use third-party tools that make them too complicated to use. The following table shows some of the known Hadoop on Docker projects and their problems. Faster and easier to change the number of nodes in a Hadoop cluster In addition, when adding nodes to the Alvinhenrick Hadoop-Mutino project, you need to manually modify the Hadoop configuration file, rebuild the Hadoop-nn-dn image, and then modify the container startup script to add nodes.

And I implemented it automatically through a shell script, and I was able to rebuild the hadoop-master image in less than 1 minute, and then run it immediately! This project starts a Hadoop cluster with 3 nodes by default and supports a Hadoop cluster with any number of nodes. In addition, starting Hadoop, running wordcount, and rebuilding images are all automated using shell scripts.

This makes the use and development of the whole project very convenient and fast. Development & Test Environment Operating System: Ubuntu and Ubuntu Kernel Version:

docker version: and friends, the hard disk is not enough, the memory is not enough, especially the kernel version is too low, which will cause the operation to fail.

It's a lot, and you can't send a link to the Q&A, otherwise I'll give you a link. For example, Hadoop and other Liangxing open slag-type source big data projects, programming languages, the following is about the underlying technology of big data. Simply guessing the technology of Yonghong Technology, there are four aspects, in fact, it also represents some of the underlying technologies of general big data:

The z-suite has high-performance big data analysis capabilities.

Yes, you can use a Linux image instance to have multiple micro servers at the same time, and you can practice clustering.

Kafka is a high-throughput, distributed, publish-and-subscribe based consumption system designed by LinkedIn, written in Scala, which is widely used for its horizontal scaling, reliability, asynchronous communication, and high throughput. At present, more and more open source distributed processing systems support integration with Kafka.

The SSHD method is probably the easiest way to connect to containers, and most users are used to connecting to VMs via SSH. Also, you don't need to use root access to connect to containers. However, there is still a lot of controversy about whether a container should manage more than one process.

This approach ultimately results in an additional SSHD process ,.. per container

OpenStack manages the underlying resources, and it can manage virtual machines for KVM, Xen, HyperV, and now Docker containers. Docker can only achieve the effect of containers on a single machine, if you manage all containers in the cluster, you still need OpenStack or OpenStack-like management platform.