{"id":3137,"date":"2021-03-19T14:08:13","date_gmt":"2021-03-19T19:08:13","guid":{"rendered":"https:\/\/microsoftgeek.com\/?p=3137"},"modified":"2021-04-01T12:42:00","modified_gmt":"2021-04-01T17:42:00","slug":"how-to-set-up-and-run-kafka-on-kubernetes","status":"publish","type":"post","link":"https:\/\/microsoftgeek.com\/?p=3137","title":{"rendered":"How To Set Up And Run Kafka On Kubernetes"},"content":{"rendered":"\n

What is Apache Kafka?<\/strong><\/p>\n\n\n\n

Kafka is a messaging system that collects and processes extensive amounts of data in real-time, making it a vital integrating component for applications running in a Kubernetes cluster. The efficiency of applications deployed in a cluster can be further augmented with an event-streaming platform such as Apache Kafka<\/strong>.<\/p>\n\n\n\n

This in-depth tutorial shows you how to configure a Kafka server on a Kubernetes cluster.<\/strong><\/p>\n\n\n\n

How Does Apache Kafka Work?<\/h2>\n\n\n\n

Apache Kafka is based on a publish-subscribe model:<\/p>\n\n\n\n

  1. Producers<\/strong> produce messages and publish them to topics<\/strong>.<\/li>
  2. Kafka categorizes the messages into topics <\/strong>and stores them so that they are immutable.<\/li>
  3. Consumers subscribe to a specific topic<\/strong> and absorb the messages provided by the producers.<\/em><\/strong><\/li><\/ol>\n\n\n\n

    Producers and Consumers in this context represent applications that produce event-driven messages and applications that consume those messages. The messages are stored on Kafka brokers, sorted by user-defined topics<\/strong>.<\/p>\n\n\n\n

    \"Basic<\/figure><\/div>\n\n\n\n

    Zookeeper<\/a> is an indispensable component of a Kafka configuration. It coordinates Kafka producers, brokers, consumers, and cluster memberships.<\/p>\n\n\n\n

    Deploy Zookeeper<\/h2>\n\n\n\n

    Kafka cannot function without Zookeeper. The Kafka service keeps restarting until a working Zookeeper deployment is detected.<\/p>\n\n\n\n

    Deploy Zookeeper beforehand, by creating a YAML file zookeeper.yml<\/em><\/strong>. This file starts a service and deployment that schedule Zookeeper pods on a Kubernetes cluster.<\/p>\n\n\n\n

    \n\n\n\n

    Use your preferred text editor to add the following fields to zookeeper.yml<\/em><\/strong>:<\/p>\n\n\n\n

    apiVersion: v1\nkind: Service\nmetadata:\n  name: zk-s\n  labels:\n    app: zk-1\nspec:\n  ports:\n  - name: client\n    port: 2181\n    protocol: TCP\n  - name: follower\n    port: 2888\n    protocol: TCP\n  - name: leader\n    port: 3888\n    protocol: TCP\n  selector:\n    app: zk-1\n    ---\nkind: Deployment\napiVersion: extensions\/v1beta1\nmetadata:\n  name: zk-deployment-1\nspec:\n  template:\n    metadata:\n      labels:\n        app: zk-1\n    spec:\n      containers:\n      - name: zk1\n        image: bitnami\/zookeeper\n        ports:\n        - containerPort: 2181\n        env:\n        - name: ZOOKEEPER_ID\n          value: \"1\"\n        - name: ZOOKEEPER_SERVER_1\n          value: zk1\n<\/pre>\n\n\n\n
    Run the following command on your Kubernetes cluster to create the definition file:<\/p>\n\n\n\n
    kubectl create -f zookeeper.yml<\/pre>\n\n\n\n
    Create Kafka Service<\/h2>\n\n\n\n
    We now need to create a Kafka Service definition file. This file manages Kafka Broker deployments by load-balancing new Kafka pods. A basic kafka-service.yml<\/em><\/strong> file contains the following elements:<\/p>\n\n\n\n
     apiVersion: v1 \n kind: Service \n metadata:  \n   labels:  \n     app: kafkaApp \n   name: kafka\n spec:  \n   ports:  \n     -  \n       port: 9092 \n       targetPort: 9092\n      protocol: TCP\n     -  \n       port: 2181 \n       targetPort: 2181 \n   selector:  \n     app: kafkaApp \n   type: LoadBalancer\n<\/pre>\n\n\n\n
    Once you have saved the file, create the service by entering the following command:<\/p>\n\n\n\n
    kubectl create -f kafka-service.yml<\/pre>\n\n\n\n
    \n\n\n\n
    Define Kafka Replication Controller<\/h2>\n\n\n\n
    Create an additional .yml<\/em><\/strong> file to serve as a replication controller for Kafka. A replication controller file, in our example kafka-repcon.yml, <\/em><\/strong>contains the following fields:<\/p>\n\n\n\n
    ---  \n apiVersion: v1 \n kind: ReplicationController \n metadata:  \n   labels:  \n     app: kafkaApp \n   name: kafka-repcon \n spec:  \n   replicas: 1\n   selector:  \n     app: kafkaApp \n   template:  \n     metadata:  \n       labels:  \n         app: kafkaApp \n     spec:  \n       containers:  \n-\n           command:  \n             - zookeeper-server-start.sh \n             - \/config\/zookeeper.properties \n           image: \"wurstmeister\/kafka\" \n           name: zk1 \n           ports:  \n             -  \n               containerPort: 2181\n<\/pre>\n\n\n\n
    Save the replication controller definition file and create it by using the following command:<\/p>\n\n\n\n
    kubectl create -f kafka-repcon.yml<\/pre>\n\n\n\n
    Start Kafka Server<\/h2>\n\n\n\n
    The configuration properties for a Kafka server are defined in the config\/server.properties<\/em><\/strong> file. As we have already configured the Zookeeper server, start the Kafka server with:<\/p>\n\n\n\n
    kafka-server-start.sh config\/server.properties<\/pre>\n\n\n\n
    How to Create Kafka Topic<\/h3>\n\n\n\n
    Kafka has a command-line utility called kafka-topics.sh<\/em><\/strong>. Use this utility to create topics on the server. Open a new terminal window and type:<\/p>\n\n\n\n
    kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Topic-Name<\/pre>\n\n\n\n
    We created a topic named Topic-Name<\/em><\/strong> with a single partition and one replica instance.<\/p>\n\n\n\n
    How to Start a Kafka Producer<\/h3>\n\n\n\n
    The config\/server.properties<\/em><\/strong> file contains the broker port id. The broker in the example is listening on port 9092. It is possible to specify the listening port directly using the command line:<\/p>\n\n\n\n
    kafka-console-producer.sh --topic kafka-on-kubernetes --broker-list localhost:9092 --topic Topic-Name <\/pre>\n\n\n\n
    Now use the terminal to add several lines of messages.<\/p>\n\n\n\n
    How to Start a Kafka Consumer<\/h3>\n\n\n\n
    As with the Producer properties, the default Consumer settings are specified in config\/consumer.properties<\/em><\/strong> file. Open a new terminal window and type the command for consuming messages:<\/p>\n\n\n\n
    kafka-console-consumer.sh --topic Topic-Name --from-beginning --zookeeper localhost:2181 <\/pre>\n\n\n\n
    The --from-beginning<\/code><\/strong> command lists messages chronologically. You are now able to enter messages from the producer\u2019s terminal and see them appearing in the consumer\u2019s terminal.<\/p>\n\n\n\n
    How to Scale a Kafka Cluster<\/h2>\n\n\n\n
    Use the command terminal and directly administer the Kafka Cluster using kubectl<\/strong><\/em>. Enter the following command and scale your Kafka cluster quickly by increasing the number of pods from one (1) to six (6):<\/p>\n\n\n\n
    kubectl scale rc kafka-rc --replicas=6<\/pre>\n\n\n\n
    Conclusion<\/strong><\/p>\n\n\n\n
    By following the instructions in this tutorial, you have successfully installed Kafka on Kubernetes. A single Kafka broker can process an impressive amount of reads and writes from a multitude of clients simultaneously.<\/p>\n\n\n\n
    If you are deploying applications within a Kubernetes cluster, use Kafka to improve the capacity of your apps to exchange information in real-time.<\/p>\n","protected":false},"excerpt":{"rendered":"
    What is Apache Kafka? Kafka is a messaging system that collects and processes extensive amounts of data in real-time, making it a vital integrating component for applications running in a Kubernetes cluster. The efficiency of applications deployed in a cluster can be further augmented with an event-streaming platform such as Apache Kafka. This in-depth tutorial shows you how to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[78,77,82,80,79,13,81],"tags":[],"class_list":["post-3137","post","type-post","status-publish","format-standard","hentry","category-containers","category-devops","category-docker","category-kafka","category-kubernetes","category-linux","category-zookeeper"],"_links":{"self":[{"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=\/wp\/v2\/posts\/3137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3137"}],"version-history":[{"count":3,"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=\/wp\/v2\/posts\/3137\/revisions"}],"predecessor-version":[{"id":3143,"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=\/wp\/v2\/posts\/3137\/revisions\/3143"}],"wp:attachment":[{"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3137"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3137"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/microsoftgeek.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}