kafka学习

一、搭建zk和kafka集群

ZooKeeper 是一个分布式和谐服务，主要用于办理分布式系统中的一致性、配置管理、向导者选举、分布式锁等题目。它像分布式系统的“电话簿”或“管理员”，帮助各个节点协同工作，确保系统的高可用性和一致性。
搭建zk集群

1、下载zk的安装包，比方：apache-zookeeper-3.8.4-bin.tar.gz
2、将安装包上传并解压到/usr/local目录
3、创建目录生存zk的数据和日志，mkdir /usr/local/apache-zookeeper-3.8.4-bin/{datas, logs}
4、修改配置，重定名 mv /usr/local/apache-zookeeper-3.8.4-bin/conf/zoo_sample.cfg /usr/local/apache-zookeeper-3.8.4-bin/conf/zoo.cfg
改配置 zoo.cfg

1. # 心跳时间间隔（单位：毫秒）
2. tickTime=2000
3. ​
4. # 集群初始化超时（tickTime倍数）
5. initLimit=10
6. ​
7. # 数据同步超时（tickTime倍数）
8. syncLimit=5
9. ​
10. # 内存快照目录
11. dataDir=/usr/local/apache-zookeeper-3.8.4-bin/datas
12. ​
13. # 事务日志目录
14. dataLogDir=/usr/local/apache-zookeeper-3.8.4-bin/logs
15. ​
16. # 客户端连接端口
17. clientPort=2181
18. ​
19. # 集群节点列表
20. server.100=192.168.1.100:2888:3888
21. server.101=192.168.1.101:2888:3888
22. server.102=192.168.1.102:2888:3888

复制代码

节点拷贝
scp -r /usr/locak/apache-zookeeper-3.8.4-bin root@ip:/usr/local/apache-zookeeper-3.8.4-bin
创建myid
每个zk节点dataDir指定目录中都需要创建一个名为myid的文件，其内容为上述配置文件中server.X的X值

1. tee /usr/local/apache-zookeeper-3.8.4-bin/datas/myid <<-'EOF'
2. 100
3. EOF

复制代码

开启端口
firewall-cmd --zone=public --add-port=2181/tcp --add-port=2888/tcp --add-port=3888 --permanent
启动/关闭/状态节点
/usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh start

1. tee /usr/local/apache-zookeeper-3.8.4-bin/zkStart.sh <<-'EOF'
2. #! /bin/bash
3. echo "zookeeper start 100, 101, 102"
4. ssh 192.168.1.100 "source /usr/local/apache-zookeeper-3.8.4-bin/zkStart.sh start"
5. ssh 192.168.1.101 "source /usr/local/apache-zookeeper-3.8.4-bin/zkStart.sh start"
6. ssh 192.168.1.102 "source /usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh start"
7. EOF
8. ​
9. tee /usr/local/apache-zookeeper-3.8.4-bin/zkStop.sh <<-'EOF'
10. #! /bin/bash
11. echo "zookeeper start 100, 101, 102"
12. ssh 192.168.1.100 "source /usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh stop"
13. ssh 192.168.1.101 "source /usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh stop"
14. ssh 192.168.1.102 "source /usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh stop"
15. EOF
16. ​
17. tee /usr/local/apache-zookeeper-3.8.4-bin/zkStatus.sh <<-'EOF'
18. #! /bin/bash
19. echo "zookeeper start 100, 101, 102"
20. ssh 192.168.1.100 "source /usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh status"
21. ssh 192.168.1.101 "source /usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh status"
22. ssh 192.168.1.102 "source /usr/local/apache-zookeeper-3.8.4-bin/zkServer.sh status"
23. EOF

复制代码

设置权限

1. chmod +x /usr/local/apache-zookeeper-3.8.4-bin/zkStart.sh
2. ​
3. chmod +x /usr/local/apache-zookeeper-3.8.4-bin/zkStop.sh
4. ​
5. chmod +x /usr/local/apache-zookeeper-3.8.4-bin/zkStatus.sh

复制代码

执行文件
/usr/local/apache-zookeeper-3.8.4-bin/zkStart.sh
访问节点
/usr/local/apache-zookeeper-3.8.4-bin/zkCli.sh -server 192.168.1.100:2181
搭建kafka集群

下载kafka安装包，比方kafka_2.12.3.7.0.tgz
解压到所有节点/usr/local路径下
创建logs文件，生存日志
mkdir /usr/local/kafka_2.12-3.7.0/logs
修改配置，/usr/local/kafka_2.12-3.7.0/config/server.properties

1. tee /usr/local/=kafka_2.12-3.7.0/config/server.properties <<-'EOF'
2. # 同一kafka集群下broker.id必须不同
3. broker.id=1
4. advertised.listeners=PLAINTEXT://192.168.1.100:9092
5. log.dirs=/usr/local/kafka_2.12-3.7.0/logs
6. zookeeper.connect=192.168.1.100:2181,192.168.1.101:2181,192.168.1.102:2181
7. EOF

复制代码

情况变量

1. tee -a /etc/profile <<-'EOF'
2. export KAFKA_HOME=/usr/local/kafka_2.12-3.7.0
3. export PATH=$PATH:$KAFKA_HOME/bin
4. EOF

复制代码

开放端口

1. firewall-cmd --zone=public --add-port=9092/tcp --parmanent
2. firewall-cmd --reload

复制代码

启动节点
启动zk：/usr/local/apache-zookeeper-3.8.4-bin/bin/zkStart.sh
逐个启动kafka节点：kafka-server-start.sh $KAFKA_HOME/config/server.properties