[Kafka] 基本安裝方式 - 手動安裝

Apache Kafka是一個分散式的訊息處理framework, 透過publish來發佈message，以及subscribe來訂閱取得message, 這裡提供簡單的安裝方式.

從架構上來看，Kafka可以分為下面幾種角色，1. producer: 發佈(publish)message to topic2. consumer: 訂閱(subscribe)topic以取得message3. broker: 簡單的說就是server，由一台以上的broker組成一個cluster4. topic: message的分類5. zookeeper: 嚴格來說，zookeeper不算是Kafka的一部分，但Kafka卻得倚靠zookeeper來做到sync。

==========================
安裝apache Kafka(Ubuntu Linux)
==========================
參考

download kafka

*解壓縮

tar -xvf kafka_2.12-0.10.1.1.tgz
把解壓縮檔案重新命名 => kafka

*移動到目標資料夾

sudo mkdir /usr/local/kafka
sudo mv kafka /usr/local/

*download zookeeper

http://apache.stu.edu.tw/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

*設定環境變數
sudo gedit ~/.bashrc

export KAFKA_HOME=/usr/local/kafka
export PATH=$PATH:$KAFKA_HOME/bin
export ZOOKEEPER_HOME=/usr/local/kafka/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin

source ~/.bashrc

*設定kafka設定檔(如果叢集每台都要設定, 可使用cp語法到別台)

broker.id=1 (每台的唯一值)
zookeeper.connect=192.168.0.41:2181(多台的話用,切開)
log.dirs=/usr/local/kafka/logs
delete.topic.enable=true
listeners=PLANTTEXT://192.168.0.41:2181( OR 9092 Broker要用這個port送出去,才不會有exception)

*設定zookeeper設定檔

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/kafka/zookeeper //Zookeeper 資料儲存路徑
clientPort=2182 //使用Zookeeper Port

*Start zookeeper

要先啟動zookeeper，原因是前面有說過，
Kafka倚靠zookeeper做message的sync
cd /usr/local/kafka
bin/zookeeper-server-start.sh config/zookeeper.properties &
netstat -tnlp 看一下2181port有沒有被啟起來

*Start Kafka server(broker)

bin/kafka-server-start.sh config/server.properties &

*Create a topic : 有了server以後，接著要create一個topic

bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 2 –partitions 2 –topic test

*查詢topic資訊

bin/kafka-topics.sh –describe –zookeeper localhost:2181 –topic test
ReplicationFactor是2份，
Replicate在0, 1, 2這三台broker上面，
topic的leader是broker 2，leader負責partition的read and write。
Isr的意思是有哪些broker正在sync，簡單的說可以知道哪些broker是活著的。

*Publish: send message to topic 寫入訊息到topic

$ bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test
…
test message 1
test message 2
^C

*Subscribe: get message from topic 讀取訊息

bin/kafka-console-consumer.sh –zookeeper localhost:2181 –topic test –from-beginning

*關閉服務

bin/zookeeper-server-stop.sh
bin/kafka-server-stop.sh

*整合eclipse

Ref: http://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html

*下載 spark-streaming-kafka-0-8_2.11然後將檔案移到kafka/libs

wget http://search.maven.org/remotecontent?filepath=org/apache/spark/spark-streaming-kafka-0-8-assembly_2.11/2.0.2/spark-streaming-kafka-0-8-assembly_2.11-2.0.2.jar

spark-subimt要加入 –jars /usr/local/kafka/libs/spark-streaming-kafka-0-8-assembly_2.11-2.0.2.jar

==================
安裝Kafka UI Tool
==================

1. yahoo kafka tool
2. kafka offset monitor
https://github.com/quantifind/KafkaOffsetMonitor

java -cp KafkaOffsetMonitor-assembly-0.2.1.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
–zk 192.168.0.41 \
–port 8089 \
–refresh 10.seconds \
–retain 2.days