centos6下redis cluster集群部署過程
一般來說,redis主從和mysql主從目的差不多,但redis主從配置很簡單,主要在從節點配置文件指定主節點ip和端口,比如:slaveof 192.168.10.10 6379,然後啟動主從,主從就搭建好了。redis主從中如果主節點發生故障,不會自動切換,需要借助redis的Sentinel(哨兵模式)或者keepalive來實現主的故障轉移。
今天介紹下redis cluster集群模式:
redis集群是一個無中心的分布式redis存儲架構,可以在多個節點之間進行數據共享,解決了redis高可用、可擴展等問題,redis集群提供了以下兩個好處:
1)將數據自動切分(split)到多個節點
2)當集群中的某一個節點故障時,redis還可以繼續處理客戶端的請求。
一個 Redis 集群包含 16384 個哈希槽(hash slot),數據庫中的每個數據都屬於這16384個哈希槽中的一個。集群使用公式 CRC16(key) % 16384 來計算鍵 key 屬於哪個槽。集群中的每一個節點負責處理一部分哈希槽。
集群中的主從復制
集群中的每個節點都有1個至N個復制品,其中一個為主節點,其余的為從節點,如果主節點下線了,集群就會把這個主節點的一個從節點設置為新的主節點,繼續工作。這樣集群就不會因為一個主節點的下線而無法正常工作。
廢話不多說,下面記錄下搭建redis cluster集群:
由於最小的redis集群需要3個主節點,一臺機器可運行多個redis實例,我搭建時使用兩臺機器,6個redis實例,其中三個主節點,三個從節點作為備份。很多案例使用單臺服務器開6個端口,操作差不多,只是配置基本相對簡單點,多臺服務器更接近生產環境。
本案例redis cluster節點信息:
redis01
172.16.51.175:7000
172.16.51.175:7001
172.16.51.175:7002
redis02
172.16.51.176:7003
172.16.51.176:7004
172.16.51.176:7005
redis03
172.16.51.178:7006
172.16.51.178:7007
172.16.51.178:7008
先說下redis01節點的部署過程(其他兩臺節點部署過程一致)
個人運維習慣,會專門創建一個app賬號,用戶部署應用程序。本案例應用程序都部署在/data目錄下,將/data權限設置成app [root@bl-redis01 ~]# useradd app [root@bl-redis01 ~]# passwd app [root@bl-redis01 ~]# chown -R app.app /data 前提準備 1)安裝 GCC 編譯工具 不然會有編譯不過的問題 [root@bl-redis01 ~]# yum install -y gcc g++ make gcc-c++ kernel-devel automake autoconf libtool make wget tcl vim ruby rubygems unzip git 2)升級所有的包,防止出現版本過久不兼容問題 [root@bl-redis01 ~]# yum -y update 3)關閉防火墻 節點之前需要開放指定端口,為了方便,生產不要禁用 [root@bl-redis01 ~]# /etc/init.d/iptables stop [root@bl-redis01 ~]# setenforce 0 [root@bl-redis01 ~]# vim /etc/sysconfig/selinux ...... SELINUX=disabled ...... redis cluster集群部署 4)下載並編譯安裝redis [root@bl-redis01 ~]# su - app [app@bl-redis01 ~]$ mkdir /data/software/ [app@bl-redis01 software]$ wget http://download.redis.io/releases/redis-4.0.1.tar.gz [app@bl-redis01 software]$ tar -zvxf redis-4.0.1.tar.gz [app@bl-redis01 software]$ mv redis-4.0.1 /data/ [app@bl-redis01 software]$ cd /data/redis-4.0.1/ [app@bl-redis01 redis-4.0.1]$ make -------------------------------------------------------------------------------------- 如果因為上次編譯失敗,有殘留的文件,做法如下: [app@bl-redis01 redis-4.0.1]$ make distclean -------------------------------------------------------------------------------------- 5)創建節點 首先在172.16.51.175機器(redis01)上/data/redis-4.0.1目錄下創建redis-cluster目錄 [app@bl-redis01 redis-4.0.1]$ mkdir /opt/redis-4.0.1/redis-cluster 接著在redis-cluster目錄下,創建名為7000、7001、7002的目錄 [app@bl-redis01 redis-4.0.1]$ mkdir 7000 [app@bl-redis01 redis-4.0.1]$ mkdir 7001 [app@bl-redis01 redis-4.0.1]$ mkdir 7002 分別修改這三個配置文件redis.conf [app@bl-redis01 redis-4.0.1]$ cd redis-cluster/ [app@bl-redis01 redis-cluster]$ ll total 12 drwxrwxr-x 2 app app 4096 Nov 16 17:38 7000 drwxrwxr-x 2 app app 4096 Nov 16 17:39 7001 drwxrwxr-x 2 app app 4096 Nov 16 17:39 7002 [app@bl-redis01 redis-cluster]$ cat 7000/redis.conf port 7000 bind 172.16.51.175 daemonize yes pidfile /var/run/redis_7000.pid cluster-enabled yes cluster-config-file nodes_7000.conf cluster-node-timeout 10100 appendonly yes [app@bl-redis01 redis-cluster]$ cat 7001/redis.conf port 7001 bind 172.16.51.175 daemonize yes pidfile /var/run/redis_7001.pid cluster-enabled yes cluster-config-file nodes_7001.conf cluster-node-timeout 10100 appendonly yes [app@bl-redis01 redis-cluster]$ cat 7002/redis.conf port 7002 bind 172.16.51.175 daemonize yes pidfile /var/run/redis_7002.pid cluster-enabled yes cluster-config-file nodes_7002.conf cluster-node-timeout 10100 appendonly yes ---------------------------------------------------------------------------------------------------- redis.conf的配置說明: #端口7000,7001,7002 port 7000 #默認ip為127.0.0.1,需要改為其他節點機器可訪問的ip,否則創建集群時無法訪問對應的端口,無法創建集群 bind 172.16.51.175 #redis後臺運行 daemonize yes #pidfile文件對應7000,7001,7002 pidfile /var/run/redis_7000.pid #開啟集群,把註釋#去掉 cluster-enabled yes #集群的配置,配置文件首次啟動自動生成 7000,7001,7002 cluster-config-file nodes_7000.conf #請求超時,默認15秒,可自行設置 cluster-node-timeout 10100 #aof日誌開啟,有需要就開啟,它會每次寫操作都記錄一條日誌 appendonly yes ---------------------------------------------------------------------------------------------------- 接著在另外兩臺機器上(172.16.51.176,172.16.51.178)重復以上三步,只是把目錄改為7003、7004、7005和7006、7007、7008,對應的配置文件也按照這個規則修改即可(即修改redis.conf文件中的端口就行了) 6)啟動集群(依次啟動7000-7008端口) #第一個節點機器上執行 3個節點 [app@bl-redis01 redis-cluster]$ for((i=0;i<=2;i++)); do /data/redis-4.0.1/src/redis-server /data/redis-4.0.1/redis-cluster/700$i/redis.conf; done #第二個節點機器上執行 3個節點 [app@bl-redis01 redis-cluster]$ for((i=3;i<=5;i++)); do /data/redis-4.0.1/src/redis-server /data/redis-4.0.1/redis-cluster/700$i/redis.conf; done #第三個節點機器上執行 3個節點 [app@bl-redis01 redis-cluster]$ for((i=6;i<=8;i++)); do /data/redis-4.0.1/src/redis-server /data/redis-4.0.1/redis-cluster/700$i/redis.conf; done 7)檢查服務 檢查各 Redis 各個節點啟動情況 [app@bl-redis01 redis-cluster]$ ps -ef | grep redis app 2564 2405 0 20:13 pts/0 00:00:00 grep redis app 15197 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7000 [cluster] app 15199 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7001 [cluster] app 15201 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7002 [cluster] [app@bl-redis01 redis-cluster]$ ps -ef | grep redis app 2566 2405 0 20:13 pts/0 00:00:00 grep redis app 15197 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7000 [cluster] app 15199 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7001 [cluster] app 15201 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7002 [cluster] 8)安裝 Ruby(需要切換到root賬號下進行安裝,app賬號下權限不夠) [root@bl-redis01 ~]# yum -y install ruby ruby-devel rubygems rpm-build [root@bl-redis01 ~]# gem install redis ----------------------------------------------------------------------------------------------------- 註意:在centos6.x下執行上面的"gem install redis"操作可能會報錯,坑很多! 默認yum安裝的ruby版本是1.8.7,版本太低,需要升級到ruby2.2以上,否則執行上面安裝會報錯! 首先安裝rvm [root@bl-redis01 ~]# curl -L get.rvm.io | bash -s stable //可能會報錯,需要安裝提示進行下面一步操作 [root@bl-redis01 ~]# curl -sSL https://rvm.io/mpapis.asc | gpg2 --import - [root@bl-redis01 ~]# find / -name rvm.sh /etc/profile.d/rvm.sh [root@bl-redis01 ~]# source /etc/profile.d/rvm.sh [root@bl-redis01 ~]# rvm requirements 然後升級ruby到2.3 [root@bl-redis01 ~]# rvm install ruby 2.3.1 [root@bl-redis01 ~]# ruby -v ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux] 列出所有ruby版本 [root@bl-redis01 ~]# rvm list 設置默認的版本 [root@bl-redis01 ~]# rvm --default use 2.3.1 更新下載源 [root@bl-redis01 ~]# gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org https://gems.ruby-china.org/ added to sources source https://rubygems.org not present in cache [root@bl-redis01 ~]# gem sources *** CURRENT SOURCES *** https://rubygems.org/ https://gems.ruby-china.org/ 最後就能順利安裝了 [root@bl-redis01 src]# gem install redis Successfully installed redis-4.0.1 Parsing documentation for redis-4.0.1 Done installing documentation for redis after 1 seconds 1 gem installed ----------------------------------------------------------------------------------------------------- 9)創建集群 千萬註意:在任意一臺上運行即可,不要在每臺機器上都運行,一臺就夠了!!!! Redis 官方提供了 redis-trib.rb 這個工具,就在解壓目錄的 src 目錄中 [root@bl-redis01 ~]# su - app [app@bl-redis01 ~]$ /data/redis-4.0.1/src/redis-trib.rb create --replicas 1 172.16.51.175:7000 172.16.51.175:7001 172.16.51.175:7002 172.16.51.176:7003 172.16.51.176:7004 172.16.51.176:7005 172.16.51.178:7006 172.16.51.178:7007 172.16.51.178:7008 出現下面信息: >>> Creating cluster >>> Performing hash slots allocation on 9 nodes... Using 4 masters: 172.16.51.175:7000 172.16.51.176:7003 172.16.51.178:7006 172.16.51.175:7001 Adding replica 172.16.51.176:7004 to 172.16.51.175:7000 Adding replica 172.16.51.178:7007 to 172.16.51.176:7003 Adding replica 172.16.51.175:7002 to 172.16.51.178:7006 Adding replica 172.16.51.176:7005 to 172.16.51.175:7001 Adding replica 172.16.51.178:7008 to 172.16.51.175:7000 M: 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf 172.16.51.175:7000 slots:0-4095 (4096 slots) master M: 44c81c15b01d992cb9ede4ad35477ec853d70723 172.16.51.175:7001 slots:12288-16383 (4096 slots) master S: 38f03c27af39723e1828eb62d1775c4b6e2c3638 172.16.51.175:7002 replicates f1abb62a8c9b448ea14db421bdfe3f1d8075189c M: 987965baf505a9aa43e50e46c76189c51a8f17ec 172.16.51.176:7003 slots:4096-8191 (4096 slots) master S: 6555292fed9c5d52fcf5b983c441aff6f96923d5 172.16.51.176:7004 replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf S: 2b5ba254a0405d4efde4c459867b15176f79244a 172.16.51.176:7005 replicates 44c81c15b01d992cb9ede4ad35477ec853d70723 M: f1abb62a8c9b448ea14db421bdfe3f1d8075189c 172.16.51.178:7006 slots:8192-12287 (4096 slots) master S: eb4067373d36d8a8df07951f92794e67a6aac022 172.16.51.178:7007 replicates 987965baf505a9aa43e50e46c76189c51a8f17ec S: 2919e041dd3d1daf176d6800dcd262f4e727f366 172.16.51.178:7008 replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf Can I set the above configuration? (type ‘yes‘ to accept): yes 輸入 yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join......... >>> Performing Cluster Check (using node 172.16.51.175:7000) M: 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf 172.16.51.175:7000 slots:0-4095 (4096 slots) master 2 additional replica(s) S: 6555292fed9c5d52fcf5b983c441aff6f96923d5 172.16.51.176:7004 slots: (0 slots) slave replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf M: 44c81c15b01d992cb9ede4ad35477ec853d70723 172.16.51.175:7001 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 2919e041dd3d1daf176d6800dcd262f4e727f366 172.16.51.178:7008 slots: (0 slots) slave replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf M: f1abb62a8c9b448ea14db421bdfe3f1d8075189c 172.16.51.178:7006 slots:8192-12287 (4096 slots) master 1 additional replica(s) S: eb4067373d36d8a8df07951f92794e67a6aac022 172.16.51.178:7007 slots: (0 slots) slave replicates 987965baf505a9aa43e50e46c76189c51a8f17ec S: 38f03c27af39723e1828eb62d1775c4b6e2c3638 172.16.51.175:7002 slots: (0 slots) slave replicates f1abb62a8c9b448ea14db421bdfe3f1d8075189c S: 2b5ba254a0405d4efde4c459867b15176f79244a 172.16.51.176:7005 slots: (0 slots) slave replicates 44c81c15b01d992cb9ede4ad35477ec853d70723 M: 987965baf505a9aa43e50e46c76189c51a8f17ec 172.16.51.176:7003 slots:4096-8191 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 10)關閉集群 推薦做法: [app@bl-redis01 ~]$ pkill redis [app@bl-redis02 ~]$ pkill redis [app@bl-redis03 ~]$ pkill redis 或者循環節點逐個關閉 [app@bl-redis01 ~]$ for((i=0;i<=2;i++)); do /opt/redis-4.0.1/src/redis-cli -c -h 172.16.51.175 -p 700$i shutdown; done [app@bl-redis02 ~]$ for((i=3;i<=5;i++)); do /opt/redis-4.0.1/src/redis-cli -c -h 172.16.51.176 -p 700$i shutdown; done [app@bl-redis03 ~]$ for((i=6;i<=8;i++)); do /opt/redis-4.0.1/src/redis-cli -c -h 172.16.51.178 -p 700$i shutdown; done 11)集群驗證 連接集群測試 參數-C可連接到集群,因為redis.conf將bind改為了ip地址,所以-h參數不可以省略,-p參數為端口號 可以先在172.16.51.175機器redis 7000 的節點set一個key [app@bl-redis01 ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> set name www.ymq.io -> Redirected to slot [5798] located at 172.16.51.176:7003 OK 172.16.51.176:7003> get name "www.ymq.io" 172.16.51.176:7003> 由上面信息可發現redis set name 之後重定向到172.16.51.176機器 redis 7003 這個節點 然後在172.16.51.178機器redis 7008 的節點get一個key [app@bl-redis03 ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.178 -c -p 7008 172.16.51.178:7008> get name -> Redirected to slot [5798] located at 172.16.51.176:7003 "www.ymq.io" 172.16.51.176:7003> 發現redis get name 重定向到172.16.51.176機器 redis 7003 這個節點. 如果看到這樣的現象,說明redis cluster集群已經是可用的了!!!!!! 12)檢查集群狀態 [app@bl-redis01 ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> [app@bl-redis01 ~]$ /data/redis-4.0.1/src/redis-trib.rb check 172.16.51.175:7000 >>> Performing Cluster Check (using node 172.16.51.175:7000) M: 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 172.16.51.175:7000 slots:0-4095 (4096 slots) master 2 additional replica(s) M: c64b0839e0199f73c5c192cc8c90f12c999f79b2 172.16.51.175:7001 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 81347f01cf38d8f0faef1ad02676ebb4cffbec9e 172.16.51.176:7005 slots: (0 slots) slave replicates c64b0839e0199f73c5c192cc8c90f12c999f79b2 M: da5dde3f2f02c232784bf3163f5f584b8cf046f2 172.16.51.178:7006 slots:8192-12287 (4096 slots) master 1 additional replica(s) M: b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 172.16.51.176:7003 slots:4096-8191 (4096 slots) master 1 additional replica(s) S: 0420c49fbc9f1fe16066d189265cca2f5e71c86e 172.16.51.178:7007 slots: (0 slots) slave replicates b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 S: 5ad89453fb36e50ecc4560de6b4acce1dbbb78b3 172.16.51.176:7004 slots: (0 slots) slave replicates 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 S: bbd1f279b99b95cf00ecbfab22b6b8dd5eb05989 172.16.51.178:7008 slots: (0 slots) slave replicates 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 S: e95407b83bfeb30e3cc537161eadc372d6aa1fa2 172.16.51.175:7002 slots: (0 slots) slave replicates da5dde3f2f02c232784bf3163f5f584b8cf046f2 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 13)列出集群節點 列出集群當前已知的所有節點(node),以及這些節點的相關信息 [app@bl-redis01 ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> cluster nodes 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 172.16.51.175:7000@17000 myself,master - 0 1510836027000 1 connected 0-4095 c64b0839e0199f73c5c192cc8c90f12c999f79b2 172.16.51.175:7001@17001 master - 0 1510836030068 2 connected 12288-16383 81347f01cf38d8f0faef1ad02676ebb4cffbec9e 172.16.51.176:7005@17005 slave c64b0839e0199f73c5c192cc8c90f12c999f79b2 0 1510836031000 6 connected da5dde3f2f02c232784bf3163f5f584b8cf046f2 172.16.51.178:7006@17006 master - 0 1510836031000 7 connected 8192-12287 b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 172.16.51.176:7003@17003 master - 0 1510836030000 4 connected 4096-8191 0420c49fbc9f1fe16066d189265cca2f5e71c86e 172.16.51.178:7007@17007 slave b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 0 1510836029067 8 connected 5ad89453fb36e50ecc4560de6b4acce1dbbb78b3 172.16.51.176:7004@17004 slave 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 0 1510836032672 5 connected bbd1f279b99b95cf00ecbfab22b6b8dd5eb05989 172.16.51.178:7008@17008 slave 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 0 1510836031000 9 connected e95407b83bfeb30e3cc537161eadc372d6aa1fa2 172.16.51.175:7002@17002 slave da5dde3f2f02c232784bf3163f5f584b8cf046f2 0 1510836031672 7 connected 14)打印集群信息 [app@bl-redis01 ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:9 cluster_size:4 cluster_current_epoch:9 cluster_my_epoch:1 cluster_stats_messages_ping_sent:8627 cluster_stats_messages_pong_sent:8581 cluster_stats_messages_sent:17208 cluster_stats_messages_ping_received:8573 cluster_stats_messages_pong_received:8626 cluster_stats_messages_meet_received:8 cluster_stats_messages_received:17207 15)集群命令 語法格式 redis-cli -c -p port 集群 cluster info :打印集群的信息 cluster nodes :列出集群當前已知的所有節點( node),以及這些節點的相關信息。 節點 cluster meet <ip> <port> :將 ip 和 port 所指定的節點添加到集群當中,讓它成為集群的一份子。 cluster forget <node_id> :從集群中移除 node_id 指定的節點。 cluster replicate <node_id> :將當前節點設置為 node_id 指定的節點的從節點。 cluster saveconfig :將節點的配置文件保存到硬盤裏面。 槽(slot) cluster addslots <slot> [slot ...] :將一個或多個槽( slot)指派( assign)給當前節點。 cluster delslots <slot> [slot ...] :移除一個或多個槽對當前節點的指派。 cluster flushslots :移除指派給當前節點的所有槽,讓當前節點變成一個沒有指派任何槽的節點。 cluster setslot <slot> node <node_id> :將槽 slot 指派給 node_id 指定的節點,如果槽已經指派給另一個節點,那麽先讓另一個節點刪除該槽>,然後再進行指派。 cluster setslot <slot> migrating <node_id> :將本節點的槽 slot 遷移到 node_id 指定的節點中。 cluster setslot <slot> importing <node_id> :從 node_id 指定的節點中導入槽 slot 到本節點。 cluster setslot <slot> stable :取消對槽 slot 的導入( import)或者遷移( migrate)。 鍵 cluster keyslot <key> :計算鍵 key 應該被放置在哪個槽上。 cluster countkeysinslot <slot> :返回槽 slot 目前包含的鍵值對數量。 cluster getkeysinslot <slot> <count> :返回 count 個 slot 槽中的鍵 。
centos6下redis cluster集群部署過程