Redis叢集的搭建（redis安裝，環境部署，建立，管理，新增，移除，修復）
阿新 • • 發佈：2019-01-04
拓撲結構
| client 192.168.4.50 | |||||
| PORT 6351 | PORT 6352 | PORT 6353 | PORT 6354 | PORT 6355 | PORT 6356 |
| RedisA 4.51 | RedisB 4.52 | RedisC 4.53 | RedisD4.54 | RedisE 4.55 | RedisF 4.56 |
部署叢集
在六臺redis伺服器安裝redis服務（51-56同操作）
[[email protected] ~]# yum - y install gcc
[ 啟動51-56的叢集服務(51為例）
[[email protected] ~]# vim /etc/redis/6379.conf
######################### LUA SCRIPTING ################
815 cluster-enabled yes //啟用叢集
823 cluster-config-file nodes-6351.conf //指定叢集資訊檔案，為了區分這裡把節點改成對應埠號
829 cluster-node-timeout 5000 //請求超時5s
[[email protected] ~]# /etc/init.d/redis_6379 restart
Stopping ...
Redis stopped
Starting Redis server...
[[email protected] ~]# netstat -pntul | grep redis-server
tcp 0 0 192.168.4.51:6351 0.0.0.0:* LISTEN 2279/redis-server 1
tcp 0 0 192.168.4.51:16351 0.0.0.0:* LISTEN 2279/redis-server 1
//叢集服務預設埠號16351（預設是埠號加10000）
[[email protected] ~]# ls /var/lib/redis/6379/ //檢視叢集檔案
dump.rdb nodes-6353.conf
檢視叢集狀態資訊，未建立前的狀態如下：
[[email protected] ~]# redis-cli -h 192.168.4.51 -p 6351
192.168.4.51:6351> CLUSTER info //檢視叢集資訊
cluster_state:fail //未叢集前的狀態
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
192.168.4.51:6351> CLUSTER nodes
394473afaa2d6fae7e0f621e557f0ea8addb4a9b :[email protected] myself,master - 0 0 0 connected
在管理主機51（可以隨機選）上部署ruby指令碼執行環境
（號外：Ruby，面向物件的指令碼語言，由日本人松本行弘(Yukihiro Matsumoto)開發，遵守GPL協議和Ruby License。由 Ruby 語言本身還發展出了JRuby（Java平臺）、IronRuby（.NET平臺）等其他平臺的 Ruby 語言替代品。因為Perl發音與6月誕生石pearl（珍珠）相同，因此Ruby以7月誕生石ruby（紅寶石）命名）
[[email protected] redis-cluster]# yum -y install ruby //安裝依賴環境
已安裝:
ruby.x86_64 0:2.0.0.648-30.el7
作為依賴被安裝:
libyaml.x86_64 0:0.1.4-11.el7_0
ruby-irb.noarch 0:2.0.0.648-30.el7
ruby-libs.x86_64 0:2.0.0.648-30.el7
rubygem-bigdecimal.x86_64 0:1.2.0-30.el7
rubygem-io-console.x86_64 0:0.4.2-30.el7
rubygem-json.x86_64 0:1.7.7-30.el7
rubygem-psych.x86_64 0:2.0.0-30.el7
rubygem-rdoc.noarch 0:4.0.0-30.el7
rubygems.noarch 0:2.0.14.1-30.el7 //這個作為依賴包也會被安裝
[[email protected] redis-cluster]# yum -y install ruby-devel-2.0.0.648-30.el7.x86_64.rpm
[[email protected] redis-cluster]# gem install redis-3.2.1.gem
Successfully installed redis-3.2.1
Parsing documentation for redis-3.2.1
Installing ri documentation for redis-3.2.1
1 gem installed
[[email protected] ~]# cd redis-4.0.8/src
[[email protected] src]# ls
... redis-trib.rb ... //ruby執行命令
[[email protected] src]# mkdir /root/bin
[[email protected] src]# cp redis-trib.rb /root/bin/ //將命令放入在這個目錄下，可以在命令列直接輸入redis-trip.rb 命令
[[email protected] src]# ls -ld /root/bin/
drwxr-xr-x. 2 root root 27 12月 22 10:37 /root/bin/
建立叢集
[[email protected] src]# redis-trib.rb create --replicas 1 192.168.4.51:6351 192.168.4.52:6352 192.168.4.53:6353 192.168.4.54:6354 192.168.4.55:6355 192.168.4.56:6356
//replicas 數字 自動為每一個節點分配一個給定的數值的從庫
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.4.51:6351
192.168.4.52:6352
192.168.4.53:6353
Adding replica 192.168.4.55:6355 to 192.168.4.51:6351
Adding replica 192.168.4.56:6356 to 192.168.4.52:6352
Adding replica 192.168.4.54:6354 to 192.168.4.53:6353
M: 394473afaa2d6fae7e0f621e557f0ea8addb4a9b 192.168.4.51:6351
slots:0-5460 (5461 slots) master
M: c91b387a09ad23f70ba24b7f1668ea1e37b2db63 192.168.4.52:6352
slots:5461-10922,16287 (5463 slots) master //這裡是錯誤，16287槽點不相連
M: e00875106197cdf9e41f5bb00ca0686b212c0290 192.168.4.53:6353
slots:10923-16383 (5461 slots) master
S: 4eb3de604b688d02fb6feeec514b9fda75c3d897 192.168.4.54:6354
replicates e00875106197cdf9e41f5bb00ca0686b212c0290
S: 5417a13f0b845cd68c12f3f9139cfd22d6806a6e 192.168.4.55:6355
replicates 394473afaa2d6fae7e0f621e557f0ea8addb4a9b
S: 32576b4e2fc456f0adb7b2cd80da49e6950576c5 192.168.4.56:6356
replicates c91b387a09ad23f70ba24b7f1668ea1e37b2db63
Can I set the above configuration? (type 'yes' to accept): yes
/usr/local/share/gems/gems/redis-3.2.1/lib/redis/client.rb:113:in `call': ERR Slot 0 is already busy (Redis::CommandError) //報錯資訊
[[email protected] ~]# rm -rf /var/lib/redis/6379/nodes-6351.conf //需要將每臺機器的這個檔案刪除
[[email protected] ~]# /etc/init.d/redis_6379 restart //重啟檢視狀態
[[email protected] ~]# vim /var/lib/redis/6379/nodes-6351.conf
641146d06001e3b101e6d1a79e5d0fcec1bd3fa3 :[email protected] myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0 //顯示為零才正確
[[email protected] ~]# redis-trib.rb create --replicas 1 192.168.4.51:6351 192.168.4.52:6352 192.168.4.53:6353 192.168.4.54:6354 192.168.4.55:6355 192.168.4.56:6356
....
M: 641146d06001e3b101e6d1a79e5d0fcec1bd3fa3 192.168.4.51:6351
slots:0-5460 (5461 slots) master
M: ea1bff01e1cb40cffa751bbe25e0ec39662b5e66 192.168.4.52:6352
slots:5461-10922 (5462 slots) master //這種才為正確
M: 09b2b77ad7b575f6d0feb0ce03553ac044e0866c 192.168.4.53:6353
slots:10923-16383 (5461 slots) master
....
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
....
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered. //完成
[[email protected] ~]# redis-cli -h 192.168.4.51 -p 6351
192.168.4.51:6351> cluster info //檢視資訊
cluster_state:ok //顯示OK
cluster_slots_assigned:16384
cluster_slots_ok:16384
驗證叢集：（redis-cli -c -h ip 地址 -p 埠 ） // -c選項是連結叢集必須的選項
[[email protected] ~]# redis-cli -c -h 192.168.4.52 -p 6352
192.168.4.52:6352> set hary 110
-> Redirected to slot [12751] located at 192.168.4.53:6353 //存到了53上
OK
192.168.4.53:6353>
[[email protected] ~]# redis-cli -h 192.168.4.53 -p 6353 //在53上檢視
192.168.4.53:6353> get harry //取值需要進入叢集
(error) MOVED 890 192.168.4.51:6351
192.168.4.53:6353> keys * //檢視值
1) "hary"
[[email protected] ~]# redis-cli -h 192.168.4.54 -p 6354 //登入53對應從庫54上
192.168.4.54:6354> keys *
1) "hary"
192.168.4.54:6354> get harry /取值需要連結主庫
(error) MOVED 890 192.168.4.51:6351
工作原理：
在三臺主庫上有hash槽：51是0-5460 52 是5161-10922 53是10923-16383 共計16384個
存入資料的時候會通過內部演算法（CRC16演算法對key值計算），得到一個hash值，然後這個值和hash槽的總數16384取餘，得到的值對應哪個redis伺服器就存到哪個redis主庫伺服器！
（ps：CRC是通訊領域中用於校驗資料傳輸正確性的最常用機制，也是Hash演算法的一個典型應用，Hash一般翻譯為“雜湊”，也可直接音譯為“雜湊”，就是把任意長度的輸入（又叫做預對映，pre-image）通過雜湊演算法變換成固定長度的輸出，該輸出就是雜湊值。）
192.168.4.51:6351> set hary 130
-> Redirected to slot [12751] located at 192.168.4.53:6353
OK
192.168.4.52:6352> set hary 110 //可以發現hash值只和變數有關，和值無關
-> Redirected to slot [12751] located at 192.168.4.53:6353
OK
192.168.4.53:6353> set harry 110
-> Redirected to slot [890] located at 192.168.4.51:6351
OK
管理命令
| 命令 | 選項 | ||||
| redis-cli | -h 幫助 | -h IP地址 | -p 埠 | -c 叢集模式 | -a 密碼 |
| redis-trib.rb 選項 引數 | add-node 新增master主機 | check 檢測叢集 | reshard 重新分片 |
add-node --slave 新增slave主機 |
del-node 刪除主機 |
選舉master主機
master宕機後對應的slave自動被選舉為master ，重啟後會自動配置為當前的master的slave並且同步資料
[[email protected] ~]# redis-cli -h 192.168.4.52 -p 6352
192.168.4.52:6352> keys *
1) "sec"
192.168.4.52:6352> shutdown //關閉52主機
[[email protected] ~]# redis-cli -c -h 192.168.4.52 -p 6352
192.168.4.52:6352> set hary 110
192.168.4.51:6351> set marry 123
-> Redirected to slot [8237] located at 192.168.4.56:6356 //可以看到資料存到了52的從庫56上面，說明56主機已經自動成為主庫
OK
[[email protected] ~]# redis-cli -h 192.168.4.56 -p 6356
192.168.4.56:6356> info replication
# Replication
role:master //連結到56上可以看到已經是主庫了
connected_slaves:0
192.168.4.56:6356> keys *
1) "sec"
2) "marry" //資料已經在56主機上了
[[email protected] ~]# /etc/init.d/redis_6379 start
[[email protected] ~]# netstat -pantul | grep :6352
tcp 0 0 192.168.4.52:6352 0.0.0.0:* LISTEN 5194/redis-server 1
[[email protected] ~]# redis-cli -h 192.168.4.52 -p 6352
192.168.4.52:6352> info replication //檢視主從資訊
# Replication
role:slave
master_host:192.168.4.56 //可以看到自動稱為56的從庫了
master_port:6356
192.168.4.52:6352> keys *
1) "sec"
2) "marry" 資料已經同步
管理叢集
新建2臺虛擬機器並執行redis服務（啟用叢集模式）
| 192.168.4.57 port 6357 redis57 | 192.168.4.58 port 5358 redis58 |
[[email protected] redis-4.0.8]# netstat -pantul | grep 6357
tcp 0 0 192.168.4.57:6357 0.0.0.0:* LISTEN 5163/redis-server 1
tcp 0 0 192.168.4.57:16357 0.0.0.0:* LISTEN 5163/redis-server 1
[[email protected] redis-4.0.8]# netstat -pantul | grep 6358
tcp 0 0 192.168.4.58:6358 0.0.0.0:* LISTEN 6857/redis-server 1
tcp 0 0 192.168.4.58:16358 0.0.0.0:* LISTEN 6857/redis-server 1
將57作為master新增到叢集51-56中 在管理主機上操作（51上） redis-trib.rb add-node 新的主機 :埠 叢集中的任一主機:埠
[[email protected] ~]# redis-trib.rb add-node 192.168.4.57:6357 192.168.4.51:6351
>>> Adding node 192.168.4.57:6357 to cluster 192.168.4.51:6351
>>> Performing Cluster Check (using node 192.168.4.51:6351)
...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.4.57:6357 to make it join the cluster.
[OK] New node added correctly.
[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
>>> Performing Cluster Check (using node 192.168.4.51:6351)
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
重新分配hash槽slots {slot *[slɒt] 【計】 槽; 儲存槽}
redis-trib.rb reshard 叢集內的主機:埠 指定移除hash槽個數 指定接收槽的id 指定移除hash槽的主機ID
注：資料也會隨著hash slots的重新分配而分配到對應的主機
[[email protected] ~]# redis-trib.rb reshard 192.168.4.51:6351
...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096（平均分配）
What is the receiving node ID? ca8be3ee5726215b63d2d38513df11ce3732d706
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all
[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
...
[OK] All 16384 slots covered.
[[email protected] ~]# redis-cli -c -h 192.168.4.57 -p 6357
192.168.4.57:6357> keys *
1) "harry"
2) "hello"
3) "ninhao"
將58作為slave主機，新增到叢集中並作為57的從庫
redis-trib.rb add-node --slave [ --master-id id值 ] ip地址:埠 叢集主機:埠 注：如果不指定主節點，會把新的節點隨機新增為節點最少的主的從庫
[[email protected] ~]# redis-trib.rb add-node --slave 192.168.4.58:6358 192.168.4.51:6351
...
>> Send CLUSTER MEET to node 192.168.4.58:6358 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.4.57:6357.
[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
...
S: 585408f1e615c16ecb6670341777288e642b1309 192.168.4.58:6358
slots: (0 slots) slave
replicates ca8be3ee5726215b63d2d38513df11ce3732d706
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
1 additional replica(s)
...
[[email protected] redis-4.0.8]# redis-cli -h 192.168.4.58 -p 6358 //會自動同步資料
192.168.4.58:6358> keys *
1) "hello"
2) "harry"
3) "ninhao"
移除slave節點 ：因為節點沒有槽位，可以直接移除
redis-trib.rb del-node 叢集ip:埠 主機id值
移除master節點：重新分片釋放佔用的hash槽 ，移除master主機
redis-trib.rb reshard 叢集主機：ip值
redis-trib.rb del-node 叢集主機：埠 master主機id值 移出hash槽個數 接收hash槽的主機ID 移除hash槽主機ID
[[email protected] ~]# redis-trib.rb del-node 192.168.4.51:6351 585408f1e615c16ecb6670341777288e642b1309 //移除192.168.4.58節點
>>> Removing node 585408f1e615c16ecb6670341777288e642b1309 from cluster 192.168.4.51:6351
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s) //可以看出57已經沒有從庫
[[email protected] ~]# redis-trib.rb reshard 192.168.4.51:6351
How many slots do you want to move (from 1 to 16384)? 1365 //可以分配任意槽數給任意主機
What is the receiving node ID? 641146d06001e3b101e6d1a79e5d0fcec1bd3fa3
Source node #1:ca8be3ee5726215b63d2d38513df11ce3732d706
Source node #2:done
....
//這裡是分配1365給51 1366給56（因為上面宕機了一次52，主庫變為了56） 1365給53
[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots: (0 slots) master //可以看出57已經沒有槽了
0 additional replica(s)
...
[[email protected] ~]# redis-trib.rb check 192.168.4.51:6351 //再次檢視已經刪除了
把修復的redis伺服器再新增到叢集中
新增redis伺服器57 redis57進入redis---cluster reset（重置叢集，這步可以省略）---刪除/var/lib/redis/6379/nodes-xxx.conf 檔案---重啟---連結叢集伺服器，新增57主機----並重新分片（從庫，不需要分片）
[[email protected] redis-4.0.8]# /etc/init.d/redis_6379 start
Starting Redis server...
[[email protected] redis-4.0.8]# redis-cli -h 192.168.4.57 -p 6357
192.168.4.57:6357> CLUSTER RESET //重置叢集資訊
OK
[[email protected] redis-4.0.8]# cat /var/lib/redis/6379/nodes-6357.conf
ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:[email protected] myself,master - 0 1545470578840 9 connected /裡面的值不為零，需要刪除檔案在重啟
vars currentEpoch 12 lastVoteEpoch 0
[[email protected] redis-4.0.8]# rm -rf /var/lib/redis/6379/nodes-6357.conf
[[email protected] redis-4.0.8]# /etc/init.d/redis_6379 restart
[[email protected] redis-4.0.8]# cat /var/lib/redis/6379/nodes-6357.conf
d38ebde0b68a65e9f8abcacab1ca54548c0ef381 192.168.4.57:[email protected] myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0
[[email protected] ~]# redis-trib.rb add-node 192.168.4.57:6357 192.168.4.51:6351
//再次新增 並重新分片redis叢集單點故障——沒有從庫的主庫宕掉，設定一主兩從。當然也可以更多，不過這個足矣了，如果兩個從都壞了，估計你就離火車票不遠了！！！