Redis學習之路（四）之Redis叢集
阿新 • • 發佈：2020-08-20
Redis學習之路（四）之Redis叢集
Redis叢集
1、Redis Cluster簡介
Redis Cluster為Redis官方提供的一種分散式叢集解決方案。它支援線上節點增加和減少。 叢集中的節點角色可能是主，也可能是從，但需要保證每個主節點都要有對應的從節點， 這樣保證了其高可用。
Redis Cluster採用了分散式系統的分片（分割槽）的思路，每個主節點為一個分片，這樣也就意味著 儲存的資料是分散在所有分片中的。當增加節點或刪除主節點時，原儲存在某個主節點中的資料 會自動再次分配到其他主節點。
如下圖，各節點間是相互通訊的，通訊埠為各節點Redis服務埠+10000，這個埠是固定的，所以注意防火牆設定， 節點之間通過二進位制協議通訊，這樣的目的是減少頻寬消耗。
在Redis Cluster中有一個概念slot，我們翻譯為槽。Slot數量是固定的，為16384個。這些slot會均勻地分佈到各個 節點上。另外Redis的鍵和值會根據hash演算法儲存在對應的slot中。簡單講，對於一個鍵值對，存的時候在哪裡是通過 hash演算法算出來的，那麼取得時候也會算一下，知道值在哪個slot上。根據slot編號再到對應的節點上去取。
Redis Cluster無法保證資料的強一致性，這是因為當資料儲存時，只要在主節點上存好了，就會告訴客戶端存好了， 如果等所有從節點上都同步完再跟客戶端確認，那麼會有很大的延遲，這個對於客戶端來講是無法容忍的。所以， 最終Redis Cluster只好放棄了資料強一致性，而把效能放在了首位。
2、Redis Cluster環境說明
Redis Cluster至少需要三個節點，即一主二從，本實驗中我們使用6個節點搭建。
| 主機名 | IP+Port | 角色 |
|---|---|---|
| redis-master | 192.168.56.11:6379 | Redis Master |
| redis-slave01 | 192.168.56.12:6379 | Redis Master |
| redis-slave02 | 192.168.56.13:6379 | Redis Master |
| redis-master | 192.168.56.11:6380 | Redis Slave |
| redis-slave01 | 192.168.56.12:6380 | Redis Slave |
| redis-slave02 | 192.168.56.13:6380 | Redis Slave |
3、Redis部署
這裡使用三臺虛擬機器，每臺虛擬機器執行兩個Redis例項，埠號分別為6379和6380，在進行修改配置檔案時，需要將之前的哨兵模式和主從配置項取消掉，並且需要將原來的資料儲存目錄中的資料清空。否則Redis是無法啟動叢集模式的。下面給出其中一主一從的6379和6380的部分需要修改的配置，其餘兩個節點採用一樣的配置即可。
[root@redis-master ~]# grep -Ev "^$|#" /usr/local/redis/redis.conf
bind 192.168.56.11
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
pidfile "/var/run/redis_6379.pid"
logfile "/var/log/redis.log"
dir "/var/redis"
cluster-enabled yes #開啟叢集
cluster-config-file nodes-6379.conf #叢集的配置檔案，首次啟動會自動建立
cluster-node-timeout 15000 #叢集節點連線超時時間，15秒
......
[root@redis-master ~]# grep -Ev "^$|#" /usr/local/redis/redis_6380.conf
bind 192.168.56.11
protected-mode yes
port 6380
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
pidfile "/var/run/redis_6380.pid"
logfile "/var/log/redis_6380.log"
dir "/var/redis_6380"
cluster-enabled yes #開啟叢集
cluster-config-file nodes-6380.conf #叢集的配置檔案，首次啟動會自動建立
cluster-node-timeout 15000 #叢集節點連線超時時間，15秒
......
[root@redis-master ~]# mkdir /var/redis_6380 #建立6380例項的資料目錄
4、啟動Redis
[root@redis-master ~]# systemctl start redis
[root@redis-master ~]# redis-server /usr/local/redis/redis_6380.conf
[root@redis-master ~]# ps -ef |grep redis
root 3536 1 0 09:33 ? 00:00:00 /usr/local/redis/src/redis-server 192.168.56.11:6379 [cluster]
root 3543 1 0 09:33 ? 00:00:00 redis-server 192.168.56.11:6380 [cluster]
[root@redis-slave01 ~]# systemctl start redis
[root@redis-slave01 ~]# redis-server /usr/local/redis/redis_6380.conf
[root@redis-slave01 ~]# ps axu |grep redis
root 3821 0.5 0.7 153832 7692 ? Ssl 09:35 0:00 /usr/local/redis/src/redis-server 192.168.56.12:6379 [cluster]
root 3826 0.5 0.6 153832 6896 ? Ssl 09:35 0:00 redis-server 192.168.56.12:6380 [cluster]
[root@redis-slave02 ~]# systemctl start redis
[root@redis-slave02 ~]# redis-server /usr/local/redis/redis_6380.conf
[root@redis-slave02 ~]# ps axu |grep redis
root 3801 0.7 0.7 153832 7696 ? Ssl 09:36 0:00 /usr/local/redis/src/redis-server 192.168.56.13:6379 [cluster]
root 3806 1.4 0.7 153832 7692 ? Ssl 09:36 0:00 redis-server 192.168.56.13:6380 [cluster]
5、部署Reids Cluster
如果虛擬機器上開啟了firewalld，所有機器需要增加如下規則，簡單粗暴的方式是直接systemctl stop firewalld
firewall-cmd --permanent --add-port 6379-6380/tcp
firewall-cmd --permanent --add-port 16379-16380/tcp
firewall-cmd --reload
當前已經啟動了6個節點的Redis服務：192.168.56.11:6379192.168.56.11:6380192.168.56.12:6379192.168.56.12:6380192.168.56.13:6379192.168.56.13:6380
下面在任一節點上執行以下構建叢集的命令，將這裡面的6個節點組建叢集模式，--cluster-replicas 1表示每個主對應一個從。
[root@redis-master ~]# redis-cli --cluster create 192.168.56.11:6379 192.168.56.11:6380 192.168.56.12:6379 192.168.56.12:6380 192.168.56.13:6379 192.168.56.13:6380 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460 #槽位的分配
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.56.12:6380 to 192.168.56.11:6379 #一主對應一從
Adding replica 192.168.56.11:6380 to 192.168.56.12:6379
Adding replica 192.168.56.13:6380 to 192.168.56.13:6379
>>> Trying to optimize slaves allocation for anti-affinity
[OK] Perfect anti-affinity obtained!
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
replicates 31886d2098fb1e627bd71b5af000957a1e252787
Can I set the above configuration? (type 'yes' to accept): yes #是否接受這樣的配置，填yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
到此提示，叢集建立成功！！！！！
6、連線叢集
這裡使用redis-cli去連線任一節點的Redis服務，可以進行一系列的日常操作，如下：
[root@redis-master ~]# redis-cli -c -h 192.168.56.11 -p 6380
192.168.56.11:6380> set k1 123 #k1鍵值儲存定位在13這個節點上的6379例項
-> Redirected to slot [12706] located at 192.168.56.13:6379
OK
192.168.56.13:6379> set k2 abc #k2鍵值儲存定位在11這個節點上的6379例項
-> Redirected to slot [449] located at 192.168.56.11:6379
OK
192.168.56.11:6379> set k3 efg #k3鍵值儲存在本節點的例項上
OK
192.168.56.11:6379> KEYS * #同樣可以獲取鍵值資料來檢視資料的儲存位置
1) "k2"
2) "k3"
192.168.56.11:6379> get k1
-> Redirected to slot [12706] located at 192.168.56.13:6379
"123"
192.168.56.13:6379> get k2
-> Redirected to slot [449] located at 192.168.56.11:6379
"abc"
192.168.56.11:6379> get k3
"efg"
7、管理叢集
7.1、檢視叢集狀態資訊
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379
192.168.56.11:6379 (31886d20...) -> 2 keys | 5461 slots | 1 slaves.
192.168.56.12:6379 (8cd40e6a...) -> 0 keys | 5462 slots | 1 slaves.
192.168.56.13:6379 (587adfa0...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 3 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
7.2、新增叢集節點
新增叢集節點，在redis-master節點上，複製配置檔案，增加6381和6382埠的例項，如下：
[root@redis-master redis]# cp redis_6380.conf redis_6381.conf
[root@redis-master redis]# cp redis_6380.conf redis_6382.conf
[root@redis-master redis]# vim redis_6381.conf #修改6381例項配置檔案
Port 6381
pidfile "/var/run/redis_6381.pid"
logfile "/var/log/redis_6381.log"
dir "/var/redis_6381"
[root@redis-master redis]# vim redis_6382.conf #修改6382例項配置檔案
Port 6382
pidfile "/var/run/redis_6382.pid"
logfile "/var/log/redis_6382.log"
dir "/var/redis_6382"
[root@redis-master redis]# mkdir /var/redis_6381 #建立新增例項的資料目錄
[root@redis-master redis]# mkdir /var/redis_6382
[root@redis-master ~]# redis-server /usr/local/redis/redis_6381.conf #啟動6381例項
[root@redis-master ~]# redis-server /usr/local/redis/redis_6382.conf #啟動6382例項
[root@redis-master ~]# ps axu |grep redis
root 3536 0.3 0.7 156392 7712 ? Ssl 09:34 1:07 /usr/local/redis/src/redis-server 192.168.56.11:6379 [cluster]
root 3543 0.3 0.3 169192 3388 ? Ssl 09:34 1:21 redis-server 192.168.56.11:6380 [cluster]
root 4189 0.2 0.2 153832 2852 ? Ssl 15:29 0:00 redis-server 192.168.56.11:6381 [cluster]
root 4194 0.1 0.2 153832 2852 ? Ssl 15:29 0:00 redis-server 192.168.56.11:6382 [cluster]
這裡新增叢集節點有幾種方式，可以將節點新增為主，也可以新增節點為從節點，也可以為節點指定給某個master節點作為從節點，下面是三種方式的不同增加方法，如下：
#將192.168.56.11:6381節點增加為叢集的主節點，命令如下：
[root@redis-master ~]# redis-cli --cluster add-node 192.168.56.11:6381 192.168.56.11:6379
>>> Adding node 192.168.56.11:6381 to cluster 192.168.56.11:6379
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.56.11:6381 to make it join the cluster.
[OK] New node added correctly.
#檢查叢集狀態，可以看到192.168.56.11:6381已經新增為叢集中的master
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379
192.168.56.11:6379 (31886d20...) -> 2 keys | 5461 slots | 1 slaves.
192.168.56.12:6379 (8cd40e6a...) -> 0 keys | 5462 slots | 1 slaves.
192.168.56.11:6381 (d04ed6a7...) -> 0 keys | 0 slots | 0 slaves.
192.168.56.13:6379 (587adfa0...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots: (0 slots) master
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
上面將192.168.56.11:6381例項新增為叢集的master，現在將6382例項新增為叢集的從節點，要知道的是，上面新增的master節點是沒有從節點的，那麼接下來新增的slave節點，會自動給master作為從節點，如下：
#將192.168.56.11:6382節點增加為叢集的從節點，命令如下：
[root@redis-master ~]# redis-cli --cluster add-node 192.168.56.11:6382 192.168.56.11:6379 --cluster-slave
>>> Adding node 192.168.56.11:6382 to cluster 192.168.56.11:6379
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots: (0 slots) master
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 192.168.56.11:6381
>>> Send CLUSTER MEET to node 192.168.56.11:6382 to make it join the cluster.
Waiting for the cluster to join
>>> Configure node as replica of 192.168.56.11:6381.
[OK] New node added correctly.
#檢查叢集節點狀態資訊，可以看到6382的master節點id為：d04ed6a7d3dbf0aaff0643b045fe22efc7c34500，即6381例項。
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379
192.168.56.11:6379 (31886d20...) -> 2 keys | 5461 slots | 1 slaves.
192.168.56.12:6379 (8cd40e6a...) -> 0 keys | 5462 slots | 1 slaves.
192.168.56.11:6381 (d04ed6a7...) -> 0 keys | 0 slots | 1 slaves.
192.168.56.13:6379 (587adfa0...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 91c803049aa0a12678e26521959aba25d4b06913 192.168.56.11:6382
slots: (0 slots) slave
replicates d04ed6a7d3dbf0aaff0643b045fe22efc7c34500
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots: (0 slots) master
1 additional replica(s)
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
同樣的，也可以為指定的master節點新增從節點，如將6382例項指定新增為6381例項的從節點：
redis-cli --cluster add-node 192.168.56.11:6382 192.168.56.11:6379 --cluster-slave --cluster-master-id d04ed6a7d3dbf0aaff0643b045fe22efc7c34500
7.3、分配slot
從上面的操作上，我們可以看到新增的master 6381例項上的slot為0，那麼這裡我們先給這個master進行分配一些slot，需要注意的是，reshard後面跟的ip，可以是叢集中任一master ip。
[root@redis-master ~]# redis-cli --cluster reshard 192.168.56.11:6379
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 91c803049aa0a12678e26521959aba25d4b06913 192.168.56.11:6382
slots: (0 slots) slave
replicates d04ed6a7d3dbf0aaff0643b045fe22efc7c34500
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots: (0 slots) master
1 additional replica(s)
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 100 #定義要分配多少slot，這裡分配100個
What is the receiving node ID? d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 #定義接收slot的node id，即新的master id
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: 31886d2098fb1e627bd71b5af000957a1e252787 #定義第一個源master的id，如果想在所有master上拿slot，直接敲all
Source node #2: 587adfa041d0c0a14aa1a875bdec219a56b10201 ##定義第二個源master的id，如果不再繼續有新的源，直接敲done
Source node #3: done
Ready to move 100 slots.
Source nodes:
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
Destination node:
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots: (0 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 0 from 31886d2098fb1e627bd71b5af000957a1e252787
Moving slot 1 from 31886d2098fb1e627bd71b5af000957a1e252787
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 0 from 192.168.56.11:6379 to 192.168.56.11:6381:
Moving slot 1 from 192.168.56.11:6379 to 192.168.56.11:6381:
Moving slot 2 from 192.168.56.11:6379 to 192.168.56.11:6381:
Moving slot 3 from 192.168.56.11:6379 to 192.168.56.11:6381:
......
#重新檢視叢集中的slot，可以看到新的master上已經分配了100個slot
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379
......
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots:[0-49],[10923-10972] (100 slots) master
1 additional replica(s)
......
slot的分配，也可以作為slot的遷移，在當前某個master節點出現故障時，我們可以通過slot的分配對故障節點的slot進行遷移，命令和分配的命令是一樣的，如下：
redis-cli --cluster reshard 192.168.56.11:6379
7.4、刪除叢集節點
有新增，就有刪除，刪除叢集中的節點，本身節點上的Redis必須不帶有slot，如果有slot，需要先進行移除，否則會報錯。被刪除的node重啟後，依然會記得叢集中的其他節點，這就需要執行cluster forget nodeid來忘記其他節點。
#這裡先刪除6382這個從節點例項，因為該例項上沒有slot，是可以直接移除的
[root@redis-master ~]# redis-cli --cluster del-node 192.168.56.11:6382 91c803049aa0a12678e26521959aba25d4b06913
>>> Removing node 91c803049aa0a12678e26521959aba25d4b06913 from cluster 192.168.56.11:6382
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379 #重新檢視是已經沒有了6382例項了
192.168.56.11:6379 (31886d20...) -> 2 keys | 5411 slots | 1 slaves.
192.168.56.12:6379 (8cd40e6a...) -> 0 keys | 5462 slots | 1 slaves.
192.168.56.11:6381 (d04ed6a7...) -> 0 keys | 100 slots | 0 slaves.
192.168.56.13:6379 (587adfa0...) -> 1 keys | 5411 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[50-5460] (5411 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots:[0-49],[10923-10972] (100 slots) master
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10973-16383] (5411 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
#那麼現在再來刪除帶有slot的6381例項，則會出現報錯，如下
[root@redis-master ~]# redis-cli --cluster del-node 192.168.56.11:6381 d04ed6a7d3dbf0aaff0643b045fe22efc7c34500
>>> Removing node d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 from cluster 192.168.56.11:6381
[ERR] Node 192.168.56.11:6381 is not empty! Reshard data away and try again.
#提示該例項不為空，需要Reshard掉資料後再重試，那麼現在先將slot進行遷移掉
[root@redis-master ~]# redis-cli --cluster reshard 192.168.56.11:6381
>>> Performing Cluster Check (using node 192.168.56.11:6381)
M: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 192.168.56.11:6381
slots:[0-49],[10923-10972] (100 slots) master
......
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 100
What is the receiving node ID? 587adfa041d0c0a14aa1a875bdec219a56b10201
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: d04ed6a7d3dbf0aaff0643b045fe22efc7c34500
Source node #2: done
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379
192.168.56.11:6379 (31886d20...) -> 2 keys | 5411 slots | 1 slaves.
192.168.56.12:6379 (8cd40e6a...) -> 0 keys | 5462 slots | 1 slaves.
192.168.56.11:6381 (d04ed6a7...) -> 0 keys | 0 slots | 0 slaves. #slots數量已經為0
192.168.56.13:6379 (587adfa0...) -> 1 keys | 5511 slots | 1 slaves.
......
#再次進行刪除6381叢集節點，即可順利完成刪除
[root@redis-master ~]# redis-cli --cluster del-node 192.168.56.11:6381 d04ed6a7d3dbf0aaff0643b045fe22efc7c34500
>>> Removing node d04ed6a7d3dbf0aaff0643b045fe22efc7c34500 from cluster 192.168.56.11:6381
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
7.5、slot數量平衡
先通過reshard功能，模擬slot不均勻分配，然後檢查叢集狀態中，可以看到各個master上的slot數量不均衡，如下：
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379
192.168.56.11:6379 (31886d20...) -> 1 keys | 4416 slots | 1 slaves.
192.168.56.12:6379 (8cd40e6a...) -> 0 keys | 4457 slots | 1 slaves.
192.168.56.13:6379 (587adfa0...) -> 2 keys | 7511 slots | 1 slaves.
......
當我們使用的slot數量分佈不均勻時，同樣也可以使用平衡功能，將各個節點的slot進行重新均衡分配：
[root@redis-master ~]# redis-cli --cluster rebalance --cluster-threshold 1 192.168.56.11:6379
>>> Performing Cluster Check (using node 192.168.56.11:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 3 nodes. Total weight = 3.00
Moving 1046 slots from 192.168.56.13:6379 to 192.168.56.11:6379
###################################################################################################
Moving 1004 slots from 192.168.56.13:6379 to 192.168.56.12:6379
###################################################################################################
[root@redis-master ~]# redis-cli --cluster check 192.168.56.11:6379
192.168.56.11:6379 (31886d20...) -> 2 keys | 5462 slots | 1 slaves.
192.168.56.12:6379 (8cd40e6a...) -> 0 keys | 5461 slots | 1 slaves.
192.168.56.13:6379 (587adfa0...) -> 1 keys | 5461 slots | 1 slaves.
......
#需要注意的是，所有節點的平衡閾值在1%，當各個節點的slot數值相差不到1%時，進行balance是會報以下提示的：
*** No rebalancing needed! All nodes are within the 1.00% threshold.
7.6、叢集外部資料匯入
先解除6381例項的叢集模式，重新啟動6381例項，並寫入資料，如下
[root@redis-master ~]# redis-cli -h 192.168.56.11 -p 6381
192.168.56.11:6381> KEYS *
(empty list or set)
192.168.56.11:6381> set k0 aaa
OK
192.168.56.11:6381> set k5 bbb
OK
192.168.56.11:6381> exit
再將6381例項的資料匯入到叢集中，如下
[root@redis-master ~]# redis-cli --cluster import 192.168.56.11:6379 --cluster-from 192.168.56.11:6381 --cluster-copy
>>> Importing data from 192.168.56.11:6381 to cluster 192.168.56.11:6379
>>> Performing Cluster Check (using node 192.168.56.11:6379)
M: 31886d2098fb1e627bd71b5af000957a1e252787 192.168.56.11:6379
slots:[0-5461] (5462 slots) master
1 additional replica(s)
S: 55a6f654dcb87c6a8017c8619f0ce8763a92abd6 192.168.56.13:6380
slots: (0 slots) slave
replicates 31886d2098fb1e627bd71b5af000957a1e252787
M: 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446 192.168.56.12:6379
slots:[5462-10922] (5461 slots) master
1 additional replica(s)
S: be0ef4a1b1a60cee781afe5c2b8b5cbd7b68b4e6 192.168.56.11:6380
slots: (0 slots) slave
replicates 8cd40e6a31c12e0b9c01f20056b9ecaa4db51446
S: eb812dc6051776e151bf69cd328bd0a66a20de01 192.168.56.12:6380
slots: (0 slots) slave
replicates 587adfa041d0c0a14aa1a875bdec219a56b10201
M: 587adfa041d0c0a14aa1a875bdec219a56b10201 192.168.56.13:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
*** Importing 2 keys from DB 0
Migrating k5 to 192.168.56.13:6379: OK
Migrating k0 to 192.168.56.12:6379: OK
[root@redis-master ~]# redis-cli -c -h 192.168.56.11 -p 6379 #登入叢集檢視資料
192.168.56.11:6379> KEYS *
1) "k2"
2) "k3"
192.168.56.11:6379> get k0
-> Redirected to slot [8579] located at 192.168.56.12:6379
"aaa"
192.168.56.12:6379> get k5
-> Redirected to slot [12582] located at 192.168.56.13:6379
"bbb"
這裡需要注意的是：Cluster-from後面跟外部redis的ip和port如果只使用cluster-copy，則要匯入叢集中的key不能在，否則如下： 如果叢集中已有同樣的key，如果需要替換，可以cluster-copy和cluster-replace聯用，這樣叢集中的key就會被替換為外部的。而在生產環境當中，為了保證Redis的高可用性，通常會配置一主兩從的模式，也就是說在建立叢集時該引數提示的副本數為2，如：--cluster-replicas 2