Day27 Redis架构实战-Redis丛集Slot分流机制
Redis丛集Slot分流机制
# Redis丛集配置
client
|
V
--------------------------------------------------------------
Redis Redis Redis
(Master) (Master) (Master)
| | | | | |
V V V V V V
Redis Redis Redis Redis Redis Redis
(Replica) (Replica) (Replica) (Replica) (Replica) (Replica)
--------------------------------------------------------------
示意 1/3 Data 1/3 Data 1/3 Data
- 资料Slot分流机制
- Redis Server丛集将Key Space分成16384个slots，这些slots可以分散到最多16384个主节点上．
- 由於Redis Server丛集采用Gossip协议实施无中心式丛集，如果超过1000个主节点时因为Gossip协议需要频繁的讯息传输所以效率会变差，主要是因为网路频宽的问题，故不建议部署超过1000个主节点．
- hash_slot = CRC16(key) mod 16384
# 依次序set key value 操作Redis Server
set book1 abc
set book2 def
set book3 ghi
set book4 jkl
set book5 mno
set book6 pqr
set book7 stu
# Redis丛集资料Slot分流机制
client
| (依上面次序进行set操作)
V
CRC16(key) mod 16384
--------------------------------------------------------------
Redis Redis Redis
(Master) (Master) (Master)
| | | | | |
V V V V V V
Redis Redis Redis Redis Redis Redis
(Replica) (Replica) (Replica) (Replica) (Replica) (Replica)
--------------------------------------------------------------
book1=abc book2=def book3=ghi
book4=jkl book5=mno book6=pqr
book7=stu
Redis丛集架构实际配置
在配置Redis Server丛集架构时，基本上会配置三个Ｍaster节点，Master节点各自搭配一个以上的Replica节点，下面实际配置的内容会在单机进行，故会将Port分配来模拟丛集．
因需要在同一台机器模拟Redis Server丛集配置，故有以下前置作业需要先完成．
- 丛集一
- 6310-->Reids Server (Master) V
- 6311-->Reids Server (Replica) V
- 6312-->Reids Server (Replica) V
- 丛集二
- 6320-->Reids Server (Master) V
- 6321-->Reids Server (Replica) V
- 6322-->Reids Server (Replica) V
- 丛集三
- 6330-->Reids Server (Master) V
- 6331-->Reids Server (Replica) V
- 6332-->Reids Server (Replica) V
# 建立Redis丛集组态设定档
# redis_cluster_6310.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6310.pid"
logfile "/tmp/redis_cluster_6310.log"
port 6310
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6310.rdb"
appendonly yes
appendfilename "appendonly_6310.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6310.conf
cluster-node-timeout 15000
# redis_cluster_6311.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6311.pid"
logfile "/tmp/redis_cluster_6311.log"
port 6311
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6311.rdb"
appendonly yes
appendfilename "appendonly_6311.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6311.conf
cluster-node-timeout 15000
# redis_cluster_6312.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6312.pid"
logfile "/tmp/redis_cluster_6312.log"
port 6312
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6312.rdb"
appendonly yes
appendfilename "appendonly_6312.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6312.conf
cluster-node-timeout 15000
# redis_cluster_6320.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6320.pid"
logfile "/tmp/redis_cluster_6320.log"
port 6320
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6320.rdb"
appendonly yes
appendfilename "appendonly_6320.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6320.conf
cluster-node-timeout 15000
# redis_cluster_6321.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6321.pid"
logfile "/tmp/redis_cluster_6321.log"
port 6321
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6321.rdb"
appendonly yes
appendfilename "appendonly_6321.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6321.conf
cluster-node-timeout 15000
# redis_cluster_6322.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6322.pid"
logfile "/tmp/redis_cluster_6322.log"
port 6322
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6322.rdb"
appendonly yes
appendfilename "appendonly_6322.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6322.conf
cluster-node-timeout 15000
# redis_cluster_6330.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6330.pid"
logfile "/tmp/redis_cluster_6330.log"
port 6330
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6330.rdb"
appendonly yes
appendfilename "appendonly_6330.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6330.conf
cluster-node-timeout 15000
# redis_cluster_6331.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6331.pid"
logfile "/tmp/redis_cluster_6331.log"
port 6331
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6331.rdb"
appendonly yes
appendfilename "appendonly_6331.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6331.conf
cluster-node-timeout 15000
# redis_cluster_6332.conf
# 服务器组态档
bind 127.0.0.1 -::1
daemonize yes
dir "/usr/local/redis/bin"
pidfile "/var/run/redis_6332.pid"
logfile "/tmp/redis_cluster_6332.log"
port 6332
protected-mode yes
# RDB+AOF持久化设定
save 3600 1 300 100 60 10000
dbfilename "dump_6332.rdb"
appendonly yes
appendfilename "appendonly_6332.aof"
aof-use-rdb-preamble yes
# 丛集设定
cluster-enabled yes
cluster-config-file cluster_nodes_6332.conf
cluster-node-timeout 15000
# 启动三个sentinel
./redis-server redis_cluster_6310.conf
./redis-server redis_cluster_6311.conf
./redis-server redis_cluster_6312.conf
./redis-server redis_cluster_6320.conf
./redis-server redis_cluster_6321.conf
./redis-server redis_cluster_6322.conf
./redis-server redis_cluster_6330.conf
./redis-server redis_cluster_6331.conf
./redis-server redis_cluster_6332.conf
# 确认是否正常运作
ps -ef|grep redis-server
# 目前每个Redis Server还是独立的主节点
./redis-cli -h 127.0.0.1 -p 6310
127.0.0.1:6310> role
1) "master"
2) (integer) 0
3) (empty array)
./redis-cli -h 127.0.0.1 -p 6311
127.0.0.1:6311> role
1) "master"
2) (integer) 0
./redis-cli -h 127.0.0.1 -p 6312
127.0.0.1:6312> role
1) "master"
2) (integer) 0
- 将三个Master加入丛集
./redis-cli -h 127.0.0.1 -p 6310
# 加入Redis Server Master 6320
127.0.0.1:6310> cluster meet 127.0.0.1 6320
OK
# 加入Redis Server Master 6330
127.0.0.1:6310> cluster meet 127.0.0.1 6330
OK
# 确认丛集Master节点状况
127.0.0.1:6310> cluster nodes
1b9af54b8e0f609e0a066530b3f12a7852589fc3 127.0.0.1:6310@16310 myself,master - 0 1634049085000 1 connected
1307d911eb56f3079e2490dcab53b3131a0f8f81 127.0.0.1:6330@16330 master - 0 1634049087204 2 connected
7a0716a69a2e1af35eb353fadae030f6fc116864 127.0.0.1:6320@16320 master - 0 1634049086186 0 connected
# cluster_state:fail 的原因为资料分流的slot尚未分配完
127.0.0.1:6310> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:0
cluster_current_epoch:2
cluster_my_epoch:1
cluster_stats_messages_ping_sent:87
cluster_stats_messages_pong_sent:92
cluster_stats_messages_meet_sent:2
cluster_stats_messages_sent:181
cluster_stats_messages_ping_received:92
cluster_stats_messages_pong_received:89
cluster_stats_messages_received:181
- 分配资料分流的slot给各Master节点 (区间可自行分配)
- 丛集一 0 ~ 5400
- 6310-->Reids Server (Master) V
- 6311-->Reids Server (Replica)
- 6312-->Reids Server (Replica)
- 丛集二 5401 ~ 10800
- 6320-->Reids Server (Master) V
- 6321-->Reids Server (Replica)
- 6322-->Reids Server (Replica)
- 丛集三 10801 ~ 16383
- 6330-->Reids Server (Master) V
- 6331-->Reids Server (Replica)
- 6332-->Reids Server (Replica)
# 分配0 ~ 5400 slots 给6310-->Reids Server (Master)
./redis-cli -h 127.0.0.1 -p 6310 cluster addslots {0..5400}
OK
# 分配5400 ~ 10800 slots 给6320-->Reids Server (Master)
./redis-cli -h 127.0.0.1 -p 6320 cluster addslots {5401..10800}
OK
# 分配10801 ~ 16383 slots 给6330-->Reids Server (Master)
./redis-cli -h 127.0.0.1 -p 6330 cluster addslots {10801..16383}
OK
# 再次确认丛集的状态
./redis-cli -h 127.0.0.1 -p 6310
127.0.0.1:6310> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:2
cluster_my_epoch:1
cluster_stats_messages_ping_sent:711
cluster_stats_messages_pong_sent:699
cluster_stats_messages_meet_sent:2
cluster_stats_messages_sent:1412
cluster_stats_messages_ping_received:699
cluster_stats_messages_pong_received:713
cluster_stats_messages_received:1412
# 确认slots分配是否如预期
127.0.0.1:6310> cluster slots
1) 1) (integer) 0
2) (integer) 5400
3) 1) "127.0.0.1"
2) (integer) 6310
3) "1b9af54b8e0f609e0a066530b3f12a7852589fc3"
2) 1) (integer) 5401
2) (integer) 10800
3) 1) "127.0.0.1"
2) (integer) 6320
3) "7a0716a69a2e1af35eb353fadae030f6fc116864"
3) 1) (integer) 10801
2) (integer) 16383
3) 1) "127.0.0.1"
2) (integer) 6330
3) "1307d911eb56f3079e2490dcab53b3131a0f8f81"
- 将Master加入Replica增加高可用机制
- 丛集一 0 ~ 5400
- 6310-->Reids Server (Master)
- 6311-->Reids Server (Replica) V
- 6312-->Reids Server (Replica) V
- 丛集二 5401 ~ 10800
- 6320-->Reids Server (Master)
- 6321-->Reids Server (Replica) V
- 6322-->Reids Server (Replica) V
- 丛集三 10801 ~ 16383
- 6330-->Reids Server (Master)
- 6331-->Reids Server (Replica) V
- 6332-->Reids Server (Replica) V
# 取的个主节点的master node id
./redis-cli -h 127.0.0.1 -p 6310
127.0.0.1:6310> cluster nodes
1b9af54b8e0f609e0a066530b3f12a7852589fc3 127.0.0.1:6310@16310 myself,master - 0 1634050264000 1 connected 0-5400
1307d911eb56f3079e2490dcab53b3131a0f8f81 127.0.0.1:6330@16330 master - 0 1634050265828 2 connected 10801-16383
7a0716a69a2e1af35eb353fadae030f6fc116864 127.0.0.1:6320@16320 master - 0 1634050266837 0 connected 5401-10800
# 6310-->Reids Server (Master)
# master node id => 1b9af54b8e0f609e0a066530b3f12a7852589fc3
# 将Reids Server 6311 6312 加入丛集
./redis-cli -h 127.0.0.1 -p 6310
127.0.0.1:6310> cluster meet 127.0.0.1 6311
OK
127.0.0.1:6310> cluster meet 127.0.0.1 6312
OK
# 将Reids Server 6311 成为 6310 的 Replica
./redis-cli -h 127.0.0.1 -p 6311
127.0.0.1:6311> cluster replicate 1b9af54b8e0f609e0a066530b3f12a7852589fc3
OK
127.0.0.1:6311> exit
# 将Reids Server 6312 成为 6310 的 Replica
./redis-cli -h 127.0.0.1 -p 6312
127.0.0.1:6312> cluster replicate 1b9af54b8e0f609e0a066530b3f12a7852589fc3
OK
127.0.0.1:6312> exit
# 6320-->Reids Server (Master)
# master node id => 7a0716a69a2e1af35eb353fadae030f6fc116864
# 将Reids Server 6321 6322 加入丛集
./redis-cli -h 127.0.0.1 -p 6320
127.0.0.1:6320> cluster meet 127.0.0.1 6321
OK
127.0.0.1:6320> cluster meet 127.0.0.1 6322
OK
# 将Reids Server 6321 成为 6320 的 Replica
./redis-cli -h 127.0.0.1 -p 6321
127.0.0.1:6321> cluster replicate 7a0716a69a2e1af35eb353fadae030f6fc116864
OK
127.0.0.1:6321> exit
# 将Reids Server 6322 成为 6320 的 Replica
./redis-cli -h 127.0.0.1 -p 6322
127.0.0.1:6322> cluster replicate 7a0716a69a2e1af35eb353fadae030f6fc116864
OK
127.0.0.1:6322> exit
# 6330-->Reids Server (Master)
# master node id => 1307d911eb56f3079e2490dcab53b3131a0f8f81
# 将Reids Server 6323 6332 加入丛集
./redis-cli -h 127.0.0.1 -p 6330
127.0.0.1:6330> cluster meet 127.0.0.1 6331
OK
127.0.0.1:6330> cluster meet 127.0.0.1 6332
OK
# 将Reids Server 6331 成为 6330 的 Replica
./redis-cli -h 127.0.0.1 -p 6331
127.0.0.1:6331> cluster replicate 1307d911eb56f3079e2490dcab53b3131a0f8f81
OK
127.0.0.1:6331> exit
# 将Reids Server 6332 成为 6330 的 Replica
./redis-cli -h 127.0.0.1 -p 6332
127.0.0.1:6332> cluster replicate 1307d911eb56f3079e2490dcab53b3131a0f8f81
OK
127.0.0.1:6332> exit
- 确认Redis丛集配置
./redis-cli -h 127.0.0.1 -p 6310
127.0.0.1:6310> cluster nodes
d20a8414e90365895a2432ad98d35b42d02228c1 127.0.0.1:6322@16322 slave 7a0716a69a2e1af35eb353fadae030f6fc116864 0 1634051854937 3 connected
d27292294e3880171277b21e1488b454e9d0d716 127.0.0.1:6332@16332 slave 1307d911eb56f3079e2490dcab53b3131a0f8f81 0 1634051854000 2 connected
1307d911eb56f3079e2490dcab53b3131a0f8f81 127.0.0.1:6330@16330 master - 0 1634051853000 2 connected 10801-16383
48b1b701f528817c932ef7cc17b899c428284d88 127.0.0.1:6321@16321 slave 1307d911eb56f3079e2490dcab53b3131a0f8f81 0 1634051852000 2 connected
7a0716a69a2e1af35eb353fadae030f6fc116864 127.0.0.1:6320@16320 master - 0 1634051854000 3 connected 5401-10800
82013de2c546919a43c9b6597a2b6035bb1ac930 127.0.0.1:6312@16312 slave 1b9af54b8e0f609e0a066530b3f12a7852589fc3 0 1634051854000 1 connected
e3b71c1c0414ea6ec90b5d37a0ff2c963e2a93ec 127.0.0.1:6331@16331 slave 1307d911eb56f3079e2490dcab53b3131a0f8f81 0 1634051857991 2 connected
eca9f94125a621846fe9c5e92d60003539f0a512 127.0.0.1:6311@16311 slave 1b9af54b8e0f609e0a066530b3f12a7852589fc3 0 1634051856973 1 connected
1b9af54b8e0f609e0a066530b3f12a7852589fc3 127.0.0.1:6310@16310 myself,master - 0 1634051853000 1 connected 0-5400
<<: 卡夫卡的藏书阁【Book28】- Kafka - MirroMaker
DAY2 练习文件内容
以下就是我请学长帮我写的练习专案内容 一、功能列表(Flex 按钮) 1.注册:存取使用者姓名和组别...
D29-(9/29)-广达(2382)-有肉松之称的电脑代工厂
注:发文日和截图的日期不一定是同一天，所以价格计算上和当日不同，是很正常的。 声明:这一系列文章并无...
Day 29: Tensorflow分类 分类图像衣物 (四)
Tensorflow 衣物图像分类 (四) 辅助阅读: Basic classification: ...
[Day 28] Final Project (4/5) — 部署模型到 Google AI Platform
前情提要 昨天我们成功的让 App 在本机端运作，但按下 开始预测! 後却出现了错误： 这意味着虽然...
码农最後的哄擡价格，高级操作：说出一口聚合分析（上？！）
dog.com来源 这就是现在目前这个业界奇怪的地方 面试时手撕tomcat，要你喷出一堆莫名奇妙...