docker叢集管理之swarm mode
當有多臺物理機的時候,就要考慮使用叢集的模式了,那麼docker如何來使用叢集來進行管理呢?在這裡主要使用的是docker自帶的swarm mode,也就是docker叢集的管理和編排。所謂的編排就是指多臺叢集的管理,主機的配置,容器的排程等。
swarm mode是docker engine中自帶的一種模式,很容易使用,並且無須安裝其他的軟體。
swarm mode的使用在使用swarm mode的時候,幾臺主機上都要先安裝好docker,架構如下所示:
將主機名為docker-ce的機器作為manager節點,也就是管理節點,而docker1和docker2作為工作節點。
1、 建立swarm叢集
[[email protected] swarm]# docker swarm init --advertise-addr 192.168.1.222 (初始化叢集,節點之間相互通訊的ip地址為192.168.1.222,預設埠為2377)
Swarm initialized: current node (pk4p936t4e03cpse3izuws07s) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-60h71geyd7z297jfy2icektmq3ha3n5nego2znytgrzqix768e-f36psbhrnrdn9h0bop6np22xm 192.168.1.222:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
[[email protected] swarm]# docker info|grep -i swarm(swarm模式已經啟用)
Swarm: active
[[email protected] swarm]# netstat -tnlp|grep docker(預設監聽兩個埠,tcp2377埠為叢集的管理埠,tcp7946為節點之間的通訊埠)
tcp6 0 0 :::2377 :::* LISTEN 66488/dockerd
tcp6 0 0 :::7946 :::* LISTEN 66488/dockerd
[[email protected] swarm]# docker network ls(預設會建立一個overlay的網路ingress,還會建立一個橋接的網路docker_gwbridge)
NETWORK ID NAME DRIVER SCOPE
641eeb86f6a4 bridge bridge local
c23afa61afaa docker_gwbridge bridge local
65f6eed9f144 host host local
n8i6cpizzlww ingress overlay swarm
b4d6492a85d5 none null local
[[email protected] swarm]# docker node ls(檢視叢集中的節點,當有多個manager節點的時候,是通過raft協議來選取主節點,也就是leader節點)
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
pk4p936t4e03cpse3izuws07s * docker-ce Ready Active Leader
[[email protected] swarm]# ls -l(swarm的配置檔案都在/var/lib/docker/swarm目錄中,會有相關的證書和manager的配置檔案,使用的是raft協議)
total 8
drwxr-xr-x. 2 root root 75 Jan 26 10:13 certificates(使用的tls來進行安全通訊)
-rw-------. 1 root root 151 Jan 26 10:13 docker-state.json(用來記錄通訊的地址和埠,也會記錄本地的地址和埠)
drwx------. 4 root root 55 Jan 26 10:13 raft(raft協議)
-rw-------. 1 root root 69 Jan 26 10:13 state.json(manager的ip和埠)
drwxr-xr-x. 2 root root 22 Jan 26 10:13 worker(記錄工作節點下發的任務資訊)
[[email protected] ~]# docker swarm join --token SWMTKN-1-60h71geyd7z297jfy2icektmq3ha3n5nego2znytgrzqix768e-f36psbhrnrdn9h0bop6np22xm 192.168.1.222:2377(其他的機器加入swarm叢集)
This node joined a swarm as a worker.
當忘記了加入叢集的token的時候,可以使用如下的指令找到token,然後在node節點上直接執行,就可以加入worker節點或者是manager節點。
檢視叢集如下:
節點之間的角色可以隨時進行變換(使用update進行更新):
2、 開放防火牆
在各個節點進行通訊的時候,必須開放相關的防火牆策略,其中包括通訊的tcp的2377埠,tcp和udp的7946埠,還有網路overlay的udp埠4789埠。
[[email protected] ~]# firewall-cmd --add-port tcp/2377 --permanent
[[email protected] ~]# firewall-cmd --add-port tcp/7946 --permanent
[[email protected] ~]# firewall-cmd --add-port udp/7946 --permanent
[[email protected] ~]# firewall-cmd --add-port udp/4789--permanent
[[email protected] ~]# systemctl restart firewalld
3、 執行服務
服務為service,也就是一組task的集合,而一個task則表示為一個容器,從基本概念來說,執行一個service,可能有幾個task,例如執行幾個nginx的服務,從而會拆解為幾個nginx的容器在各個節點上進行執行。
[[email protected] ~]# docker service create --name web nginx(create表示建立一個服務,名稱為web,映象為nginx)
oy2y8sb31c2jpn9owk6gdt7nk
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service create --name frontweb --mode global nginx(建立一個名稱問frontweb的服務,模式為global,映象為nginx)
ld835zsd9x1x4rdaj6u1i1rfy
overall progress: 3 out of 3 tasks
pk4p936t4e03: running [==================================================>]
xvkxa7z22v75: running [==================================================>]
6xum2o1iqmya: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service ls(檢視執行的服務)
ID NAME MODE REPLICAS IMAGE PORTS
ld835zsd9x1x frontweb global 3/3 nginx:latest
oy2y8sb31c2j web replicated 1/1 nginx:latest
[[email protected] ~]# docker service ps web(檢視執行的詳細資訊,預設情況下manager節點也可以執行容器)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
li2bfdt1dfjs web.1 nginx:latest docker-ce Running Running 13 minutes ago
[[email protected] ~]# docker service ps frontweb(檢視執行的詳細資訊)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
s96twac1s4av frontweb.6xum2o1iqmyaun2khb4b5z57h nginx:latest docker2 Running Running 34 seconds ago
qtr35ehwuu26 frontweb.xvkxa7z22v757jnptndvtcc4t nginx:latest docker1 Running Running 37 seconds ago
jujtu01q49o2 frontweb.pk4p936t4e03cpse3izuws07s nginx:latest docker-ce Running Running 55 seconds ago
在建立服務的時候,會經過幾個狀態,一個是prepared,表示準備,主要是從倉庫拉取映象,然後啟動容器,也就是starting,最後會進行驗證容器狀態,從而最後變成running狀態。
在檢視服務的時候,會出現一個mode,也就是服務的型別,可以分為兩種,一種replicated,表示副本,預設情況下是使用replicated模式,並且預設情況只會建立一個副本,主要使用的目的是為了高可用;另外一種為global,也就是必須在每個機器上執行一個task也就是容器,可以看到在使用global的模式的時候建立了三個容器。
4、服務的擴縮容
在使用服務的時候,由於是叢集,那麼就必然會涉及到高可用,從而會有服務的擴容和縮容,在swarm中還是很容易的。
[[email protected] ~]# docker service scale web=3(擴容為3,也就是執行三個容器)
web scaled to 3
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service ls(檢視服務,可以看到replicas副本數量為3個)
ID NAME MODE REPLICAS IMAGE PORTS
oy2y8sb31c2j web replicated 3/3 nginx:latest
[[email protected] ~]# docker service ps web(可以看到三個節點上都運行了每個都各運行了一個容器task)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
li2bfdt1dfjs web.1 nginx:latest docker-ce Running Running 25 minutes ago
8dsrshssyd6t web.2 nginx:latest docker2 Running Running 46 seconds ago
4i7vgzspdpts web.3 nginx:latest docker1 Running Running 46 seconds ago
在預設情況下,管理的機器也是可以執行容器的,從而在manager節點上也執行一個容器。
[[email protected] ~]# docker service scale web=2(將web服務縮容為2個)
web scaled to 2
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service ps web(檢視執行的容器)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
4i7vgzspdpts web.3 nginx:latest docker1 Running Running 9 minutes ago
56s441jtydq4 web.5 nginx:latest docker-ce Running Running about a minute ago
當要讓swarm的manager節點不執行容器的時候,只要更改節點的狀態,從Active變成Drain即可,如果在manager上執行容器,那麼當manager宕機的時候,如果不是多節點的manager,會導致此服務無法進行排程。
[[email protected] ~]# docker node update --availability drain docker-ce(將manager節點的狀態修改為drain狀態,從而不會執行相關的task任務)
docker-ce
[[email protected] ~]# docker node ls(檢視節點的狀態從active變成drain)
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
xvkxa7z22v757jnptndvtcc4t docker1 Ready Active
6xum2o1iqmyaun2khb4b5z57h docker2 Ready Active
pk4p936t4e03cpse3izuws07s * docker-ce Ready Drain Leader
[[email protected] ~]# docker service ps web(本來執行在docker-ce上的容器會被關閉,然後自動遷移到其他的worker節點上)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
4i7vgzspdpts web.3 nginx:latest docker1 Running Running 12 minutes ago
x2w8qdxuv2y5 web.5 nginx:latest docker2 Running Running about a minute ago
56s441jtydq4 \_ web.5 nginx:latest docker-ce Shutdown Shutdown about a minute ago
5、 自動故障轉移
在叢集中,當有機器發生宕機了咋辦,swarm可以做到自動遷移,但是在生產環境中需要考慮的一個問題就是,如果出現了自動的failover,那麼其他的機器是否有足夠的資源來建立這些容器,所以,在進行執行容器的時候,就要考慮剩餘資源的問題,例如cpu和記憶體。
[[email protected] ~]# docker service ps web(檢視服務的分佈)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
nrkfvb2js6p1 web.1 nginx:latest docker1 Running Running 10 seconds ago
q17kbbwd3ewr web.2 nginx:latest docker2 Running Running 10 seconds ago
yvpijmfr4qrm web.3 nginx:latest docker2 Running Running 10 seconds ago
[[email protected] ~]# systemctl stop docker(關閉docker服務,模擬機器宕機)
[[email protected] ~]# docker node ls(檢視node的資訊,發現docker2主機標記為down,也就是主機宕機)
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
xvkxa7z22v757jnptndvtcc4t docker1 Ready Active
6xum2o1iqmyaun2khb4b5z57h docker2 Down Active
pk4p936t4e03cpse3izuws07s * docker-ce Ready Drain Leader
[[email protected] ~]# docker service ps web(檢視自動遷移後的服務,會將宕機的服務全部標記為shutdown關閉)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
nrkfvb2js6p1 web.1 nginx:latest docker1 Running Running about a minute ago
i76d3p7bmdht web.2 nginx:latest docker1 Running Running 28 seconds ago
q17kbbwd3ewr \_ web.2 nginx:latest docker2 Shutdown Running 50 seconds ago
ohg093dh9zvt web.3 nginx:latest docker1 Running Running 28 seconds ago
yvpijmfr4qrm \_ web.3 nginx:latest docker2 Shutdown Running 50 seconds ago
[[email protected] ~]# systemctl start docker(將主機進行上線)
[[email protected] ~]# docker node ls(主機狀態變成ready,表示可執行task任務)
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
xvkxa7z22v757jnptndvtcc4t docker1 Ready Active
6xum2o1iqmyaun2khb4b5z57h docker2 Ready Active
pk4p936t4e03cpse3izuws07s * docker-ce Ready Drain Leader
[[email protected] ~]# docker service ps web(再次檢視服務的分佈,服務不會再次進行遷移到不同的機器上,維持原狀)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
nrkfvb2js6p1 web.1 nginx:latest docker1 Running Running 5 minutes ago
i76d3p7bmdht web.2 nginx:latest docker1 Running Running 4 minutes ago
q17kbbwd3ewr \_ web.2 nginx:latest docker2 Shutdown Shutdown about a minute ago
ohg093dh9zvt web.3 nginx:latest docker1 Running Running 4 minutes ago
yvpijmfr4qrm \_ web.3 nginx:latest docker2 Shutdown Shutdown about a minute ago
6、 訪問服務
訪問服務的時候,主要分為兩種,一種是內部訪問的服務,也就是不對外開放埠,一種是對外的服務,會向外開放埠,也就是主機對映埠。
[[email protected] ~]# docker service create --name web --replicas=2 httpd(建立副本為2的httpd服務)
60c9i7de4mu4x9n3ia03nrh52
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service ps web(檢視執行容器的主機)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
wb56gj6zbtlw web.1 httpd:latest docker2 Running Running 18 seconds ago
n6hupzwlchss web.2 httpd:latest docker1 Running Running 18 seconds ago
[[email protected] ~]# docker ps (登入本地主機,檢視執行的容器)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
19a6ed6ec945 httpd:latest "httpd-foreground" 23 seconds ago Up 22 seconds 80/tcp web.2.n6hupzwlchss01fbcec6jsp27
[[email protected] ~]# docker exec 19a6ed6ec945 ip addr show(檢視執行容器的ip地址)
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
201: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
[[email protected] ~]# curl 172.17.0.2(根據ip地址訪問,只能在本節點上進行訪問,屬於內部網路,也就是docker_gwbrige網路)
<html><body><h1>It works!</h1></body></html>
[[email protected] ~]# docker service update --publish-add 8000:80 web
web(新增主機對映埠,從而將服務對外放開,從而外部能訪問此服務)
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service ps web(將原來的容器關閉,重新執行容器)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
pktheo7u3178 web.1 httpd:latest docker2 Running Running 11 seconds ago
wb56gj6zbtlw \_ web.1 httpd:latest docker2 Shutdown Shutdown 13 seconds ago
j5mrcqlozlhz web.2 httpd:latest docker1 Running Running 16 seconds ago
n6hupzwlchss \_ web.2 httpd:latest docker1 Shutdown Shutdown 20 seconds ago
[[email protected] ~]# curl 192.168.1.222:8000(無論是manager節點還是worker節點,均監聽了8000埠,均可以訪問)
<html><body><h1>It works!</h1></body></html>
[[email protected] ~]# curl 192.168.1.32:8000
<html><body><h1>It works!</h1></body></html>
[[email protected] ~]# curl 192.168.1.33:8000
<html><body><h1>It works!</h1></body></html>
[[email protected] ~]# netstat -ntlp|grep 8000(叢集中的每個機器都會監聽8000埠,無論是否運行了這個容器)
tcp6 0 0 :::8000 :::* LISTEN 66488/dockerd
在這裡主要使用了routing mesh的功能,在swarm內部實現了負載均衡,使用的網路為swarm自動建立的overlay網路。
當使用publish埠的時候,最大的壞處就是對外暴露了埠號,而且在使用的時候,如果進行了故障轉移,那麼多個服務執行在同一個主機上面,會造成端口占用衝突。
7、 服務發現
在使用叢集的時候,如果進行了自動轉移,那麼ip地址會發生變化,如果指定了ip地址,那麼就會影響其他服務的使用,從而需要服務發現的功能,也就是自動將服務進行dns的解析,然後負載到正確的服務中。
[[email protected] ~]# docker network create --driver overlay kel(建立overlay網路,預設的ingress網路未實現服務發現的功能)
nomp3f50to1s4gn6ke4zpxn8n
[[email protected] ~]# docker service create --name web --replicas=2 --network=kel nginx(將服務掛載到自建立的overlay網路中,從而能實現名稱解析)
uo5kuxad0wojn5j24moudi4l5
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service create --name ds --network=kel busybox sleep 100000(建立另外一個服務依賴於其他的服務)
hwcn4esjpgqyae7r9bqcn8a9m
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service ps ds(檢視執行的服務)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
7gknpd69eti1 ds.1 busybox:latest docker2 Running Running 12 seconds ago
[[email protected] ~]# docker exec 66ece5551fc5 ping -c 2 web(自動實現了dns的解析功能)
PING web (10.0.0.5): 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=0.833 ms
64 bytes from 10.0.0.5: seq=1 ttl=64 time=0.323 ms
--- web ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.323/0.578/0.833 ms
[[email protected] ~]# docker exec 66ece5551fc5 nslookup web(檢視vip地址)
Server: 127.0.0.11
Address 1: 127.0.0.11
Name: web
Address 1: 10.0.0.5
[[email protected] ~]# docker exec 66ece5551fc5 nslookup tasks.web(檢視各個主機的ip地址)
Server: 127.0.0.11
Address 1: 127.0.0.11
Name: tasks.web
Address 1: 10.0.0.7 b2f14e41a97a.kel
Address 2: 10.0.0.6 web.1.78a2eugjholl3c35xa361kdaj.kel
8、滾動更新rolling update
當需要進行更新的時候,swarm可以設定相應的策略來進行更新,例如並行更新多少個容器,更新之間的間隔時間等。
[[email protected] ~]# docker service update --image nginx:1.10 web(更新web的映象)
web
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service ps web(更新的時候先關閉,然後再更新)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
512sgoen7yu9 web.1 nginx:1.10 docker1 Running Running 6 minutes ago
rz9thj8s1iv6 \_ web.1 nginx:1.9 docker1 Shutdown Shutdown 6 minutes ago
5hsl7q2486iz web.2 nginx:1.10 docker2 Running Running 6 minutes ago
czgdnq3lmhgu \_ web.2 nginx:1.9 docker2 Shutdown Shutdown 7 minutes ago
[[email protected] ~]# docker service update --image nginx:1.11 web
web
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service update --update-parallelism 2 --update-delay 1m web(更新的時候,設定更新的併發數和更新的間隔時間)
web
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
[[email protected] ~]# docker service inspect web --pretty(檢視具體的資訊,可以看到更新的一些引數配置)
ID: cxsk18saisqsycc9bn1m768xx
Name: web
Service Mode: Replicated
Replicas: 2
Placement:
UpdateConfig:
Parallelism: 2
Delay: 1m0s
On failure: pause
Monitoring Period: 5s
Max failure ratio: 0
Update order: stop-first
RollbackConfig:
Parallelism: 1
On failure: pause
Monitoring Period: 5s
Max failure ratio: 0
Rollback order: stop-first
ContainerSpec:
Image: nginx:[email protected]:e6693c20186f837fc393390135d8a598a96a833917917789d63766cab6c59582
Resources:
Endpoint Mode: vip
9、label控制service執行的節點
主要是用來控制每個task執行的節點,從而可以為每個節點設定屬性label,然後根據label來指定task執行的位置(生產環境中可能要手動將容器分佈在不同的機器上,從而達到高可用的目的)。
[[email protected] ~]# docker node update --label-add ncname=docker1 docker1(更新節點的屬性,為節點新增標籤)
docker1
[[email protected] ~]# docker node update --label-add ncname=docker2 docker2(更新節點的屬性,為節點新增標籤)
docker2
[[email protected] ~]# docker node inspect docker1 --pretty(查看設定的屬性)
ID: xvkxa7z22v757jnptndvtcc4t
Labels:
- ncname=docker1
[[email protected] ~]# docker service create --name web --constraint node.labels.ncname==docker1 --replicas 2 nginx(指定機器來執行相關的任務,主要是根據label的值)
m0126tvjof5owsn9crke00w63
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged