用 Consul 和 Traefik 实现 Docker 容器的服务注册与发现

docker 实现应用的容器化
consul 集群实现服务的注册、发现
traefik 处理外部流量的负载均衡与路由

启动 consul 集群与 docker

通过 vagrant 起三台虚拟机实现基本的 consul 集群环境（为了节约资源把 docker 也运行在这上面了）。
consul 的 vagrant 配置文件如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|

$script = <<SCRIPT

echo "Installing"

yum install -y wget
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum clean all
yum makecache

yum install -y jq unzip vim wget net-tools bind-utils dnsmasq

sudo cp /vagrant/consul /usr/bin/consul

echo "Installing docker.." 
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io
sudo systemctl start docker

echo "success" 

SCRIPT
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "centos/7"
  config.vm.provision "shell", inline: $script

  config.vm.define "node1" do |node1|
    node1.vm.hostname = "node1"
    node1.vm.network "private_network", ip: "172.17.17.11"

  end

  config.vm.define "node2" do |node2|
    node2.vm.hostname = "node2"
    node2.vm.network "private_network", ip: "172.17.17.12"
  end

  config.vm.define "node3" do |node3|
    node3.vm.hostname = "node3"
    node3.vm.network "private_network", ip: "172.17.17.13"
  end
  # Disable automatic box update checking. If you disable this, then
  # boxes will only be checked for updates when the user runs
  # `vagrant box outdated`. This is not recommended.
  config.vm.box_check_update = false

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine. In the example below,
  # accessing "localhost:8080" will access port 80 on the guest machine.
  # NOTE: This will enable public access to the opened port
  # config.vm.network "forwarded_port", guest: 80, host: 8080

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine and only allow access
  # via 127.0.0.1 to disable public access
  # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  # config.vm.network "private_network", ip: "192.168.33.10"

  # Create a public network, which generally matched to bridged network.
  # Bridged networks make the machine appear as another physical device on
  # your network.
  # config.vm.network "public_network"

  # Share an additional folder to the guest VM. The first argument is
  # the path on the host to the actual folder. The second argument is
  # the path on the guest to mount the folder. And the optional third
  # argument is a set of non-required options.
  # config.vm.synced_folder "../data", "/vagrant_data"

  # Provider-specific configuration so you can fine-tune various
  # backing providers for Vagrant. These expose provider-specific options.
  # Example for VirtualBox:
  #
   config.vm.provider "virtualbox" do |vb|
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #
  #   # Customize the amount of memory on the VM:
     vb.memory = "1024"
   end
  #
  # View the documentation for the provider you are using for more
  # information on available options.

  # Enable provisioning with a shell script. Additional provisioners such as
  # Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
  # documentation for more information about their specific syntax and use.
  # config.vm.provision "shell", inline: <<-SHELL
  #   apt-get update
  #   apt-get install -y apache2
  # SHELL
end

为了节省时间我直接下载好了 consul 的可执行文件放到了 vagrant 配置文件同目录下，vagrant 会把当前目录下的文件都复制进虚拟机的 /vagrant 目录下，还有三个节点的 consul 配置文件。

node1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "datacenter":"dc1",
    "primary_datacenter":"dc1",
    "bootstrap_expect":3,
    "advertise_addr": "172.17.17.11",
    "bind_addr": "172.17.17.11",
    "client_addr":"0.0.0.0",
    "server":true,
    "node_name":"node1",
    "ui":true,
    "data_dir":"/opt/consul",
    "enable_script_checks":true,
    "enable_local_script_checks":true,
    "log_file":"/opt/consul/",
    "log_level":"info",
    "log_rotate_duration":"24h"
}

node2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
    "datacenter":"dc1",
    "primary_datacenter":"dc1",
    "advertise_addr": "172.17.17.12",
    "bind_addr": "172.17.17.12",
    "client_addr":"0.0.0.0",
    "server":true,
    "node_name":"node2",
    "ui":true,
    "data_dir":"/opt/consul",
    "enable_script_checks":true,
    "enable_local_script_checks":true,
    "log_file":"/opt/consul/",
    "log_level":"info",
    "log_rotate_duration":"24h",
    "start_join":[
        "172.17.17.11"
    ],
    "retry_join":[
        "172.17.17.11"
    ]
}

node3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
    "datacenter":"dc1",
    "primary_datacenter":"dc1",
    "advertise_addr": "172.17.17.13",
    "bind_addr": "172.17.17.13",
    "client_addr":"0.0.0.0",
    "server":true,
    "node_name":"node3",
    "ui":true,
    "data_dir":"/opt/consul",
    "enable_script_checks":true,
    "enable_local_script_checks":true,
    "log_file":"/opt/consul/",
    "log_level":"info",
    "log_rotate_duration":"24h",
    "start_join":[
        "172.17.17.11"
    ],
    "retry_join":[
        "172.17.17.11"
    ]

}

三个节点分别是

  • node1 172.17.17.11
  • node2 172.17.17.12
  • node3 172.17.17.13

运行 vagrant up --provider=virtualbox 把三个节点启动。

分别进入三个节点先 sudo su - 切换成 root 用户，然后把 consul 起来，再 exit 退出就会后台运行了。

1
2
3
[root@node1 ~]# consul agent -config-file /vagrant/node1.json &
[root@node2 ~]# consul agent -config-file /vagrant/node2.json &
[root@node3 ~]# consul agent -config-file /vagrant/node3.json &

在三个节点上分别通过 docker 运行 web 服务，对外端口是 32768

1
[root@node1 ~]# docker run -d -p 32768:80 --name test containous/whoami

服务注册

在三个节点上分别把刚才的 web 服务注册到 consul，这一步在实际项目中可以写到 docker image build 中，然后在容器启动的时候就注册到 consul。

1
[root@node1 ~]# curl --request PUT --data @web.json http://127.0.0.1:8500/v1/agent/service/register?replace-existing-checks=true

web.json 内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
  "ID": "web",
  "Name": "consul_web",
  "Tags": [
    "whoami"
  ],
  "Check": {
    "Args": ["curl", "172.17.17.11:32768"],
    "Interval": "10s",
    "Timeout": "3s"
  },
  "Address": "172.17.17.11",
  "Port": 32768
}

其中增加了简单的健康检查, ip 部分每个节点修改成自己的。

现在通过访问 http://172.17.17.11:8500/ 可以看到如下的信息

随便找一个节点通过 consul 自带的 dns 查询下服务，可以看到返回了所有健康的服务节点。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@node1 ~]# dig @127.0.0.1 -p 8600 consul_web.service.consul

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> @127.0.0.1 -p 8600 consul_web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11034
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul_web.service.consul.	IN	A

;; ANSWER SECTION:
consul_web.service.consul. 0	IN	A	172.17.17.11
consul_web.service.consul. 0	IN	A	172.17.17.13
consul_web.service.consul. 0	IN	A	172.17.17.12

;; ADDITIONAL SECTION:
consul_web.service.consul. 0	IN	TXT	"consul-network-segment="
consul_web.service.consul. 0	IN	TXT	"consul-network-segment="
consul_web.service.consul. 0	IN	TXT	"consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Feb 06 13:10:12 UTC 2020
;; MSG SIZE  rcvd: 210

服务发现 dns

通过 dnsmasq 将节点的 dns 修改为 consul 的，先创建配置文件 /etc/dnsmasq.d/10-consul 内容为

1
2
server=/consul/127.0.0.1#8600
server=223.5.5.5

然后运行 systemctl start dnsmasq 启动 dnsmasq，再把 /etc/resolv.conf dns 服务器地址改成 127.0.0.1。

通过 curl 访问

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@node1 ~]# curl http://consul_web.service.consul:32768 -i
HTTP/1.1 200 OK
Date: Thu, 06 Feb 2020 13:19:43 GMT
Content-Length: 178
Content-Type: text/plain; charset=utf-8

Hostname: 2cc06cae3b6e
IP: 127.0.0.1
IP: 172.18.0.2
RemoteAddr: 172.17.17.11:54108
GET / HTTP/1.1
Host: consul_web.service.consul:32768
User-Agent: curl/7.29.0
Accept: */*

把 docker 中 dns 配置修改为当前节点的，配置文件 /etc/docker/daemon.json

1
2
3
4
{
    "dns": ["172.17.17.11"],
    "dns-search": ["service.consul"],
}

这样在 docker 容器内部就可以通过 consul 的 dns 来进行服务发现了。现在服务的注册与内部的服务发现都已经完成了。

traefik

对于外部流量的进入与路由使用 Traefik 来管理。

起一台跑 traefik 的 vagrant 配置文件如下，同样下载了 traefik 的可执行文件到当前目录。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|

$script = <<SCRIPT

yum install -y wget
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum clean all
yum makecache

yum install -y jq unzip vim wget net-tools bind-utils dnsmasq

sudo cp /vagrant/consul /usr/bin/consul
sudo cp /vagrant/traefik /usr/bin/traefik

echo "success"

SCRIPT
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "centos/7"
  config.vm.provision "shell", inline: $script

  config.vm.define "traefik" do |traefik|
      traefik.vm.hostname = "traefik"
      traefik.vm.network "private_network", ip: "172.17.17.21"
  end

  # Disable automatic box update checking. If you disable this, then
  # boxes will only be checked for updates when the user runs
  # `vagrant box outdated`. This is not recommended.
  config.vm.box_check_update = false

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine. In the example below,
  # accessing "localhost:8080" will access port 80 on the guest machine.
  # NOTE: This will enable public access to the opened port
  # config.vm.network "forwarded_port", guest: 80, host: 8080

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine and only allow access
  # via 127.0.0.1 to disable public access
  # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  # config.vm.network "private_network", ip: "192.168.33.10"

  # Create a public network, which generally matched to bridged network.
  # Bridged networks make the machine appear as another physical device on
  # your network.
  # config.vm.network "public_network"

  # Share an additional folder to the guest VM. The first argument is
  # the path on the host to the actual folder. The second argument is
  # the path on the guest to mount the folder. And the optional third
  # argument is a set of non-required options.
  # config.vm.synced_folder "../data", "/vagrant_data"

  # Provider-specific configuration so you can fine-tune various
  # backing providers for Vagrant. These expose provider-specific options.
  # Example for VirtualBox:
  #
   config.vm.provider "virtualbox" do |vb|
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #
  #   # Customize the amount of memory on the VM:
     vb.memory = "1024"
   end
  #
  # View the documentation for the provider you are using for more
  # information on available options.

  # Enable provisioning with a shell script. Additional provisioners such as
  # Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
  # documentation for more information about their specific syntax and use.
  # config.vm.provision "shell", inline: <<-SHELL
  #   apt-get update
  #   apt-get install -y apache2
  # SHELL
end

节点为

  • traefik 172.17.17.21

在 traefik 上运行一个 client 模式的 consul 的 agent，加入 consul 的集群

1
[root@traefik ~]# consul agent -data-dir /opt/consul -node=agent_traefik -bind=172.17.17.21 -join 172.17.17.11 &

然后运行 traefik

1
[root@traefik ~]# traefik --configFile=/vagrant/config/traefik.toml

配置文件 /vagrant/config/traefik.toml 内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[global]
  checkNewVersion = true
  sendAnonymousUsage = false

[api]
  insecure = true
  debug = true
  dashboard = true

[log]
  filePath = "/vagrant/logs/traefik.log"
  level = "DEBUG"
#  format = "json"
[accessLog]
  filePath = "/vagrant/logs/access.log"

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.web-secure]
    address = ":443"

#[certificatesResolvers.sample.acme]
#  email = ""
#  storage = "acme.json"
#  [certificatesResolvers.sample.acme.httpChallenge]
#    entryPoint = "web"
#  caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
#  [certificatesResolvers.sample.acme.tlsChallenge]

[providers]
  [providers.consulCatalog]
    exposedByDefault = false
    refreshInterval = "5s"
  [providers.file]
    filename = "/vagrant/config/static.toml"
    watch = true

配置文件 /vagrant/config/static.toml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# 强制所有 http 请求转 https
#[http.routers.http-catchall]
#  rule = "hostregexp(`{host:.+}`)"
#  entrypoints = ["web"]
#  service = "api@internal"
#  middlewares = ["toHttps"]

# https 路由
#[http.routers.my-api]
#  rule = "Host(`traefik.haozy.com`)"
#  service = "api@internal"
#  middlewares = ["auth"]
#  [http.routers.my-api.tls]
#    certResolver = "sample"
#    [[http.routers.my-api.tls.domains]]
#      main = "traefik.haozy.com"

#[http.middlewares.toHttps.redirectScheme]
#    scheme = "https"

# http 路由
[http.routers.my-api-http]
  rule = "Host(`traefik.haozy.com`)"
  entrypoints = ["web"]
  service = "api@internal"
  middlewares = ["auth"]

[http.middlewares.auth.basicAuth]
  # 密码生成 echo $(htpasswd -nb haozy 123456)
  users = [
    "haozy:$$apr1$$pfkpgu.w$$jHQtt8T96PdvyojTBgh5E/",
  ]

将域名 traefik.haozy.com host 指到 172.17.17.21 ，访问 traefik.haozy.com 因为配了 auth 验证，输入用户名密码后就可以看到管理界面了。https 也很简单，因为本地测试先注释了对应的配置。

现在将 node1、node2、node3 节点上注册的 web 服务删除，在 web.json 中增加 traefik 相关的 tag，再重新注册。

web.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
  "ID": "web",
  "Name": "consul_web",
  "Tags": [
    "whoami",
    "traefik.enable=true",
    "traefik.http.routers.consul_web.rule=Host(`web.haozy.com`)",
    "traefik.http.routers.consul_web.entrypoints=web",
    "traefik.http.services.consul_web.loadbalancer.passhostheader=true"
  ],
  "Check": {
    "Args": ["curl", "172.17.17.11:32768"],
    "Interval": "10s",
    "Timeout": "3s"
  },
  "Address": "172.17.17.11",
  "Port": 32768
}
1
2
[root@node1 ~]# curl --request PUT http://127.0.0.1:8500/v1/agent/service/deregister/web
[root@node1 ~]# curl --request PUT --data @web.json http://127.0.0.1:8500/v1/agent/service/register?replace-existing-checks=true

现在通过 traefik 管理界面可以看到 web.consul.com 的路由信息，以及对应的后端服务。



将域名 web.haozy.com 指向 172.17.17.21，请求会轮训请求所有后端服务。至此外部流量的负载均衡与路由就完成了，当然 traefik 中的 Routers 和 Middlewares 还可以实现很多复杂的处理。

再多说一点，traefik 2.0 开始分成静态配置文件和动态配置，在静态文件中配置的 Routers、Middlewares 之类的是可以在动态配置中调用的，省去了很多定义，比如调用静态配置文件中的 auth。
traefik.http.routers.consul_web.middlewares=auth@file