Etcd+coredns分布式高可用dns集群搭建

Sep 18, 2018 in LINUX
coredns dns linux
5 min read

一. 搭建etcd集群

因为是测试环境，所以整个集群三个节点都在同一台服务器，只是更换了一下端口

etcd	host
etcd0	172.25.52.206
etcd1	172.25.52.206
etcd2	172.25.52.206

1. 下载

// 下载
# wget https://github.com/coreos/etcd/releases/download/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz
    
// 解压后
# ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md

其中 etcd 就是可执行文件，etcdctl 是客户端文件

2. 配置集群

目前支持三种发现方式：Static，etcd Discovery，DNS Discovery。
● Static适用于有固定IP的主机节点
● etcd Discovery适用于DHCP环境
● DNS Discovery依赖DNS SRV记录
这里我们采用Static方式，创建etcd0脚本，方便配置etcd启动参数

# mkdir -p /root/Etcd/data/etcd0 /root/Etcd/data/etcd1 /root/Etcd/data/etcd2
    
# mkdir -p /root/Etcd/log/etcd0 /root/Etcd/log/etcd1 /root/Etcd/log/etcd2

B. etcd 初始化

// etcd0 
# /root/Etcd/etcd/etcd --name etcd0 --data-dir /root/Etcd/data/etcd0 --advertise-client-urls http://172.25.52.206:2379,http://172.25.52.206:4001 --listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 --initial-advertise-peer-urls http://172.25.52.206:2380 --listen-peer-urls http://0.0.0.0:2380 --initial-cluster-token etcd-cluster-1 --initial-cluster etcd0=http://172.25.52.206:2380,etcd1=http://172.25.52.206:3380,etcd2=http://172.25.52.206:4380 --initial-cluster-state new
    
// etcd1 端口在ectd0的基础上全部加 1000
# /root/Etcd/etcd/etcd --name etcd1 --data-dir /root/Etcd/data/etcd1 --advertise-client-urls http://172.25.52.206:3379,http://172.25.52.206:5001 --listen-client-urls http://0.0.0.0:3379,http://0.0.0.0:5001 --initial-advertise-peer-urls http://172.25.52.206:3380 --listen-peer-urls http://0.0.0.0:3380 --initial-cluster-token etcd-cluster-1 --initial-cluster etcd0=http://172.25.52.206:2380,etcd1=http://172.25.52.206:3380,etcd2=http://172.25.52.206:4380 --initial-cluster-state new
    
// etcd2 端口在ectd0的基础上全部加 2000
# /root/Etcd/etcd/etcd --name etcd2 --data-dir /root/Etcd/data/etcd2 --advertise-client-urls http://172.25.52.206:4379,http://172.25.52.206:6001 --listen-client-urls http://0.0.0.0:4379,http://0.0.0.0:6001 --initial-advertise-peer-urls http://172.25.52.206:4380 --listen-peer-urls http://0.0.0.0:4380 --initial-cluster-token etcd-cluster-1 --initial-cluster etcd0=http://172.25.52.206:2380,etcd1=http://172.25.52.206:3380,etcd2=http://172.25.52.206:4380 --initial-cluster-state new

参数说明：
● –data-dir 指定节点的数据存储目录，若不指定，则默认是当前目录。这些数据包括节点ID，集群ID，集群初始化配置，Snapshot文件，若未指定–wal-dir，还会存储WAL文件
● –wal-dir 指定节点的was文件存储目录，若指定了该参数，wal文件会和其他数据文件分开存储
● –name 节点名称
● –initial-advertise-peer-urls 告知集群其他节点的URL，tcp2380端口用于集群通信
● –listen-peer-urls 监听URL，用于与其他节点通讯
● –advertise-client-urls 告知客户端的URL, 也就是服务的URL，tcp2379端口用于监听客户端请求
● –initial-cluster-token 集群的ID
● –initial-cluster 集群中所有节点
● –initial-cluster-state 集群状态，new为新创建集群，existing为已存在的集群

当三个节点都协商达到一致的时候，集群就正常了，这个时候需要重启一下节点，把集群初始化的一些参数去掉

// etcd0 再次重启，去掉init参数
# nohup /root/Etcd/etcd/etcd --name etcd0 --data-dir /root/Etcd/data/etcd0 --advertise-client-urls http://172.25.52.206:2379,http://172.25.52.206:4001 --listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 --listen-peer-urls http://0.0.0.0:2380 > /root/Etcd/log/etcd0/all.log 2>&1 &
    
// etcd1再次重启，去掉init参数
# nohup /root/Etcd/etcd/etcd --name etcd1 --data-dir /root/Etcd/data/etcd1 --advertise-client-urls http://172.25.52.206:3379,http://172.25.52.206:5001 --listen-client-urls http://0.0.0.0:3379,http://0.0.0.0:5001 --listen-peer-urls http://0.0.0.0:3380 > /root/Etcd/log/etcd1/all.log 2>&1 &
    
// etcd2再次重启，去掉init参数
# nohup /root/Etcd/etcd/etcd --name etcd2 --data-dir /root/Etcd/data/etcd2 --advertise-client-urls http://172.25.52.206:4379,http://172.25.52.206:6001 --listen-client-urls http://0.0.0.0:4379,http://0.0.0.0:6001 --listen-peer-urls http://0.0.0.0:4380 > /root/Etcd/log/etcd2/all.log 2>&1 &

3.集群使用

// 首先这是环境变量，让API版本为2,因为etcd 插件只支持v2版本（coredns 2 版本后的etc插件支持 etcd3了）
# echo "export ETCDCTL_API=2" >> /etc/profile && source /etc/profile
    
// 存入数据
# etcdctl put cluster CoreDNS
ok
    
// 取出
# etcdctl get cluster
cluster
CoreDNS

详细的使用查看 etcdctl -h

二. 结合coredns

1. 搭建coredns 请查看coredns的安装篇

2. 配置coredns

这里主要使用的是coredns 的 etcd 插件

# cat Corefile
    
.:1053 {
    # prometheus :9153    # prometheus url 监控插件 http://host:9153/metrics
    
    etcd yourdomaintech.com {
        stubzones
        fallthrough     # 把ectd无法解析的子域名交给proxy
        path /example
        endpoint http://172.25.52.206:2379 http://172.25.52.206:3379 http://172.25.52.206:4379
        upstream 8.8.8.8:53 8.8.4.4:53
    }
    errors
    whoami
    log
    cache 30 yourdomaintech.com    # 缓存 example.com 域名的解析300s
    proxy . 114.114.114.114 8.8.8.8
}
    
    
// 或者 配置为
    
yourdomaintech.com:1053 {
    log
    errors
    proxy . 114.114.114.114 8.8.8.8
    etcd {
        stubzones
        fallthrough
        path /yourdomaintech
        endpoint http://172.25.52.206:2379 http://172.25.52.206:3379 http://172.25.52.206:4379
        upstream 8.8.8.8:53 8.8.4.4:53
    }
}
    
.:1053 {
    # prometheus :9153    # prometheus url http://host:9153/metrics
    errors
    whoami
    log
    cache 30 yourdomaintech.com
    proxy . 114.114.114.114 8.8.8.8
}

这里所有域名为 example.com 以及他的子域名都会匹配到 etcd中，其他不匹配的则使用proxy中的dns
etcd 插件的详细使用查看etcd pligin

etcd 参数

etcd [ZONES...] {
stubzones
fallthrough [ZONES...]
path PATH
endpoint ENDPOINT...
upstream [ADDRESS...]
tls CERT KEY CACERT
}

stubzones 表示支持子域名

path 是etcd 中的path

endpoint 就是etcd 中的客户端入口

upstream

fallthrough 如果区域匹配但不能生成记录，则将请求传递给下一个插件,比如，只想解析 bi.yourdomaintech.com, api.yourdomaintech.com 交给公网解析，就需要用到这个

3. 启动coredns ，并测试

启动coredns

# ./coredns -conf Corefile
    
mydomain.com.:1053
.:1053
2018/07/10 20:54:55 [INFO] CoreDNS-1.1.4
2018/07/10 20:54:55 [INFO] linux/amd64, go1.10.3, f78f3023
CoreDNS-1.1.4
linux/amd64, go1.10.3, f78f3023

向etcd 集群插入数据，域名解析

// 插入数据
# etcdctl set /yourdomaintech/com/yourdomaintech/www '{"host":"172.25.52.206"}'
{"host":"172.25.52.206"}
    
// 测试
# dig @172.25.52.206 -p 1053 +short www.yourdomaintech.com
172.25.52.206
    
// coredns 输出
172.25.52.206:21949 - [10/Jul/2018:12:40:54 +0800] 21535 "A IN www.yourdomaintech.com. udp 45 false 4096" NOERROR qr,aa,rd,ra 76 0.000303906s
    
// 修改dns记录
# etcdctl update /yourdomaintech/com/yourdomaintech/www '{"host":"172.25.52.207"}'
{"host":"172.25.52.207"}
    
// 测试
# dig @172.25.52.206 -p 1053 +short www.example.com
172.25.52.207
    
// 测试bi.yourdomaintech.com 和 www.baidu.com
# dig @172.25.52.206 -p 53 +short bi.yourdomaintech.com
183.60.177.248
    
# dig @172.25.52.206 -p 53 +short www.baidu.com
www.a.shifen.com.
14.215.177.39
14.215.177.38

可以看到，只要不是etcd数据库里面的域名其他都会通过公网正常解析所以只要保证etcd集群正常，部署多个dns服务器就可以实现高可用的dns服务器。同时可以用调用api的方式进行插入和修改

4. 测试cache 插件

启动cache 30s，配置如下

.:53 {
    # prometheus :9153    # prometheus url 监控插件 http://host:9153/metrics
    
    etcd {
        stubzones
        fallthrough     # 把ectd无法解析的子域名交给proxy
        path /yourdomaintech
        endpoint http://172.25.52.206:2379 http://172.25.52.206:3379 http://172.25.52.206:4379
        upstream /etc/resolv.conf
    }
    errors
    whoami
    log
    cache 30    # 缓存域名的解析300s
    proxy . /etc/resolv.conf
}

dig 测试域名“www.baidu.com”

// 首次测试
    
# dig  www.baidu.com

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7_5.1 <<>> www.baidu.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15302
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
    
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.baidu.com.         IN  A
    
;; ANSWER SECTION:
www.baidu.com.      30  IN  CNAME   www.a.shifen.com.
www.a.shifen.com.   30  IN  A   14.215.177.39
www.a.shifen.com.   30  IN  A   14.215.177.38
    
;; Query time: 31 msec
;; SERVER: 172.25.52.206#53(172.25.52.206)
;; WHEN: Thu Sep 13 20:44:17 CST 2018
;; MSG SIZE  rcvd: 149

第二次测试

#dig  www.baidu.com
    
; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7_5.1 <<>> www.baidu.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4866
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
    
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.baidu.com.         IN  A
    
;; ANSWER SECTION:
www.baidu.com.      27  IN  CNAME   www.a.shifen.com.
www.a.shifen.com.   27  IN  A   14.215.177.39
www.a.shifen.com.   27  IN  A   14.215.177.38
    
;; Query time: 0 msec
;; SERVER: 172.25.52.206#53(172.25.52.206)
;; WHEN: Thu Sep 13 20:44:21 CST 2018
;; MSG SIZE  rcvd: 149

可以看到Query time 为 0 msec ,说明缓存成功了

5. 附带etcd 3 版本，带tls的配置文件

// Corefile
.:53 {
    # prometheus :9153    # prometheus url 监控插件 http://host:9153/metrics
    
    etcd yourdomaintech.com {
        stubzones
        fallthrough     # 把ectd无法解析的子域名交给proxy
        endpoint https://172.25.52.237:2379 https://172.25.52.238:2379 https://172.25.52.239:2379
        path /yourdomaindns 
        tls /etc/etcd/ssl/member-gz-52-237.pem /etc/etcd/ssl/member-gz-52-237-key.pem /etc/etcd/ssl/ca.pem
        upstream /etc/resolv.conf
    }
    debug
    errors
    whoami
    log
    cache 30
    proxy . /etc/resolv.conf
}

注意 endpoint 是https

ectdctl v3 带tls的部分操作指令

# export ETCDCTL_API=3
    
// 写入
# /Etcd/etcd-3.3/etcdctl --endpoints=https://172.25.52.237:2379 --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/member-gz-52-237.pem --key=/etc/etcd/ssl/member-gz-52-237-key.pem put /coredns/com/yourdomaintech/sky '{"host":"1.1.1.1","ttl":60}'
    
// 获取
/Etcd/etcd-3.3/etcdctl --endpoints=https://172.25.52.237:2379 --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/member-gz-52-237.pem --key=/etc/etcd/ssl/member-gz-52-237-key.pem get /coredns/com/yourdomaintech/sky
    
// 删除
/Etcd/etcd-3.3/etcdctl --endpoints=https://172.25.52.237:2379 --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/member-gz-52-237.pem --key=/etc/etcd/ssl/member-gz-52-237-key.pem del /coredns/com/yourdomaintech/sky
    
// 例举
Etcd/etcd-3.3/etcdctl --endpoints=https://172.25.52.237:2379 --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/member-gz-52-237.pem --key=/etc/etcd/ssl/member-gz-52-237-key.pem get /coredns --keys-only --prefix

5. 将coredns 做成systemd 服务

服务文件： /etc/systemd/system/coredns.service

[Unit]
Description=Coredns
After=network-online.target
    
[Service]
Type=simple
ExecStart=/usr/local/bin/coredns/coredns -conf /usr/local/bin/coredns/Corefile
Restart=always
ExecStop=/bin/kill -9 $MAINPID
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=coredns
    
    
[Install]
WantedBy=multi-user.target

rsyslog 文件： /etc/rsyslog.d/coredns.conf

if $programname == 'coredns' then /var/log/coredns/coredns.log

最后重启一下 rsyslog