Luck's Blog

创建高可用etcd集群

2018/04/02 基于k8s的私有容器云从0到1的建设之路

新增hosts解析

10.80.231.151   etcd-host0
10.80.230.57    etcd-host1
10.81.50.223    etcd-host2

使用变量(集群所有节点)

#!/usr/bin/bash

# TLS Bootstrapping 使用的 Token，可以使用命令 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成
BOOTSTRAP_TOKEN="f12b6181f97c303ae8a849edb2fe2b57"

# 最好使用 主机未用的网段 来定义服务网段和 Pod 网段

# 服务网段 (Service CIDR），部署前路由不可达，部署后集群内使用IP:Port可达
SERVICE_CIDR="10.254.0.0/16"

# POD 网段 (Cluster CIDR），部署前路由不可达，**部署后**路由可达(flanneld保证)
CLUSTER_CIDR="172.30.0.0/16"

# 服务端口范围 (NodePort Range)
export NODE_PORT_RANGE="8400-9000"

# etcd 集群服务地址列表
export ETCD_ENDPOINTS="https://10.80.231.151:2379,https://10.80.230.57:2379,https://10.81.50.223:2379"

# flanneld 网络配置前缀
export FLANNEL_ETCD_PREFIX="/kubernetes/network"

# kubernetes 服务 IP (一般是 SERVICE_CIDR 中第一个IP)
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"

# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
export CLUSTER_DNS_SVC_IP="10.254.0.2"

# 集群 DNS 域名
export CLUSTER_DNS_DOMAIN="cluster.local."

# ------------etcd-------------
export NODE_NAME="etcd-host1"  # 当前部署的机器名称(随便定义，只要能区分不同机器即可)
export NODE_IP="10.80.230.57"     # 当前部署的机器 IP
export NODE_IPS="10.80.231.151 10.80.230.57 10.81.50.223"   # etcd 集群所有机器 IP 
export ETCD_NODES="etcd-host0=https://10.80.231.151:2380,etcd-host1=https://10.80.230.57:2380,etcd-host2=https://10.81.50.223:2380"   # etcd 集群间通信的IP和端口

下载二进制文件

到 https://github.com/coreos/etcd/releases 页面下载最新版本的二进制文件：

$ wget https://github.com/coreos/etcd/releases/download/v3.1.6/etcd-v3.1.6-linux-amd64.tar.gz
$ tar -xvf etcd-v3.1.6-linux-amd64.tar.gz
$ scp ca.pem hz-k8s-cluster-n01:/etc/kubernetes/ssl/
$ scp ca.pem hz-k8s-cluster-n02:/etc/kubernetes/ssl/
$ scp ca.pem hz-k8s-cluster-n03:/etc/kubernetes/ssl/
$ scp etcd-v3.2.15-linux-amd64/etcd* hz-k8s-cluster-n01:/usr/local/bin
$ scp etcd-v3.2.15-linux-amd64/etcd* hz-k8s-cluster-n02:/usr/local/bin
$ scp etcd-v3.2.15-linux-amd64/etcd* hz-k8s-cluster-n03:/usr/local/bin

创建 TLS 秘钥和证书

为了保证通信安全，客户端(如 etcdctl) 与 etcd 集群、etcd 集群之间的通信需要使用 TLS 加密， 拷贝之前准备好的证书文件：

$ mkdir -p /etc/etcd/ssl
$ scp etcd*.pem hz-k8s-cluster-n01:/etc/etcd/ssl
$ scp etcd*.pem hz-k8s-cluster-n02:/etc/etcd/ssl
$ scp etcd*.pem hz-k8s-cluster-n03:/etc/etcd/ssl

创建 etcd 的 systemd unit 文件

$ mkdir -p /var/lib/etcd # 必须先创建工作目录
$ cat > etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --name=${NODE_NAME} \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://${NODE_IP}:2380 \
  --listen-peer-urls=https://${NODE_IP}:2380 \
  --listen-client-urls=https://${NODE_IP}:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://${NODE_IP}:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=${ETCD_NODES} \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
  • 指定 etcd 的工作目录和数据目录为 /var/lib/etcd，需在启动服务前创建这个目录；
  • 为了保证通信安全，需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书（trusted-ca-file）；
  • –initial-cluster-state 值为 new 时，–name 的参数值必须位于 –initial-cluster 列表中；

启动etcd服务

mv etcd.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd
systemctl status etcd

最先启动的 etcd 进程会卡住一段时间，等待其它节点上的 etcd 进程加入集群，为正常现象。

验证服务

在任一 kubernetes master 机器上执行如下命令：

[root@hz-k8s-cluster-n03 etcd]# etcdctl   --ca-file=/etc/kubernetes/ssl/ca.pem   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   cluster-health
member 3329fc9340cc2c66 is healthy: got healthy result from https://10.81.50.223:2379
member b012fbc7fa1577b9 is healthy: got healthy result from https://10.80.231.151:2379
member cb56718925e9801b is healthy: got healthy result from https://10.80.230.57:2379
cluster is healthy
[root@hz-k8s-cluster-n03 etcd]# for ip in ${NODE_IPS}; do
   ETCDCTL_API=3 /usr/local/bin/etcdctl \
   --endpoints=https://${ip}:2379  \
   --cacert=/etc/kubernetes/ssl/ca.pem \
   --cert=/etc/etcd/ssl/etcd.pem \
   --key=/etc/etcd/ssl/etcd-key.pem \
   endpoint health; done
https://10.80.231.151:2379 is healthy: successfully committed proposal: took = 1.470028ms
https://10.80.230.57:2379 is healthy: successfully committed proposal: took = 2.066554ms
https://10.81.50.223:2379 is healthy: successfully committed proposal: took = 1.696224ms

结果最后一行为 cluster is healthy 时表示集群服务正常。

Search

    Table of Contents