Evalexp's Digital Garden

证书中心

8分钟阅读

证书中心

对于需要对外暴露的服务，此时我们申请的SSL证书一般是被信任的，所以我们需要保管好这份证书，通过它为我们的网站进行加密传输。

通过acme.sh我们可以申请泛域名证书，也就是SAN证书。那么考虑一个场景，现在有若干个服务，服务1-10在香港地域、服务11-20在大陆地域，服务1-10在香港的n个服务器上，服务11-20在大陆的m个服务器上，如果证书过期，此时我们需要手动替换n+m个服务器的证书，这实际上是一个很大的工作量。

ETCD

既然使用了Traefik，那么我们就可以使用Traefik的ETCD Provider来作为动态配置来源；也就是说对于上面的场景，我们直接使用一个etcd来存储所有的证书，然后将Traefik接入etcd即可实现动态自动更新证书了。

ETCD 证书

创建一个ssl的目录，然后在里面准备创建证书：

evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl genrsa -out ca-key.pem 4096
evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl req -new -x509 -days 3650 -key ca-key.pem -out ca.pem -subj "/C=CN/ST=HK/L=HongKong/O=evalexp.top/OU=IT/CN=traefik-etcd-ca"

注意这里先创建了Etcd的CA证书。

接下来生成服务器的证书，首先我们为该证书创建一个配置server-csr.conf：

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
 
[req_distinguished_name]
C = CN
ST = HK
L = HongKong
O = evalexp.top
OU = IT
CN = traefik-etcd.evalexp.top
 
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
 
[alt_names]
DNS.1 = localhost
DNS.2 = etcd
DNS.3 = traefik-etcd
DNS.4 = traefik-etcd.evalexp.top
DNS.5 = evalexp.top
DNS.6 = *.evalexp.top

以该配置申请证书，然后使用CA证书签名：

# 申请
evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl genrsa -out server-key.pem 4096
evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl req -new -key server-key.pem -out server.csr -config server-csr.conf
# 签名
evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl x509 -req -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server.pem -days 3650 -extensions v3_req -extfile server-csr.conf
Certificate request self-signature ok
subject=C = CN, ST = HK, L = HongKong, O = evalexp.top, OU = IT, CN = traefik-etcd.evalexp.top

服务端搞定了，接下来生成客户端的证书，注意服务端需要配置相关的属性，而客户端则不需要这么麻烦了，只需要使用CA对其签名即可：

evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl genrsa -out client-key.pem 4096
evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl req -new -key client-key.pem -out client.csr -subj "/C=CN/ST=HK/L=HongKong/O=evalexp.top/OU=IT/CN=traefik-etcd-client"
evalexp@VM-8-6-debian:~/service/app/traefik-etcd/ssl$ openssl x509 -req -in client.csr -CA ca.pem -CAkey ca-key.pem -CACreateserial -out client.pem -days 3650 -sha256
Certificate request self-signature ok
subject=C = CN, ST = HK, L = HongKong, O = evalexp.top, OU = IT, CN = traefik-etcd-client

ETCD 配置

接下来创建一个config目录，然后来对etcd进行配置，写入etcd.yaml文件：

name: etcd-evalexp
data-dir: /var/lib/etcd
listen-client-urls: https://0.0.0.0:2379
advertise-client-urls: https://traefik-etcd.evalexp.top:2379
listen-peer-urls: https://0.0.0.0:2380
initial-advertise-peer-urls: https://traefik-etcd.evalexp.top:2380
 
# 单节点模式
initial-cluster: etcd-evalexp=https://traefik-etcd.evalexp.top:2380
initial-cluster-token: etcd-evalexp-token
initial-cluster-state: new
 
# TLS 客户端配置
client-transport-security:
  cert-file: /etc/etcd/ssl/server.pem
  key-file: /etc/etcd/ssl/server-key.pem
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ssl/ca.pem
 
# TLS 节点间通信配置（单节点也需要）
peer-transport-security:
  cert-file: /etc/etcd/ssl/server.pem
  key-file: /etc/etcd/ssl/server-key.pem
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ssl/ca.pem
 
# 性能优化
heartbeat-interval: 100
election-timeout: 1000
max-snapshots: 5
max-wals: 5
snapshot-count: 10000
 
# 配额设置 (2GB)
quota-backend-bytes: 2147483648
 
# 日志配置
log-level: info
logger: zap
log-outputs: ['/var/log/etcd/etcd.log', 'stderr']
 
# 指标端点
metrics: extensive
listen-metrics-urls: http://0.0.0.0:2381
 
# 自动压缩
auto-compaction-mode: periodic
auto-compaction-retention: "1h"

ETCD Docker

接下来我们通过docker部署etcd，使用compose来完成部署：

services:
  traefik-etcd:
    image: quay.io/coreos/etcd:v3.6.4
    container_name: traefik-etcd
    hostname: traefik-etcd.evalexp.top
    restart: always
    volumes:
      - /datapool/encryption/traefik_etcd_data:/var/lib/etcd:rw
      - ./ssl:/etc/etcd/ssl:ro
      - ./config/etcd.yml:/etc/etcd/etcd.yml:ro
      - traefik-etcd-logs:/var/log/etcd:rw
    ports:
      - "2379:2379"
    environment:
      - ETCD_CONFIG_FILE=/etc/etcd/etcd.yml
    command:
      - etcd
      - --config-file=/etc/etcd/etcd.yml
    networks:
      - app_net
    healthcheck:
      test: ["CMD", "etcdctl", "--endpoints=https://localhost:2379", "--cert=/etc/etcd/ssl/client.pem", "--key=/etc/etcd/ssl/client-key.pem", "--cacert=/etc/etcd/ssl/ca.pem", "endpoint", "health"]
      interval: 60s
      timeout: 10s
      retries: 3
      start_period: 60s
 
volumes:
  traefik-etcd-logs:

直接启动即可。

考虑到减少暴露面，我们将2379端口也削减掉，直接通过Traefik的TCP路由+TLS直通，复用443端口来进行etcd的暴露。

将etcd接入Traefik代理中：

services:
  traefik-etcd:
    image: quay.io/coreos/etcd:v3.6.4
    container_name: traefik-etcd
    hostname: traefik-etcd.evalexp.top
    restart: always
    volumes:
      - /datapool/encryption/traefik_etcd_data:/var/lib/etcd:rw
      - ./ssl:/etc/etcd/ssl:ro
      - ./config/etcd.yml:/etc/etcd/etcd.yml:ro
      - traefik-etcd-logs:/var/log/etcd:rw
    #ports:
    #  - "2379:2379"
    environment:
      - ETCD_CONFIG_FILE=/etc/etcd/etcd.yml
    command:
      - etcd
      - --config-file=/etc/etcd/etcd.yml
    networks:
      - app_net
    healthcheck:
      test: ["CMD", "etcdctl", "--endpoints=https://localhost:2379", "--cert=/etc/etcd/ssl/client.pem", "--key=/etc/etcd/ssl/client-key.pem", "--cacert=/etc/etcd/ssl/ca.pem", "endpoint", "health"]
      interval: 60s
      timeout: 10s
      retries: 3
      start_period: 60s
    labels:
      - "traefik.enable=true"
      - "traefik.tcp.routers.traefik-etcd.rule=HostSNI(`traefik-etcd.evalexp.top`) || HostSNI(`traefik-etcd`)"
      - "traefik.tcp.routers.traefik-etcd.entrypoints=https"
      - "traefik.tcp.routers.traefik-etcd.service=traefik-etcd"
      - "traefik.tcp.routers.traefik-etcd.tls=true"
      - "traefik.tcp.routers.traefik-etcd.tls.passthrough=true"
      - "traefik.tcp.services.traefik-etcd.loadbalancer.server.port=2379"
 
volumes:
  traefik-etcd-logs:

注意务必启用TLS并且开启TLS直通；这样才能让etcd自行处理证书认证。

Traefik 接入

需加入与Router - Traefik同一网络

接下来我们将Traefik接入这里的etcd，然后动态更新etcd即可完成证书的动态更新。

首先将ca.pem、client.pem和client-key.pem放入certs的目录中；然后我们就可以直接引入这个Provider：

services:
  traefik:
    image: traefik:v3.4
    container_name: traefik
    security_opt:
      - no-new-privileges:true
    networks:
      - app_net
    command:
      - "--api.insecure=false"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=app_net"
      - "--providers.file.directory=/etc/traefik/dynamic"
      - "--entryPoints.http.address=:80"
      - "--entryPoints.https.address=:443"
      - "--entryPoints.https.http.tls=true"
      - "--entryPoints.http.http.redirections.entryPoint.to=https"
      - "--entryPoints.http.http.redirections.entryPoint.scheme=https"
      - "--providers.etcd.endpoints=traefik-etcd:2379"
      - "--providers.etcd.rootkey=traefik"
      - "--providers.etcd.tls.ca=/certs/etcd-ca.pem"
      - "--providers.etcd.tls.cert=/certs/etcd-client.pem"
      - "--providers.etcd.tls.key=/certs/etcd-client-key.pem"
 
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./certs:/certs:ro"
      - "./dynamic:/etc/traefik/dynamic:ro"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.evalexp.top`)"
      - "traefik.http.routers.dashboard.entrypoints=https"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.middlewares=dashboard-auth"
      - "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$2y$$05$$HaY5fUzjZxKwK6AqKVZyjOMLOIw.BLGtFuHHecSBRocWu6AxeLBDu"

注意这样启用后，如果etcd中没有这样的prefix，也就是rootkey，那么traefik会一直有报错日志。

所有其实很简单，如果要将一个Traefik引入证书中心以便实现证书自动更新，只需要添加：

      - "--providers.etcd.endpoints=traefik-etcd.evalexp.top:443"
      - "--providers.etcd.rootkey=traefik"
      - "--providers.etcd.tls.ca=/certs/etcd-ca.pem"
      - "--providers.etcd.tls.cert=/certs/etcd-client.pem"
      - "--providers.etcd.tls.key=/certs/etcd-client-key.pem"

注意这里要变为traefik-etcd.evalexp.top，是因为在上面的配置中，etcd本身就启动在Docker的内置网络中，使用服务名进行解析。

这样即便是有n+m个traefik，也可以快速地自动更新好证书。

SSL证书更新

注意我这里统一重命名前面加了etcd-的前缀；接下来我们将证书放入etcd中，首先我们在etcd的docker服务下面创建一个shell脚本用于快速操作：

#!/bin/bash
CERT_FILE="/etc/etcd/ssl/client.pem"
KEY_FILE="/etc/etcd/ssl/client-key.pem"
CA_FILE="/etc/etcd/ssl/ca.pem"
 
docker exec traefik-etcd etcdctl --endpoints=https://localhost:2379 --cert=$CERT_FILE --key=$KEY_FILE --cacert=$CA_FILE "$@"

然后我们添加证书：

evalexp@VM-8-6-debian:~/service/app/traefik-etcd$ ./etcd-cli.sh put traefik/tls/certificates/0/certFile -- "$(cat /home/evalexp/out/evalexp.top_ecc/fullchain.cer)"
OK
evalexp@VM-8-6-debian:~/service/app/traefik-etcd$ ./etcd-cli.sh put traefik/tls/certificates/0/keyFile -- "$(sudo cat /home/evalexp/out/evalexp.top_ecc/evalexp.top.key)"
OK
evalexp@VM-8-6-debian:~/service/app/traefik-etcd$ ./etcd-cli.sh put traefik/tls/certificates/1/certFile -- "$(cat /home/evalexp/out/fnos.evalexp.top_ecc/fullchain.cer)"
OK
evalexp@VM-8-6-debian:~/service/app/traefik-etcd$ ./etcd-cli.sh put traefik/tls/certificates/1/keyFile -- "$(sudo cat /home/evalexp/out/fnos.evalexp.top_ecc/fnos.evalexp.top.key)"
OK

当然我们直接用一个脚本会更快：

#!/bin/bash
CERT_FILE="/etc/etcd/ssl/client.pem"
KEY_FILE="/etc/etcd/ssl/client-key.pem"
CA_FILE="/etc/etcd/ssl/ca.pem"
 
etcdctl(){
  docker exec traefik-etcd etcdctl --endpoints=https://localhost:2379 --cert="$CERT_FILE" --key="$KEY_FILE" --cacert="$CA_FILE" "$@"
}
 
BASE="/home/evalexp/out/"
i=0
for dir in "$BASE"/*/; do
  domain=$(basename "$dir" | sed 's/_ecc//')
  cert="$dir/fullchain.cer"
  key="$dir/$domain.key"
 
  if [ -f "$cert" ] && [ -f "$key" ]; then
    key_content=$(cat "$key" 2>/dev/null)
    if [ -z "$key_content" ]; then
      echo "Failed to read content of $key" >&2
      continue
    fi
    cert_content=$(cat "$cert" 2>/dev/null)
    if [ -z "$cert_content" ]; then
      echo "Failed to read content of $cert" >&2
      continue
    fi
    echo "Adding to traefik/tls/certificates/$i/ ..."
    etcdctl put "traefik/tls/certificates/$i/certFile" -- "$cert_content"
    etcdctl put "traefik/tls/certificates/$i/keyFile" -- "$key_content"
    i=$((i+1))
  fi
done

将这个脚本保存为auto-cert.sh，它将自动遍历位于/home/evalexp/out下的证书并添加到etcd中完整自动更新。

关系图谱

反向链接