Background of the problem

The k8s cluster is deployed overseas, and the Ingress nodes in the cluster have a slow network access speed from China. It is too expensive to increase bandwidth and exchange for high-quality lines, so we start to choose a roundabout way to save the country. 😂

Initial idea of ​​the problem

  • Buy a Linux machine with a good line but low configuration (1C2G)
  • Deploy OpenResty to reverse the domain name of Ingress, so that the client can access the domain name after the proxy to speed up

Put the idea into practice

The problem of slow access has been solved, but another problem has arisen: The configuration of the proxy machine is too low, and it is not in the same network as the k8s cluster, so it cannot join the k8s cluster. This has brought another tedious problem:

After the k8s Ingress configuration is updated, the OpenResty configuration needs to be manually updated. I am too lazy. And manually updating the configuration is tedious and prone to errors

Further optimization

Because the initial requirements were relatively simple, it was just to automate the tedious and manual work. Reduce manual involvement or only manually review the application to make it effective. Therefore, the ideal solution is:

  • Use python's kubernetes library to connect to k8s
  • The program continuously monitors the k8s api to automatically render the nginx configuration for the newly added or modified Ingress rules
  • Perform keyword search on the changed rules, and only the rules that match the keywords will be rendered
  • Manually review the rendered configuration; and perform nginx -s reload

Program writing

Monitoring configuration changes

Using the watch module in the kubernetes library, we can monitor changes in Ingress in the Kubernetes cluster. By comparing the Ingress rules before and after, we can find out which rules have changed.

The following is an example of dynamically obtaining the ingress metadata name under the specified namespace

from kubernetes import client, config, watch

def watch_k8s_config(namespace):
    """kube config认证文件"""
    config.load_kube_config('~/.kube/config')
    networking_v1 = client.NetworkingV1Api()

    # 监听k8s资源变化方法
    w = watch.Watch()
    print("开始持续监听中...")
    for events in w.stream(networking_v1.list_namespaced_ingress, namespace=namespace):
        ingress = events['object']
         # 获取元数据名称
        rules_name = ingress.metadata.name
        print(f"已经捕获到: {rules_name}")

 # 调用方法传入default命名空间
watch_k8s_config("default")

After running, the existing ingress rules will be printed out immediately. If you add or modify ingress rules, they will also be printed synchronously.

Functional Improvement

When planning the rules for kubectl apply ingress, capture the desired fields and return them in json format, such as the data type in the following format:

{
    'name': 'ingress名称',
    'rules': [
        {
            'domain_name': '前置代理域名',
            'https_enable': False
        }
    ],
    'source_domain': '后置源域名',
    'source_path': '后置源路径',
    'secret_name': '证书名称'
}

Nginx configuration rendering method

For rendering configuration, we use JinJa2 template engine. Now we plan the configuration template. Then we generate the template by passing in json data.

JinJa2模板(template.j2):

server {
    listen 80;
    {% if https_enable %}listen 443 ssl; {% endif %}
    server_name {{domain_name}};
    client_max_body_size 4096m;
    {% if https_enable %}
    # ssl配置
    ssl_certificate     /etc/nginx/ssl/{{secret_name}}/ssl.pem;
    ssl_certificate_key /etc/nginx/ssl/{{secret_name}}/ssl.key;
    ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;

    if ($server_port !~ 443){
        rewrite ^(/.*)$ https://$host$1 permanent;
    }
    {% endif %}
    location / {
        {% if https_enable %}proxy_pass https://{{source_domain}}{{source_path}};{% else %}proxy_pass http://{{source_domain}}{{source_path}};{% endif %}
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    access_log  /etc/nginx/logs/{{domain_name}}.access.log;
    error_log   /etc/nginx/logs/{{domain_name}}.error.log;
}

渲染配置的代码片段

import os,sys
from jinja2 import Environment, FileSystemLoader

def generate_openresty_config(data):

    # 这里用前面的template.j2配置模板
    template = Environment(loader=FileSystemLoader(searchpath="./")).get_template("template.j2")

    for rule in data["rules"]:
        """
        渲染的字段含义:
            domain_name: 前置代理名称
            source_domain: 后置源域名
            source_path: 后置源路径
            https_enable: 是否启用HTTPS
            secret_name: HTTPS证书名称
        """

        domain_name = rule["domain_name"]
        source_domain = data["source_domain"]
        https_enable = rule["https_enable"]
        source_path = data["source_path"]
        secret_name = data["secret_name"]

        nginx_config = template.render(
            secret_name=secret_name,
            domain_name=domain_name,
            source_path=source_path,
            https_enable=https_enable,
            source_domain=source_domain,
        )

        # 进行渲染配置后的内容写入到当前路径下{domain_name}.conf文件内
        with open(f"./{domain_name}.conf", "w") as f:
            f.write(nginx_config)

Combine program code

Merge all the above codes into a program (main.py):

import os,sys
import logging
from kubernetes import client, config, watch
from jinja2 import Environment, FileSystemLoader

previous_rules = {}

"""
容易变动的配置通过环境变量来维护:

- keyword:  检索关键字
- namespace: 命名空间
- log_level: 日志输出等级
"""
keyword = os.environ.get('keyword') or ''
namespace = os.environ.get('namespace') or 'default'
log_level = os.environ.get('log_level') or 'INFO'

# 设置日志
logging.basicConfig(
    level=logging.getLevelName(log_level),
    datefmt="%d/%m/%Y %H:%M:%S",
    format="[%(asctime)s][%(levelname)s] ==> %(message)s",
    handlers=[logging.StreamHandler(stream=sys.stdout)],
)

logger = logging.getLogger(__name__)

# 检查是否启用了https
def tls_check(ingress, host):
    for tls_rule in ingress.spec.tls:
        if host in tls_rule.hosts:
            return True
    return False

# host关键字匹配
def keyword_matching(rule): 
    return keyword in rule.host

# 生成nginx配置规则
def generate_openresty_config(data):
    template = Environment(loader=FileSystemLoader(searchpath="./")).get_template("template.j2")

    for rule in data["rules"]:
        domain_name = rule["domain_name"]
        source_domain = data["source_domain"]
        https_enable = rule["https_enable"]
        source_path = data["source_path"]
        secret_name = data["secret_name"]

        nginx_config = template.render(
            secret_name=secret_name,
            domain_name=domain_name,
            source_path=source_path,
            https_enable=https_enable,
            source_domain=source_domain,
        )

        with open(f"./{domain_name}.conf", "w") as f:
            f.write(nginx_config)

# 监听配置变更方法
def watch_k8s_config(namespace):
    """
    input: kube confugration
    return: json data: new_ingress_info
    """

    # kube config的认证文件
    config.load_kube_config('~/.kube/config')
    networking_v1 = client.NetworkingV1Api()

    # 监听k8s资源变化方法
    w = watch.Watch()

    logger.info(f"事件监听中 namespace: {namespace}")
    if keyword != '':
        logger.info(f"匹配host关键字: {keyword}")

    for events in w.stream(networking_v1.list_namespaced_ingress, namespace=namespace):
        ingress = events['object']
        rules_name = ingress.metadata.name

        rules = []
        for rule in ingress.spec.rules:
            if keyword_matching(rule):
                tls_enabled = tls_check(ingress, rule.host)
                rule_info = {"domain_name": rule.host, "https_enable": tls_enabled}
                rules.append(rule_info)

        new_ingress_info = {"name": rules_name, "rules": rules, "source_domain": ingress.spec.tls[0].hosts[0],"source_path": rule.http.paths[0].path,"secret_name": ingress.spec.tls[0].secret_name}
        if rules_name in previous_rules:
            # 拿之前的配置和新配置进行对比
            if previous_rules[rules_name] != new_ingress_info:
                logger.info(f"配置规则{rules_name}已监听到更新")
                generate_openresty_config(new_ingress_info)
                if True:
                    logger.info(f"配置规则{rules_name}已生成配置文件")
                    logger.info(f"前置代理域名: {rule_info['domain_name']} 后置源域名: {new_ingress_info['source_domain']} 后置源路径: {new_ingress_info['source_path']} 是否启用TLS: {rule_info['https_enable']}")
                else:
                    logger.error(f"配置规则{rules_name}生成配置失败")

        previous_rules[rules_name] = new_ingress_info

if __name__ == "__main__":
    if  keyword == '':
        logger.warning("未指定匹配关键字,则生成全部规则配置") 
    watch_k8s_config(namespace)

需要将config.load_kube_config('~/.kube/config')的k8s客户端认证文件修改为具体的路径,接着运行程序,传入环境变量配置,在配置发生变更时来查看运行日志,下面是一个运行的日志例子:

$ export namespace=private # 监听private命名空间下的变更
$ export keyword=proxy     # 只有host配置的域名包含proxy关键字才会被捕获

$ python3 main.py
[01/02/2024 14:10:47][INFO] ==> 事件监听中 namespace: private
[01/02/2024 14:10:47][INFO] ==> 匹配host关键字: proxy
[01/02/2024 14:12:13][INFO] ==> 配置规则tele-stream-bot已监听到更新
[01/02/2024 14:12:13][INFO] ==> 配置规则tele-stream-bot已生成配置文件
[01/02/2024 14:12:13][INFO] ==> 前置代理域名: tele-xxxx 后置源域名: tele-xxxx 后置源路径: / 是否启用TLS: True

如果运行结果如上所示,那么在当前目录下,就会生成一个nginx的配置文件 xxx.conf

Container operation

I don't like to install various environments in the server, I just want to run a docker container to meet my needs.

If running in a container, you need to pay special attention to the paths of several files in the main.py program:

    # 生成的nginx配置路径
    with open(f"/app/configs/{domain_name}.conf", "w") as f:
        f.write(nginx_config)

    # k8s认证文件的路径
    config.load_kube_config('/app/k8s_config')

准备dockerfile:

FROM python:3.7-alpine

WORKDIR /app/
RUN pip3 install --no-cache-dir kubernetes jinja2

ADD template.j2 main.py /app/

CMD ["python3", "main.py"]

构建容器镜像

docker build . -t svc_to_openresty:1.0

运行容器

docker rm -f svc_to_openresty
docker run -itd --name=svc_to_openresty \
-e keyword=检索关键字 \
-e namespace=命名空间 \
-v /app/openresty/conf:/app/configs/ \
-v /root/.kube/config:/app/k8s_config \
-v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime \
svc_to_openresty:1.0
docker logs -f --tail=200 svc_to_openresty
Last modification:September 5, 2024
如果觉得我的文章对你有用,请随意赞赏