cozy-proxy
A simple kube-proxy addon for 1:1 NAT services in Kubernetes using an NFT backend.
This project ensures a one-to-one mapping between a service and a pod in Kubernetes.
Why
At Cozystack, we strive to follow the standard Kubernetes network architecture by separating the pod network, service networks, and external load balancers. However, our platform also runs virtual machines that sometimes require an external IP address.
There are several ways to achieve this:
- Using a separate Kube-OVN subnet and exposing it via BGP with kube-ovn-speaker.
- Adding a secondary interface with Multus.
- Using native Kubernetes services with externalIPs and exposing them via MetalLB.
The last option is the simplest and most flexible, but it has a limitation: Kubernetes services do not forward all traffic,
but only traffic on specific ports (see: Kubernetes Issue #23864).
Additionally, kube-proxy does not perform SNAT, which causes outgoing traffic from the pod to use the default gateway of the host where it is running.
To address these issues, we have added an additional controller that performs NAT for services carrying the service.kubernetes.io/service-proxy-name: cozy-proxy label.
How It Works
cozy-proxy is a simple Kubernetes controller that watches for services labeled with service.kubernetes.io/service-proxy-name: cozy-proxy — the standard Kubernetes mechanism for delegating a service to a non-default proxy. kube-proxy skips services carrying this label, so cozy-proxy becomes the sole handler and no rules collide.
When it finds such a service, it creates NFT rules that forward traffic from the service's external IP to the pod's IP and vice versa, performing source-IP preservation for egress traffic.
This controller can be used together with kube-proxy and Cilium in kube-proxy replacement mode.
The label is the only selector. Services that do not carry it are not managed by cozy-proxy, regardless of any annotations they may have.
Ingress mode
The optional networking.cozystack.io/wholeIP annotation selects the ingress mode for a managed service:
| Value |
Behavior |
"true" |
Whole-IP passthrough. All TCP/UDP traffic to the LoadBalancer IP is forwarded to the backend pod. |
"false" |
Per-port filtering (default). Only TCP/UDP traffic to ports listed in Service.spec.ports is forwarded; rest dropped. |
| absent |
Same as "false" — per-port filtering. |
In both modes, egress traffic from the backend pod is SNATed to the
LoadBalancer IP for source-IP preservation.
The optional networking.cozystack.io/allowICMP: "true" annotation, only
meaningful in port-filter mode (wholeIP: "false"), accepts ICMP traffic
toward the backend pod IP that would otherwise be dropped by the per-port
filter. Without it, all ICMP to a port-filtered pod is dropped — which also
blocks ping, PMTU discovery (ICMP "fragmentation needed"), and ICMP
unreachable signalling. Recommended for any service where path-MTU mismatches
or observability matter.
Datapath
The nftables ruleset placed in table ip cozy_proxy consists of:
- Chain
egress_snat at priority raw (-300): rewrites packet source IP via
the pod_svc map for outbound traffic from managed pods. Runs before
conntrack so the recorded tuple has saddr=LB_IP.
- Chain
ingress_dnat at priority mangle (-150): rewrites packet
destination IP via the svc_pod map for inbound traffic to a LoadBalancer
IP. Runs after conntrack so reply packets of egress flows are matched
correctly.
- Chain
port_filter at priority filter (0): for Services in port-filter
mode (wholeIP: "false"), drops ingress packets whose
(daddr, l4proto, dport) is not in allowed_ports. The chain accepts
packets in conntrack states established or related first, so reply
packets of egress flows bypass the filter even when their dport is the VM's
ephemeral source port. ICMP is dropped by default; if the
allowICMP: "true" annotation is set, the pod IP is added to
icmp_allowed_pods and ICMP toward it is accepted before the drop rule.
Rule scoping
The two rewrites are deliberately not scoped alike. Every controller instance
watches every Service, but programs:
| Object |
Scope |
Why |
svc_pod (ingress_dnat), allowed_ports, icmp_allowed_pods |
the node hosting the backend pod |
A non-hosting node would rewrite the destination before the packet has even left it. The hosting node then records a conntrack tuple of (client -> podIP), while the reply leaves the pod and gets saddr rewritten to the service IP by egress_snat at priority raw, before conntrack. (svcIP -> client) matches nothing, so the reply is not established and port_filter drops it. |
pod_svc (egress_snat) |
every node |
An intra-cluster client is source-NATed by the CNI to its own node address, which the overlay knows how to reach directly. The backend's reply is then tunnelled straight to that node and never traverses the hosting node's netfilter hooks, so the client's node is the only place left where the pod IP can still be turned back into the service IP. Without the entry there, the reply arrives with the wrong source and the client answers with a RST — a cross-node connection that hangs while the same-node one works. |
NODE_NAME carries the node identity; the chart injects it from
spec.nodeName. When it is unset the ingress scope check is disabled and
everything is programmed everywhere, so the binary still runs under a
deployment that does not inject it.
Installation
Install controller using Helm-chart:
helm install cozy-proxy charts/cozy-proxy -n kube-system
make image builds a linux/amd64 and linux/arm64 image index, which needs a buildx builder that can produce multi-platform output (the docker-container driver, or a Docker daemon with the containerd image store). Pass a single platform such as PLATFORMS=linux/amd64 to build only that one, for example for a local LOAD=true.
Usage
Create a LoadBalancer service with the service.kubernetes.io/service-proxy-name: cozy-proxy label. The default mode is per-port filtering, so to forward every port to the backend pod (whole-IP passthrough) add the networking.cozystack.io/wholeIP: "true" annotation:
apiVersion: v1
kind: Service
metadata:
name: example-service
labels:
service.kubernetes.io/service-proxy-name: cozy-proxy
annotations:
networking.cozystack.io/wholeIP: "true"
spec:
allocateLoadBalancerNodePorts: false
externalTrafficPolicy: Local
ports:
- port: 65535 # any
selector:
app: nginx
type: LoadBalancer
---
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: docker.io/library/nginx:alpine
Check that the service has an external IP:
Example output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
example-service LoadBalancer 10.96.195.46 1.2.3.4 65535/TCP 84s
Now try to access the service using icmp and tcp; both should work:
ping 1.2.3.4
curl 1.2.3.4
Check external IP from inside the pod:
kubectl exec -ti nginx -- curl icanhazip.com
Example output would be the same as the service external IP:
Environment
This controller was developed primarily for the Cozystack platform and has been tested in the following environment:
- OS: Talos Linux
- CNI: Kube-OVN with Cilium in chaining mode.
- Kube-proxy: Cilium in kube-proxy replacement mode.
- LoadBalancer: MetalLB in L2 mode with
externalTrafficPolicy: Local.
If you have tested it in other environments, please let us know.
Credits