Policy Routing for Container Orchestration

Implement dynamic route updates in response to container lifecycle events.

In this lesson, you will learn how to bridge container lifecycle events with policy-based routing to ensure traffic egresses through the correct network path as containers spin up or down.

The Problem: Ephemeral Interfaces

In container orchestration (like Kubernetes or custom containerized environments), interfaces like veth pairs are created and destroyed dynamically. When a new container is deployed, its traffic might need to be isolated to a specific VRF or routed via a unique policy table (e.g., to force traffic through a specific VPN tunnel or dedicated gateway). Relying on static configuration is impossible because the container’s IP and interface index change with every lifecycle event.

Automating Route Updates

To manage this, we hook into the container runtime’s lifecycle (via CNI plugins or post-start hooks). When a container starts, we program the kernel to associate its specific interface or source IP with a secondary routing table defined by our ip rule policies.

For example, when a container veth (e.g., veth123) comes up, your orchestration logic must trigger a netlink command to insert a rule:

# Direct traffic from the container's source subnet to a specific routing table
ip rule add from 10.200.1.5 table 100
# Ensure that table 100 has a gateway for container egress
ip route add default via 192.168.10.1 dev eth1 table 100

Handling Lifecycle Events

The challenge is cleanup. If the container is destroyed but the routing rule remains, you risk "black-holing" traffic or leaking stale routing entries that can cause conflicts when a new container is assigned the same IP.

A robust implementation uses a Netlink listener or an orchestration hook that tracks the ifindex of the container’s interface. When the container stops, the hook performs a reverse cleanup:

ip rule del from 10.200.1.5
ip route flush table 100 (or specifically remove the dynamic entries associated with that container)

By integrating these commands into the container's "up" and "down" scripts, you transform the kernel’s static routing tables into a dynamic, intent-based network fabric that evolves alongside your application.

Key Takeaway: Dynamic policy routing in orchestration requires pairing CNI lifecycle hooks with atomic netlink operations to ensure that routing rules are created and purged in strict synchronization with container interfaces.