Showing posts with label networking. Show all posts
Showing posts with label networking. Show all posts

Saturday, April 18, 2026

6G Networks: What Developers Need to Know Before 2027

The first time I tried to ship a product that depended on 5G's advertised low latency, I learned that marketing latency and engineering latency are two different things. The promise was 1 ms. What we measured in production across a major US carrier was 28 ms median, with a long tail of 80+ ms spikes whenever the user walked between cell sites. We ended up redesigning the app around the assumption that the network was no better than 4G LTE with a slightly faster peak. That experience taught me to read mobile standards the way a product manager reads a vendor whitepaper: with a lot of respect for the spec and a healthy skepticism about the deployment. I'm bringing the same posture to 6G in this post.

If you're a developer, you've probably skimmed past 5G headlines for years thinking "this doesn't affect me." But 6G is different — and the reason has nothing to do with faster phone calls.

6G is shaping up to be the infrastructure layer that unlocks the next wave of applications: real-time AI inference at the edge, truly immersive extended reality, and autonomous systems that communicate faster than human reflexes. By 2027, the first 6G deployments will go live in South Korea and Japan. If you're building software that touches mobile, IoT, edge computing, or latency-sensitive systems, understanding what 6G means for developers is no longer optional.

This guide breaks down what 6G actually is, how it differs from 5G, and — most importantly — what it means for how you'll build applications in the next few years.


What Problem Does 6G Solve?

To understand 6G, you need to understand where 5G fell short.

5G promised three things: ultra-fast speeds (up to 10 Gbps), ultra-low latency (under 1ms in ideal conditions), and massive device density (up to 1 million devices per square kilometer). In lab conditions, 5G delivers on all three. In the real world, most users get a slightly faster 4G experience with better coverage — and developers got an infrastructure they couldn't reliably design for.

The gap between 5G's theoretical capabilities and practical performance comes from physics and deployment reality: high-frequency mmWave signals that can't penetrate walls, coverage gaps in rural areas, network slicing complexity that few carriers have fully implemented, and backhaul bottlenecks that limit edge compute performance.

6G addresses these limitations structurally, not incrementally:

Terahertz (THz) spectrum. While 5G mmWave tops out at ~100 GHz, 6G targets the 100 GHz–10 THz range. This unlocks theoretical peak speeds of 1 Tbps — 100x faster than 5G's best case. The tradeoff is range: THz signals are absorbed by oxygen and moisture. The solution involves intelligent reflective surfaces (IRS) — programmable panels that act like mirrors for radio waves, redirecting signals around obstacles. This is a hardware innovation with significant deployment implications.

Sub-millisecond latency. 5G targets 1ms; 6G targets 0.1ms (100 microseconds). This isn't just a spec sheet improvement. It's the threshold below which round-trip network communication becomes imperceptible to human senses. Applications that were previously impractical — surgical robotics, haptic feedback over distance, real-time collaborative holograms — become feasible.

Native AI integration. This is the biggest shift for developers. 5G is a pipe; AI is bolted on. 6G is being designed from the ground up with AI as a first-class citizen: networks that self-optimize, predict congestion before it happens, and allocate spectrum dynamically. The 6G standard includes "AI/ML-native" architecture as a core requirement, not an afterthought.

Sensing as a service. 6G radios will double as environmental sensors. The same signal that carries your data can detect motion, map physical spaces, measure environmental conditions, and even perform rudimentary imaging. This "ISAC" (Integrated Sensing and Communication) capability means your network becomes a distributed sensing grid — relevant for robotics, smart cities, and any application that needs real-world context.

flowchart LR subgraph Five["5G (advertised vs real)"] A1[Peak: 10 Gbps] A2[Latency target: 1 ms] A3[Real median: ~28 ms] end subgraph Six["6G (target)"] B1[Peak: 1 Tbps] B2[Latency target: 0.1 ms] B3[Sensing + AI native] end subgraph App["What it unlocks"] C1[Remote haptics] C2[Holographic AR collab] C3[Edge LLM inference] C4[Network-as-sensor APIs] end Six --> App style Five fill:#1e293b,stroke:#f87171,color:#f8fafc style Six fill:#1e293b,stroke:#4ade80,color:#f8fafc style App fill:#0f172a,stroke:#60a5fa,color:#f8fafc

The 6G Timeline: What's Actually Happening

6G is not vaporware. It has a concrete development timeline with real funding and regulatory activity:

2020–2024: Research phase. The ITU (International Telecommunication Union) kicked off IMT-2030 standardization — the formal process that defines what 6G must deliver. Samsung, Nokia, Ericsson, Huawei, and dozens of university research labs published competing visions. The US, EU, South Korea, Japan, and China each launched national 6G initiatives with billions in public funding.

2025–2026: Standards convergence. The 3GPP (the standards body that defines mobile networks) begins formal 6G specification work in Release 21, expected to land in 2028. Meanwhile, early prototype hardware is being tested by NTT DOCOMO (Japan), SK Telecom (South Korea), and Ericsson in Europe.

2027–2028: First deployments. South Korea and Japan are targeting limited 6G network launches in time for the 2028 Los Angeles Olympics. Early deployments will use sub-6 GHz and mmWave spectrum, with THz bands arriving later as hardware matures.

2030+: Mass adoption. Mainstream 6G coverage in dense urban areas. Consumer devices with 6G chipsets. The same trajectory as 4G (deployed 2010, mainstream by 2015) and 5G (deployed 2019, mainstream by 2023).

For developers, this means: you have 2–3 years before you need to write 6G-aware code, but you should understand the architecture now so you're not redesigning systems from scratch when it arrives.

timeline title 6G Development Timeline 2020-2024 : ITU IMT-2030 research phase : National initiatives launch : Vendor whitepapers published 2025-2026 : 3GPP Release 21 specification work begins : Prototype hardware testing : DOCOMO / SK Telecom / Ericsson trials 2027-2028 : First limited deployments (South Korea, Japan) : Sub-6 GHz + mmWave rollout : Dev APIs enter beta 2029-2030 : Urban 6G coverage expands : THz bands begin consumer rollout : Standard edge + sensing APIs stabilize 2031+ : Mainstream consumer 6G : Low-cost chipsets : Ecosystem maturity

What Changes for Developers

Latency-first application design becomes viable

Today, even with 5G, developers building interactive applications on mobile networks assume ~20–50ms round-trip latency as a realistic floor. Applications that need genuinely low latency (gaming, real-time collaboration, AR overlays) push compute to the cloud edge and accept that the last mile is a bottleneck.

With 6G's 0.1ms target, the last-mile bottleneck shrinks by 90%+. Applications that cache aggressively, batch operations, or prefetch to hide latency can be redesigned to trust the network for near-real-time round trips. This enables:

  • Remote haptic interfaces: A surgeon's hand movements transmitted to a robot with zero perceptible delay
  • Synchronous AR collaboration: Multiple users interacting with shared AR objects that update in real time across devices
  • Tight IoT control loops: Industrial machinery controlled over the network with the same responsiveness as a local connection

The implication for backend architects: service meshes and API design will need to handle much higher-frequency, lower-latency request patterns. The "chatty API" anti-pattern becomes less of a problem. New patterns emerge for continuous state synchronization.

Edge computing gets a second act

5G was supposed to make edge computing mainstream. It hasn't — not because edge compute is a bad idea, but because the economics and tooling weren't there. 6G's "network as a platform" model changes this.

6G standards include Multi-access Edge Computing (MEC) as a native feature, not an add-on. Edge servers within 6G base stations will be standardized, discoverable, and programmable through APIs. For developers, this means:

  • Standard APIs for offloading compute to the nearest edge node
  • Seamless failover between edge and cloud
  • Location-aware routing baked into the network layer

The developer experience for edge deployment will look more like deploying to a managed cloud function than configuring carrier-specific hardware. Think AWS Lambda but running 50ms from your user, not 200ms.

AI inference moves to the radio edge

Today, running AI inference close to users requires significant infrastructure: edge servers, careful caching of model weights, optimized runtimes. With 6G's native AI capabilities and THz bandwidth, a new pattern becomes viable: streaming model computation across the network.

Instead of downloading and running a model locally, a device sends raw sensor data to an intelligent edge node that runs inference and returns results — all within the 0.1ms window. For developers building on-device AI (think camera-based AR features, real-time audio processing, computer vision in field applications), 6G removes the constraint that the model must fit on the device.

This has profound implications for the AI application layer: you can deploy larger, more capable models to edge users without requiring high-end hardware on the device itself.

Sensing APIs become a new platform primitive

ISAC (Integrated Sensing and Communication) in 6G means the network itself generates spatial and environmental data. Imagine a standard API call that returns: "here are the detected objects in a 50-meter radius of this device." Smart city applications, indoor navigation, proximity-based features, and safety systems could query network-generated sensing data instead of deploying dedicated sensor hardware.

From a developer perspective, this is a new category of platform primitive — similar to how GPS turned location from a hardware problem into an API call. The standardization of ISAC APIs is still early, but developers should watch this space.

flowchart TB Dev["Your App"] -->|"sensing API call"| NetSrv[6G Network Services] Dev -->|"edge compute
offload"| MEC[MEC Node
at base station] Dev -->|"AI inference
over THz"| AIEdge[AI-Native
Inference Service] NetSrv --> ISAC[ISAC Radios
sensing + comms] ISAC --> Scene[Scene Graph
objects, motion, range] Scene --> Dev MEC --> RegCache[Regional Cache
model weights, media] RegCache --> Dev AIEdge --> GPUNode[GPU / NPU
pool] GPUNode --> Dev style Dev fill:#1e293b,stroke:#fb923c,color:#f8fafc style Scene fill:#1e293b,stroke:#60a5fa,color:#f8fafc style RegCache fill:#1e293b,stroke:#4ade80,color:#f8fafc style GPUNode fill:#1e293b,stroke:#a78bfa,color:#f8fafc

What You Should Do Now

You're not building for 6G today. But there are concrete actions that position you well:

1. Understand the 5G capabilities you're probably underusing. Network slicing, edge compute APIs through AWS Wavelength or Azure Edge Zones, and 5G's high-bandwidth low-latency modes are already available and underused. Building applications that take advantage of these today is both useful now and a learning exercise for 6G patterns.

2. Design systems that degrade gracefully across connectivity. 6G will coexist with 5G, 4G, and WiFi for years. Applications that assume a specific latency or bandwidth profile will break. Progressive enhancement — designing for the lowest common denominator and unlocking features as connectivity improves — is the right architectural posture.

3. Follow the 3GPP and ITU standards process. The organizations defining 6G publish their working documents publicly. You don't need to read every specification, but following the high-level decisions (which spectrum, which use cases, which APIs) gives you 18-month advance notice on where the platform is going. Subscribe to the ITU IMT-2030 mailing list.

4. Watch the edge compute tooling landscape. Companies like Cloudflare, Fastly, and AWS are already building the developer experience layer for edge compute. The patterns they establish for 5G edge will extend to 6G. Get comfortable with edge-first deployment patterns now.

5. Think about what your application would do with 0.1ms latency and 1 Tbps bandwidth. This is a useful design exercise. If the network were not a constraint, what would you build differently? The answers often reveal opportunities to simplify your architecture when 6G arrives.


What 5G Taught Us the Hard Way (and Why It Matters for 6G)

I mentioned the 5G low-latency disappointment in the intro. Let me make that concrete because the lessons carry directly into how you should evaluate 6G claims.

Carrier deployment reality lags vendor spec by 3-5 years. 5G's 1 ms URLLC (Ultra-Reliable Low Latency) mode requires the carrier to have deployed a dedicated network slice, to have MEC nodes within a few kilometres of the user, and to have configured prioritized scheduling. In the US, fewer than 15% of 5G cell sites had the full URLLC stack as of late 2024. The headline "1 ms" was meaningful in a lab; in the field, you had to call your carrier's enterprise team, negotiate an SLA, and pay for dedicated capacity to get anywhere close. 6G will follow the same pattern. Design for graceful degradation.

Progressive enhancement wins every network generation. The apps that survived and thrived through the 3G/4G/5G transitions were the ones that measured actual connectivity characteristics and adapted. Here's the pattern I recommend, which works today on 5G and will extend cleanly to 6G:

import time
import statistics
from dataclasses import dataclass

@dataclass
class LinkProfile:
    median_rtt_ms: float
    p95_rtt_ms: float
    bandwidth_mbps: float
    capability_class: str  # "low" | "standard" | "premium"

def probe_link(probe_url: str, samples: int = 10) -> LinkProfile:
    """Measure real RTT over the actual link, not what the OS reports."""
    latencies = []
    for _ in range(samples):
        start = time.monotonic()
        requests.get(probe_url, timeout=2)
        latencies.append((time.monotonic() - start) * 1000)
    median = statistics.median(latencies)
    p95 = sorted(latencies)[int(samples * 0.95) - 1]
    # Bandwidth test omitted for brevity
    bw = estimate_bandwidth(probe_url)

    if median < 5 and bw > 500:
        tier = "premium"   # 6G territory
    elif median < 30 and bw > 50:
        tier = "standard"  # real-world 5G / wired
    else:
        tier = "low"       # degraded mobile
    return LinkProfile(median, p95, bw, tier)

def configure_app(profile: LinkProfile):
    if profile.capability_class == "premium":
        enable_realtime_sync()
        enable_stream_inference()
    elif profile.capability_class == "standard":
        enable_debounced_sync(ms=250)
        use_cached_inference()
    else:
        use_offline_mode()
        defer_nonessential_sync()

This pattern gives you a single code path that works well on 4G, light it up on 5G, and automatically takes advantage of 6G when it arrives. You don't need separate 6G SDKs; you need honest measurement and adaptive behaviour.

Trust real numbers, not spec sheets. When evaluating any new network generation, insist on measurement traces from real deployments before you commit to an architecture that depends on the advertised latency or throughput. The 3GPP standard for URLLC and the real-world median latency on a US carrier in 2024 were separated by roughly an order of magnitude. The same gap will exist for 6G until at least 2028.

The Skeptic's Corner

Is 6G overhyped? Absolutely, in some ways.

The 1 Tbps peak speed and 0.1ms latency will require ideal conditions — short distances, line of sight, and THz hardware that is currently expensive and power-hungry. Mass-market 6G for a typical smartphone user in 2030 will be fast and low-latency, but probably not "1 Tbps" fast.

The THz spectrum challenges are real. Water vapor, rain, and building materials absorb THz signals aggressively. Making THz-based 6G work in dense urban environments requires the intelligent reflective surface technology to work at scale — which is technically possible but commercially unproven.

And the "AI-native" network vision assumes a level of carrier infrastructure investment and standardization cooperation that has historically been slower than the spec sheets suggest.

The realistic scenario: 6G will deliver meaningful improvements over 5G — perhaps 10x better latency in practice, 5-10x better throughput in real conditions — with genuinely new capabilities (sensing, tighter edge integration) that create real developer opportunities. The revolutionary applications will take a decade after first deployment to reach mainstream scale, just like every previous generation.

Plan for 6G as infrastructure that changes what's architecturally possible, not as a magic wand that arrives at a specific date.


A Concrete Prep Checklist for 2026-2027

The question I get most often is "what should I actually do before 6G lands?" My answer has four items, and they are things you can start this quarter.

Instrument your current app's network characteristics. You probably don't actually know the median and p95 RTT your users are experiencing, broken down by carrier and connection type. Ship a lightweight telemetry probe that records these, with user consent and proper sampling. When 6G starts showing up on traces, you'll know on day one rather than months later when somebody notices. This data also tells you which 5G features you're already entitled to and should be using.

Pick one edge-compute platform and ship something on it. Cloudflare Workers, AWS Wavelength, Azure Edge Zones — they all preview the 6G edge developer experience. You don't need to pick the "right" one; you need to get past the "have deployed nothing at the edge" line. The patterns transfer, and the tooling maturity gap between edge and cloud is closing faster than most backend teams realize.

Separate latency-sensitive and latency-tolerant paths in your architecture now. Even if you're not on 6G yet, the code that will benefit from sub-millisecond networks is almost always the code that has a clear interaction-loop semantic: input → immediate visible response. Refactoring your app so that these paths are explicit (separate services, separate metrics, separate SLOs) pays off today on 5G and will be a unlock on 6G. Apps that conflate interaction-critical and batch-eligible operations will be stuck with 4G-era behaviour long after the underlying network is capable of better.

Watch the standards bodies, lightly. You don't need to read 3GPP specs. You need one or two technical analysts in your RSS feed who summarize what's happening. Ericsson Technology Review, Nokia Bell Labs blog, and the Linux Foundation's O-RAN technical updates are a good starter set. Budget 30 minutes a month on 6G news. That's enough to spot architectural shifts before your competitors do, without making it a distraction.

Conclusion

6G represents the third major inflection point in mobile infrastructure for developers (after 3G's "always-on internet" moment and 4G's "mobile app ecosystem" moment). The sub-millisecond latency, terahertz bandwidth, native AI, and integrated sensing capabilities aren't incremental improvements — they enable categories of applications that are currently impractical.

You have a 2-3 year window before 6G becomes a real deployment target. Use it to understand the architecture, track the standards, and build on 5G edge capabilities that preview the 6G developer experience.

The developers who understand this shift early will design better systems and spot opportunities others miss. Start now.


Sources

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-18 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

Friday, April 17, 2026

Kubernetes Networking: Services, Ingress, Network Policies, and Service Mesh

Hero image

Introduction

Kubernetes networking is the layer that most engineers understand just enough to debug obvious failures — and not enough to prevent subtle ones. Pod-to-pod communication works by default, but "works by default" means all pods can reach all other pods with no access controls. A compromised pod in your frontend namespace can reach your database service. A misconfigured Ingress routes production traffic to a test deployment. A missing network policy allows a compromised dependency to exfiltrate data.

This post covers Kubernetes networking at production depth: how Services work at the IP table level, why ClusterIP vs NodePort vs LoadBalancer matters beyond YAML syntax, Ingress controllers and their performance characteristics, Network Policies as the foundation of zero-trust networking in Kubernetes, the CNI layer (what Cilium actually does), service mesh trade-offs, and the automation stack (ExternalDNS, cert-manager) that makes production cluster management manageable. Each section includes the debugging approach for when things go wrong — because in production, they always do eventually.

The Kubernetes Network Model

Kubernetes mandates three properties of its network model:
1. Every pod gets a unique IP address
2. Pods on the same node communicate without NAT
3. Pods on different nodes communicate without NAT

How this is implemented depends on the CNI (Container Network Interface) plugin: Flannel (simple VXLAN overlay), Calico (BGP + iptables), Cilium (eBPF), or Weave. The API is uniform; the implementation varies significantly in performance and features.

Pod IPs are ephemeral — they change when pods restart. Services provide stable virtual IPs (ClusterIPs) that route to healthy pod endpoints. The kube-proxy component (or its replacement) implements this routing.

Architecture diagram

Services: ClusterIP, NodePort, and LoadBalancer

# ClusterIP: stable virtual IP, only reachable within cluster
apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments
spec:
  type: ClusterIP
  selector:
    app: payment-api        # routes to pods with this label
  ports:
  - port: 80               # service port (what callers use)
    targetPort: 8080       # pod port (what your app listens on)
    protocol: TCP
---
# NodePort: exposes service on each node's IP at a static port
# Use for local development, not production (bypasses Ingress/LoadBalancer)
apiVersion: v1
kind: Service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080        # 30000-32767 range; if omitted, auto-assigned
---
# LoadBalancer: provisions a cloud load balancer (NLB/ELB)
# Each LoadBalancer service costs money (one cloud LB per service)
# Use Ingress to multiplex many services behind one LoadBalancer
apiVersion: v1
kind: Service
spec:
  type: LoadBalancer
  ports:
  - port: 443
    targetPort: 8443

How ClusterIP routing works: kube-proxy (or eBPF in Cilium) watches the Endpoints object (list of healthy pod IPs) for each Service. It programs iptables rules that DNAT the ClusterIP:port to one of the healthy pod IPs using random or round-robin selection. When a pod fails its liveness probe and is removed from Endpoints, it's removed from the iptables rules within seconds. The Service IP is stable; the underlying pod IPs rotate.

The iptables implementation has an O(n) lookup time — at 10,000 services, iptables rules become a performance bottleneck. Cilium's eBPF implementation uses hash tables for O(1) lookup and is the reason large clusters migrate away from kube-proxy.

Ingress: HTTP Routing and TLS Termination

An Ingress resource defines HTTP routing rules: host-based and path-based routing to Services. One Ingress controller (typically one LoadBalancer service) multiplexes traffic to hundreds of backend services.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    # nginx-specific annotations
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/rate-limit-window: "1m"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls-cert  # TLS cert stored as Secret
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /api/v1/payments(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: payment-api
            port:
              number: 80
      - path: /api/v1/orders(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: order-api
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 80

Ingress controller choices in 2026:
- nginx-ingress: most widely deployed, extensive annotation-based configuration, well-documented
- Traefik: native Let's Encrypt integration, middleware chain, better UI
- Kong: API gateway features (auth plugins, rate limiting, request transformation)
- Gateway API (standard): the successor to Ingress, more expressive, better multi-team support

The Gateway API (now stable as of Kubernetes 1.28+) separates cluster-level infrastructure (GatewayClass, Gateway) from application-level routing (HTTPRoute), enabling better multi-tenant control. The Ingress API remains supported but new features are being added to Gateway API.

# Gateway API: replacing Ingress
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-route
spec:
  parentRefs:
  - name: prod-gateway           # references the Gateway (cluster-level)
  hostnames:
  - "api.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api/v1/payments
    backendRefs:
    - name: payment-api
      port: 80
      weight: 90               # 90% traffic to stable
    - name: payment-api-canary
      port: 80
      weight: 10               # 10% traffic to canary — traffic splitting

The Gateway API's traffic splitting capability enables canary deployments at the Ingress layer — weight-based routing without a service mesh.

Network Policies: Zero-Trust Kubernetes Networking

Without Network Policies, all pods in a cluster can reach all other pods. A compromised pod in your CDN service can connect to your database. Network Policies define which pods can talk to which pods.

# Default deny: no ingress or egress from the payments namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}              # applies to all pods in the namespace
  policyTypes:
  - Ingress
  - Egress
---
# Allow: payments pods can receive traffic from API gateway only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-gateway-ingress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: api-gateway     # only from api-gateway namespace
    - podSelector:
        matchLabels:
          app: api-gateway      # and only from api-gateway pods
    ports:
    - port: 8080
      protocol: TCP
---
# Allow: payments pods can reach the database and DNS only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payment-service-egress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: databases
    ports:
    - port: 5432               # PostgreSQL
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - port: 53                 # DNS — don't forget this or pod DNS breaks
      protocol: UDP
    - port: 53
      protocol: TCP

The default-deny-all then explicit-allow pattern is the correct approach for production. Start locked down; add permissions as needed. The common mistake is writing podSelector: {} (allow-all) for both ingress and egress because it "just works" — then you have no network segmentation at all.

Network Policy enforcement requires a CNI that supports it. Flannel does not. Calico, Cilium, and Weave do. Verify your CNI supports Network Policies before applying them — on Flannel clusters, the policies are accepted by the API server but silently not enforced.

Comparison visual

EndpointSlices and Large-Scale Service Routing

The Endpoints object has a 1MB size limit — a single Service with more than ~5,000 pod IPs exceeds this limit and breaks. EndpointSlices (stable since Kubernetes 1.21) shard the endpoint data: each EndpointSlice holds up to 100 endpoints, and kube-proxy watches all EndpointSlices for a Service.

EndpointSlices also reduce kube-proxy CPU load: when a single pod is added to a Service, only the affected EndpointSlice is updated — not the entire endpoint list. At 1,000-pod Services, this reduces the update payload by ~99%.

Topology-aware routing (also called topology hints) assigns EndpointSlices to the same zone as the requesting pod. This reduces cross-zone data transfer costs (in cloud providers, cross-zone traffic is billed at ~$0.01/GB) and latency for services with geographically distributed nodes:

# Enable topology-aware routing on a Service
apiVersion: v1
kind: Service
metadata:
  name: recommendation-api
  annotations:
    service.kubernetes.io/topology-mode: "auto"  # prefer same-zone endpoints
spec:
  selector:
    app: recommendation-api
  ports:
  - port: 80
    targetPort: 8080

With topology-mode: auto, kube-proxy (or Cilium) preferentially routes requests to endpoints in the same availability zone. If no in-zone endpoints are available, it falls back to any endpoint. The annotation works best when endpoints are roughly evenly distributed across zones — severely imbalanced distributions disable the optimization automatically.

Cilium: eBPF-Based Networking

Cilium replaces kube-proxy and extends Network Policies with capabilities not possible in iptables:

# Cilium NetworkPolicy: L7-aware policies (iptables only does L3/L4)
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-l7-policy
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: api-gateway
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/api/v1/charge"   # only allow this specific endpoint
        - method: "GET"
          path: "/api/v1/status"   # and this one
        # All other paths blocked at the network layer

L7-aware policies block specific HTTP paths or gRPC methods at the network layer — not the application layer. A compromised internal service cannot call DELETE /api/v1/all_orders even if it can reach the payment service's pod IP.

Cilium's eBPF implementation also provides:
- Network performance: O(1) service lookup vs O(n) iptables, measurable improvement at >1000 services
- Hubble observability: real-time flow visibility, service dependency maps, network policy verification
- Transparent encryption: WireGuard-based pod-to-pod encryption without application changes
- Bandwidth management: per-pod egress bandwidth limits

The operational trade-off: Cilium requires a more recent Linux kernel (5.4+ for most features, 5.10+ for advanced features) and a more complex installation than vanilla kube-proxy. In 2026, most managed Kubernetes offerings (EKS, GKE, AKS) support or default to Cilium.

ExternalDNS: Automating DNS Record Management

Manually updating DNS records when LoadBalancer IPs change is error-prone. ExternalDNS watches Services and Ingresses and automatically updates Route 53, Cloud DNS, or Cloudflare when load balancer IPs are assigned:

# ExternalDNS deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  template:
    spec:
      containers:
      - name: external-dns
        image: registry.k8s.io/external-dns/external-dns:v0.14.0
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=example.com      # only manage example.com records
        - --provider=aws                   # Route 53
        - --policy=upsert-only             # never delete records (safer)
        - --aws-zone-type=public
        - --log-level=info

When you create an Ingress with host: api.example.com, ExternalDNS automatically creates a Route 53 A record pointing to the Ingress controller's LoadBalancer IP. When the LoadBalancer IP changes (during cluster migration, for example), ExternalDNS updates the record automatically.

Combined with cert-manager (automatic TLS certificate provisioning from Let's Encrypt), you get fully automated HTTPS endpoint management:

# cert-manager ClusterIssuer for Let's Encrypt
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          class: nginx

# Ingress with cert-manager annotation: auto-provisions certificate
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod  # triggers cert provisioning
spec:
  tls:
  - hosts:
    - api.example.com
    secretName: api-example-com-tls   # cert-manager stores cert here
  rules:
  - host: api.example.com
    ...

The combination of ExternalDNS + cert-manager + nginx-ingress is the standard production Kubernetes HTTP stack: deploy an application, annotate the Ingress, and within 2 minutes it's live with a valid HTTPS certificate and DNS record — fully automated.

Service Mesh: When You Need It

A service mesh (Istio, Linkerd, Cilium Service Mesh) adds a sidecar proxy (or eBPF) to each pod that handles:
- Mutual TLS (mTLS) between all services
- Traffic management (canary deployments, circuit breakers, retries)
- Observability (distributed traces, metrics, per-service dashboards)

The value is real. The cost is also real: Istio adds 7-15MB memory per sidecar, 1-3ms latency per hop, and significant operational complexity (CRDs, control plane management, certificate rotation).

Use a service mesh when:
- You require mTLS for compliance (SOC 2, PCI DSS) and can't add TLS to each service individually
- You need circuit breaking, retries, and timeout policies applied consistently across all services
- You want distributed tracing without application-level instrumentation (OpenTelemetry auto-instrumentation is usually sufficient)
- You have 20+ services that need consistent traffic management

Don't use a service mesh when:
- You have fewer than 10 services (overhead exceeds benefit)
- Your team doesn't have operational experience with the service mesh control plane
- You can achieve the same goals with Cilium Network Policies + OpenTelemetry + Gateway API traffic splitting

Linkerd has a much simpler operational model than Istio (no complex VirtualService/DestinationRule CRDs, lower resource overhead, automatic mTLS with near-zero config). If mTLS compliance is the primary driver, Linkerd is the better choice. If advanced traffic management is needed, Istio's capabilities justify its complexity.

Note that Cilium Service Mesh (available when Cilium is your CNI) provides mTLS and basic traffic management via eBPF without sidecars — zero sidecar overhead. For clusters already running Cilium, this is worth evaluating before adopting Istio or Linkerd, since the infrastructure is already in place.

Debugging Kubernetes Network Issues

A systematic approach to the most common networking failures:

Pod can't reach another pod by Service name:

# Step 1: Verify DNS resolves
kubectl exec -it debug-pod -- nslookup payment-api.payments.svc.cluster.local
# If fails: CoreDNS problem or pod DNS config issue

# Step 2: Verify Service has endpoints
kubectl get endpoints payment-api -n payments
# If ADDRESS column is empty: selector doesn't match pod labels

# Step 3: Test direct pod-to-pod connectivity (bypass Service)
kubectl get pods -n payments -o wide          # get pod IP
kubectl exec -it debug-pod -- curl http://10.244.1.5:8080/health
# If succeeds but Service fails: iptables/kube-proxy issue

# Step 4: Check Network Policy isn't blocking
kubectl exec -n payments debug-pod -- curl http://payment-api:80
# If denied: check NetworkPolicy objects in both namespaces

# Cilium policy troubleshooting with Hubble
hubble observe --namespace payments --follow
# Shows dropped packets with reason: network policy rule, etc.

Ingress not routing correctly:

# Check Ingress controller logs
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller | grep payment-api

# Verify Ingress backend resolves
kubectl describe ingress api-ingress -n production
# Look for: "Default backend: default-http-backend:80 (<error>)"

# Check that the backend Service and its pods are healthy
kubectl get svc payment-api -n payments
kubectl get endpoints payment-api -n payments  # must have addresses

# Test with curl from inside the cluster (bypasses Ingress, tests Service)
kubectl run test --image=curlimages/curl --rm -it -- \
    curl http://payment-api.payments.svc.cluster.local/health

Connection timeout to external services:

# Check egress Network Policy allows the external IP/port
kubectl get networkpolicy -n payments

# Test DNS resolution of external service
kubectl exec -it pod -- nslookup api.stripe.com

# Check if traffic is being NATted correctly
kubectl exec -it pod -- curl -v https://api.stripe.com/v1/balance \
    -H "Authorization: Bearer sk_test_..."

Intermittent connection failures under load:
This is often a conntrack table overflow (the kernel's connection tracking table is full). Symptoms: nf_conntrack: table full, dropping packet in node kernel logs. Mitigation:

# On each node
sysctl net.netfilter.nf_conntrack_max
# Increase if close to limit under load
# Also check: sysctl net.netfilter.nf_conntrack_count

With Cilium (eBPF), conntrack is handled in eBPF maps with much higher limits than the kernel conntrack table — this is one of the scaling advantages.

DNS in Kubernetes: CoreDNS and Service Discovery

CoreDNS is the default DNS server in Kubernetes. Services are resolvable via DNS within the cluster:

# DNS name format: <service>.<namespace>.svc.cluster.local
# Short forms also work within the same namespace:
# - <service>                            (same namespace)
# - <service>.<namespace>                (any namespace)
# - <service>.<namespace>.svc            (any namespace)
# - <service>.<namespace>.svc.cluster.local (full FQDN)

# From within payments namespace:
# payment-api              → resolves to ClusterIP
# payment-api.payments     → same
# postgres.databases       → cross-namespace

# Headless services (ClusterIP: None): DNS returns pod IPs
# Used for StatefulSets where you need to reach specific pods

CoreDNS performance issues at scale: with many pods making many DNS lookups, CoreDNS can become a bottleneck. Mitigations:
- ndots:5 default causes each DNS lookup to try 5 search domain suffixes before returning. Set ndots:2 for services that use FQDNs
- NodeLocal DNSCache: DNS cache on each node, reduces CoreDNS load by ~70%
- Increase CoreDNS replicas: HPA on CoreDNS based on requests/second

Production Kubernetes Networking Stack: Recommendations by Scale

Small clusters (1-20 nodes, <100 services):
- CNI: Flannel or Calico (simple, well-understood)
- Service routing: kube-proxy (iptables overhead not visible at this scale)
- Ingress: nginx-ingress (most documentation, easiest debugging)
- Network Policies: Calico provides enforcement even with Flannel overlay
- Service mesh: skip (overhead exceeds benefit)
- DNS automation: ExternalDNS + cert-manager

Medium clusters (20-200 nodes, 100-1000 services):
- CNI: Cilium (eBPF performance advantage becomes measurable, Hubble for visibility)
- Service routing: Cilium replaces kube-proxy
- Ingress: nginx-ingress or Gateway API
- Network Policies: Cilium NetworkPolicy (L7-aware)
- Service mesh: Linkerd if mTLS compliance required; skip otherwise
- Topology-aware routing for cross-zone cost reduction

Large clusters (200+ nodes, 1000+ services):
- CNI: Cilium (iptables simply doesn't scale here)
- EndpointSlices: enabled and monitored
- Ingress: Gateway API (mature at this cluster size)
- Service mesh: likely required (traffic management at this scale benefits from centralized control)
- NodeLocal DNSCache: required (CoreDNS becomes bottleneck without it)

Conclusion

Kubernetes networking has multiple layers, each with distinct concerns. Services provide stable virtual IPs for ephemeral pods. Ingress (and increasingly, Gateway API) multiplexes HTTP traffic with host and path routing. Network Policies provide the access controls that make multi-tenant clusters safe. Cilium's eBPF implementation delivers L7-aware policies and O(1) service lookup at scale. Service meshes add mTLS and advanced traffic management — valuable for large deployments, over-engineered for small ones.

The security posture that production clusters should target: default-deny Network Policies per namespace, explicit-allow ingress and egress for every service, CoreDNS always permitted. Combined with Cilium for enforcement and Hubble for visibility, you have network segmentation comparable to a traditional firewall — but dynamic and programmable through Kubernetes manifests.

The automation stack of ExternalDNS plus cert-manager plus an Ingress controller eliminates manual DNS and certificate management — the operational overhead that previously made Kubernetes networking painful to manage. In 2026, a new service can be deployed, DNS-assigned, and HTTPS-terminated in minutes with no manual intervention. When things go wrong, the debugging workflow follows the layers: DNS resolution, Service endpoints, Network Policy rules, Ingress configuration — in that order. Cilium's Hubble CLI makes network policy debugging particularly tractable, showing dropped packets with the exact policy rule that blocked them.

Sources

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-17 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

Bigger Is Not the Same as Better. The Job That Moved Is the Phone, Not the Lab.

Bigger is a plan. The phone is the receipt. The brief for this cycle is a question: does bigger always mean better in AI? The 2026 answer i...