Service Mesh

Moving service-to-service networking concerns (mTLS, retries, traffic shifting, telemetry) into an infrastructure layer of proxies, and when that complexity pays off.

Intent

A service mesh provides a dedicated infrastructure layer for east-west (service-to-service) traffic. A data plane of proxies (sidecars per pod, or node-level/ambient proxies) intercepts traffic; a control plane configures them.

Capabilities: mutual TLS and workload identity, authorization policies, retries/timeouts/outlier detection, traffic splitting for canaries, fault injection, and uniform telemetry (metrics, traces) without code changes.

flowchart LR
  subgraph Pod A
    A[Order service] <--> PA[Proxy]
  end
  subgraph Pod B
    B[Payment service] <--> PB[Proxy]
  end
  PA <-- mTLS --> PB
  CP[Control plane<br/>config, certificates, policy] -.-> PA
  CP -.-> PB
  PA & PB --> TEL[(Metrics & traces)]

Options

Mesh Notes (version-sensitive)
Istio Most feature-rich; sidecar mode and ambient mode (node-level ztunnel + optional waypoint proxies) to reduce per-pod overhead
Linkerd Simpler, lightweight Rust proxy
Cilium Service Mesh eBPF-based data plane
Consul Multi-platform (VMs + Kubernetes)
Cloud-managed Google Cloud Service Mesh, AWS VPC Lattice (application networking), Istio-based add-on for AKS

Example: canary traffic split (Istio)

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: payment }
spec:
  hosts: [payment]
  http:
    - route:
        - destination: { host: payment, subset: v1 }
          weight: 90
        - destination: { host: payment, subset: v2 }
          weight: 10
      timeout: 2s
      retries: { attempts: 2, perTryTimeout: 800ms, retryOn: "5xx,reset,connect-failure" }

Pros and cons

Pros Cons
Uniform mTLS and policy without code changes Operational complexity and upgrade burden
Language-agnostic resilience and telemetry Latency and resource overhead (especially sidecars)
Fine-grained traffic control for releases Debugging through proxies is harder
Foundation for zero trust inside the cluster Overkill for a handful of services

Interview questions

When is a service mesh worth it?

Many services in several languages, strict zero-trust/mTLS requirements, need for consistent traffic management and telemetry, and a platform team to run it. For a few services in one language, libraries and Kubernetes network policies are often enough.

What is ambient mesh?

An Istio data-plane mode without per-pod sidecars: a per-node proxy (ztunnel) handles L4 mTLS, and optional waypoint proxies handle L7 features, reducing resource use and simplifying upgrades.