Tracing Data Missing from Zipkin After Istio Upgrade? Fix Here

When This Happens

You've just updated Istio from 1.12.x to 1.14.x. The service mesh is running fine. Prometheus metrics work. Kiali shows your graph. But when you open Zipkin, you see nothing. Zero traces. No spans. The Zipkin UI just stares back at you with empty query results. This happens right after the upgrade, not before. It's not a network issue – you can curl the Zipkin endpoint from inside a pod and get a 200 OK. The data just isn't reaching it.

What's Actually Going On

Istio 1.14 changed how envoy handles tracing. In 1.12, envoy used the old envoy.filters.network.http_connection_manager tracing config that just forwarded trace headers automatically. In 1.14, they switched to the newer envoy.filters.http.router approach, but the default trace_sampling setting got flipped to 0.0 (meaning no traces). The reason: Istio's devs wanted to reduce overhead by default. But they didn't update the default Zipkin integration to send traces when sampling is zero. So envoy receives trace headers, but it drops them because it thinks you don't want to trace anything.

The key thing: Zipkin itself is fine. Prometheus shows Zipkin is up. The problem is that envoy isn't sending spans to it. And since your upgrade didn't change any MeshConfig or EnvoyFilter that explicitly sets sampling, you get the default zero-sampling behavior.

The Fix

You need to set envoy's trace sampling to a non-zero value. The cleanest way is through Istio's mesh config, not per-pod annotations. Here's how:

Check your current mesh config

kubectl get configmap -n istio-system istio -o yaml | grep -A 10 defaultConfig

Look for tracing: and sampling: lines. If you see sampling: 0 or no tracing block at all, that's your problem.

Edit the istio ConfigMap

kubectl edit configmap -n istio-system istio

Add or modify the defaultConfig section. Here's the block you need:

defaultConfig:
  tracing:
    sampling: 100
    zipkin:
      address: zipkin.istio-system:9411

Sampling value 100 means 100% of requests get traced. That's fine for dev. For production, pick 10 (10%) or 1 (1%) depending on traffic volume. 100 is safe for most small setups because envoy batches spans anyway.

Restart your proxies

Envoy reads config only on startup. So you must restart all pods in the mesh. The quickest way:

kubectl rollout restart deployment -n your-namespace --all

Wait 10-15 seconds per pod for the new config to load.

Verify traces

Hit an endpoint in your mesh. Then check Zipkin after 30 seconds. You should see spans now. If not, move to the next section.

If It Still Fails

Three things to check:

EnvoyFilter blocking tracing: Some older EnvoyFilter patches might explicitly disable tracing. Look for filters that modify the HTTP connection manager. Run kubectl get envoyfilter -A and inspect each one. If any has tracing: {} or disabled: true, that's overriding your mesh config.
Zipkin address mismatch: In Istio 1.14, the default Zipkin address changed from zipkin:9411 to zipkin.istio-system:9411. If you have a custom namespace, update the address in step 2.
Envoy sidecar logs: Check logs of a pod that should send traces. Run kubectl logs -n your-namespace your-pod -c istio-proxy | grep -i trace. If you see trace sampling is disabled, your config change didn't apply – maybe you edited the wrong ConfigMap or forgot to restart the pod.

The real fix here is that envoy's default changed silently. Istio 1.14's documentation mentions the sampling change in the release notes, but buried under "tracing improvements." Most people miss it.

One More Thing

If you're using Kiali alongside Zipkin, you'll also notice Kiali's traces tab shows nothing. That's the same root cause – Kiali queries Zipkin, which gets no data. Fixing the sampling fixes both.

This exact problem hit me after upgrading a small Kubernetes cluster running a Node.js microservice. The Zipkin UI showed the service graph but no spans. Took me two hours to trace it back to the sampling default. Now I check that first on every upgrade.