Jaeger is not showing any trace results - kubectl

Question

I am able to run kiali fine. But it Jaeger is not showing any results. I'm using virtualbox for this exercise. In order for me to view it in my local browser I'm using port forwarding.
I think this is communication issue between pods.
Below is what I'm using.
Virtualbox
Minimal install of CentOS_8.4.2105
istio-1.11.4
Docker version 20.10.9, build c2ea9bc
minikube version: v1.23.2
[centos#centos8 bin]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:32:41Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
here is my Kiali
Below is my Jaeger
[centos#centos8 warmup-exercise]$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-7bdcf77687-w5hvt 1/1 Running 0 3h58m
istio-egressgateway-5547fcc8fc-qsd2l 1/1 Running 0 3h58m
istio-ingressgateway-8f568d595-j6wzd 1/1 Running 0 3h58m
istiod-6659979bdf-9chbn 1/1 Running 0 3h58m
jaeger-5c7c5c8d87-p5678 1/1 Running 0 3h58m
kiali-7fd9f6f484-vlxms 1/1 Running 0 3h58m
prometheus-f5f544b59-br5n4 2/2 Running 0 3h58m
[centos#centos8 warmup-exercise]$ kubectl --namespace istio-system describe pod/jaeger-5c7c5c8d87-p5678
Name: jaeger-5c7c5c8d87-p5678
Namespace: istio-system
Priority: 0
Node: minikube/192.168.49.2
Start Time: Thu, 21 Oct 2021 12:42:47 -0400
Labels: app=jaeger
pod-template-hash=5c7c5c8d87
Annotations: prometheus.io/port: 14269
prometheus.io/scrape: true
sidecar.istio.io/inject: false
Status: Running
IP: 172.17.0.6
IPs:
IP: 172.17.0.6
Controlled By: ReplicaSet/jaeger-5c7c5c8d87
Containers:
jaeger:
Container ID: docker://3e155e7909f9f9976184b0b8f72880307d6bb7f8810d98c25d2dd8f18df342bb
Image: docker.io/jaegertracing/all-in-one:1.20
Image ID: docker-pullable://jaegertracing/all-in-one#sha256:54c2ea315dab7215c51c1b06b111c666f594e90317584f84eabbc59aa5856b13
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 21 Oct 2021 12:49:26 -0400
Ready: True
Restart Count: 0
Requests:
cpu: 10m
Liveness: http-get http://:14269/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:14269/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
BADGER_EPHEMERAL: false
SPAN_STORAGE_TYPE: badger
BADGER_DIRECTORY_VALUE: /badger/data
BADGER_DIRECTORY_KEY: /badger/key
COLLECTOR_ZIPKIN_HTTP_PORT: 9411
MEMORY_MAX_TRACES: 50000
QUERY_BASE_PATH: /jaeger
Mounts:
/badger from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tj4pj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-tj4pj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 51m (x3 over 102m) kubelet Readiness probe failed: Get "http://172.17.0.6:14269/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 51m (x5 over 160m) kubelet Liveness probe failed: Get "http://172.17.0.6:14269/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning NodeNotReady 51m node-controller Node is not ready
[centos#centos8 warmup-exercise]$ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
api-gateway-5cd5c547c6-lrt6k 2/2 Running 0 3h30m 172.17.0.13 minikube <none> <none>
photo-service-7c79458679-trblk 2/2 Running 0 3h30m 172.17.0.11 minikube <none> <none>
position-simulator-6c7b7949f8-k2z7t 2/2 Running 0 3h30m 172.17.0.14 minikube <none> <none>
position-tracker-cbbc8b7f6-dl4gz 2/2 Running 0 3h30m 172.17.0.12 minikube <none> <none>
staff-service-6597879677-7zh2c 2/2 Running 0 3h30m 172.17.0.15 minikube <none> <none>
vehicle-telemetry-c8fcb46c6-n9764 2/2 Running 0 3h30m 172.17.0.10 minikube <none> <none>
webapp-85fd946885-zdjck 2/2 Running 0 3h30m 172.17.0.16 minikube <none> <none>
I'm still learning devops. Let me know if I missed something.

Answer 1

The default sample rate for Jaeger is 1% : https://istio.io/latest/docs/tasks/observability/distributed-tracing/jaeger/