istio:error installer PersistentVolumeClaim "istio-jaeger-pvc" is invalid - istio

Question

I'm trying to run istioctl install istio-config.yaml command within CodeBuild on AWS but I get this error:
error installer PersistentVolumeClaim "istio-jaeger-pvc" is invalid:
spec.resources.requests.storage: Forbidden: field can not be less than
previous value
even though I don't have the path spec.resources.requests.storage in my configuration file!
This is the content of my file:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
addonComponents:
grafana:
enabled: true
k8s:
replicaCount: 1
istiocoredns:
enabled: false
kiali:
enabled: true
k8s:
replicaCount: 1
prometheus:
enabled: true
k8s:
replicaCount: 1
tracing:
enabled: true
components:
base:
enabled: true
citadel:
enabled: false
k8s:
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
cni:
enabled: false
egressGateways:
- enabled: true
k8s:
resources:
requests:
cpu: 10m
memory: 40Mi
name: istio-egressgateway
ingressGateways:
- enabled: true
k8s:
resources:
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: status-port
port: 15020
targetPort: 15020
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn/to/cert"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
name: istio-ingressgateway
istiodRemote:
enabled: false
pilot:
enabled: true
k8s:
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: GODEBUG
value: gctrace=1
- name: PILOT_TRACE_SAMPLING
value: "100"
- name: CONFIG_NAMESPACE
value: istio-config
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 1
periodSeconds: 3
timeoutSeconds: 5
resources:
requests:
cpu: 10m
memory: 100Mi
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
policy:
enabled: false
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-policy
resources:
requests:
cpu: 10m
memory: 100Mi
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
telemetry:
enabled: false
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: GOMAXPROCS
value: "6"
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-telemetry
replicaCount: 1
resources:
limits:
cpu: 4800m
memory: 4G
requests:
cpu: 50m
memory: 100Mi
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
hub: docker.io/istio
meshConfig:
accessLogFile: /dev/stdout
defaultConfig:
tracing:
sampling: 100
proxyMetadata: {}
disablePolicyChecks: false
enablePrometheusMerge: false
profile: demo
tag: 1.6.3
values:
base:
validationURL: ""
clusterResources: true
gateways:
istio-egressgateway:
autoscaleEnabled: false
env: {}
name: istio-egressgateway
secretVolumes:
- mountPath: /etc/istio/egressgateway-certs
name: egressgateway-certs
secretName: istio-egressgateway-certs
- mountPath: /etc/istio/egressgateway-ca-certs
name: egressgateway-ca-certs
secretName: istio-egressgateway-ca-certs
type: ClusterIP
zvpn: {}
istio-ingressgateway:
applicationPorts: ""
autoscaleEnabled: false
debug: info
domain: ""
env: {}
meshExpansionPorts:
- name: tcp-pilot-grpc-tls
port: 15011
targetPort: 15011
- name: tcp-istiod
port: 15012
targetPort: 15012
- name: tcp-citadel-grpc-tls
port: 8060
targetPort: 8060
- name: tcp-dns-tls
port: 853
targetPort: 8853
name: istio-ingressgateway
secretVolumes:
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
secretName: istio-ingressgateway-certs
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
secretName: istio-ingressgateway-ca-certs
type: LoadBalancer
zvpn: {}
global:
arch:
amd64: 2
ppc64le: 2
s390x: 2
configValidation: true
controlPlaneSecurityEnabled: true
defaultNodeSelector: {}
defaultPodDisruptionBudget:
enabled: true
defaultResources:
requests:
cpu: 10m
enableHelmTest: false
imagePullPolicy: ""
imagePullSecrets: []
istioNamespace: istio-system
istiod:
enableAnalysis: false
enabled: true
jwtPolicy: first-party-jwt
logAsJson: false
logging:
level: default:info
meshExpansion:
enabled: false
useILB: false
meshNetworks: {}
mountMtlsCerts: false
multiCluster:
clusterName: ""
enabled: false
network: ""
omitSidecarInjectorConfigMap: false
oneNamespace: false
operatorManageWebhooks: false
pilotCertProvider: istiod
priorityClassName: ""
proxy:
autoInject: enabled
clusterDomain: cluster.local
componentLogLevel: misc:error
enableCoreDump: false
envoyStatsd:
enabled: false
excludeIPRanges: ""
excludeInboundPorts: ""
excludeOutboundPorts: ""
image: proxyv2
includeIPRanges: '*'
logLevel: warning
privileged: false
readinessFailureThreshold: 30
readinessInitialDelaySeconds: 1
readinessPeriodSeconds: 2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
statusPort: 15020
tracer: zipkin
proxy_init:
image: proxyv2
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 10m
memory: 10Mi
sds:
token:
aud: istio-ca
sts:
servicePort: 0
tracer:
datadog:
address: $(HOST_IP):8126
lightstep:
accessToken: ""
address: ""
stackdriver:
debug: false
maxNumberOfAnnotations: 200
maxNumberOfAttributes: 200
maxNumberOfMessageEvents: 200
zipkin:
address: ""
trustDomain: cluster.local
useMCP: false
grafana:
accessMode: ReadWriteMany
contextPath: /grafana
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- disableDeletion: false
folder: istio
name: istio
options:
path: /var/lib/grafana/dashboards/istio
orgId: 1
type: file
datasources:
datasources.yaml:
apiVersion: 1
env: {}
envSecrets: {}
image:
repository: grafana/grafana
tag: 7.1.3
nodeSelector: {}
persist: false
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
security:
enabled: false
passphraseKey: passphrase
secretName: grafana
usernameKey: username
service:
annotations: {}
externalPort: 3000
name: http
type: ClusterIP
storageClassName: ""
tolerations: []
istiocoredns:
coreDNSImage: coredns/coredns
coreDNSPluginImage: istio/coredns-plugin:0.2-istio-1.1
coreDNSTag: 1.6.2
istiodRemote:
injectionURL: ""
kiali:
contextPath: /kiali
createDemoSecret: false
dashboard:
auth:
strategy: login
grafanaInClusterURL: http://grafana:3000
jaegerInClusterURL: http://tracing/jaeger
passphraseKey: passphrase
secretName: kiali
usernameKey: username
viewOnlyMode: false
hub: quay.io/kiali
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
security:
cert_file: /kiali-cert/cert-chain.pem
enabled: false
private_key_file: /kiali-cert/key.pem
service:
annotations: {}
tag: v1.22.1
mixer:
adapters:
kubernetesenv:
enabled: true
prometheus:
enabled: true
metricsExpiryDuration: 10m
stackdriver:
auth:
apiKey: ""
appCredentials: false
serviceAccountPath: ""
enabled: false
tracer:
enabled: false
sampleProbability: 1
stdio:
enabled: true
outputAsJson: false
useAdapterCRDs: false
policy:
adapters:
kubernetesenv:
enabled: true
useAdapterCRDs: false
autoscaleEnabled: false
image: mixer
sessionAffinityEnabled: false
telemetry:
autoscaleEnabled: false
env:
GOMAXPROCS: "6"
image: mixer
loadshedding:
latencyThreshold: 100ms
mode: enforce
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
replicaCount: 1
sessionAffinityEnabled: false
tolerations: []
pilot:
appNamespaces: []
autoscaleEnabled: false
autoscaleMax: 5
autoscaleMin: 1
configMap: true
configNamespace: istio-config
cpu:
targetAverageUtilization: 80
enableProtocolSniffingForInbound: true
enableProtocolSniffingForOutbound: true
env: {}
image: pilot
keepaliveMaxServerConnectionAge: 30m
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
policy:
enabled: false
replicaCount: 1
tolerations: []
traceSampling: 1
prometheus:
contextPath: /prometheus
hub: docker.io/prom
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provisionPrometheusCert: true
retention: 6h
scrapeInterval: 15s
security:
enabled: true
tag: v2.15.1
tolerations: []
sidecarInjectorWebhook:
enableNamespacesByDefault: false
injectLabel: istio-injection
objectSelector:
autoInject: true
enabled: false
rewriteAppHTTPProbe: true
telemetry:
enabled: true
v1:
enabled: false
v2:
enabled: true
metadataExchange: {}
prometheus:
enabled: true
stackdriver:
configOverride: {}
enabled: false
logging: false
monitoring: false
topology: false
tracing:
jaeger:
hub: docker.io/jaegertracing
memory:
max_traces: 100000
tag: "1.16"
persist: true
spanStorageType: badger
storageClassName: "gp2"
accessMode: ReadWriteOnce
nodeSelector: {}
opencensus:
exporters:
stackdriver:
enable_tracing: true
hub: docker.io/omnition
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 200m
memory: 400Mi
tag: 0.1.9
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provider: jaeger
service:
annotations: {}
externalPort: 9411
name: http-query
type: ClusterIP
zipkin:
hub: docker.io/openzipkin
javaOptsHeap: 700
maxSpans: 500000
node:
cpus: 2
probeStartupDelay: 10
queryPort: 9411
resources:
limits:
cpu: 1000m
memory: 2048Mi
requests:
cpu: 150m
memory: 900Mi
tag: 2.20.0
version: ""
and this is the whole log of the command:
$ istioctl install istio-config.yaml
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
- Processing resources for Istio core.
✔ Istio core installed
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
- Processing resources for Istiod.
✔ Istiod installed
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
- Processing resources for Egress gateways, Ingress gateways.
✔ Egress gateways installed
- Processing resources for Ingress gateways.
✔ Ingress gateways installed
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.2021-03-08T09:26:21.151700Z error installer PersistentVolumeClaim "istio-jaeger-pvc" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
- Processing resources for Addons.
✘ Addons encountered an error: PersistentVolumeClaim "istio-jaeger-pvc" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value
- Pruning removed resourcesError: failed to apply manifests: errors occurred during operation
This is more details about the pvc istio-jaeger-pvc:
$ kubectl get persistentvolumeclaim/istio-jaeger-pvc -n istio-system -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-provisioner":"kubernetes.io/aws-ebs","volume.kubernetes.io/storage-resizer":"kubernetes.io/aws-ebs"},"finalizers":["kubernetes.io/pvc-protection"],"labels":{"app":"jaeger","install.operator.istio.io/owning-resource":"installed-state","install.operator.istio.io/owning-resource-namespace":"istio-system","operator.istio.io/component":"AddonComponents","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.6.8","release":"istio"},"name":"istio-jaeger-pvc","namespace":"istio-system"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"15Gi"}},"storageClassName":"gp2","volumeMode":"Filesystem"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
volume.kubernetes.io/selected-node: ip-10-8-121-54.eu-central-1.compute.internal
volume.kubernetes.io/storage-resizer: kubernetes.io/aws-ebs
creationTimestamp: "2021-01-11T10:37:18Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app: jaeger
install.operator.istio.io/owning-resource: installed-state
install.operator.istio.io/owning-resource-namespace: istio-system
operator.istio.io/component: AddonComponents
operator.istio.io/managed: Reconcile
operator.istio.io/version: 1.6.8
release: istio
name: istio-jaeger-pvc
namespace: istio-system
resourceVersion: "47732761"
selfLink: /api/v1/namespaces/istio-system/persistentvolumeclaims/istio-jaeger-pvc
uid: 66beac27-7ddb-46ce-9061-af0578bd4b89
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15Gi
storageClassName: gp2
volumeMode: Filesystem
volumeName: pvc-66beac27-7ddb-46ce-9061-af0578bd4b89
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 15Gi
phase: Bound
$ kubectl describe persistentvolumeclaim/istio-jaeger-pvc -n istio-system
Name: istio-jaeger-pvc
Namespace: istio-system
StorageClass: gp2
Status: Bound
Volume: pvc-66beac27-7ddb-46ce-9061-af0578bd4b89
Labels: app=jaeger
install.operator.istio.io/owning-resource=installed-state
install.operator.istio.io/owning-resource-namespace=istio-system
operator.istio.io/component=AddonComponents
operator.istio.io/managed=Reconcile
operator.istio.io/version=1.6.8
release=istio
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-provisioner":"kubernetes.i...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
volume.kubernetes.io/selected-node: ip-10-8-121-54.eu-central-1.compute.internal
volume.kubernetes.io/storage-resizer: kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 15Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: istio-tracing-85fb6445f-qs7qp
Events: <none>

Answer 1

The solution for this one is to simply increase the memory size in the istio-config.yaml file.
in my case, I'm updating the PVC and it looks like it's already filled with data and decreasing it wasn't an option for istio, so I increased it in the config file instead:
tracing:
jaeger:
hub: docker.io/jaegertracing
memory:
max_traces: 100000
tag: "1.16"
persist: true
spanStorageType: badger
storageClassName: "gp2"
accessMode: ReadWriteOnce
nodeSelector: {}
opencensus:
exporters:
stackdriver:
enable_tracing: true
hub: docker.io/omnition
resources:
limits:
cpu: "1"
memory: 15Gi # I increased this one
requests:
cpu: 200m
memory: 15Gi # and this one