In my setup we send all the calls going out of cluster to an Internal Load Balancer in GCP. We do this by creating a egress service and manually adding endpoints to this service. The endpoint to this service is the IP of the Internal load balancer.
[sourabh.w#K9-MAC-035 r19-3]$ k get svc,ep -n egproxy-lle
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/egproxy ClusterIP 10.206.180.135 80/TCP,443/TCP 4d
NAME ENDPOINTS AGE
endpoints/egproxy 10.207.132.8:30443,10.207.132.8:30080 4d
[sourabh.w#K9-MAC-035 r19-3]$
For all micro-services in my setup, they have to run an "openssl s_Client" command at startup. This command is failing for me.
openssl s_client -servername ae17-api.kohlsecommerce.com -connect ae17-api.kohlsecommerce.com:443 -debug -state
While doing this I make sure ae17-api.kohlsecommerce.com is mapped to egproxy service's IP(10.206.180.135) in /etc/hosts.
Here is the o/p when I run openssl inside pod:
root#product-26-655f4f55b6-g2bpq:/# openssl s_client -servername ae17-api.kohlsecommerce.com -connect ae17-api.kohlsecommerce.com:443 -state -debug
CONNECTED(00000003)
SSL_connect:before SSL initialization
write to 0x556dc50b2860 [0x556dc50c3a20] (212 bytes => 212 (0xD4))
0000 - 16 03 01 00 cf 01 00 00-cb 03 03 43 59 24 26 31 ...........CY$&1
0010 - 4f 13 80 47 f2 09 25 f7-ec 74 40 57 7c d0 bc c6 O..G..%..t#W|...
0020 - 18 9b a7 a3 3c 38 80 d6-f4 99 62 00 00 38 c0 2c ....<8....b..8.,
0030 - c0 30 00 9f cc a9 cc a8-cc aa c0 2b c0 2f 00 9e .0.........+./..
0040 - c0 24 c0 28 00 00 c0 23-c0 88 00 67 c0 0a c0 14 .$.(.k.#.'.g....
0050 - 00 39 c0 09 c0 00 00 33-00 9d 00 9c 00 3d 00 3c .9.....3.....=.<
0060 - 00 35 00 2f 00 ff 01 00-66 6a 00 00 00 20 00 1e .5./.....j... ..
0070 - 00 00 1b 61 65 31 37 2d-61 70 69 2e 6b 6f 68 6c ...ae17-api.
0080 - 73 65 63 6f 6d 6d 65 72-63 65 2e 63 6f 6d 00 0b ecommerce.com..
0090 - 00 04 03 00 01 02 00 8a-00 0a 00 08 00 1d 00 17 ................
00a0 - 00 19 00 18 00 00 00 66-00 16 00 00 00 17 00 00 .....#..........
00b0 - 00 0d 00 20 00 00 06 01-06 02 06 03 05 01 05 02 ... ............
00c0 - 05 03 04 01 04 02 04 03-03 01 03 02 03 03 02 01 ................
00d0 - 02 02 02 03 ....
SSL_connect:SSLv3/TLS write client hello
read from 0x556dc50b2860 [0x556dc50ba803] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
SSL_connect:error in SSLv3/TLS write client hello
write:errno=104
no peer certificate available
No client certificate CA names sent
SSL handshake has read 0 bytes and written 212 bytes
Verification: OK
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1553126020
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
I tried creating various set of serviceentries and virtualservices but nothing worked:
With Load Balancer IP:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
generation: 1
name: egproxy-ext
namespace: r19-3-mui-qa
spec:
addresses:
- 10.207.132.8/32
endpoints:
- address: 10.207.132.8
hosts:
- istio-ilb.lle-mcommerce.com
location: MESH_INTERNAL
ports:
- name: http
number: 30080
protocol: HTTP
- name: https
number: 30443
protocol: HTTPS
resolution: STATIC
with egproxy serivce FQDN:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: egproxy-headless-service-fqdn-ext
namespace: r19-3-mui-qa
spec:
addresses:
- 10.206.117.116/32
endpoints:
- address: 10.207.132.8
hosts:
- egproxy.egproxy-lle.svc.cluster.local
location: MESH_INTERNAL
ports:
- name: http
number: 30080
protocol: HTTP
- name: https
number: 30443
protocol: HTTPS
resolution: STATIC
Destination rule for Load balancer:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egress-gateway
namespace: default
spec:
host: istio-ilb.lle-mcommerce.com
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 30443
tls:
mode: SIMPLE
Destination rule for egproxy service
Want the communication to work from microservices to ILB via headless egproxy service.
Workarounds like "egress-gateway" is also a viable option but for that also facing problems in putting together correct config to make it work.
In Istio, to access a service, you need to configure either Kubernetes Service or Istio ServiceEntry. You may need to disable mutual TLS. See this preliminary example https://deploy-preview-3899--preliminary-istio.netlify.com/docs/examples/advanced-gateways/egress-kubernetes-services/.
Related
I have the following value:
stdout_lines: [
[
"iso.3.6.1.2.1.17.4.3.1.1.0.80.121.102.104.4 \"00 50 79 66 68 04 \""
],
[
"iso.3.6.1.2.1.17.4.3.1.1.0.80.121.102.104.6 \"00 50 79 66 68 06 \""
],
[
"iso.3.6.1.2.1.17.4.3.1.1.0.80.121.102.104.8 \"00 50 79 66 68 08 \""
]
]
I want to get the MAC address values in the following form:
00:50:79:66:68:04
00:50:79:66:68:06
00:50:79:66:68:08
That's what I'm trying to do in my playbook:
- set_fact:
mac: "{{ stdout_lines|first|regex_replace(_regex, _replace)|trim }}"
vars:
_regex: '.*"(.*)"'
_replace: '\1'
- set_fact:
matched: "{{ matched|d([]) + [item[2:]|join(':')] }}"
with_items:
- "{{ mac }}"
It turns out some nonsense. What am I doing wrong?
flatten the data then map regex_replace and trim. For example
- set_fact:
mac: "{{ stdout_lines|
flatten|
map('regex_replace', _regex, _replace)|
map('trim')|
map('split')|
map('join', ':')|
list }}"
vars:
_regex: '.*"(.*)"'
_replace: '\1'
gives
mac:
- 00:50:79:66:68:04
- 00:50:79:66:68:06
- 00:50:79:66:68:08
try this playbook: flatten the list, trap the right part of string with regex_search, trim and replace the space by :
- name: "make this working"
hosts: localhost
vars:
mac:
- - iso.3.6.1.2.1.17.4.3.1.1.0.80.121.102.104.4 "00 50 79 66 68 04 "
- - iso.3.6.1.2.1.17.4.3.1.1.0.80.121.102.104.6 "00 50 79 66 68 06 "
- - iso.3.6.1.2.1.17.4.3.1.1.0.80.121.102.104.8 "00 50 79 66 68 08 "
tasks:
- set_fact:
result: "{{ result | d([]) + [reg] }}"
loop: "{{ mac | flatten }}"
vars:
reg: "{{ item | regex_search('(\\d\\d ){6}') | trim | replace(' ',':')}}"
- debug:
var: result
result:
ok: [localhost] => {
"result": [
"00:50:79:66:68:04",
"00:50:79:66:68:06",
"00:50:79:66:68:08"
]
}
My AWS EC2 Ubuntu / Django container ran fine for 21 months, then 26th Dec 6:30am stopped.
I took a snapshot, attached it to another container, and downloaded the logs from /etc/log and searched for 'error', 'fail', etc.
The most likely error is in /var/log/kern.log.1:
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 0.000000] You might have to change the root device
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 0.000000] in your root= kernel command line option
...
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 7.907183] new mount options do not match the existing superblock, will be ignored
/etc/fstab contains just:
LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0
As suggested in this answer, I compared the contents of my snapshot /var/fstab with my fresh new container fstab. The only difference was that the final number was zero instead of one. As expected, when I changed it to one, and re-attached it as sda1, the container still failed to boot.
Another error reported, in the logs displayed when you click on the EC2 instance in AWS console, then Monitor and Troubleshoot -> Get System Log, is:
Failed to start Service for snap ap…amazon-ssm-agent.amazon-ssm-agent
I think this in hibernate.log explains it:
ERROR Health ping failed with error - EC2RoleRequestError: no EC2 instance role found
This seems to be generated when the SSM Agent cannot find IAM credentials. I assume this is caused by the disk error above? Although I don't understand how anything is logged in the first place if there's no disk access.
Is there anything persistent on the original container instance that is not in the EBS volume? I ask because I haven't dared stop the original container - I have just taken a snapshot, converted it to a volume, assigned it to /dev/xvdf1 on a brand new Ubuntu 18 micro container (same as original), edited fstab, then re-assigned it to the new container as /dev/sda1, which seems to then be treated as the boot device.
I do get that I should be Dockerized and a container fail should be a non-event; I work with elastic ECS clusters on other, larger projects. If approved by our Ministry of Education this'll shoot past all of them to 40mn users, at which time I'll Dockerize! But in the meantime I'm self-funded on a shoestring so cutting all available corners.
As much of /var/log/kern.log.1 as stackoverflow allows:
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.187911] acpiphp: Slot [23] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.191920] acpiphp: Slot [24] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.195973] acpiphp: Slot [25] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.200038] acpiphp: Slot [26] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.203937] acpiphp: Slot [27] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.207896] acpiphp: Slot [28] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.211921] acpiphp: Slot [29] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.215931] acpiphp: Slot [30] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.219956] acpiphp: Slot [31] registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.223910] PCI host bridge to bus 0000:00
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.227411] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.231413] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.235413] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.239413] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.243413] pci_bus 0000:00: root bus resource [bus 00-ff]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.247651] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.254759] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.259402] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.262039] pci 0000:00:01.1: reg 0x20: [io 0xc100-0xc10f]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.264462] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.267410] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.271417] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.275415] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.280563] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.283472] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.283472] * this clock source is slow. Consider trying other clock sources
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.290913] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.293774] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.296348] pci 0000:00:02.0: reg 0x10: [mem 0xf0000000-0xf1ffffff pref]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.299906] pci 0000:00:02.0: reg 0x14: [mem 0xf3000000-0xf3000fff]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.308109] pci 0000:00:03.0: [5853:0001] type 00 class 0xff8000
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.312703] pci 0000:00:03.0: reg 0x10: [io 0xc000-0xc0ff]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.315997] pci 0000:00:03.0: reg 0x14: [mem 0xf2000000-0xf2ffffff pref]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.332618] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.335725] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.339727] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.343720] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.359284] xen:balloon: Initialising balloon driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.359501] iommu: Default domain type: Translated
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.363549] SCSI subsystem initialized
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.367464] libata version 3.00 loaded.
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.367580] pci 0000:00:02.0: vgaarb: setting as boot VGA device
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.371403] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.371413] pci 0000:00:02.0: vgaarb: bridge control possible
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.375413] vgaarb: loaded
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.378661] ACPI: bus type USB registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.379440] usbcore: registered new interface driver usbfs
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.383425] usbcore: registered new interface driver hub
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.387423] usbcore: registered new device driver usb
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.391445] pps_core: LinuxPPS API ver. 1 registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.395410] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti#linux.it>
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.399417] PTP clock support registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.403487] EDAC MC: Ver: 3.0.0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.407980] PCI: Using ACPI for IRQ routing
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.411413] PCI: pci_cache_line_size set to 64 bytes
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.412104] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.412230] NetLabel: Initializing
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.415410] NetLabel: domain hash size = 128
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.419411] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.423436] NetLabel: unlabeled traffic allowed by default
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.427552] hpet: 3 channels of 0 reserved for per-cpu timers
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.431421] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.435409] hpet0: 3 comparators, 64-bit 62.500000 MHz counter
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.441465] clocksource: Switched to clocksource xen
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.457119] *** VALIDATE bpf ***
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.461224] VFS: Disk quotas dquot_6.6.0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.466176] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.474200] *** VALIDATE ramfs ***
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.478173] *** VALIDATE hugetlbfs ***
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.482131] AppArmor: AppArmor Filesystem Enabled
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.486935] pnp: PnP ACPI init
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.490705] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.497810] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.497902] system 00:01: [io 0x08a0-0x08a3] has been reserved
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.503120] system 00:01: [io 0x0cc0-0x0ccf] has been reserved
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.508523] system 00:01: [io 0x04d0-0x04d1] has been reserved
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513842] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513874] xen: --> pirq=17 -> irq=8 (gsi=8)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513890] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513916] xen: --> pirq=18 -> irq=12 (gsi=12)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513930] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513951] xen: --> pirq=19 -> irq=1 (gsi=1)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513961] pnp 00:04: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513981] xen: --> pirq=20 -> irq=6 (gsi=6)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513982] pnp 00:05: [dma 2]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.513994] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.514021] xen: --> pirq=21 -> irq=4 (gsi=4)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.514033] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.514086] system 00:07: [io 0x10c0-0x1141] has been reserved
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.520268] system 00:07: [io 0xb044-0xb047] has been reserved
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.527019] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.538353] pnp: PnP ACPI: found 8 devices
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.543610] thermal_sys: Registered thermal governor 'fair_share'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.543611] thermal_sys: Registered thermal governor 'bang_bang'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.549630] thermal_sys: Registered thermal governor 'step_wise'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.555825] thermal_sys: Registered thermal governor 'user_space'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.561805] thermal_sys: Registered thermal governor 'power_allocator'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.572322] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.587520] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.593297] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.599606] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.606415] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.613339] NET: Registered protocol family 2
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.618000] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.625663] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.633738] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.641541] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.648563] TCP: Hash tables configured (established 8192 bind 8192)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.655582] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.662660] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.669898] NET: Registered protocol family 1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.674699] NET: Registered protocol family 44
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.679748] pci 0000:00:01.0: PIIX3: Enabling Passive Release
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.686040] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.691783] pci 0000:00:01.0: Activating ISA DMA hang workarounds
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.698107] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.706446] PCI: CLS 0 bytes, default 64
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 1.710323] Trying to unpack rootfs image as initramfs...
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.005772] Freeing initrd memory: 20584K
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.010438] check: Scanning for low memory corruption every 60 seconds
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.016818] Initialise system trusted keyrings
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.021163] Key type blacklist registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.024937] workingset: timestamp_bits=36 max_order=18 bucket_order=0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.191161] zbud: loaded
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.194718] squashfs: version 4.0 (2009/01/31) Phillip Lougher
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.200798] fuse: init (API version 7.31)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.204963] *** VALIDATE fuse ***
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.208296] *** VALIDATE fuse ***
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.212263] Platform Keyring initialized
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.220327] Key type asymmetric registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.224328] Asymmetric key parser 'x509' registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.229003] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 244)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.235919] io scheduler mq-deadline registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.240902] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.247469] intel_idle: Please enable MWAIT in BIOS SETUP
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.247592] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.254319] ACPI: Power Button [PWRF]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.258321] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.265503] ACPI: Sleep Button [SLPF]
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.274729] xen: --> pirq=22 -> irq=28 (gsi=28)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.274878] xen:grant_table: Grant tables using version 1 layout
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.280458] Grant table initialized
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.284017] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.290075] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.333228] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.342248] Linux agpgart interface v0.103
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.443466] loop: module loaded
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.447771] Invalid max_queues (4), will use default max: 1.
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.455722] ata_piix 0000:00:01.1: version 2.13
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.457270] scsi host0: ata_piix
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.462581] scsi host1: ata_piix
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.467378] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.474736] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.486334] libphy: Fixed MDIO Bus: probed
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.491567] tun: Universal TUN/TAP device driver, 1.6
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.497536] PPP generic driver version 2.4.2
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.502914] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.510801] ehci-pci: EHCI PCI platform driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.517292] ehci-platform: EHCI generic platform driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.523953] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.531590] ohci-pci: OHCI PCI platform driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.537455] ohci-platform: OHCI generic platform driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.544598] uhci_hcd: USB Universal Host Controller Interface driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.552431] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.565496] serio: i8042 KBD port at 0x60,0x64 irq 1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.571530] serio: i8042 AUX port at 0x60,0x64 irq 12
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.578006] mousedev: PS/2 mouse device common for all mice
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.586224] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.596276] blkfront: xvda: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.607036] rtc_cmos 00:02: registered as rtc0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.611669] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.619503] device-mapper: uevent: version 1.0.3
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.625356] device-mapper: ioctl: 4.41.0-ioctl (2019-09-16) initialised: dm-devel#redhat.com
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.635175] platform eisa.0: Probing EISA bus 0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.638713] platform eisa.0: EISA: Cannot allocate resource for mainboard
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.647013] platform eisa.0: Cannot allocate resource for EISA slot 1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.655615] platform eisa.0: Cannot allocate resource for EISA slot 2
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.663198] xvda: xvda1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.666858] platform eisa.0: Cannot allocate resource for EISA slot 3
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.674483] platform eisa.0: Cannot allocate resource for EISA slot 4
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.682619] platform eisa.0: Cannot allocate resource for EISA slot 5
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.690633] platform eisa.0: Cannot allocate resource for EISA slot 6
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.697216] platform eisa.0: Cannot allocate resource for EISA slot 7
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.703807] platform eisa.0: Cannot allocate resource for EISA slot 8
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.710494] platform eisa.0: EISA: Detected 0 cards
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.717001] intel_pstate: CPU model not supported
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.724375] drop_monitor: Initializing network drop monitor service
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.733417] NET: Registered protocol family 10
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.743737] Segment Routing with IPv6
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.749681] NET: Registered protocol family 17
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.754856] Key type dns_resolver registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.759478] RAS: Correctable Errors collector initialized.
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.764934] *** VALIDATE rdt ***
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.768803] resctrl: L3 allocation detected
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.773572] IPI shorthand broadcast: enabled
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.778278] sched_clock: Marking stable (2067715763, 710545130)->(3227957399, -449696506)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.786706] registered taskstats version 1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.791310] Loading compiled-in X.509 certificates
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.797014] Loaded X.509 cert 'Build time autogenerated kernel key: 2937964050fa7e9ded592347912fc3fa79bbb107'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.806996] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.817402] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.827202] zswap: loaded using pool lzo/zbud
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.832084] Key type ._fscrypt registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.836916] Key type .fscrypt registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.844234] Key type big_key registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.849923] Key type encrypted registered
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.854132] AppArmor: AppArmor sha1 policy hashing enabled
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.860047] ima: No TPM chip found, activating TPM-bypass!
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.865891] ima: Allocated hash algorithm: sha1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.871732] ima: No architecture policies found
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.877725] evm: Initialising EVM extended attributes:
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.884451] evm: security.selinux
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.888512] evm: security.SMACK64
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.892742] evm: security.SMACK64EXEC
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.897787] evm: security.SMACK64TRANSMUTE
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.903102] evm: security.SMACK64MMAP
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.907734] evm: security.apparmor
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.911827] evm: security.ima
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.915811] evm: security.capability
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.920478] evm: HMAC attrs: 0x1
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.925235] xenbus_probe_frontend: Device with no driver: device/vif/0
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.933397] PM: Magic number: 1:524:368
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.938702] rtc_cmos 00:02: setting system clock to 2021-12-27T06:22:37 UTC (1640586157)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.950506] Freeing unused decrypted memory: 2040K
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.957449] Freeing unused kernel image memory: 2656K
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.963719] Write protecting the kernel read-only data: 22528k
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.971212] Freeing unused kernel image memory: 2008K
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.977588] Freeing unused kernel image memory: 1484K
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 2.993183] x86/mm: Checked W+X mappings: passed, no W+X pages found.
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.001984] x86/mm: Checking user space page tables
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.018117] tsc: Refined TSC clocksource calibration: 2400.002 MHz
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.027256] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983938a92, max_idle_ns: 440795216168 ns
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.040828] x86/mm: Checked W+X mappings: passed, no W+X pages found.
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.049352] Run /init as init process
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.262764] xen_netfront: Initialising Xen virtual ethernet driver
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.282477] cryptd: max_cpu_qlen set to 1000
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.328241] AVX2 version of gcm_enc/dec engaged.
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 3.333653] AES CTR mode by8 optimization enabled
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.175419] raid6: avx2x4 gen() 23237 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.227416] raid6: avx2x4 xor() 14129 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.279418] raid6: avx2x2 gen() 20103 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.331413] raid6: avx2x2 xor() 12258 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.383415] raid6: avx2x1 gen() 17503 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.435416] raid6: avx2x1 xor() 11955 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.487417] raid6: sse2x4 gen() 12725 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.539416] raid6: sse2x4 xor() 7896 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.591418] raid6: sse2x2 gen() 10688 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.643426] raid6: sse2x2 xor() 7083 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.695418] raid6: sse2x1 gen() 9197 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.747416] raid6: sse2x1 xor() 6511 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.752299] raid6: using algorithm avx2x4 gen() 23237 MB/s
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.758123] raid6: .... xor() 14129 MB/s, rmw enabled
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.763565] raid6: using avx2x2 recovery algorithm
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.771239] xor: automatically using best checksumming function avx
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.781240] async_tx: api initialized (async)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.854695] Btrfs loaded, crc32c=crc32c-intel
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 5.891555] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 6.588912] Loading iSCSI transport class v2.0-870.
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 6.658308] iscsi: registered transport (tcp)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 6.725913] EXT4-fs (xvda1): re-mounted. Opts: discard
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 6.845798] iscsi: registered transport (iser)
Dec 27 06:22:42 ip-172-31-42-17 kernel: [ 7.907183] new mount options do not match the existing superblock, will be ignored
I've been trying to create an EKS cluster with vpc-cni addon due to the pod restrictions for m5.xlarge VMs (57). After creation I can see it is passed to the launchtemplate object but when doing a node describe it still can allocate the previous (wrong?) number
ClusterConfig:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: exchange-develop
region: us-east-1
version: '1.21'
managedNodeGroups:
- name: default
labels:
worker: default
instanceType: m5.xlarge
desiredCapacity: 2
minSize: 2
maxSize: 4
tags:
'k8s.io/cluster-autoscaler/enabled': 'true'
'k8s.io/cluster-autoscaler/exchange-develop': 'owned'
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::658464581062:policy/eks-csi-driver-policy
- arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
- arn:aws:iam::658464581062:policy/ALBIngressControllerIAMPolicy
- arn:aws:iam::658464581062:policy/ExternalDNSPlicy
- arn:aws:iam::658464581062:policy/eks-cluster-autoscaler
maxPodsPerNode: 110
availabilityZones: ['us-east-1c', 'us-east-1d']
iam:
withOIDC: true
vpc:
cidr: 10.10.0.0/16
#autoAllocateIPv6: true
# disable public access to endpoint and only allow private access
clusterEndpoints:
publicAccess: true
privateAccess: true
addons:
- name: vpc-cni
version: '1.10.1'
Launch template with redacted data:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=***
--
Content-Type: text/x-shellscript
Content-Type: charset="us-ascii"
#!/bin/sh
set -ex
sed -i -E "s/^USE_MAX_PODS=\"\\$\{USE_MAX_PODS:-true}\"/USE_MAX_PODS=false/" /etc/eks/bootstrap.sh
KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq ".maxPods=110" $KUBELET_CONFIG)" > $KUBELET_CONFIG
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=<>
API_SERVER_URL=<>
K8S_CLUSTER_DNS_IP=<>
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=exchange-develop,alpha.eksctl.io/nodegroup-name=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=default,worker=default,eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d --max-pods=58' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false
Node description:
Name: ip-10-10-19-34.ec2.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=exchange-develop
alpha.eksctl.io/nodegroup-name=default
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m5.xlarge
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=default
eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd
eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c
kubernetes.io/arch=amd64
kubernetes.io/hostname=<<
kubernetes.io/os=linux
node.kubernetes.io/instance-type=m5.xlarge
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1c
worker=default
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 02 Dec 2021 10:22:20 -0300
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:40 -0300 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.10.19.34
ExternalIP: <<
Hostname: <<
InternalDNS: <<
ExternalDNS: <<
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 4
ephemeral-storage: 83873772Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 15921236Ki
pods: 58
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 3920m
ephemeral-storage: 76224326324
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 14904404Ki
pods: 58
System Info:
Machine ID: ec28ac2717ec395cdf5b4e37f7672569
System UUID: ec28ac27-17ec-395c-df5b-4e37f7672569
Boot ID: 50b3d3d9-5dfa-40b6-99c8-20873632c7ca
Kernel Version: 5.4.156-83.273.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.7
Kubelet Version: v1.21.5-eks-bc4871b
Kube-Proxy Version: v1.21.5-eks-bc4871b
ProviderID: aws:///<<<
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-9z7pw 25m (0%) 0 (0%) 0 (0%) 0 (0%) 61m
kube-system kube-proxy-2slc8 100m (2%) 0 (0%) 0 (0%) 0 (0%) 61m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 125m (3%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
See allocatable pods 58...
So, what is the correct way of using eksctl to create a cluster with vpc-cni and pass the maxPodsPerNode argument to ec2 launch template?
EDIT:
Other things I've tried:
Create cluster from scratch with vpc-cni addon 2 managedNodeGroups
with maxPodsPerNode and without, both will take the value of 58 and
not 110
Add another nodegroup with eksctl create nodegroup, still 58
Add another nodegroup with EKS AWS UI, still 58
Eksctl version 0.75.0
Kubectl version 1.21.2
For managedNodeGroup you need to specify the AMI ID:
aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id --region us-east-1 --query "Parameter.Value" --output text
managedNodeGroups:
- name: default
...
maxPodsPerNode: 110
ami: ami-00836a7940260f6dd
overrideBootstrapCommand: |
#!/bin/bash
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd'
I'm following blog post about using ELK stack. Machine for instalation is an amazon small Ubuntu instance.
I got to the point when I need to install Kibana service so I run:
sudo apt-get install kibana
Then I changed in /etc/kibana/kibana.yml
server.port: 5601
elasticsearch.url: "0.0.0.0:9200"
since I can get response from elasticsearch sudo curl 0.0.0.0:9200
then I run
sudo service kibana start
And after running sudo service kibana status I receiving:
x#ip-xx-xx-xx-xx:/$ sudo service kibana status
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2016-12-02 13:52:55 UTC; 13ms ago
Main PID: 5921 (node)
Tasks: 6
Memory: 1.1M
CPU: 3ms
CGroup: /system.slice/kibana.service
└─5921 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
Dec 02 13:52:55 ip-xx-xx-xx-xx systemd[1]: Started Kibana.
x#ip-xx-xx-xx-xx:/$ sudo service kibana status
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Dec 02 13:52:56 ip-xx-xx-xx-xx kibana[5921]: buildSha: '8f2ace746d1b84702bb618308efa65dc0c3f8a34' },
Dec 02 13:52:56 ip-xx-xx-xx-xx kibana[5921]: dev: { basePathProxyTarget: 5603 },
Dec 02 13:52:56 ip-xx-xx-xx-xx kibana[5921]: pid: { exclusive: false },
Dec 02 13:52:56 ip-xx-xx-xx-xx systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Dec 02 13:52:56 ip-xx-xx-xx-xx systemd[1]: kibana.service: Unit entered failed state.
Dec 02 13:52:56 ip-xx-xx-xx-xx systemd[1]: kibana.service: Failed with result 'exit-code'.
Dec 02 13:52:57 ip-xx-xx-xx-xx systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Dec 02 13:52:57 ip-xx-xx-xx-xx systemd[1]: Stopped Kibana.
Dec 02 13:52:57 ip-xx-xx-xx-xx systemd[1]: kibana.service: Start request repeated too quickly.
Dec 02 13:52:57 ip-xx-xx-xx-xx systemd[1]: Failed to start Kibana.
Unfortunatelly log is not created under directory /var/log/kibanaeven after setting rights by chown kibana:kibana /var/log/kibana:
ll /var/log/kibana/
total 8
drwxr-xr-x 2 kibana kibana 4096 Dec 2 10:20 ./
drwxrwxr-x 9 root syslog 4096 Dec 2 09:50 ../
First of all I wish to see Kibana log (whole problem resolution will be even better :) )
I was ready to use SES for production so I had my sending limits increased. This is the email from AWS:
"Congratulations! After reviewing your case, we have increased your sending quota to 50,000 messages per day and your maximum send rate to 14 messages per second in AWS Region US East (N. Virginia). Your account has also been moved out of the sandbox, so you no longer need to verify recipient addresses."
I configured sSMTP so I can send email using the mail command, using AWS endpoint and generated SMTP credentials. I send an email and I get this:
"Oct 17 14:08:10 ia sSMTP[20486]: 554 Message rejected: Email address is not verified. The following identities failed the check in region US-EAST-1: root , root#ia.internal.vdopia.com"
The SMTP endpoint is: email-smtp.us-east-1.amazonaws.com:587
What am I doing wrong?
Updated
Output of syslog:
Output of syslog:
Oct 19 07:29:44 ia sSMTP[427]: Creating SSL connection to host
Oct 19 07:29:44 ia sSMTP[427]: 220 email-smtp.amazonaws.com ESMTP SimpleEmailService-1652178317 ANxvvoY79LhkdX5l8cYI
Oct 19 07:29:44 ia sSMTP[427]: EHLO ia.internal.vdopia.com
Oct 19 07:29:44 ia sSMTP[427]: 250 Ok
Oct 19 07:29:44 ia sSMTP[427]: STARTTLS
Oct 19 07:29:44 ia sSMTP[427]: 220 Ready to start TLS
Oct 19 07:29:44 ia sSMTP[427]: SSL connection using RSA_AES_128_CBC_SHA1
Oct 19 07:29:44 ia sSMTP[427]: EHLO ia.internal.vdopia.com
Oct 19 07:29:44 ia sSMTP[427]: 250 Ok
Oct 19 07:29:44 ia sSMTP[427]: AUTH LOGIN
--- removing some lines
Oct 19 07:29:44 ia sSMTP[427]: 235 Authentication successful.
Oct 19 07:29:44 ia sSMTP[427]: MAIL FROM: <root#ia.internal.vdopia.com>
Oct 19 07:29:44 ia sSMTP[427]: 250 Ok
Oct 19 07:29:44 ia sSMTP[427]: RCPT TO:<ayush.sharma#vdopia.com>
Oct 19 07:29:44 ia sSMTP[427]: 250 Ok
Oct 19 07:29:44 ia sSMTP[427]: DATA
Oct 19 07:29:44 ia sSMTP[427]: 354 End data with <CR><LF>.<CR><LF>
Oct 19 07:29:44 ia sSMTP[427]: Received: by ia.internal.vdopia.com (sSMTP sendmail emulation); Wed, 19 Oct 2016 07:29:44 +0000
Oct 19 07:29:44 ia sSMTP[427]: From: "root" <root#ia.internal.vdopia.com>
Oct 19 07:29:44 ia sSMTP[427]: Date: Wed, 19 Oct 2016 07:29:44 +0000
Oct 19 07:29:44 ia sSMTP[427]: Subject: testing
Oct 19 07:29:44 ia sSMTP[427]: To: <ayush.sharma#vdopia.com>
Oct 19 07:29:44 ia sSMTP[427]: X-Mailer: mail (GNU Mailutils 2.99.98)
Oct 19 07:29:44 ia sSMTP[427]:
Oct 19 07:29:45 ia sSMTP[427]: .
Oct 19 07:29:45 ia sSMTP[427]: 554 Message rejected: Email address is not verified. The following identities failed the check in region US-EAST-1: root <root#ia.internal.vdopia.com>, root#ia.internal.vdopia.com
SMTP Response Codes Returned by Amazon SES
When your account is moved out of sand box, you do not have to verify the recipients' addresses. But you still have to verify the sender's address or domain. From your post, it appears you have not verified the sender's address. Remember to verify the address/domain that appears in:
From
Source
Sender / Return-Path
Can you post your actual mail command/script that you are using to send the mail?
Nevertheless this question is answered correctly, maybe this tip helps out other people. After the activation and switch from AWS SES Sandbox to Production mode (support request of limit increase) I realized, that using the old "SMTP IAM User" caused the same problem. Just create a new "SMTP IAM User" after production grant. I really cannot explain it but that worked several times now.