EC2 Bootstrap script not executed on Ubuntu - amazon-web-services

I am trying to set bootstrap script for EC2 Ubuntu machine with below CLI commands:
#cloud-boothook
#!/bin/bash
sudo chmod 750 -R /home/
Above text is set via AWS console in user data.
Once machine has started running, change mode still not amended by above user data script.
Tried alternative command as:
Content-Type: text/cloud-boothook
Content-Type: text/x-shellscript
sudo chmod 750 -R /home/
Still directory permission is not changed.
My objective is to set user home directory with 750 change mode on machine startup.
I would appreciate any solution to this.
Here is the error message appearing for me:
[[0;32m OK [0m] Started Initial cloud-init job (pre-networking).
[[0;32m OK [0m] Reached target Network (Pre).
[[0;32m OK [0m] Started ifup for eth0.
Starting Raise network interfaces...
[[0;32m OK [0m] Started Raise network interfaces.
[[0;32m OK [0m] Reached target Network.
Starting Initial cloud-init job (metadata service crawler)...
[ 7.535726] cloud-init[1003]: Cloud-init v. 0.7.9 running 'init' at Sun, 29 Oct 2017 19:44:35 +0000. Up 7.23 seconds.
[ 7.548260] cloud-init[1003]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++
[ 7.565098] cloud-init[1003]: ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
[ 7.580238] cloud-init[1003]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 7.592760] cloud-init[1003]: ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
[ 7.602150] cloud-init[1003]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 7.613465] cloud-init[1003]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 7.625404] cloud-init[1003]: ci-info: | eth0 | True | 172.31.19.221 | 255.255.240.0 | . | 06:ad:ba:ed:47:10 |
[ 7.635005] cloud-init[1003]: ci-info: | eth0 | True | fe80::4ad:baff:feed:4710/64 | . | link | 06:ad:ba:ed:47:10 |
[ 7.648580] cloud-init[1003]: ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
[ 7.663022] cloud-init[1003]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
[ 7.674192] cloud-init[1003]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
[ 7.682430] cloud-init[1003]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[ 7.690935] cloud-init[1003]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
[ 7.703027] cloud-init[1003]: ci-info: | 0 | 0.0.0.0 | 172.31.16.1 | 0.0.0.0 | eth0 | UG |
[ 7.712719] cloud-init[1003]: ci-info: | 1 | 172.31.16.0 | 0.0.0.0 | 255.255.240.0 | eth0 | U |
[ 7.719950] cloud-init[1003]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
**[ 7.728024] cloud-init[1003]: 2017-10-29 19:44:35,859 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'Content-Type: text/cloud'...'**
[[0;32m OK [0m] Started Initial cloud-init job (metadata service crawler).
[[0;32m OK [0m] Reached target Cloud-config availability.
[[0;32m OK [0m] Reached target System Initialization.
[[0;32m OK [0m] Listening on D-Bus System Message Bus Socket.
Starting Socket activation for snappy daemon.

You are assigning 750 permission to /home recursively.
750 means:
The Owner may read, write and execute.
The Group may read and execute. (but not write)
The world may not do anything with this file.
The /home directory itself is owned by root and is in the root group. Therefore, after recursively applying 750, only root can access the contents of /home (including other users' home directories).
Therefore, you really don't want to do that.

Related

unable to access my http service on 8080 deployed on ec2 instance

I have an ec2 instance running a sample web service on http 8080 port. I am unable to access it from my mac(outside ec2).
I have done the following after going through many stackoverflow questions.
i have deployed my spring boot application on ec2 and is running successfully. I was also able to hit that service from localhost:8080 and got the expected response.
[ec2-user#ip-172-31-34-45 ~]$
[ec2-user#ip-172-31-34-45 ~]$
[ec2-user#ip-172-31-34-45 ~]$
[ec2-user#ip-172-31-34-45 ~]$ cd samplespringboot/
[ec2-user#ip-172-31-34-45 samplespringboot]$ java -jar spring-boot-hello-1.0.jar &
[1] 438
[ec2-user#ip-172-31-34-45 samplespringboot]$
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.7.6)
2022-12-02 06:59:11.948 INFO 438 --- [ main] com.example.demo.DemoApplication : Starting DemoApplication v1.0 using Java 1.8.0_342 on ip-172-31-34-45.ap-south-1.compute.internal with PID 438 (/home/ec2-user/samplespringboot/spring-boot-hello-1.0.jar started by ec2-user in /home/ec2-user/samplespringboot)
2022-12-02 06:59:11.960 INFO 438 --- [ main] com.example.demo.DemoApplication : No active profile set, falling back to 1 default profile: "default"
2022-12-02 06:59:14.739 INFO 438 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-12-02 06:59:14.766 INFO 438 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-12-02 06:59:14.766 INFO 438 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.69]
2022-12-02 06:59:14.931 INFO 438 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-12-02 06:59:14.932 INFO 438 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2825 ms
2022-12-02 06:59:16.405 INFO 438 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2022-12-02 06:59:16.433 INFO 438 --- [ main] com.example.demo.DemoApplication : Started DemoApplication in 5.574 seconds (JVM running for 6.602)
[ec2-user#ip-172-31-34-45 samplespringboot]$
[ec2-user#ip-172-31-34-45 samplespringboot]$
[ec2-user#ip-172-31-34-45 samplespringboot]$ curl localhost:8080
2022-12-02 06:59:41.048 INFO 438 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2022-12-02 06:59:41.053 INFO 438 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2022-12-02 06:59:41.055 INFO 438 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 2 ms
Hello World!
Hello World, Spring Boot![ec2-user#ip-172-31-34-45 samplespringboot]$
when i try to hit the same from local mac, i am getting connection timed out..
vinays#Vinays-MacBook-Pro target % curl ec2-x-y-z-133.ap-south-1.compute.amazonaws.com:8080
curl: (28) Failed to connect to ec2-x-yz-133.ap-south-1.compute.amazonaws.com port 8080 after 75008 ms: Operation timed out
so i have the following outbound and inbound rules of my ec2 instance
And i have also added NACLs entries like this..
I am not sure what i am doing wrong. Any help is appreciated. Thanks.
[UPDATE]:
I cannot specify port 8080 in edit inbound rules. by default it is 80.
In the inbound rule, you have to add 8080 port as allowed. Otherwise, any host cannot reach to your EC2 node from outside world.
Allow 8080 port from your real IP address would do the trick as well. If you want to enable more secure way, then allow only your real IP for 8080 port, and it will be reachable.

Unable to connect browser to any of my docker images

I downloaded cookiecutter django to start a new project the other day. I spun it up (along with postgres, redis, etc) inside docker containers. The configuration files should be fine because they were all generated by coockicutter.
However, once I build and turn on the containers I am unable to see the "hello world" splash page when I connect to my localhost:8000. But there is something going wrong between the applications and the containers because I am able to connect to them via telnet and through docker exec -it commands etc. The only thing I can think of is some sort of permissions issue? So I gave all the files/directors 777 permissions to test that but that hasnt changed anything.
logs
% docker compose -f local.yml up
[+] Running 8/0
⠿ Container dashboard_local_docs Created 0.0s
⠿ Container dashboard_local_redis Created 0.0s
⠿ Container dashboard_local_mailhog Created 0.0s
⠿ Container dashboard_local_postgres Created 0.0s
⠿ Container dashboard_local_django Created 0.0s
⠿ Container dashboard_local_celeryworker Created 0.0s
⠿ Container dashboard_local_celerybeat Created 0.0s
⠿ Container dashboard_local_flower Created 0.0s
Attaching to dashboard_local_celerybeat, dashboard_local_celeryworker, dashboard_local_django, dashboard_local_docs, dashboard_local_flower, dashboard_local_mailhog, dashboard_local_postgres, dashboard_local_redis
dashboard_local_postgres |
dashboard_local_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
dashboard_local_postgres |
dashboard_local_postgres | 2022-07-07 14:36:15.969 UTC [1] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
dashboard_local_postgres | 2022-07-07 14:36:15.992 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
dashboard_local_postgres | 2022-07-07 14:36:15.992 UTC [1] LOG: listening on IPv6 address "::", port 5432
dashboard_local_postgres | 2022-07-07 14:36:15.995 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
dashboard_local_postgres | 2022-07-07 14:36:15.999 UTC [26] LOG: database system was shut down at 2022-07-07 14:35:47 UTC
dashboard_local_postgres | 2022-07-07 14:36:16.004 UTC [1] LOG: database system is ready to accept connections
dashboard_local_mailhog | 2022/07/07 14:36:16 Using in-memory storage
dashboard_local_mailhog | 2022/07/07 14:36:16 [SMTP] Binding to address: 0.0.0.0:1025
dashboard_local_mailhog | 2022/07/07 14:36:16 Serving under http://0.0.0.0:8025/
dashboard_local_mailhog | [HTTP] Binding to address: 0.0.0.0:8025
dashboard_local_mailhog | Creating API v1 with WebPath:
dashboard_local_mailhog | Creating API v2 with WebPath:
dashboard_local_docs | sphinx-autobuild -b html --host 0.0.0.0 --port 9000 --watch /app -c . . ./_build/html
dashboard_local_docs | [sphinx-autobuild] > sphinx-build -b html -c . /docs /docs/_build/html
dashboard_local_redis | 1:C 07 Jul 2022 14:36:17.057 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
dashboard_local_redis | 1:C 07 Jul 2022 14:36:17.057 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=1, just started
dashboard_local_redis | 1:C 07 Jul 2022 14:36:17.057 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.057 * monotonic clock: POSIX clock_gettime
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.057 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.057 * Running mode=standalone, port=6379.
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.057 # Server initialized
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.057 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.058 * Loading RDB produced by version 6.2.7
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.058 * RDB age 30 seconds
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.058 * RDB memory usage when created 0.78 Mb
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.058 # Done loading RDB, keys loaded: 3, keys expired: 0.
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.058 * DB loaded from disk: 0.000 seconds
dashboard_local_redis | 1:M 07 Jul 2022 14:36:17.058 * Ready to accept connections
dashboard_local_docs | Running Sphinx v5.0.1
dashboard_local_celeryworker | PostgreSQL is available
dashboard_local_celerybeat | PostgreSQL is available
dashboard_local_docs | loading pickled environment... done
dashboard_local_docs | building [mo]: targets for 0 po files that are out of date
dashboard_local_docs | building [html]: targets for 0 source files that are out of date
dashboard_local_docs | updating environment: 0 added, 0 changed, 0 removed
dashboard_local_docs | looking for now-outdated files... none found
dashboard_local_docs | no targets are out of date.
dashboard_local_docs | build succeeded.
dashboard_local_docs |
dashboard_local_docs | The HTML pages are in _build/html.
dashboard_local_docs | [I 220707 14:36:18 server:335] Serving on http://0.0.0.0:9000
dashboard_local_celeryworker | [14:36:18] watching "/app" and reloading "celery.__main__.main" on changes...
dashboard_local_docs | [I 220707 14:36:18 handlers:62] Start watching changes
dashboard_local_docs | [I 220707 14:36:18 handlers:64] Start detecting changes
dashboard_local_django | PostgreSQL is available
dashboard_local_celerybeat | celery beat v5.2.7 (dawn-chorus) is starting.
dashboard_local_flower | PostgreSQL is available
dashboard_local_celerybeat | __ - ... __ - _
dashboard_local_celerybeat | LocalTime -> 2022-07-07 09:36:19
dashboard_local_celerybeat | Configuration ->
dashboard_local_celerybeat | . broker -> redis://redis:6379/0
dashboard_local_celerybeat | . loader -> celery.loaders.app.AppLoader
dashboard_local_celerybeat | . scheduler -> django_celery_beat.schedulers.DatabaseScheduler
dashboard_local_celerybeat |
dashboard_local_celerybeat | . logfile -> [stderr]#%INFO
dashboard_local_celerybeat | . maxinterval -> 5.00 seconds (5s)
dashboard_local_celerybeat | [2022-07-07 09:36:19,658: INFO/MainProcess] beat: Starting...
dashboard_local_celeryworker | /usr/local/lib/python3.9/site-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
dashboard_local_celeryworker | absolutely not recommended!
dashboard_local_celeryworker |
dashboard_local_celeryworker | Please specify a different user using the --uid option.
dashboard_local_celeryworker |
dashboard_local_celeryworker | User information: uid=0 euid=0 gid=0 egid=0
dashboard_local_celeryworker |
dashboard_local_celeryworker | warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
dashboard_local_celeryworker |
dashboard_local_celeryworker | -------------- celery#e1ac9f770cbd v5.2.7 (dawn-chorus)
dashboard_local_celeryworker | --- ***** -----
dashboard_local_celeryworker | -- ******* ---- Linux-5.4.0-96-generic-x86_64-with-glibc2.31 2022-07-07 09:36:19
dashboard_local_celeryworker | - *** --- * ---
dashboard_local_celeryworker | - ** ---------- [config]
dashboard_local_celeryworker | - ** ---------- .> app: dashboard:0x7fd9dcaeb1c0
dashboard_local_celeryworker | - ** ---------- .> transport: redis://redis:6379/0
dashboard_local_celeryworker | - ** ---------- .> results: redis://redis:6379/0
dashboard_local_celeryworker | - *** --- * --- .> concurrency: 8 (prefork)
dashboard_local_celeryworker | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
dashboard_local_celeryworker | --- ***** -----
dashboard_local_celeryworker | -------------- [queues]
dashboard_local_celeryworker | .> celery exchange=celery(direct) key=celery
dashboard_local_celeryworker |
dashboard_local_celeryworker |
dashboard_local_celeryworker | [tasks]
dashboard_local_celeryworker | . dashboard.users.tasks.get_users_count
dashboard_local_celeryworker |
dashboard_local_django | Operations to perform:
dashboard_local_django | Apply all migrations: account, admin, auth, authtoken, contenttypes, django_celery_beat, sessions, sites, socialaccount, users
dashboard_local_django | Running migrations:
dashboard_local_django | No migrations to apply.
dashboard_local_flower | INFO 2022-07-07 09:36:20,646 command 7 140098896897856 Visit me at http://localhost:5555
dashboard_local_flower | INFO 2022-07-07 09:36:20,652 command 7 140098896897856 Broker: redis://redis:6379/0
dashboard_local_flower | INFO 2022-07-07 09:36:20,655 command 7 140098896897856 Registered tasks:
dashboard_local_flower | ['celery.accumulate',
dashboard_local_flower | 'celery.backend_cleanup',
dashboard_local_flower | 'celery.chain',
dashboard_local_flower | 'celery.chord',
dashboard_local_flower | 'celery.chord_unlock',
dashboard_local_flower | 'celery.chunks',
dashboard_local_flower | 'celery.group',
dashboard_local_flower | 'celery.map',
dashboard_local_flower | 'celery.starmap',
dashboard_local_flower | 'dashboard.users.tasks.get_users_count']
dashboard_local_flower | INFO 2022-07-07 09:36:20,663 mixins 7 140098817644288 Connected to redis://redis:6379/0
dashboard_local_celeryworker | [2022-07-07 09:36:20,792: INFO/SpawnProcess-1] Connected to redis://redis:6379/0
dashboard_local_celeryworker | [2022-07-07 09:36:20,794: INFO/SpawnProcess-1] mingle: searching for neighbors
dashboard_local_flower | WARNING 2022-07-07 09:36:21,700 inspector 7 140098800826112 Inspect method active_queues failed
dashboard_local_flower | WARNING 2022-07-07 09:36:21,710 inspector 7 140098766993152 Inspect method reserved failed
dashboard_local_flower | WARNING 2022-07-07 09:36:21,712 inspector 7 140098784040704 Inspect method scheduled failed
dashboard_local_flower | WARNING 2022-07-07 09:36:21,714 inspector 7 140098758600448 Inspect method revoked failed
dashboard_local_flower | WARNING 2022-07-07 09:36:21,715 inspector 7 140098792433408 Inspect method registered failed
dashboard_local_flower | WARNING 2022-07-07 09:36:21,715 inspector 7 140098276423424 Inspect method conf failed
dashboard_local_flower | WARNING 2022-07-07 09:36:21,715 inspector 7 140098809218816 Inspect method stats failed
dashboard_local_flower | WARNING 2022-07-07 09:36:21,716 inspector 7 140098775648000 Inspect method active failed
dashboard_local_celeryworker | [2022-07-07 09:36:21,802: INFO/SpawnProcess-1] mingle: all alone
dashboard_local_celeryworker | [2022-07-07 09:36:21,811: WARNING/SpawnProcess-1] /usr/local/lib/python3.9/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
dashboard_local_celeryworker | leak, never use this setting in production environments!
dashboard_local_celeryworker | warnings.warn('''Using settings.DEBUG leads to a memory
dashboard_local_celeryworker |
dashboard_local_celeryworker | [2022-07-07 09:36:21,811: INFO/SpawnProcess-1] celery#e1ac9f770cbd ready.
dashboard_local_django | Watching for file changes with StatReloader
dashboard_local_django | INFO 2022-07-07 09:36:22,862 autoreload 9 140631340287808 Watching for file changes with StatReloader
dashboard_local_django | Performing system checks...
dashboard_local_django |
dashboard_local_django | System check identified no issues (0 silenced).
dashboard_local_django | July 07, 2022 - 09:36:23
dashboard_local_django | Django version 3.2.14, using settings 'config.settings.local'
dashboard_local_django | Starting development server at http://0.0.0.0:8000/
dashboard_local_django | Quit the server with CONTROL-C.
dashboard_local_celeryworker | [2022-07-07 09:36:25,661: INFO/SpawnProcess-1] Events of group {task} enabled by remote.
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
69591187e44d dashboard_local_flower "/entrypoint /start-…" 11 minutes ago Up 2 minutes 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp dashboard_local_flower
15914b6b91e0 dashboard_local_celerybeat "/entrypoint /start-…" 11 minutes ago Up 2 minutes dashboard_local_celerybeat
e1ac9f770cbd dashboard_local_celeryworker "/entrypoint /start-…" 11 minutes ago Up 2 minutes dashboard_local_celeryworker
6bbfc900c346 dashboard_local_django "/entrypoint /start" 11 minutes ago Up 2 minutes 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp dashboard_local_django
b8bec3422bae redis:6 "docker-entrypoint.s…" 11 minutes ago Up 2 minutes 6379/tcp dashboard_local_redis
2b7c3d9eabe3 dashboard_production_postgres "docker-entrypoint.s…" 11 minutes ago Up 2 minutes 5432/tcp dashboard_local_postgres
0249aaaa040c mailhog/mailhog:v1.0.0 "MailHog" 11 minutes ago Up 2 minutes 1025/tcp, 0.0.0.0:8025->8025/tcp, :::8025->8025/tcp dashboard_local_mailhog
d5dd94cbb070 dashboard_local_docs "/start-docs" 11 minutes ago Up 2 minutes 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp dashboard_local_docs
the ports are listening
telnet 127.0.0.1 8000
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^]
% sudo netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 29532/smbd
tcp 0 0 127.0.0.1:43979 0.0.0.0:* LISTEN 31867/BlastServer
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 963/rpcbind
tcp 0 0 0.0.0.0:46641 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:51857 0.0.0.0:* LISTEN 4149/rpc.statd
tcp 0 0 0.0.0.0:5555 0.0.0.0:* LISTEN 14326/docker-proxy
tcp 0 0 0.0.0.0:6100 0.0.0.0:* LISTEN 31908/Xorg
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 973/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 29295/sshd
tcp 0 0 0.0.0.0:8025 0.0.0.0:* LISTEN 13769/docker-proxy
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 30117/master
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 882/sshd: noakes#no
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN 29532/smbd
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 14272/docker-proxy
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 13850/docker-proxy
tcp6 0 0 :::139 :::* LISTEN 29532/smbd
tcp6 0 0 :::40717 :::* LISTEN -
tcp6 0 0 :::41423 :::* LISTEN 4149/rpc.statd
tcp6 0 0 :::111 :::* LISTEN 963/rpcbind
tcp6 0 0 127.0.0.1:41265 :::* LISTEN 30056/java
tcp6 0 0 :::5555 :::* LISTEN 14333/docker-proxy
tcp6 0 0 :::6100 :::* LISTEN 31908/Xorg
tcp6 0 0 :::22 :::* LISTEN 29295/sshd
tcp6 0 0 :::13782 :::* LISTEN 2201/xinetd
tcp6 0 0 :::13783 :::* LISTEN 2201/xinetd
tcp6 0 0 :::8025 :::* LISTEN 13779/docker-proxy
tcp6 0 0 ::1:25 :::* LISTEN 30117/master
tcp6 0 0 ::1:6010 :::* LISTEN 882/sshd: noakes#no
tcp6 0 0 :::13722 :::* LISTEN 2201/xinetd
tcp6 0 0 :::6556 :::* LISTEN 2201/xinetd
tcp6 0 0 :::445 :::* LISTEN 29532/smbd
tcp6 0 0 :::8000 :::* LISTEN 14278/docker-proxy
tcp6 0 0 :::1057 :::* LISTEN 2201/xinetd
tcp6 0 0 :::7778 :::* LISTEN 2201/xinetd
tcp6 0 0 :::7779 :::* LISTEN 2201/xinetd
tcp6 0 0 :::9000 :::* LISTEN 13860/docker-proxy
local.yml
version: '3'
volumes:
dashboard_local_postgres_data: {}
dashboard_local_postgres_data_backups: {}
services:
django: &django
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
#user: "root:root"
image: dashboard_local_django
container_name: dashboard_local_django
platform: linux/x86_64
depends_on:
- postgres
- redis
- mailhog
volumes:
- .:/app:z
env_file:
- ./.envs/.local/.django
- ./.envs/.local/.postgres
ports:
- "8000:8000"
command: /start
postgres:
build:
context: .
dockerfile: ./compose/production/postgres/Dockerfile
image: dashboard_production_postgres
container_name: dashboard_local_postgres
volumes:
- dashboard_local_postgres_data:/var/lib/postgresql/data:Z
- dashboard_local_postgres_data_backups:/backups:z
env_file:
- ./.envs/.local/.postgres
docs:
image: dashboard_local_docs
container_name: dashboard_local_docs
platform: linux/x86_64
build:
context: .
dockerfile: ./compose/local/docs/Dockerfile
env_file:
- ./.envs/.local/.django
volumes:
- ./docs:/docs:z
- ./config:/app/config:z
- ./dashboard:/app/dashboard:z
ports:
- "9000:9000"
command: /start-docs
mailhog:
image: mailhog/mailhog:v1.0.0
container_name: dashboard_local_mailhog
ports:
- "8025:8025"
redis:
image: redis:6
container_name: dashboard_local_redis
celeryworker:
<<: *django
image: dashboard_local_celeryworker
container_name: dashboard_local_celeryworker
depends_on:
- redis
- postgres
- mailhog
ports: []
command: /start-celeryworker
celerybeat:
<<: *django
image: dashboard_local_celerybeat
container_name: dashboard_local_celerybeat
depends_on:
- redis
- postgres
- mailhog
ports: []
command: /start-celerybeat
flower:
<<: *django
image: dashboard_local_flower
container_name: dashboard_local_flower
ports:
- "5555:5555"
command: /start-flower

AWC EC2 Amazon Linux 2 Instances failed to boot after applying os updates

Yesterday we lost contact with 10 identically configured servers, after some investigation the conclusion was that a reboot after security updates had failed.
We have so far not been able to get any of the servers back online, but were lucky enough to be able to reinstall the instances without data loss.
I will paste the console log below, can anyone help me determine the root cause and perhaps give me some advice on if there is a better way to configure the server to make recovery easier (like getting past the "Press Enter to continue." prompt, that it seems to hang in).
The full log is too big for SO, so I put it on pastebin and pasted a redacted version below. I have removed the escape sequences that colorize the output and removed some double new lines, but besides that it is complete.
[ 0.000000] Linux version 4.14.200-155.322.amzn2.x86_64 (mockbuild#ip-10-0-1-230) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-10) (GCC)) #1 SMP Thu Oct 15 20:11:12 UTC 2020
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.14.200-155.322.amzn2.x86_64 root=UUID=a1e1011e-e38f-408e-878b-fed395b47ad6 ro console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 LANG=en_US.UTF-8
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.7 present.
[ 0.000000] DMI: Amazon EC2 t3.micro/, BIOS 1.0 10/16/2017
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] tsc: Fast TSC calibration using PIT
[ 0.000000] e820: last_pfn = 0x3e3fa max_arch_pfn = 0x400000000
[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Using GB pages for direct mapping
[ 0.000000] RAMDISK: [mem 0x3433e000-0x36196fff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F8F80 000014 (v00 AMAZON)
[ 0.000000] ACPI: RSDT 0x000000003E3FE360 00003C (v01 AMAZON AMZNRSDT 00000001 AMZN 00000001)
[ 0.000000] ACPI: FACS 0x000000003E3FFF40 000040
[ 0.000000] ACPI: SSDT 0x000000003E3FF6C0 00087A (v01 AMAZON AMZNSSDT 00000001 AMZN 00000001)
[ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[ 0.000000] e820: [mem 0x40000000-0xdfffffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on KVM
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.14.200-155.322.amzn2.x86_64 root=UUID=a1e1011e-e38f-408e-878b-fed395b47ad6 ro console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 LANG=en_US.UTF-8
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Memory: 943540K/1019488K available (10252K kernel code, 1958K rwdata, 2780K rodata, 2088K init, 4240K bss, 75948K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Kernel/User page tables isolation: enabled
[ 0.000000] ftrace: allocating 26683 entries in 105 pages
[ 0.004000] Hierarchical RCU implementation.
[ 0.004000] RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=2.
[ 0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.004000] NR_IRQS: 524544, nr_irqs: 440, preallocated irqs: 16
[ 0.004000] Console: colour VGA+ 80x25
[ 0.004000] console [tty0] enabled
[ 0.004000] console [ttyS0] enabled
[ 0.004005] tsc: Detected 2500.000 MHz processor
[ 0.007582] Calibrating delay loop (skipped) preset value.. 5000.00 BogoMIPS (lpj=10000000)
[ 0.008002] pid_max: default: 32768 minimum: 301
[ 0.012006] ACPI: Core revision 20170728
[ 0.016560] ACPI: 2 ACPI AML tables successfully acquired and loaded
[ 0.020015] Security Framework initialized
[ 0.024002] SELinux: Initializing.
[ 0.028159] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.032082] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.036012] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.040006] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.044325] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[ 0.048003] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[ 0.052003] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[ 0.056003] Spectre V2 : Mitigation: Full generic retpoline
[ 0.060002] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[ 0.064002] Speculative Store Bypass: Vulnerable
[ 0.067720] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
[ 0.068002] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[ 0.072086] Freeing SMP alternatives memory: 24K
[ 0.076807] smpboot: Max logical packages: 1
[ 0.080264] x2apic enabled
[ 0.084003] Switched APIC routing to physical x2apic.
[ 0.088000] ..TIMER: vector=0x30 apic1=0 pin1=0 apic2=-1 pin2=-1
[ 0.088000] smpboot: CPU0: Intel(R) Xeon(R) Platinum 8175M CPU # 2.50GHz (family: 0x6, model: 0x55, stepping: 0x4)
[ 0.088074] Performance Events: unsupported p6 CPU model 85 no PMU driver, software events only.
[ 0.092046] Hierarchical SRCU implementation.
[ 0.095857] NMI watchdog: Perf event create on CPU 0 failed with -2
[ 0.096002] NMI watchdog: Perf NMI watchdog permanently disabled
[ 0.100049] smp: Bringing up secondary CPUs ...
[ 0.103696] x86: Booting SMP configuration:
[ 0.104003] .... node #0, CPUs: #1
[ 0.004000] kvm-clock: cpu 1, msr 0:3e357041, secondary cpu clock
[ 0.106853] KVM setup async PF for cpu 1
[ 0.107214] kvm-stealtime: cpu 1, msr 3e1161c0
[ 0.112307] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[ 0.116006] TAA CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/tsx_async_abort.html for more details.
[ 0.120007] smp: Brought up 1 node, 2 CPUs
[ 0.123417] smpboot: Total of 2 processors activated (10000.00 BogoMIPS)
[ 0.124320] devtmpfs: initialized
[ 0.126970] x86/mm: Memory block size: 128MB
[ 0.128137] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.132008] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 0.136156] NET: Registered protocol family 16
[ 0.139769] cpuidle: using governor ladder
[ 0.140013] cpuidle: using governor menu
[ 0.143281] ACPI: bus type PCI registered
[ 0.144000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.148144] PCI: Using configuration type 1 for base access
[ 0.156770] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.160017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.164044] ACPI: Added _OSI(Module Device)
[ 0.168007] ACPI: Added _OSI(Processor Device)
[ 0.172007] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 0.176004] ACPI: Added _OSI(Processor Aggregator Device)
[ 0.180007] ACPI: Interpreter enabled
[ 0.184011] ACPI: (supports S0 S4 S5)
[ 0.187094] ACPI: Using IOAPIC for interrupt routing
[ 0.188018] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.300750] ACPI: Enabled 16 GPEs in block 00 to 0F
[ 0.308023] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.312007] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[ 0.316010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[ 0.320007] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[ 0.328324] acpiphp: Slot [3] registered
[ 0.420040] acpiphp: Slot [31] registered
[ 0.424003] PCI host bridge to bus 0000:00
[ 0.536451] pci 0000:00:03.0: vgaarb: setting as boot VGA device
[ 0.540000] pci 0000:00:03.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 0.548009] pci 0000:00:03.0: vgaarb: bridge control possible
[ 0.551996] vgaarb: loaded
[ 0.556090] EDAC MC: Ver: 3.0.0
[ 0.559140] PCI: Using ACPI for IRQ routing
[ 0.560280] NetLabel: Initializing
[ 0.563268] NetLabel: domain hash size = 128
[ 0.568019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.571902] NetLabel: unlabeled traffic allowed by default
[ 0.576145] clocksource: Switched to clocksource kvm-clock
[ 0.586755] VFS: Disk quotas dquot_6.6.0
[ 0.590090] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.594562] pnp: PnP ACPI init
[ 0.597855] pnp: PnP ACPI: found 5 devices
[ 0.608231] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 0.614881] NET: Registered protocol family 2
[ 0.618324] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.622749] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[ 0.626965] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.631170] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.635163] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.639358] NET: Registered protocol family 1
[ 0.642779] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.646797] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.651113] pci 0000:00:03.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 0.657825] Unpacking initramfs...
[ 0.734208] Freeing initrd memory: 31076K
[ 0.737636] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x240939f1bb2, max_idle_ns: 440795263295 ns
[ 0.745181] Scanning for low memory corruption every 60 seconds
[ 0.750602] audit: initializing netlink subsys (disabled)
[ 0.754606] audit: type=2000 audit(1603879247.564:1): state=initialized audit_enabled=0 res=1
[ 0.754917] Initialise system trusted keyrings
[ 0.764927] Key type blacklist registered
[ 0.768266] workingset: timestamp_bits=36 max_order=18 bucket_order=0
[ 0.773861] zbud: loaded
[ 0.905903] Key type asymmetric registered
[ 0.909292] Asymmetric key parser 'x509' registered
[ 0.912915] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[ 0.918972] io scheduler noop registered (default)
[ 0.922543] io scheduler cfq registered
[ 0.925904] crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
[ 0.964594] crc32c_combine: 8373 self tests passed
[ 0.968628] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 1.000785] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 1.007649] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[ 1.014310] i8042: Warning: Keylock active
[ 1.018572] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 1.022414] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 1.026284] rtc_cmos 00:00: RTC can wake from S4
[ 1.030475] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
[ 1.034755] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
[ 1.038955] hidraw: raw HID events driver (C) Jiri Kosina
[ 1.042936] NET: Registered protocol family 17
[ 1.046622] mce: Using 32 MCE banks
[ 1.049627] sched_clock: Marking stable (1049607566, 0)->(1755024155, -705416589)
[ 1.056014] registered taskstats version 1
[ 1.059279] Loading compiled-in X.509 certificates
[ 1.064832] Loaded X.509 cert 'Build time autogenerated kernel key: 121ffea65ca15230f4a21fe7e5b65abaabaa433c'
[ 1.072013] zswap: loaded using pool lzo/zbud
[ 1.075526] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 1.079746] ima: Allocated hash algorithm: sha1
[ 1.083589] rtc_cmos 00:00: setting system clock to 2020-10-28 09:59:31 UTC (1603879171)
[ 1.091820] Freeing unused kernel memory: 2088K
[ 1.116102] Write protecting the kernel read-only data: 16384k
[ 1.120697] Freeing unused kernel memory: 2016K
[ 1.126528] Freeing unused kernel memory: 1316K
[ 1.160972] systemd[1]: Inserted module 'autofs4'
[ 1.176133] NET: Registered protocol family 10
[ 1.181508] Segment Routing with IPv6
[ 1.184828] systemd[1]: Inserted module 'ipv6'
[ 1.189116] random: systemd: uninitialized urandom read (16 bytes read)
[ 1.193763] random: systemd: uninitialized urandom read (16 bytes read)
[ 1.198171] random: systemd: uninitialized urandom read (16 bytes read)
[ 1.205354] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[ 1.217384] systemd[1]: Detected virtualization kvm.
[ 1.221077] systemd[1]: Detected architecture x86-64.
[ 1.224774] systemd[1]: Running in initial RAM disk.
Welcome to Amazon Linux 2 dracut-033-535.amzn2.1.3 (Initramfs)
[ 1.230712] systemd[1]: No hostname configured.
[ 1.234213] systemd[1]: Set hostname to <localhost>.
[ 1.237934] systemd[1]: Initializing machine ID from KVM UUID.
[ OK ] Reached target Swap.
[ 1.265844] systemd[1]: Reached target Swap.
[ 1.269312] systemd[1]: Starting Swap.
[ OK ] Created slice Root Slice.
[ 1.274036] systemd[1]: Created slice Root Slice.
[ OK ] Listening on Journal Socket.
[ OK ] Reached target Timers.
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Reached target Local File Systems.
[ OK ] Listening on udev Control Socket.
[ OK ] Created slice System Slice.
Starting Setup Virtual Console...
Starting Journal Service...
Starting Create list of required st... nodes for the current kernel...
Starting Apply Kernel Variables...
[ OK ] Reached target Slices.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Reached target Sockets.
Starting dracut cmdline hook...
[ OK ] Started Setup Virtual Console.
[ OK ] Started Create list of required sta...ce nodes for the current kernel.
[ OK ] Started Apply Kernel Variables.
Starting Create Static Device Nodes in /dev...
[ OK ] Started Create Static Device Nodes in /dev.
[ OK ] Started Journal Service.
[ OK ] Started dracut cmdline hook.
Starting dracut pre-udev hook...
[ 1.390579] device-mapper: uevent: version 1.0.3
[ 1.394255] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel#redhat.com
[ OK ] Started dracut pre-udev hook.
Starting udev Kernel Device Manager...
[ OK ] Started udev Kernel Device Manager.
Starting dracut pre-trigger hook...
[ OK ] Started dracut pre-trigger hook.
Starting udev Coldplug all Devices...
[ OK ] Started udev Coldplug all Devices.
Starting Show Plymouth Boot Screen...
[ OK ] Reached target System Initialization.
Starting dracut initqueue hook...
[ 1.534629] nvme nvme0: pci function 0000:00:04.0
[ OK ] Started Show Plymouth Boot Screen.
[ OK ] Reached target Paths.
[ OK ] Reached target Basic System.
[ 1.543815] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[ 1.546543] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[ 1.556607] nvme nvme1: pci function 0000:00:1f.0
[ 1.557854] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
[ 1.576394] AVX2 version of gcm_enc/dec engaged.
[ 1.580503] AES CTR mode by8 optimization enabled
[ 1.601321] alg: No test for pcbc(aes) (pcbc-aes-aesni)
[ 1.776495] nvme0n1: p1 p128
[ 1.908576] random: fast init done
[ OK ] Found device /dev/disk/by-uuid/a1e1011e-e38f-408e-878b-fed395b47ad6.
Starting File System Check on /dev/...e-e38f-408e-878b-fed395b47ad6...
[ OK ] Started File System Check on /dev/d...11e-e38f-408e-878b-fed395b47ad6.
[ OK ] Started dracut initqueue hook.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
Starting dracut pre-mount hook...
[ OK ] Started dracut pre-mount hook.
Mounting /sysroot...
[ 2.235770] SGI XFS with ACLs, security attributes, no debug enabled
[ 2.242333] XFS (nvme0n1p1): Mounting V5 Filesystem
[ 4.142597] XFS (nvme0n1p1): Ending clean mount
[ OK ] Mounted /sysroot.
[ OK ] Reached target Initrd Root File System.
Starting Reload Configuration from the Real Root...
[ OK ] Started Reload Configuration from the Real Root.
[ OK ] Reached target Initrd File Systems.
[ OK ] Reached target Initrd Default Target.
Starting dracut pre-pivot and cleanup hook...
[ OK ] Started dracut pre-pivot and cleanup hook.
Starting Cleaning Up and Shutting Down Daemons...
[ OK ] Stopped Cleaning Up and Shutting Down Daemons.
[ OK ] Stopped target Timers.
[ OK ] Stopped dracut pre-pivot and cleanup hook.
Stopping dracut pre-pivot and cleanup hook...
[ OK ] Stopped target Remote File Systems.
[ OK ] Stopped target Remote File Systems (Pre).
[ OK ] Stopped target Initrd Default Target.
Starting Plymouth switch root service...
[ OK ] Stopped dracut pre-mount hook.
Stopping dracut pre-mount hook...
[ OK ] Stopped dracut initqueue hook.
Stopping dracut initqueue hook...
[ OK ] Stopped target Basic System.
[ OK ] Stopped target Sockets.
[ OK ] Stopped target System Initialization.
[ OK ] Stopped target Swap.
[ OK ] Stopped target Local File Systems.
[ OK ] Stopped Apply Kernel Variables.
Stopping Apply Kernel Variables...
[ OK ] Stopped target Local Encrypted Volumes.
[ OK ] Stopped udev Coldplug all Devices.
Stopping udev Coldplug all Devices...
[ OK ] Stopped dracut pre-trigger hook.
Stopping dracut pre-trigger hook...
Stopping udev Kernel Device Manager...
[ OK ] Stopped target Slices.
[ OK ] Stopped target Paths.
[ OK ] Stopped udev Kernel Device Manager.
[ OK ] Stopped Create Static Device Nodes in /dev.
Stopping Create Static Device Nodes in /dev...
[ OK ] Stopped Create list of required sta...ce nodes for the current kernel.
Stopping Create list of required st... nodes for the current kernel...
[ OK ] Stopped dracut pre-udev hook.
Stopping dracut pre-udev hook...
[ OK ] Stopped dracut cmdline hook.
Stopping dracut cmdline hook...
[ OK ] Closed udev Kernel Socket.
[ OK ] Closed udev Control Socket.
Starting Cleanup udevd DB...
[ OK ] Started Cleanup udevd DB.
[ OK ] Reached target Switch Root.
[ 4.553875] systemd-journald[667]: Received SIGTERM from PID 1 (systemd).
[ OK ] Started Plymouth switch root service.
Starting Switch Root...
[ 4.885212] systemd: 30 output lines suppressed due to ratelimiting
[ 5.925390] SELinux: Disabled at runtime.
[ 5.980115] audit: type=1404 audit(1603879176.396:2): selinux=0 auid=4294967295 ses=4294967295
[ 6.083250] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 6.106470] systemd[1]: Inserted module 'ip_tables'
Welcome to Amazon Linux 2
[ OK ] Stopped Switch Root.
[ OK ] Stopped Journal Service.
Starting Journal Service...
[ OK ] Reached target Swap.
[ OK ] Listening on Delayed Shutdown Socket.
Mounting Huge Pages File System...
[ OK ] Stopped target Switch Root.
[ OK ] Stopped target Initrd Root File System.
[ OK ] Created slice system-getty.slice.
[ OK ] Listening on udev Control Socket.
[ OK ] Listening on Device-mapper event daemon FIFOs.
[ OK ] Created slice User and Session Slice.
Starting Create list of required st... nodes for the current kernel...
[ OK ] Listening on LVM2 poll daemon socket.
[ OK ] Stopped target Initrd File Systems.
[ OK ] Listening on udev Kernel Socket.
Mounting Debug File System...
[ OK ] Reached target Slices.
[ OK ] Listening on LVM2 metadata daemon socket.
Mounting POSIX Message Queue File System...
[ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice.
Starting Monitoring of LVM2 mirrors... dmeventd or progress polling...
[ OK ] Created slice system-serial\x2dgetty.slice.
Starting Read and set NIS domainname from /etc/sysconfig/network...
[ OK ] Listening on /dev/initctl Compatibility Named Pipe.
[ OK ] Set up automount Arbitrary Executab...ats File System Automount Point.
Starting Remount Root and Kernel File Systems...
[ OK ] Started Journal Service.
[ OK ] Mounted Debug File System.
[ OK ] Mounted POSIX Message Queue File System.
[ OK ] Mounted Huge Pages File System.
[ OK ] Started Create list of required sta...ce nodes for the current kernel.
[ OK ] Started Remount Root and Kernel File Systems.
[ OK ] Started Read and set NIS domainname from /etc/sysconfig/network.
Starting udev Coldplug all Devices...
Starting Configure read-only root support...
Starting Relabel kernel modules early in the boot, if needed...
Starting Create Static Device Nodes in /dev...
Starting Flush Journal to Persistent Storage...
[ OK ] Started Relabel kernel modules early in the boot, if needed.
Starting Load Kernel Modules...
[ 7.047237] systemd-journald[1398]: Received request to flush runtime journal from PID 1
[ 7.069936] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.2.10g
[ 7.084119] ena: ena device version: 0.10
[ 7.089001] ena: ena controller version: 0.0.1 implementation version 1
[ OK ] Started Configure read-only root support.
Starting Load/Save Random Seed...
[ OK ] Started Load/Save Random Seed.
[ 7.156042] ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy.
[ OK ] Started udev Coldplug all Devices.
Starting udev Wait for Complete Device Initialization...
[ 7.181318] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 0a:cf:65:4e:dd:ff
[ OK ] Started Load Kernel Modules.
Starting Apply Kernel Variables...
[ OK ] Started LVM2 metadata daemon.
Starting LVM2 metadata daemon...
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Create Static Device Nodes in /dev.
Starting udev Kernel Device Manager...
[ OK ] Started Flush Journal to Persistent Storage.
[ OK ] Started udev Kernel Device Manager.
[ OK ] Found device /dev/ttyS0.
[ 7.776329] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[ 7.783413] ACPI: Power Button [PWRF]
[ 7.786723] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input4
[ 7.793032] ACPI: Sleep Button [SLPF]
Starting Relabel kernel modules early in the boot, if needed...
[ OK ] Created slice system-ec2net\x2difup.slice.
[ OK ] Started Relabel kernel modules early in the boot, if needed.
[ 7.888784] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input5
[ 7.904661] mousedev: PS/2 mouse device common for all mice
[ OK ] Started udev Wait for Complete Device Initialization.
Starting Activation of DM RAID sets...
[ OK ] Started Activation of DM RAID sets.
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling.
[ OK ] Reached target Local File Systems (Pre).
[ 59.305661] random: crng init done
[ 59.308921] random: 7 urandom warning(s) missed due to ratelimiting
[ TIME ] Timed out waiting for device dev-sdf.device.
[DEPEND] Dependency failed for /home/storage.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Mark the need to relabel after reboot.
[DEPEND] Dependency failed for Relabel all filesystems, if necessary.
[DEPEND] Dependency failed for Migrate local... structure to the new structure.
Starting Preprocess NFS configuration...
[ OK ] Reached target Timers.
[ OK ] Reached target Network (Pre).
[ OK ] Reached target Login Prompts.
[ OK ] Reached target Cloud-init target.
Starting Initial hibernation setup job...
Starting Initial cloud-init job (metadata service crawler)...
[ OK ] Reached target Network.
[ OK ] Reached target Paths.
[ OK ] Reached target Sockets.
Starting Create Volatile Files and Directories...
[ OK ] Started Emergency Shell.
Starting Emergency Shell...
[ OK ] Reached target Emergency Mode.
Starting Tell Plymouth To Write Out Runtime Data...
[ OK ] Started Preprocess NFS configuration.
[ OK ] Started Create Volatile Files and Directories.
Mounting RPC Pipe File System...
Starting Security Auditing Service...
Starting RPC bind service...
[ 97.160193] RPC: Registered named UNIX socket transport module.
[ 97.160194] RPC: Registered udp transport module.
[ 97.160194] RPC: Registered tcp transport module.
[ 97.160195] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ OK ] Mounted RPC Pipe File System.
[ OK ] Reached target rpc_pipefs.target.
[ OK ] Reached target NFS client services.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
[ OK ] Started Tell Plymouth To Write Out Runtime Data.
[ OK ] Started RPC bind service.
[ OK ] Started Security Auditing Service.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Started Update UTMP about System Boot/Shutdown.
Starting Update UTMP about System Runlevel Changes...
[ OK ] Started Update UTMP about System Runlevel Changes.
[ 99.871085] hibinit-agent[1855]: Traceback (most recent call last):
[ 99.871339] hibinit-agent[1855]: File "/usr/bin/hibinit-agent", line 496, in <module>
[ 99.871592] hibinit-agent[1855]: main()
[ 99.872080] hibinit-agent[1855]: File "/usr/bin/hibinit-agent", line 435, in main
[ 99.872516] hibinit-agent[1855]: if not hibernation_enabled(config.state_dir):
[ 99.873017] hibinit-agent[1855]: File "/usr/bin/hibinit-agent", line 390, in hibernation_enabled
[ 99.873487] hibinit-agent[1855]: imds_token = get_imds_token()
[ 99.873793] hibinit-agent[1855]: File "/usr/bin/hibinit-agent", line 365, in get_imds_token
[ 99.875332] hibinit-agent[1855]: response = requests.put(token_url, headers=request_header)
[ 99.877065] hibinit-agent[1855]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 121, in put
[ 99.877230] hibinit-agent[1855]: return request('put', url, data=data, **kwargs)
[ 99.877959] hibinit-agent[1855]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
[ 99.878225] hibinit-agent[1855]: response = session.request(method=method, url=url, **kwargs)
[ 99.878614] hibinit-agent[1855]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
[ 99.879747] hibinit-agent[1855]: resp = self.send(prep, **send_kwargs)
[ 99.880157] hibinit-agent[1855]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
[ 99.884411] hibinit-agent[1855]: r = adapter.send(request, **kwargs)
[ 99.884728] hibinit-agent[1855]: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 419, in send
[ 99.892094] hibinit-agent[1855]: raise ConnectTimeout(e, request=request)
[ 99.892377] hibinit-agent[1855]: requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7efc029fa390>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
[FAILED] Failed to start Initial hibernation setup job.
See 'systemctl status hibinit-agent.service' for details.
[ 101.215791] cloud-init[1856]: Cloud-init v. 19.3-3.amzn2 running 'init' at Wed, 28 Oct 2020 10:01:11 +0000. Up 101.18 seconds.
[ 101.264707] cloud-init[1856]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[ 101.264940] cloud-init[1856]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[ 101.272469] cloud-init[1856]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 101.274166] cloud-init[1856]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[ 101.274497] cloud-init[1856]: ci-info: | eth0 | False | . | . | . | 0a:cf:65:4e:dd:ff |
[ 101.284890] cloud-init[1856]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
[ 101.286727] cloud-init[1856]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 101.286986] cloud-init[1856]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[ 101.291933] cloud-init[1856]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[ 101.292215] cloud-init[1856]: ci-info: +-------+-------------+---------+-----------+-------+
[ 101.294122] cloud-init[1856]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[ 101.294383] cloud-init[1856]: ci-info: +-------+-------------+---------+-----------+-------+
[ 101.294543] cloud-init[1856]: ci-info: +-------+-------------+---------+-----------+-------+
Welcome to emerg
Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.
Press Enter to continue.
Ok, shortly after posting we figured it out. Seems like a mount point has changed (I expect due to a linux kernel update) and we have not used the nofail option in /etc/fstab as described in the aws knowledge center, this caused the server to hang at boot.
Going forward we will also ensure we use UUID mounting so we are independent on the device naming in /dev/.
I think I've narrowed this down to the ec2-utils package. We had the same issue, related to devices not mounting properly that we initially thought was related to the ENA or NVMe driver. Once we ran a yum update, it was resolved.
If you downgrade the ec2-utils package to ec2-utils-1.2-2.amzn2 the issue returns. This seems to only affect nitro based instances. To fix it, you can temporarily boot as a t2 or other older instance type and update the package.

Unexpected character in wdqs-updater_1 upon restart of a Wikibase

When working with a local Wikibase I noticed that the autocomplete function stopped working in the local query service. After a shutdown and startup I get the following message:
...
Stopping phenowiki_wdqs_1 ... done
...
./startup.sh
Starting phenowiki_wdqs_1 ... done
Starting phenowiki_elasticsearch_1 ... done
Starting phenowiki_mysql_1 ... done
Starting phenowiki_wdqs-proxy_1 ... done
Starting phenowiki_wikibase_1 ... done
Starting phenowiki_quickstatements_1 ... done
Starting phenowiki_wdqs-updater_1 ... done
Starting phenowiki_wdqs-frontend_1 ... done
Attaching to phenowiki_wdqs_1, phenowiki_elasticsearch_1, phenowiki_mysql_1, phenowiki_wikibase_1, phenowiki_wdqs-updater_1, phenowiki_quickstatements_1, phenowiki_wdqs-proxy_1, phenowiki_wdqs-frontend_1
wdqs_1 | /wdqs /wdqs
wdqs_1 | Running Blazegraph from /wdqs on :9999/bigdata
wdqs_1 | OpenJDK 64-Bit Server VM warning: Cannot open file /var/log/wdqs/wdqs-blazegraph_jvm_gc.pid7-2019-07-12_07-54-42.log due to No such file or directory
wdqs_1 |
wdqs_1 | 2019-07-12 07:54:42.564:INFO::main: Logging initialized #166ms
elasticsearch_1 | [2019-07-12T07:54:44,800][INFO ][o.e.n.Node ] [] initializing ...
wdqs_1 | 2019-07-12 07:54:42.571:INFO:oejr.Runner:main: Runner
wdqs_1 | 2019-07-12 07:54:42.659:INFO:oejs.Server:main: jetty-9.2.3.v20140905
wikibase_1 | wait-for-it.sh: waiting 120 seconds for mysql.svc:3306
wdqs_1 | 2019-07-12 07:54:47.605:WARN:oeja.AnnotationConfiguration:main: ServletContainerInitializers: detected. Class hierarchy: empty
wikibase_1 | wait-for-it.sh: mysql.svc:3306 is available after 0 seconds
wikibase_1 | wait-for-it.sh: waiting 120 seconds for mysql.svc:3306
wdqs_1 | #logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} IP:%X{req.remoteHost} UA:%X{req.userAgent} - %msg%n
elasticsearch_1 | [2019-07-12T07:54:44,903][INFO ][o.e.e.NodeEnvironment ] [5sOo-00] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/nvme0n1p2)]], net usable_space [297.7gb], net total_space [878.6gb], spins? [possibly], types [ext4]
elasticsearch_1 | [2019-07-12T07:54:44,903][INFO ][o.e.e.NodeEnvironment ] [5sOo-00] heap size [494.9mb], compressed ordinary object pointers [true]
mysql_1 | 2019-07-12 7:54:45 0 [Note] mysqld (mysqld 10.3.16-MariaDB-1:10.3.16+maria~bionic) starting as process 1 ...
wikibase_1 | wait-for-it.sh: mysql.svc:3306 is available after 0 seconds
wikibase_1 | wait-for-it.sh: waiting 60 seconds for elasticsearch.svc:9200
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Using Linux native AIO
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Uses event mutexes
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
quickstatements_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.80.7. Set the 'ServerName' directive globally to suppress this message
wdqs_1 | INFO: com.bigdata.util.config.LogUtil: Configure: file:/tmp/jetty-0.0.0.0-9999-blazegraph-service-0.3.0-SNAPSHOT.war-_bigdata-any-1586101280383644326.dir/webapp/WEB-INF/classes/log4j.properties
wdqs-updater_1 | wait-for-it.sh: waiting 120 seconds for myserver.nl:80
wdqs_1 | 07:54:48.105 [main] WARN com.bigdata.Banner IP: UA: - Defaulting log level to WARN: com.bigdata
quickstatements_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.80.7. Set the 'ServerName' directive globally to suppress this message
wdqs_1 |
wdqs_1 | BlazeGraph(TM) Graph Engine
wdqs_1 |
wdqs_1 | Flexible
wdqs_1 | Reliable
wdqs_1 | Affordable
wdqs_1 | Web-Scale Computing for the Enterprise
wdqs_1 |
wdqs_1 | Copyright SYSTAP, LLC DBA Blazegraph 2006-2016. All rights reserved.
wdqs_1 |
wdqs_1 | 1a307070fc17
wdqs_1 | Fri Jul 12 07:54:48 GMT 2019
wdqs_1 | Linux/4.15.0-50-generic amd64
wdqs_1 | Intel(R) Xeon(R) Silver 4114 CPU # 2.20GHz Family 6 Model 85 Stepping 4, GenuineIntel #CPU=40
wdqs_1 | IcedTea 1.8.0_212
wdqs_1 | freeMemory=642250776
wdqs_1 | buildVersion=2.1.5-SNAPSHOT
wdqs_1 | gitCommit=8ff64aab6071b5e591e80676809e43f0ddb8ad49
wdqs_1 |
wdqs_1 | Dependency License
wdqs_1 | ICU http://source.icu-project.org/repos/icu/icu/trunk/license.html
wdqs_1 | bigdata-ganglia http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | blueprints-core https://github.com/tinkerpop/blueprints/blob/master/LICENSE.txt
wdqs_1 | colt http://acs.lbl.gov/software/colt/license.html
wdqs_1 | commons-codec http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | commons-fileupload http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | commons-io http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | commons-logging http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | dsiutils http://www.gnu.org/licenses/lgpl-2.1.html
wdqs_1 | fastutil http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | flot http://www.opensource.org/licenses/mit-license.php
wdqs_1 | high-scale-lib http://creativecommons.org/licenses/publicdomain
wdqs_1 | httpclient http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | httpclient-cache http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | httpcore http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | httpmime http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | jackson-core http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | jetty http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | jquery https://github.com/jquery/jquery/blob/master/MIT-LICENSE.txt
wdqs_1 | jsonld https://raw.githubusercontent.com/jsonld-java/jsonld-java/master/LICENCE
wdqs_1 | log4j http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | lucene http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | nanohttp http://elonen.iki.fi/code/nanohttpd/#license
wdqs_1 | rexster-core https://github.com/tinkerpop/rexster/blob/master/LICENSE.txt
wdqs_1 | river http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | semargl https://github.com/levkhomich/semargl/blob/master/LICENSE
wdqs_1 | servlet-api http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 | sesame http://www.openrdf.org/download.jsp
wdqs_1 | slf4j http://www.slf4j.org/license.html
wdqs_1 | zookeeper http://www.apache.org/licenses/LICENSE-2.0.html
wdqs_1 |
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Number of pools: 1
wdqs-updater_1 | wait-for-it.sh: myserver.nl:80 is available after 0 seconds
quickstatements_1 | [Fri Jul 12 07:54:48.712372 2019] [mpm_prefork:notice] [pid 11] AH00163: Apache/2.4.25 (Debian) PHP/5.6.40 configured -- resuming normal operations
quickstatements_1 | [Fri Jul 12 07:54:48.712425 2019] [core:notice] [pid 11] AH00094: Command line: 'apache2 -D FOREGROUND'
wdqs_1 | 07:54:49.105 [main] WARN com.bigdata.rdf.ServiceProviderHook IP: UA: - Running.
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Using SSE2 crc32 instructions
elasticsearch_1 | [2019-07-12T07:54:44,921][INFO ][o.e.n.Node ] node name [5sOo-00] derived from node ID [5sOo-00hTAC1FV3BKuzvxA]; set [node.name] to override
wdqs-updater_1 | wait-for-it.sh: waiting 120 seconds for wdqs.svc:9999
wdqs_1 | 07:54:49.159 [main] INFO o.w.q.r.b.mwapi.MWApiServiceFactory IP: UA: - Loading MWAPI service configuration from ./mwservices.json
wdqs_1 | 07:54:49.537 [main] INFO o.w.q.r.b.mwapi.MWApiServiceFactory IP: UA: - Registered 4 services.
elasticsearch_1 | [2019-07-12T07:54:44,922][INFO ][o.e.n.Node ] version[5.6.14], pid[1], build[f310fe9/2018-12-05T21:20:16.416Z], OS[Linux/4.15.0-50-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
wdqs-updater_1 | wait-for-it.sh: wdqs.svc:9999 is available after 3 seconds
elasticsearch_1 | [2019-07-12T07:54:44,922][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
wdqs-updater_1 | Updating via http://wdqs.svc:9999/bigdata/namespace/wdq/sparql
elasticsearch_1 | [2019-07-12T07:54:45,984][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [aggs-matrix-stats]
elasticsearch_1 | [2019-07-12T07:54:45,984][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [ingest-common]
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
wdqs-updater_1 | OpenJDK 64-Bit Server VM warning: Cannot open file /var/log/wdqs/wdqs-updater_jvm_gc.pid7.log due to No such file or directory
wdqs-updater_1 |
elasticsearch_1 | [2019-07-12T07:54:45,984][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [lang-expression]
wdqs-updater_1 | I> No access restrictor found, access to any MBean is allowed
elasticsearch_1 | [2019-07-12T07:54:45,984][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [lang-groovy]
wdqs-updater_1 | Jolokia: Agent started with URL http://127.0.0.1:8778/jolokia/
wdqs_1 | 07:54:49.555 [main] INFO o.w.q.r.b.WikibaseContextListener IP: UA: - Wikibase services initialized.
wdqs_1 | 07:54:49.642 [main] INFO o.w.q.r.b.t.ThrottlingFilter IP: UA: - ThrottlingFilter MBean registered as org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter:filterName=throttling-filter.
elasticsearch_1 | [2019-07-12T07:54:45,984][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [lang-mustache]
wdqs_1 | 2019-07-12 07:54:49.681:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext#2d363fb3{/bigdata,file:/tmp/jetty-0.0.0.0-9999-blazegraph-service-0.3.0-SNAPSHOT.war-_bigdata-any-1586101280383644326.dir/webapp/,AVAILABLE}{file:/wdqs/blazegraph-service-0.3.0-SNAPSHOT.war}
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Completed initialization of buffer pool
wdqs-updater_1 | #logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
elasticsearch_1 | [2019-07-12T07:54:45,985][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [lang-painless]
elasticsearch_1 | [2019-07-12T07:54:45,985][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [parent-join]
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
wdqs_1 | 2019-07-12 07:54:49.682:WARN:oejsh.RequestLogHandler:main: !RequestLog
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
elasticsearch_1 | [2019-07-12T07:54:45,985][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [percolator]
wdqs_1 | 2019-07-12 07:54:49.693:INFO:oejs.ServerConnector:main: Started ServerConnector#46e8a539{HTTP/1.1}{0.0.0.0:9999}
elasticsearch_1 | [2019-07-12T07:54:45,985][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [reindex]
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables
wdqs_1 | 2019-07-12 07:54:49.696:INFO:oejs.Server:main: Started #7339ms
elasticsearch_1 | [2019-07-12T07:54:45,985][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [transport-netty3]
elasticsearch_1 | [2019-07-12T07:54:45,985][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded module [transport-netty4]
elasticsearch_1 | [2019-07-12T07:54:45,985][INFO ][o.e.p.PluginsService ] [5sOo-00] loaded plugin [extra]
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
elasticsearch_1 | [2019-07-12T07:54:46,148][WARN ][o.e.d.s.ScriptModule ] Native scripts are deprecated. Use a custom ScriptEngine to write scripts in java.
elasticsearch_1 | [2019-07-12T07:54:47,312][INFO ][o.e.d.DiscoveryModule ] [5sOo-00] using discovery type [zen]
elasticsearch_1 | [2019-07-12T07:54:48,064][INFO ][o.e.n.Node ] initialized
elasticsearch_1 | [2019-07-12T07:54:48,065][INFO ][o.e.n.Node ] [5sOo-00] starting ...
elasticsearch_1 | [2019-07-12T07:54:48,207][INFO ][o.e.t.TransportService ] [5sOo-00] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: 10.3.16 started; log sequence number 6230267; transaction id 2553
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mysql_1 | 2019-07-12 7:54:45 0 [Note] Plugin 'FEEDBACK' is disabled.
mysql_1 | 2019-07-12 7:54:45 0 [Note] Server socket created on IP: '::'.
mysql_1 | 2019-07-12 7:54:45 0 [Warning] 'proxies_priv' entry '#% root#c16b13b97f72' ignored in --skip-name-resolve mode.
mysql_1 | 2019-07-12 7:54:45 0 [Note] Reading of all Master_info entries succeeded
mysql_1 | 2019-07-12 7:54:45 0 [Note] Added new Master_info '' to hash table
mysql_1 | 2019-07-12 7:54:45 0 [Note] mysqld: ready for connections.
mysql_1 | Version: '10.3.16-MariaDB-1:10.3.16+maria~bionic' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
mysql_1 | 2019-07-12 7:54:45 0 [Note] InnoDB: Buffer pool(s) load completed at 190712 7:54:45
elasticsearch_1 | [2019-07-12T07:54:51,287][INFO ][o.e.c.s.ClusterService ] [5sOo-00] new_master {5sOo-00}{5sOo-00hTAC1FV3BKuzvxA}{tfRuBuVIToaQhhqABeBQEg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch_1 | [2019-07-12T07:54:51,343][INFO ][o.e.h.n.Netty4HttpServerTransport] [5sOo-00] publish_address {192.168.80.3:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch_1 | [2019-07-12T07:54:51,344][INFO ][o.e.n.Node ] [5sOo-00] started
wdqs-updater_1 | 07:54:51.561 [main] INFO org.wikidata.query.rdf.tool.Update - Checking where we left off
wdqs-updater_1 | 07:54:51.563 [main] INFO o.w.query.rdf.tool.rdf.RdfRepository - Checking for left off time from the updater
wdqs-updater_1 | 07:54:52.111 [main] INFO o.w.query.rdf.tool.rdf.RdfRepository - Checking for left off time from the dump
wdqs-updater_1 | 07:54:52.149 [main] INFO org.wikidata.query.rdf.tool.Update - Defaulting start time to 90 days ago: 2019-04-13T07:54:52Z
elasticsearch_1 | [2019-07-12T07:54:52,178][INFO ][o.e.g.GatewayService ] [5sOo-00] recovered [3] indices into cluster_state
wikibase_1 | wait-for-it.sh: elasticsearch.svc:9200 is available after 5 seconds
wdqs-updater_1 | 07:54:52.324 [main] ERROR org.wikidata.query.rdf.tool.Update - Error during updater run.
wdqs-updater_1 | java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
wdqs-updater_1 | at [Source: (org.apache.http.conn.EofSensorInputStream); line: 1, column: 2]
wdqs-updater_1 | at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:270)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:301)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:314)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:139)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:33)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Updater.run(Updater.java:125)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.main(Update.java:80)
wdqs-updater_1 | Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
wdqs-updater_1 | at [Source: (org.apache.http.conn.EofSensorInputStream); line: 1, column: 2]
wdqs-updater_1 | at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1798)
wdqs-updater_1 | at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:663)
wdqs-updater_1 | at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:561)
wdqs-updater_1 | at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
wdqs-updater_1 | at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:826)
wdqs-updater_1 | at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:723)
wdqs-updater_1 | at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4129)
wdqs-updater_1 | at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3988)
wdqs-updater_1 | at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:422)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:264)
wdqs-updater_1 | ... 6 common frames omitted
wdqs-updater_1 | Exception in thread "main" java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
wdqs-updater_1 | at [Source: (org.apache.http.conn.EofSensorInputStream); line: 1, column: 2]
wdqs-updater_1 | at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:270)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:301)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:314)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:139)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:33)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Updater.run(Updater.java:125)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.main(Update.java:80)
wdqs-updater_1 | Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
wdqs-updater_1 | at [Source: (org.apache.http.conn.EofSensorInputStream); line: 1, column: 2]
wdqs-updater_1 | at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1798)
wdqs-updater_1 | at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:663)
wdqs-updater_1 | at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:561)
wdqs-updater_1 | at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
wdqs-updater_1 | at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:826)
wdqs-updater_1 | at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:723)
wdqs-updater_1 | at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4129)
wdqs-updater_1 | at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3988)
wdqs-updater_1 | at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:422)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:264)
wdqs-updater_1 | ... 6 more
wikibase_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.80.6. Set the 'ServerName' directive globally to suppress this message
wikibase_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.80.6. Set the 'ServerName' directive globally to suppress this message
wikibase_1 | [Fri Jul 12 07:54:52.423218 2019] [mpm_prefork:notice] [pid 59] AH00163: Apache/2.4.25 (Debian) PHP/7.2.19 configured -- resuming normal operations
wikibase_1 | [Fri Jul 12 07:54:52.423284 2019] [core:notice] [pid 59] AH00094: Command line: 'apache2 -D FOREGROUND'
phenowiki_wdqs-updater_1 exited with code 1
elasticsearch_1 | [2019-07-12T07:54:53,079][INFO ][o.e.c.r.a.AllocationService] [5sOo-00] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[mw_cirrus_metastore_first][0]] ...]).
Update finished
wikibase_1 | 10.2.0.24 - - [12/Jul/2019:07:57:36 +0000] "GET /wiki/Special:RecentChanges?hidebots=1&hideWikibase=1&limit=50&days=7&enhanced=1&urlversion=2&peek=1&from=20190712075407&isAnon=true&action=render HTTP/1.1" 204 472 "http://myserver.nl:8181/wiki/Special:RecentChanges?hidebots=1&hideWikibase=1&limit=50&days=7&enhanced=1&urlversion=2" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
wikibase_1 | 10.2.0.24 - - [12/Jul/2019:07:57:36 +0000] "GET /wiki/Special:RecentChanges?hidebots=1&hideWikibase=1&limit=50&days=7&enhanced=1&urlversion=2 HTTP/1.1" 200 8081 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
wikibase_1 | 10.2.0.24 - - [12/Jul/2019:07:57:37 +0000] "POST /w/api.php HTTP/1.1" 200 923 "http://myserver.nl:8181/wiki/Special:RecentChanges?hidebots=1&hideWikibase=1&limit=50&days=7&enhanced=1&urlversion=2" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
wikibase_1 | 10.2.0.24 - - [12/Jul/2019:07:57:37 +0000] "POST /w/api.php HTTP/1.1" 200 924 "http://myserver.nl:8181/wiki/Special:RecentChanges?hidebots=1&hideWikibase=1&limit=50&days=7&enhanced=1&urlversion=2" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
wikibase_1 | 10.2.0.24 - - [12/Jul/2019:07:57:37 +0000] "GET /w/opensearch_desc.php HTTP/1.1" 200 793 "-" "com.apple.Safari.SearchHelper/14607.2.6.1.1 CFNetwork/978.0.7 Darwin/18.6.0 (x86_64)"
wikibase_1 | 10.2.0.24 - - [12/Jul/2019:07:58:54 +0000] "GET /wiki/Special:RecentChanges?hidebots=1&hideWikibase=1&limit=50&days=7&enhanced=1&urlversion=2&peek=1&from=20190712075737&isAnon=true&action=render HTTP/1.1" 204 472 "http://myserver.nl:8181/wiki/Special:RecentChanges?hidebots=1&hideWikibase=1&limit=50&days=7&enhanced=1&urlversion=2" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
wdqs-proxy_1 | 192.168.80.9 - - [12/Jul/2019:07:59:29 +0000] "GET /bigdata/namespace/wdq/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D&nocache=26048639 HTTP/1.0" 200 83 "http://myserver.nl:8282/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" "-"
wdqs-frontend_1 | 10.2.0.24 - - [12/Jul/2019:07:59:29 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D&nocache=26048639 HTTP/1.1" 200 94 "http://myserver.nl:8282/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" "-"
Is anyone familiar on how this could have happend? I managed to reproduce this problem by starting from a clean docker based Wikibase after startup I did a shutdown and then start it up again.

Is there a programatic way to get all the information showm on the EC2 Service Limits page?

Could someone tell me if exists a programmatically way, or from command line, for example using awscli, to get all the information, that is showed on the "EC2 Service Limits" page?
I've used this command: ec2 describe-account-attributes
, but it gives only a general information, but I need to know as well such information as:
Running On-Demand c1.medium instances,
Running On-Demand c1.xlarge instances
etc.
The closest I could find is:
aws support describe-trusted-advisor-check-result \
--check-id eW7HH0l7J9 \
--query 'result.sort_by(flaggedResources[?status!="ok"],&metadata[2])[?starts_with(metadata[2],`On-Demand`)].metadata' \
--output table \
--region us-east-1
This seems to return only non-default limits though. My output looks like this:
---------------------------------------------------------------------------------
| DescribeTrustedAdvisorCheckResult |
+-----------+------+------------------------------------+-------+-----+---------+
| us-east-1| EC2 | On-Demand instances - c3.2xlarge | xxx | 1 | Green |
| us-east-1| EC2 | On-Demand instances - c4.2xlarge | xxx | 0 | Green |
| us-west-2| EC2 | On-Demand instances - c4.2xlarge | xxxx | 0 | Green |
| us-east-1| EC2 | On-Demand instances - c4.xlarge | xxx | 0 | Green |
| us-west-2| EC2 | On-Demand instances - c4.xlarge | xxxx | 9 | Green |
| us-east-1| EC2 | On-Demand instances - m3.xlarge | xxx | 1 | Green |
| us-east-1| EC2 | On-Demand instances - m4.large | xxx | 7 | Green |
| us-west-2| EC2 | On-Demand instances - m4.large | xxx | 8 | Green |
| us-east-1| EC2 | On-Demand instances - m4.xlarge | xxx | 34 | Green |
| us-west-2| EC2 | On-Demand instances - m4.xlarge | xxx | 97 | Green |
| us-east-1| EC2 | On-Demand instances - r4.2xlarge | xx | 8 | Green |
| us-west-2| EC2 | On-Demand instances - r4.2xlarge | xx | 8 | Green |
| us-east-1| EC2 | On-Demand instances - t2.medium | xxx | 1 | Green |
| us-west-2| EC2 | On-Demand instances - t2.medium | xxxx | 1 | Green |
| us-east-1| EC2 | On-Demand instances - t2.micro | xxx | 2 | Green |
| us-west-2| EC2 | On-Demand instances - t2.micro | xxxx | 2 | Green |
| us-west-2| EC2 | On-Demand instances - t2.nano | xxxx | 1 | Green |
| us-east-1| EC2 | On-Demand instances - t2.small | xxx | 5 | Green |
| us-west-2| EC2 | On-Demand instances - t2.small | xxxx | 8 | Green |
+-----------+------+------------------------------------+-------+-----+---------+