Problems with Startup script - Compute Engine - google-cloud-platform

I need to run a jar file when my Ubuntu 16.04 instance on GCP is ON. So I add a custom metadata:
but It's not working.
For testings, I have tried to generate files every time I turn off and on the VM, and startup script compiles them successfully.
EDITED: If I run this command from shell, it runs succesfully and port 4444 is opened.
Furthermore, the startup script ignores xvfb for the following example, but
fuego_vins.txt and fuego_vins_exit.txt are deposited on /usr/bin:
#! /bin/bash
sudo touch /usr/bin/fuego_vins.txt
sudo apt-get update
sudo xvfb-run -a java -Dwebdriver.chrome.driver=/usr/bin/chromedriver -jar selenium-server-standalone-3.13.0.jar -port 4444
sudo touch /usr/bin/fuego_vins_exit.txt

I tested it using:
xvfb version: xorg-server 1.18.4
selenium-server-standalone-3.9.1.jar
from /usr/bin/selenium
-rwxr-xr-x 1 root root 153 Nov 27 08:31 run
-rwxr-xr-x 1 user user 23428464 Feb 7 2018 selenium-server-standalone-3.9.1.jar
I created script into /home/user folder into VM then I ran it from startup-script.
startup-script:
sudo apt-get update
sudo apt-get upgrade
sudo /usr/bin/selenium/run
from the Serial port 1 (console) output for instance:
Ubuntu 16.04.5 LTS instance ttyS0
instance-2 login: Nov 27 09:00:43 instance-2 startup-script: INFO startup-script: 2018-11-27 09:00:43.766:INFO::main: Logging initialized #2790ms to org.seleniumhq.jetty9.util.log.StdErrLog
Nov 27 09:00:44 instance-2 startup-script: INFO startup-script: 2018-11-27 09:00:44.100:INFO:osjs.Server:main: jetty-9.4.7.v20170914, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
Nov 27 09:00:44 instance-2 startup-script: INFO startup-script: 2018-11-27 09:00:44.158:WARN:osjs.SecurityHandler:main: ServletContext#o.s.j.s.ServletContextHandler#45ca843{/,null,STARTING} has uncovered http methods for path: /
Nov 27 09:00:44 instance-2 startup-script: INFO startup-script: 2018-11-27 09:00:44.166:INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler#45ca843{/,null,AVAILABLE}
Nov 27 09:00:44 instance-2 startup-script: INFO startup-script: 2018-11-27 09:00:44.230:INFO:osjs.AbstractConnector:main: Started ServerConnector#93c3e76{HTTP/1.1,[http/1.1]}{0.0.0.0:4444}
Nov 27 09:00:44 instance-2 startup-script: INFO startup-script: 2018-11-27 09:00:44.231:INFO:osjs.Server:main: Started #3256ms
I could not run it, with selenium-server-standalone-3.13.0.jar; so I'll try later. Hope it could be useful to you.

Related

EC2 instance - unable to read write files though they exists

Please help with this strange problem that i am facing with aws EC2 instance. I have launched a EC2 instance & installed git to clone my repo. all good untill this, but i am unable to read/write those files though they exists in the same path. observed the same behaviour in Amazon Linux instance as well.
ubuntu#ip-172-31-54-19:~/win2win$
ubuntu#ip-172-31-54-19:~/win2win$ cd $HOME
ubuntu#ip-172-31-54-19:~$
ubuntu#ip-172-31-54-19:~$
ubuntu#ip-172-31-54-19:~$ cd win2win/
ubuntu#ip-172-31-54-19:~/win2win$
ubuntu#ip-172-31-54-19:~/win2win$ cat myscript.sh
cat: myscript.sh: No such file or directory
ubuntu#ip-172-31-54-19:~/win2win$
ubuntu#ip-172-31-54-19:~/win2win$ ls -lrt
total 28
drwxrwxr-x 2 ubuntu ubuntu 4096 Apr 5 09:21 templates
-rw-rw-r-- 1 ubuntu ubuntu 792 Apr 5 09:21 sampleIDs
-rw-rw-r-- 1 ubuntu ubuntu 2614 Apr 5 09:30 myscript.sh
ubuntu#ip-172-31-54-19:~/win2win$
ubuntu#ip-172-31-54-19:~/win2win$ pwd
/home/ubuntu/win2win
ubuntu#ip-172-31-54-19:~/win2win$
Can anyone help.
thanks in advance.

docker image is different when running from different host

During building of a 3rd party library (libtorch, if it matters) in a docker container, I came across an error of a missing include file.
The same process of building worked fine when running the build process from Ubuntu 16.04 host, but when running from an Ubuntu 18.04 host, the file was missing.
After a bit of trace back, I'm now just running the base container from NVidia, and looking for the file.
This is the outputs I get:
Ubuntu 16.04 host:
$ uname -a
Linux ub-carmel 4.15.0-123-generic #126~16.04.1-Ubuntu SMP Wed Oct 21 13:48:05 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 19.03.13, build 4484c46d9d
$ docker pull nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
11.1-cudnn8-devel-ubuntu18.04: Pulling from nvidia/cuda
Digest: sha256:c5bf5c984998cc18a3f3a741c2bd7187ed860dc6d993b6fb402d0effb9fe6579
Status: Image is up to date for nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
$ docker run -it nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
root#2ecc17248fab:/# ll /usr/lib/gcc/x86_64-linux-gnu/7/include | grep ia32
-rw-r--r-- 1 root root 7817 Dec 4 2019 ia32intrin.h
Ubuntu 18.04 host:
$ uname -a
Linux ub-carmel-18-04 5.4.0-56-generic #62~18.04.1-Ubuntu SMP Tue Nov 24 10:07:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 19.03.14, build 5eb3275d40
$ docker pull nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
11.1-cudnn8-devel-ubuntu18.04: Pulling from nvidia/cuda
Digest: sha256:c5bf5c984998cc18a3f3a741c2bd7187ed860dc6d993b6fb402d0effb9fe6579
Status: Downloaded newer image for nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
$ docker run -it nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
root#89f771e82a51:/# ll /usr/lib/gcc/x86_64-linux-gnu/7/include | grep ia32
root#89f771e82a51:/#
As you can see, the sha256 digest of the images is the same (and matches the digest from NVidia's NGC here)
At first I thought that maybe in some hidden way the includes come from the host, but the ia32intrin.h file exists in both hosts
What can cause such issue?
EDIT
Added the docker --version outputs for each host. There's a difference, but I doubt this should cause such issues
EDIT 2
Added the output for uname -a
EDIT 3
Output of docker version:
Ubuntu 16:
$ docker version
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:59 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:01:30 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Ubuntu 18:
$ docker version
Client: Docker Engine - Community
Version: 19.03.14
API version: 1.40
Go version: go1.13.15
Git commit: 5eb3275d40
Built: Tue Dec 1 19:20:17 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.14
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 5eb3275d40
Built: Tue Dec 1 19:18:45 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.9
GitCommit: ea765aba0d05254012b0b9e595e995c09186427f
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
So I tested it on a different Ubuntu machines (EC2 instances) and in that case, for both 18.04 & 16.04 the file exists. so looks like it's a problem on my machine.
Any thoughts of what can cause this?
Best guess is that the pulled layers on the Ubuntu 18.04 host are somehow corrupt. The nuclear option to clean that up is to reset docker. This will delete all images, volumes, containers, logs, networks, everything, so backup anything you want to keep before running this:
sudo -s # these commands need root
systemctl stop docker
rm -rf /var/lib/docker
systemctl start docker
exit # exit sudo

Deploying Ghost One-Click droplet on Digital Ocean: Exit code 1 when verifying domain

I'm trying to follow along with DigitalOceans intructions on creating a one-click droplet in order to set up a ghost cms for my website.
I set up my website on Netlify, and had Netlify manage the dns. When Netlify manages the dns it looks like it automatically sets up ssl on Netlify.
The domain registrar is Namecheap.
When I ssh to my droplet I get the output below and visiting https://example.com/ghost/ gives a 404 error, but I can still visit https://example.com
~ % ssh root#111.111.1.11
The authenticity of host '111.111.1.11 (111.111.1.11)' can't be established.
ECDSA key fingerprint is SHA111:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '111.111.1.11' (ECDSA) to the list of known hosts.
root#111.111.1.11's password:
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
-------------------------------------------------------------------------------
Configuring DigitalOcean 1-Click Ghost installation.
Please wait a minute while your 1-Click is configured.
Once complete, you are encouraged to run mysql_secure_installation to ready
your server for production. Passwords have been saved to:
root/.digitalocean_password
-------------------------------------------------------------------------------
Ensuring Ghost-CLI is up-to-date...
+ sudo npm i -g ghost-cli#latest
/usr/bin/ghost -> /usr/lib/node_modules/ghost-cli/bin/ghost
+ ghost-cli#1.13.1
added 59 packages from 29 contributors, removed 11 packages and updated 48 packages in 16.313s
Ghost will prompt you for two details:
1. Your domain
- Add an A Record -> 111.111.1.11 & ensure the DNS has fully propagated
- Or alternatively enter http://111.111.1.11
2. Your email address (only used for SSL)
Press enter when you're ready to get started!
+ sudo chown -R ghost-mgr:ghost-mgr /home/ghost-mgr/.config
✔ Checking system Node.js version
✔ Checking logged in user
✔ Checking current folder permissions
✔ Checking operating system compatibility
✔ Checking for a MySQL installation
✔ Checking memory availability
✔ Checking for latest Ghost version
✔ Setting up install directory
✔ Downloading and installing Ghost v3.13.4
✔ Finishing install process
? Enter your blog URL: https://example.com
✔ Configuring Ghost
✔ Setting up instance
+ sudo useradd --system --user-group ghost
+ sudo chown -R ghost:ghost /var/www/ghost/content
✔ Setting up "ghost" system user
ℹ Setting up "ghost" mysql user [skipped]
✔ Creating nginx config file at /var/www/ghost/system/files/example.com.conf
+ sudo ln -sf /var/www/ghost/system/files/example.com.conf /etc/nginx/sites-available/example.com.conf
+ sudo ln -sf /etc/nginx/sites-available/example.com.conf /etc/nginx/sites-enabled/example.com.conf
+ sudo nginx -s reload
✔ Setting up Nginx
? Enter your email (For SSL Certificate) myemail#outlook.com
+ sudo mkdir -p /etc/letsencrypt
+ sudo ./acme.sh --install --home /etc/letsencrypt
+ sudo /etc/letsencrypt/acme.sh --issue --home /etc/letsencrypt --domain example.com --webroot /var/www/ghost/system/nginx-root --reloadcmd "nginx -s reload" --accountemail myemail#outlook.com
✖ Setting up SSL
✔ Creating systemd service file at /var/www/ghost/system/files/ghost_example-com.service
+ sudo ln -sf /var/www/ghost/system/files/ghost_example-com.service /lib/systemd/system/ghost_example-com.service
+ sudo systemctl daemon-reload
✔ Setting up Systemd
+ sudo systemctl is-active ghost_example-com
+ sudo systemctl start ghost_example-com
+ sudo systemctl is-enabled ghost_example-com
+ sudo systemctl enable ghost_example-com --quiet
✔ Starting Ghost
One or more errors occurred.
1) ProcessError
Message: Command failed: /bin/sh -c sudo -S -p '#node-sudo-passwd#' /etc/letsencrypt/acme.sh --issue --home /etc/letsencrypt --domain example.com --webroot /var/www/ghost/system/nginx-root --reloadcmd "nginx -s reload" --accountemail myemail#outlook.com
[Tue Apr 21 18:30:16 UTC 2020] example.com:Verify error:Invalid response from https://example.com/.well-known/acme-challenge/vsRL-FT-yUeK0ntPgkXjl2-4vcYw8FV213estE4b1as [206.189.73.52]:
[Tue Apr 21 18:30:16 UTC 2020] Please add '--debug' or '--log' to check more details.
[Tue Apr 21 18:30:16 UTC 2020] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
[Tue Apr 21 18:30:11 UTC 2020] Create account key ok.
[Tue Apr 21 18:30:11 UTC 2020] Registering account
[Tue Apr 21 18:30:12 UTC 2020] Registered
[Tue Apr 21 18:30:12 UTC 2020] ACCOUNT_THUMBPRINT='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
[Tue Apr 21 18:30:12 UTC 2020] Creating domain key
[Tue Apr 21 18:30:12 UTC 2020] The domain key is here: /etc/letsencrypt/example.com/example.com.key
[Tue Apr 21 18:30:12 UTC 2020] Single domain='example.com'
[Tue Apr 21 18:30:12 UTC 2020] Getting domain auth token for each domain
[Tue Apr 21 18:30:13 UTC 2020] Getting webroot for domain='example.com'
[Tue Apr 21 18:30:13 UTC 2020] Verifying: example.com
Exit code: 1
Debug Information:
OS: Ubuntu, v18.04.3 LTS
Node Version: v10.16.3
Ghost Version: 3.13.4
Ghost-CLI Version: 1.13.1
Environment: production
Command: 'ghost install --auto --db=mysql --dbhost=localhost --dbname=ghost_production --dbuser=ghost --dbpass=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbb --dir=/var/www/ghost --start'
Additional log info available in: /home/ghost-mgr/.ghost/logs/ghost-cli-debug-2020-04-21T18_30_42_032Z.log
Try running ghost doctor to check your system for known issues.
You can always refer to https://ghost.org/docs/api/ghost-cli/ for troubleshooting.
------------------------------------------------------------------------------
For any further commands, please switch to the ghost-mgr user to manage Ghost.
sudo -i -u ghost-mgr
------------------------------------------------------------------------------
I did a bit of searching and it appears from this thread that you need to set up an SSL at the other end rather than using the one from Netlify:
https://community.netlify.com/t/how-to-do-dns-ssl-for-sub-domain-that-doesnt-run-on-netlify/5498/11

Manually start uwsgi with python3.6 and python2 and django 2

I just started to use uWSGI and Django. I want to invoke it manually.
I'm using python36, Django 2 and if I install uWSGI (pip install uwsgi) in venv and everything is ok.
Q1. When using outside of venv I first installed uWSGI:
pip install uwsgi -> 2.0.18 version
Installed plugins: sudo apt-get install python3-setuptools
Trying to run with python36:
uwsgi --plugin-dir=/usr/lib/uwsgi/plugins --plugin python3 --master --http :5000 --home ~/my_playground/webapps/ --chdir ~/my_playground/webapps/p_tscze/ --module p_tscze.wsgi:application
Plugins in /usr/lib/uwsgi/plugins:
/usr/lib/uwsgi/plugins$ ls | grep python3
asyncio_python36_plugin.so
asyncio_python3_plugin.so
python36_plugin.so
python3_plugin.so
Result obtained -> note python 2.7:
!!! UNABLE to load uWSGI plugin: ./python36_plugin.so: undefined symbol: uwsgi_legion_scrolls !!!
*** Starting uWSGI 2.0.18 (64bit) on [Wed May 29 18:04:24 2019] ***
compiled with version: 7.4.0 on 29 May 2019 15:30:14
os: Linux-4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019
nodename: start-tehnicka
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 1
current working directory: /usr/lib/uwsgi/plugins
detected binary path: /usr/local/bin/uwsgi
chdir() to /home/anel/my_playground/webapps/p_tscze/
your processes number limit is 3618
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :5000 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:34279 (port auto-assigned) fd 3
Python version: 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
Set PythonHome to /home/anel/my_playground/webapps/
ImportError: No module named site
So please can you let me know what I'm doing wrong?
Q2. When comparing the result of the admin page (./manage.py runserver) and result obtained using uwsgi frontend is not the same, why?
Thanks in advance
For future references:
wget https://projects.unbit.it/downloads/uwsgi-2.0.18.tar.gz
tar -xzvf uwsgi-2.0.18.tar.gz
cd uwsgi-2.0.18
python3 uwsgiconfig.py --build nolang #build binary with python3
python3 uwsgiconfig.py --plugin plugins/python nolang python36 # create a new binary
Run newly created binary uwsgi (we can link to this binary):
./uwsgi --plugin-dir=/home/anel/uwsgi-2.0.18 --plugin python36 --master --http :5000 --home ~/my_playground/webapps/ --chdir ~/my_playground/webapps/p_tscze/ --module p_tscze.wsgi:application

docker-compose on windows directory sync

Running the django tutorial for docker compose, but the command to init the django project is not working as expected.
$ docker-compose run web django-admin.py startproject composeexample .
[31mERROR[0m: Interactive mode is not yet supported on Windows.
Please pass the -d flag when using `docker-compose run`.
In windows "interactive" mode is not supported so I modified the command to run in "detached" mode.
$ docker-compose run -d web django-admin.py startproject composeexample .
Creating network "djangotest_default" with the default driver
...
Successfully built 0fb90648c1d8
[33mWARNING[0m: Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
djangotest_web_run_1
This seems to create the boxes.. but my current directory shows no new files (from the django-admin command).
$ ls
docker-compose.yml Dockerfile requirements.txt
How do I get the container output to sync with the current working directory as expected?
The fix was to uninstall Windows 10 and install a *nix system. The files then appear as per the tutorial.
~/docker/django-test $ ll
total 60
drwxr-xr-x 4096 Sep 24 00:47 ./
drwxr-xr-x 4096 Sep 24 00:40 ../
drwxr-xr-x 4096 Sep 24 00:47 composeexample/
-rw-r--r-- 209 Sep 24 00:41 docker-compose.yml
-rw-r--r-- 146 Sep 24 00:41 Dockerfile
-rwxr-xr-x 812 Sep 24 00:47 manage.py*
-rw-r--r-- 16 Sep 24 00:41 requirements.txt