In my use case, I am trying to use the $HOME variable to identify my app server path in the instance startup.
I am using Google compute engine with a startup script which uses $HOME variable. But it looks $HOME is not set or the user is not created while startup script executes in google cloud.
It throws $HOME not set error. Is there any workaround for this? Now I have to restart the instance after creating for the first time. So that the $HOME variable will be set when I restart. But this is an ugly hack for production.
Could someone help me with this?
The startup script is executed as root when the user have been not created yet and no user is logged in (you can check it running at startup $ users and comparing the output of $ cat /etc/shadow after a reboot).
Honestly I don't understand how just a reboot can make your $HOME be populated at startup time since on Linux, the HOME environment variable is set by the login program:
by login on console, telnet and rlogin sessions
by sshd for SSH
connections by gdm, kdm or xdm for graphical sessions.
However if you need to reboot and you don't want to do it manually you can reboot just once after the creation of a machine:
if [ -f flagreboot ]; then
...
your script
...
else
touch flagreboot
reboot
fi
On the other hand if you know which is going to be the $HOME path of your application you can think to simply export this variable at startup to populate it manually.
$ export HOME=/home/username
printenv
cd $HOME
touch test.txt
echo $HOME >> test.txt
echo $PWD >> test.txt
printenv > env.txt
I included the above code in my startup script. Strangely, the $HOME, $PWD and many other environment variables are not set while the startup script is runninng. Here are the contents of of the files I created during the startup.
test.txt:
/
env.txt:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/
LANG=en_US.UTF-8
SHLVL=2
_=/usr/bin/printenv
Here's the output(some values removed) of printenv command, immediately after the VM creation.
XDG_SESSION_ID=
HOSTNAME=server1
SELINUX_ROLE_REQUESTED=
TERM=xterm-256color
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=
SELINUX_USE_CURRENT_RANGE=
SSH_TTY=/dev/pts/0
USER=
LS_COLORS=
MAIL=/var/spool/mail/xyz
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/*<username>*/.local/bin:/home/*<username>*/bin
PWD=/home/*<username>*
LANG=en_US.UTF-8
SELINUX_LEVEL_REQUESTED=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/*<username>*
LOGNAME=*<username>*
SSH_CONNECTION=
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/1000
_=/usr/bin/printenv
To summarize, not all the environment variables are set at the time the startup script executes. They are populated some time after. I find that wierd, but that's how it's works.
Related
This is a continuation of this thread, posted here because it was too complicated for a comment.
TL;DR
In a Vertex AI User Managed Notebook, how does one retain the exposed kernel icons for existing venv (and conda, if possible) environments stored on the data disk, through repeated stop and start cycles?
Details
I am using User Managed Notebook Instances built off a Docker image. Once the Notebook is launched, I manually go in create a custom environment. For the moment, let's say this is a venv python environment. The environment works fine and I can expose the kernel so it shows as an icon in the Jupyter Lab's Launcher. If I shut the instance down and restart it, the icon is gone. I have been trying to create a start-up script that re-exposes the kernel, but it is not working properly. I have been trying to use method #2 proposed by #gogasca in the link above. Among other operations (which do execute correctly), my start-up script contains the following:
cat << 'EOF' > /home/jupyter/logs/exposeKernel.sh
#!/bin/bash
set -x
if [ -d /home/jupyter/envs ]; then
# For each env creation file...
for i in /home/jupyter/envs/*.sh; do
tempName="${i##*/}"
envName=${tempName%.*}
# If there is a corresponding env directory, then expose the kernel
if [ -d /home/jupyter/envs/${envName} ]; then
/home/jupyter/envs/${envName}/bin/python3 -m ipykernel install --prefix=/root/.local --name $envName &>> /home/jupyter/logs/log.txt
echo -en "Kernel created for: $envName \n" &>> /home/jupyter/logs/log.txt
else
echo -en "No kernels can be exposed\n" &>> /home/jupyter/logs/log.txt
fi
done
fi
EOF
chown root /home/jupyter/logs/exposeKernel.sh
chmod a+r+w+x /home/jupyter/logs/exposeKernel.sh
su -c '/home/jupyter/logs/exposeKernel.sh' root
echo -en "Existing environment kernels have been exposed\n\n" &>> /home/jupyter/logs/log.txt
I am attempting to log the operations, and I see in the log that the kernel is created successfully in the same location that it would be created if I were to manually activate the environment and expose the kernel from within. Despite the apparent success in the log (no errors, at least), the kernel icon does not appear. If I manually run the exposeKernel.sh script from the terminal using su -c '/home/jupyter/logs/exposeKernel.sh' root, it also works fine and the kernel is exposed correctly. #gogasca's comments on the aforementioned thread suggest that I should be using the jupyter user instead of root, but repeated testing and logging indicates that the jupyter user fails to execute the code while root succeeds (though neither create the kernel icon when called from the start-up script).
Questions:
(1) My goal is to automatically re-expose the existing environment kernels on startup. Presumably they disappear each time the VM is stopped and started because there is some kind of linking to the boot disk that is rebuilt each time. What is the appropriate strategy here? Is there a way to build the environments (interested in both conda and venv) so that their kernel icons don't vaporize on shut-down?
(2) If the answer to (1) is no, then why does the EOF-created file fail to accomplish the job when called from a start-up script?
(3) Tangentially related, am I correct in thinking that the post-startup-script executes only once during the initial Notebook instance creation process, while the the startup-script or startup-script-url executes each time the Notebook is started?
I am trying to run a script on my EC2 at startup, with an image I created that runs ubuntu.
However, the script is failing although when I connect through ssh and run the script it is working.
My user data is:
#!/bin/bash
echo '
#!/bin/bash
sleep 30
sudo apt-get update
cd /etc/apache2/sites-available
sudo sed -i 's/oldurl/newurl/g' 000-default.conf
sudo sed -i 's/oldurl/newurl/g' 000-default.conf
sudo certbot --apache -d url1 -d url2
sudo systemctl restart apache2' > init-ssl.sh
sleep 2 & init-ssl.sh
I stopped my instance and changed my user data to something simple like:
#!/bin/bash
echo 'work' > try1.txt
I didn't see an error but I also didn't see my new try1.txt file.
A script passed via User Data will only be executed on the first boot of the instance. (Actually, the first boot per Instance ID.)
If you want to debug the script, the log file is available in:
/var/log/cloud-init-output.log
Your attempt to redirect to a file with echo ' ... ' >init-ssl.sh is being thwarted by the fact that the script also contains a single quote ('), which is closing the echo early. You should use different quotes to avoid this happening. Or, as #Mornor points out, simply run the script directly. If you want to sleep for a bit up-front, then just put the sleep() at the start of the script.
I put the following commands in user data of an EC2 running RedHat 8 AMI (ami-0fc841be1f929d7d1), when they run, the mkdir tries to create .kube at root which looks to me like $HOME is not set at the time.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Following are log from /var/log/user-data.log
+ mkdir -p /.kube
+ sudo cp -i /etc/kubernetes/admin.conf /.kube/config
++ id -u
++ id -g
+ sudo chown 0:0 /.kube/config
When I SSH to the instance, the $HOME is set correctly to /home/ec2-user.
Could you advise what I did wrong here?
Thank you
When your EC2 server is provisioned, the user data script runs as user root, so $HOME is empty. What you could do, is to define the HOME env var at the top of your user data script, like this (insert your user's home directory here):
export HOME=/home/ubuntu
I've tried it and it works (I install NVM, SDKMAN, sbt, java, git, docker; all works fine). You might need to do some chown at the end of your user data script to change the owner of some files back to your user. For example, if your user data sets up some files in your home directory:
chown ubuntu ~/.foo/bar.properties
$HOME refers to the home directory of the logged in user. Userdata runs under the root user, and the root user $HOME is /. That is the result you are seeing.
Instead of the variable $HOME, your script should refer to /home as a literal.
See https://superuser.com/questions/271925/where-is-the-home-environment-variable-set
You are running as sudo which is known to change environment variables that are established with your users shell (such as $HOME) as well as shell context based such as ssh-agent.
Generally you can ensure this persists when you run sudo by adding it to the env_keep settings in your sudoers configuration by adding the below line within /etc/sudoers. More information is available here, be careful about modifying this file.
Defaults env_keep=HOME
Otherwise if you don't want to make the above change, ensure you have the permissions to carry this out without running sudo or pass an absolute path value in.
I would generally stay clear of user data for important configuration anyway,
instead build a pre-baked AMI ahead of time with the configuration how you want it, using a configuration tool such as Ansible, Chef, Puppet.
Alternatively as this is within the User Data anyway, it is unlikely you have already configured the sudoers configuration, you should instead just specify the path.
I faced the same issue. Adding this to the User Data script helped resolve it. The sub shells will have the HOME set with this change to profile.
cat > /etc/profile.d/set_home.sh << 'EOF'
export HOME=~
EOF
chmod a+x /etc/profile.d/set_home.sh
When I ssh into my EC2 Instance and run the following commands my SpringServer.jar file executes and I can access my Spring application by going to myawsaccount:8080/times. when I specify the following commands in User Data I cant access my application at myawsaccount:8080/times and im not sure why. Any help would be appreciated.
Commands
#!/bin/bash --> only in user script
sudo su
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.rpm
yum install -y jdk-8u141-linux-x64.rpm
wget https://myawsaccount.s3-eu-west-1.amazonaws.com/SpringServer-1-0.0.1-SNAPSHOT.jar
java -jar SpringServer-1-0.0.1-SNAPSHOT.jar
To troubleshoot UserData issues, the best thing to do is to login to an instance,
and inspect one of UserData log files.
Most impotently /var/log/cloud-init-output.log:
The cloud-init output log file (/var/log/cloud-init-output.log) captures console output so it is easy to debug your scripts following a launch if the instance does not behave the way you intended.
Also your UserData script will be located in /var/lib/cloud/instances/<instance-id>/. Thus, once you are in the instance you can manually try to run it and fix/debug while in the instance.
Setting environment variables using export doesn't work in user data as it only sets them for the current shell session. You can fix this by copying them to your profile configuration:
#!/bin/bash
...
echo 'export JAVA_HOME=/opt/jdk1.8.0_141' >> /etc/profile
echo 'export JRE_HOME=/opt/jdk1.8.0_141/jre' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin' >> /etc/profile
...
This way, the environment variables will be available in every session.
Boiling my issue down to the simplest case, I'm using Compute Engine with the following startup-script:
#! /bin/bash
sudo useradd -m drupal
su drupal
cd /home/drupal
touch test.txt
I can confirm the drupal user exists after this command, so does the test file. However I expect the owner of the test file to be 'drupal' (hence the su). However, when I use this as a startup script I can still confirm ROOT is the owner of the file:
meaning my
su drupal
did not work. sudo su drupal also does not make any difference. I'm using Google Container OS, but same happens on a Debian 8 image.
sudo su is not a command run within a shell -- it starts a new shell.
That new shell is no longer running your script, and the old shell that is running the script waits for the new one to exit before it continues.
The sudo su command will start a new shell. The old shell waits for the old one to exit and continues executing the rest of the code.
Your script is running in the 'old' shell, which means these commands:
cd /home/drupal
touch test.txt
are still executed as root and thus the owner of these files is root as well.
You can modify your script to this:
#! /bin/bash
sudo useradd -m drupal
sudo -u drupal bash -c 'cd ~/; touch text2.txt'
and it should work.
The -u flag executes the command as the user specified, in this case 'drupal'
I wrote some stuff underneath - but looks like this should work:
how to run script as another user without password
The other option would be to ssh into your own machine as the other user, you can use sshpass to send the password, or get your own public key.
When I test a similar script:
su [my username]
touch test.txt
It actually logs in as me, and doesn't finish until I ctrl-d
Further testing reveals that the only way to own the file is if I invoke the script from the shell, ie:
su me
touch test.txt
./test2.sh
test2.sh:
touch test2.txt
gives both files to root, even if I own both scripts.
This follows that everything YOU do is yours, you can't make something for someone else.