I have a simple bash script only does “sleep 3600” on remote host (Amazon EC2) and I am using fabric to call it via fabric.operations.run (I did NOT set any env.timeout or env.command_timeout).
If the remote bash script sleeps for 3600 seconds, fabric was NOT able to return after the bash script is done running. I printed the stack trace and it kept waiting on channel.exits_status_ready() (https://github.com/fabric/fabric/blob/master/fabric/operations.py LINE: 794) even if the script already returned.
This ONLY happens for long running process. I tried to make bash script sleep for 120 seconds and it worked fine.
I double checked the open connections using netstat, and the ssh session opened by fabric was still alive.
Help needed :) Any idea why this happens?
Figured out, just needed to use env.keepalive = 1
Related
I need to create AWS CentOS 7 instance images for a customer, and need it to automatically send the ip and instance id to our AWS server every time the instance boots. For example, this is the very basic test version of the script I need to run:
#!/bin/bash
$serverIP=""
curl "https://$serverIP"/myphp.php?id='sentid'&ip='sentip'"
If the script is run directly, it works fine and is received by the server and processed there. But I can't get it to run at boot. I cannot put the script in the "User Data" directly due to security concerns as the customer can then see it easily, it needs to be in a script in the filesystem of the image.
I've tried several things that work fine on a physical Linux server, but not on AWS. I know profile.d runs every time someone logs in but over-sending like that is fine.
/etc/profile.d/myscript.sh
This stops the AWS instance from booting. Even just
#!/bin/bash/
echo "hello world"
prevents it from booting. The instance starts, but when you go to ssh into it you get 'Network Error: connection timed out', which is the standard error if you put a wrong ip in, or upset it by leaving a service like httpd enabled.
However, a blank bash script with just #!/bin/bash will allow the instance to start. Removing the script via user data usually makes it boot, sometimes it just dies.
The first thing I tried was crontab. I did:
crontab -e
#reboot /var/ook/myscript.sh
systemctl enable crond.service
But the instance wouldn't start. So I put "systemctl disable crond.service" in the User Data and one booted, but another still stayed dead. Myscript.sh was just another echo "doob" >> file which worked fine when run directly.
I tried putting in /etc/systemd/system/my-startup.service:
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/var/ook/writedood.sh
[Install]
WantedBy=multi-user.target
then:
systemctl enable my-startup.service
But this did nothing. My script "writedood.sh" was just echo "doob" >> ./file.txt ensuring file.txt was chmod 777. At least it didn't prevent the instance from starting.
To give context, an instance won't start if httpd is left enabled on shutdown, but will if you disable it in User Data.
I wanted to have a go at putting something in init.d but I'm not sure how to simply tell it to run a script once in the background, and given the plethora of success I've had so far with the instance not restarting, I'm not holding out much hope that that would work.
Thanks in advance!
EDIT::: I realised that sometimes AWS EC2 Instances Console is causing the problem where I can't ssh in after stopping and starting. It blanks the public ipv4 address when I click stop, but when I start, it puts the old address up and hangs. If I refresh the page, or uncheck/check the instance; the ip changes to the new address. This has caused much consternation.
Crontab worked if I placed the scripts and output file in different folders. It's very finicky; any errors, such as it not being able to write to the output file, and the instance won't start. I put startscript.sh in /usr/local/src, and output.out to /tmp/ to ensure there were no permissions problems, and now the instance starts and runs the script on boot.
I then realised that sometimes AWS EC2 Instances Console is causing the problem where I can't ssh in after stopping and starting. It blanks the public ipv4 address when I click stop, but when I start, it puts the old address up and hangs. If I refresh the page, or uncheck/check the instance; the ip changes to the new address. This has caused much consternation.
here's a problem that is driving me nuts. First off, I am not a Linux expert, so I might just be missing some detail.
I am trying to restart an application (namely rpi-webrtc-streamer, but that shouldn't matter) using a shell script. The reason is that when a configuration change happens I need to update the config files and restart.
The idea is to call a bash script using system() function and pass in the pid of the current process. The script should then just kill the process using the supplied pid, and execute it again. In theory this shouldn't be a problem...
What may be complicating it is that the process needs to run with sudo. Not sure if that's the case but just thought I should mention it.
Now this is the script:
#!/bin/bash
echo "restarting streamer..."
echo "killing process with PID $1"
kill $1
# I have tried different intervals, even 10 seconds, doesn't help
sleep 2
echo "running new streamer instance"
echo "path:"
pwd
#printenv
echo "id -u"
# just to verify the script runs with sudo
id -u
./webrtc-streamer --verbose
echo "done"
The problem is that the application fails with the following error:
(direct_socket.cc:77): Failed to listen 0.0.0.0:8888.
... and then it shuts down. Well obviously it's not able open the port. It almost looks as if the previous instance of the app is still holding the port open. I have however tried tweaking the sleep amount of seconds in the script but that shouldn't be a problem, first I think the script will continue execution after the process is actually killed and second the process shuts down immediately anyway, I can see that from the logs.
If I however run the app immediately after the script fails from the shell that actually executed the initial app in the first place, it runs without any issues (being able to open the port). No matter how much seconds it waited in the sleep previously.
The only other thing I though of would be that the bash script might be running with different environment variables. I tried to print those but I don't see anything significant.
Also I verified that the app does not change the working directory, but that again should not be a problem as it actually launches. It then just exits after not being able to open the port.
I also tried adding sudo before the app execution in the script (which shouldn't be necessary AFAIK). Doesn't make a difference.
Any ideas?
As suggested by jordanm in the comments, I solved the problem by using systemd.
So, I'm using Google Cloud Platform and set below startup script
#! /bin/bash
cd /home/user
sudo ./process1
sudo ./process2
I worried about this script because process1 blocks shell and prevent to run sudo ./process2. And it really was. process1 was started successfully but process2 was not started.
I checked that script has no problem with starting process1 and process2. Execute ./process2 via SSH worked but after I close the SSH shell and process2 was stopped too.
How can I start both process in booting time(or even after)?
I tried testing your startup script in my environment,it seems the script works well.
1.You can please try checking process1 and process2 scripts.
2.If you want your process to run in the background even after the SSH session is closed, you can use “&” { your_command & }at the end of your command.
To run a command in the background, add the ampersand symbol (&) at the end of the command:
your_command &
then the script execution continues and isn't blocked. Or use linux internal means to auto run processes on boot.
I have a bash script which will take 5-6 hrs to complete and yesterday i accessed aws 12 month free tire and running ec2 (ubuntu) on it ,i want to run that bash script even after i close my main machine ...how can i do this ?
Assuming this is on linux system, you can run your script in the background using & optons. Something like this
yourBashScript.sh &
Where & tells the shell to run it in the background. So even if you close the shell or end your ssh session, it will keep running in the background till it finishes the job or crashes due to any error.
You can always check whether your script is running or not using ps command. Something like this
ps -eaf | grep yourBashScript
this may return the process information for your script, if it is in running state.
I have a bash script. I would like to run it continuously on google cloud server. I connected to my VM via SSH in browser but after I've closed my browser, script was stopped.
I tried to use Cloud Shell but if I restart my laptop, script launches from start. It doesn't work continuously!
Is it possible to launch my script in google cloud, shut down laptop and be sure what my script works?
The solution: GNU screen. This awesome little tool let's you run a process after you've ssh'ed into your remote server, and then detach from it - leaving it running like it would run in the foreground (not stopped in the background).
So after we've ssh'ed into our GCE VM, we will need to:
1. install GNU screen:
apt-get update
apt-get upgrade
apt-get install screen
type "screen". this will open up a new screen - kind of similar in look & feel to what "clear" would result in.
run the process (e.g.: ./init-dev.sh to fire up a ChicagoBoss erlang server)
type: Ctrl + A, and then Ctrl + D. This will detach your screen session but leave your processes running!
feel free to close the SSH terminal. whenever you feel like it, ssh back into your GCE VM, and type screen -r to resume your previously detached session.
to kill all detached screens, run:
screen -ls | grep pts | cut -d. -f1 | awk '{print $1}' | xargs kill
You have the following options:
1. Task schedules - which involves cron jobs. Check this sample. Via this answer;
2. Using startup scripts.
I performed the following test and it worked for me:
I created an instance in GCE, SSH-d into it and created the following script, myscript.bash:
#!/bin/bash
sleep 15s
echo Hello World > result.txt
and then, ran
$ bash myscript.bash
and immediately closed the browser window holding the SSH session.
I then waited for at least 15 seconds, re-engaged in an SSH connection with the VM in question and ran $ ls and voila:
myscript.bash result.txt
So the script ran even after closing the browser holding the SSH session.
Still, technically, I believe your solution lies with 1. or 2.
You can use
nohup yourscript.sh > output_log_file.log
I faced similar issue. I logged into Virtual Machine through google cloud command on my local machine, tried to exit by closing the terminal, It halted the script running in the instance.
Use command exit to log out of cloud consoles in local machine putty console (twice).
Make sure you have not enabled "PREEMPT INSTANCE" while creating a VM instance.
It will force to close the instance within 24 hours to reduce the costing by a huge difference.
I have a NodeJS project and I solved with pm2