I can't create a minion from the map file, no idea what's happened. A month ago my script was working correctly, right now it fails. I was trying to do some research about it but I could't find anything about it. Could someone have a look on my DEBUG log? The minion is created on DigitalOcean but the master server can't connect to it at all.
so I run:
salt-cloud -P -m /etc/salt/cloud.maps.d/production.map -l debug
The master is running on Ubuntu 16.04.1 x64, the minion also.
I use the latest saltstack's library:
echo "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial main" >> /etc/apt/sources.list.d/saltstack.list
I tested both 2016.3.2 and 2016.3.3, what is interesting, the same script was working correctly 4 weeks ago, I assume something had to change.
ERROR:
Writing /usr/lib/python2.7/dist-packages/salt-2016.3.3.egg-info
* INFO: Running install_ubuntu_git_post()
disabled
Created symlink from /etc/systemd/system/multi-user.target.wants/salt-minion.service to /lib/systemd/system/salt-minion.service.
* INFO: Running install_ubuntu_check_services()
* INFO: Running install_ubuntu_restart_daemons()
Job for salt-minion.service failed because a configured resource limit was exceeded. See "systemctl status salt-minion.service" and "journalctl -xe" for details.
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
* ERROR: No init.d support for salt-minion was found
* ERROR: Fai
[DEBUG ] led to run install_ubuntu_restart_daemons()!!!
[ERROR ] Failed to deploy 'minion-zk-0'. Error: Command 'ssh -t -t -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -oPasswordAuthentication=no -oChallengeResponseAuthentication=no -oPubkeyAuthentication=yes -oIdentitiesOnly=yes -oKbdInteractiveAuthentication=no -i /etc/salt/keys/cloud/do.pem -p 22 root#REMOVED_IP '/tmp/.saltcloud-5d18c002-e817-46d5-9fb2-d3bdb2dfe7fd/deploy.sh -c '"'"'/tmp/.saltcloud-5d18c002-e817-46d5-9fb2-d3bdb2dfe7fd'"'"' -P git v2016.3.3'' failed. Exit code: 1
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2293, in create_multiprocessing
local_master=parallel_data['local_master']
File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1281, in create
output = self.clouds[func](vm_)
File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/digital_ocean.py", line 481, in create
ret = __utils__['cloud.bootstrap'](vm_, __opts__)
File "/usr/lib/python2.7/dist-packages/salt/utils/cloud.py", line 527, in bootstrap
deployed = deploy_script(**deploy_kwargs)
File "/usr/lib/python2.7/dist-packages/salt/utils/cloud.py", line 1516, in deploy_script
if root_cmd(deploy_command, tty, sudo, **ssh_kwargs) != 0:
File "/usr/lib/python2.7/dist-packages/salt/utils/cloud.py", line 2167, in root_cmd
retcode = _exec_ssh_cmd(cmd, allow_failure=allow_failure, **kwargs)
File "/usr/lib/python2.7/dist-packages/salt/utils/cloud.py", line 1784, in _exec_ssh_cmd
cmd, proc.exitstatus
SaltCloudSystemExit: Command 'ssh -t -t -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -oPasswordAuthentication=no -oChallengeResponseAuthentication=no -oPubkeyAuthentication=yes -oIdentitiesOnly=yes -oKbdInteractiveAuthentication=no -i /etc/salt/keys/cloud/do.pem -p 22 root#REMOVED_ID '/tmp/.saltcloud-5d18c002-e817-46d5-9fb2-d3bdb2dfe7fd/deploy.sh -c '"'"'/tmp/.saltcloud-5d18c002-e817-46d5-9fb2-d3bdb2dfe7fd'"'"' -P git v2016.3.3'' failed. Exit code: 1
[DEBUG ] LazyLoaded nested.output
minion-zk-0:
----------
Error:
Command 'ssh -t -t -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -oPasswordAuthentication=no -oChallengeResponseAuthentication=no -oPubkeyAuthentication=yes -oIdentitiesOnly=yes -oKbdInteractiveAuthentication=no -i /etc/salt/keys/cloud/do.pem -p 22 root#REMOVED_IP '/tmp/.saltcloud-5d18c002-e817-46d5-9fb2-d3bdb2dfe7fd/deploy.sh -c '"'"'/tmp/.saltcloud-5d18c002-e817-46d5-9fb2-d3bdb2dfe7fd'"'"' -P git v2016.3.3'' failed. Exit code: 1
root#master-zk:/etc/salt/cloud.maps.d# salt '*' test.ping
minion-zk-0:
Minion did not return. [No response]
root#master-zk:/etc/salt/cloud.maps.d#
It is located in your cloud configuration somewhere in /etc/salt/cloud.profiles.d/, /etc/salt/cloud.providers.d/ or /etc/salt/cloud.d/. Just figure out where and change the value salt to your masters ip.
I currently do this in my providers setting like that:
hit-vcenter:
driver: vmware
user: 'foo'
password: 'secret'
url: 'some url'
protocol: 'https'
port: 443
minion:
master: 10.1.10.1
Related
connect-script = /app/connect.sh
disconnect-script = /app/disconnect.sh
I have the above configuration in my ocserv.conf in the docker container, but ocserv fails to execute /app/connect.sh when there is a connection. I cann't find the real cause from the following log, has anyone had the same issue?
ocserv[26]: main[test]:xxx.xxx.179.135:57352 user of group 'Route' authenticated (using cookie)
ocserv[29]: main[test]:xxx.xxx.179.135:57352 executing script up /app/connect.sh
ocserv[29]: main[test]:xxx.xxx.179.135:57352 main-user.c:379: Could not execute script /app/connect.sh
ocserv[26]: main[test]:xxx.xxx.179.135:57352 connect-script exit status: 1
ocserv[26]: main[test]:xxx.xxx.179.135:57352 failed authentication attempt for user 'test'
The content of /app/connect.sh:
#!/bin/bash
echo "$(date) [info] User ${USERNAME} Connected - Server: ${IP_REAL_LOCAL} VPN IP: ${IP_REMOTE} Remote IP: ${IP_REAL} Device:${DEVICE}"
Well, I figured it out myself that the docker container I created doesn't have bash, and one solution is to substitute #!/bin/bash with #!/bin/sh.
Following the steps in the AWS documentation
https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-lambda.html
using aws-stepfuncitons-local docker container
I'm getting a connection refused error at the last step
2019-05-28 12:37:05.004: arn:aws:states:us-east-1:123456789012:execution:HelloWorld:test :
{
"Type":"ExecutionFailed",
"PreviousEventId":5,
"ExecutionFailedEventDetails":
{
"Error":"Lambda.SdkClientException",
"Cause":"Unable to execute HTTP request: Connect to 127.0.0.1:3001 [/127.0.0.1] failed: Connection refused (Connection refused)"
}
}
Any help on how to resolve it would be greatly appreciated.
The docker container can't talk to services on the host network. To get it to work you need to add '--network host'.
Example:
docker run -p 8083:8083 --network host --env-file aws-stepfunctions-local-credentials.txt amazon/aws-stepfunctions-local
More details here and here
Just update LAMBDA_ENDPOINT to http://host.docker.internal:3001
It took me a few days, so that's what works for me:
on my Mac: LAMBDA_ENDPOINT=http://host.docker.internal:5000/ (5000 for moto_server, use 3001 otherwise) works.
And use docker run -p 8083:8083 --env-file aws-stepfunctions-local-credentials.txt amazon/aws-stepfunctions-local
on ubuntu-latest (github action), however, this combination works:
LAMBDA_ENDPOINT=http://127.0.0.1:5000/
docker run -p 8083:8083 --env-file aws-stepfunctions-local-credentials.txt amazon/aws-stepfunctions-local --network host
=> note the additional --network host
To handle both cases inside the same codebase (in python):
_, output = subprocess.getstatusoutput('uname -s')
if output == 'Darwin': # on a Mac
sam_step_function_local_cmd = subprocess.Popen(
[
'docker', 'run',
'-p=8083:8083',
'--env-file=aws-stepfunctions-local-credentials-mac.txt',
'amazon/aws-stepfunctions-local:1.7.3'
],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
else: # on linux
sam_step_function_local_cmd = subprocess.Popen(
[
'docker', 'run',
'-p=8083:8083',
'--env-file=aws-stepfunctions-local-credentials-linux.txt',
'--network=host',
'amazon/aws-stepfunctions-local:1.7.3'
],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
try:
mock_ssm(
func
)(*args, **kwargs)
finally:
sam_step_function_local_cmd.terminate()
aws-stepfunctions-local-credentials-linux.txt:
AWS_DEFAULT_REGION=eu-west-1
LAMBDA_ENDPOINT=http://127.0.0.1:5000/
aws-stepfunctions-local-credentials-mac.txt:
AWS_DEFAULT_REGION=eu-west-1
LAMBDA_ENDPOINT=http://host.docker.internal:5000/
This works fine on ubuntu 16.04, but not on 17.10
+ curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M100 56093 100 56093 0 0 56093 0 0:00:01 --:--:-- 0:00:01 98929
+ chmod +x ./awslogs-agent-setup.py
+ ./awslogs-agent-setup.py -n -c /etc/awslogs/awslogs.conf -r us-west-2
Step 1 of 5: Installing pip ...^[[0mlibyaml-dev does not exist in system ^[[0m^[[92mDONE^[[0m
Step 2 of 5: Downloading the latest CloudWatch Logs agent bits ... ^[[0mTraceback (most recent call last):
File "./awslogs-agent-setup.py", line 1317, in <module>
main()
File "./awslogs-agent-setup.py", line 1313, in main
setup.setup_artifacts()
File "./awslogs-agent-setup.py", line 858, in setup_artifacts
self.install_awslogs_cli()
File "./awslogs-agent-setup.py", line 570, in install_awslogs_cli
subprocess.call([AWSCLI_CMD, 'configure', 'set', 'plugins.cwlogs', 'cwlogs'], env=DEFAULT_ENV)
File "/usr/lib/python2.7/subprocess.py", line 168, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1025, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I noticed that earlier on in the process, in the AWS boilerplate it failed to install libyaml-dev but not sure if that's the only problem.
Always find the answer right after I post it...
Here's my modified CF template command:
050_install_awslogs:
command: !Sub
"/bin/bash -x\n
exec >>/var/log/cf_050_install_awslogs.log 2>&1 \n
echo 050_install_awslogs...\n
set -xe\n
# Get the CloudWatch Logs agent\n
mkdir /opt/awslogs\n
cd /opt/awslogs\n
# Needed for python3 in 17.10\n
apt-get install -y libyaml-dev python-dev \n
pip3 install awscli-cwlogs\n
# avoid it complaining about not having /var/awslogs/bin/aws binary\n
if [ ! -d /var/awslogs/bin ] ; then\n
mkdir -p /var/awslogs/bin\n
ln -s /usr/local/bin/aws /var/awslogs/bin/aws\n
fi\n
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O\n
chmod +x ./awslogs-agent-setup.py\n
# Hack for python 3.6 & old awslogs-agent-setup.py\n
sed -i 's/3,6/3,7/' awslogs-agent-setup.py\n
./awslogs-agent-setup.py -n -c /etc/awslogs/awslogs.conf -r ${AWS::Region}\n
echo 050_install_awslogs end\n
"
Not entirely sure about the need for the dir creation but I expect this is a temporary case that will get resolved soon as one still needs to fudge the python 3.6 compatibility check.
it may be installable using python 2.7 as well, but that felt like going backwards at this point as the my rationale for 17.10 was python 3.6.
Credit for the yaml package and dir creation idea to https://forums.aws.amazon.com/thread.jspa?threadID=265977 but I prefer to avoid easy_install.
I had similar issue on Ubuntu 18.04.
Instruction from AWS for standalone install worked for my case.
To download and run it standalone, use the following commands and follow the prompts:
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/AgentDependencies.tar.gz -O
tar xvf AgentDependencies.tar.gz -C /tmp/
sudo python ./awslogs-agent-setup.py --region us-east-1 --dependency-path /tmp/AgentDependencies
After installing/configuring whenever-elasticbeanstalk gem, I'm seeing the following error in /var/log/cfn-init.log on my EC2 instance after running git aws.push from my local repo.
Iam using aws elastic benastalk with rails 4.
2014-10-21 08:08:37,602 [DEBUG] Running test for command cron_01_set_leader
2014-10-21 08:08:37,744 [DEBUG] Test command output:
2014-10-21 08:08:37,745 [DEBUG] Test for command cron_01_set_leader passed
2014-10-21 08:08:38,085 [ERROR] Command cron_01_set_leader (su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER) failed
2014-10-21 08:08:38,086 [DEBUG] Command cron_01_set_leader output: bash: /usr/local/bin/bundle: No such file or directory
Traceback (most recent call last):
I have added the whenever-elasticbeanstalk
Below is my cron.config file content..
Any idea ...what am i doing wrong?
files:
# Reload the on deployment
/opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh:
mode: "00700"
owner: root
group: root
content: |
#!/usr/bin/env bash
. /opt/elasticbeanstalk/containerfiles/envvars
cd $EB_CONFIG_APP_CURRENT
su -c "/usr/local/bin/bundle exec setup_cron" $EB_CONFIG_APP_USER
# Add Bundle to the PATH
"/etc/profile.d/bundle.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
export PATH=$PATH:/usr/local/bin
encoding: plain
container_commands:
cron_01_set_leader:
test: test ! -f /opt/elasticbeanstalk/containerfiles/.cron-setup-complete
leader_only: true
cwd: /var/app/ondeck
command: su -c "/usr/local/bin/bundle exec create_cron_leader --no-update" $EB_CONFIG_APP_USER
cron_02_write_cron_setup_complete_file:
cwd: /opt/elasticbeanstalk/containerfiles
command: touch .cron-setup-complete
Which solution stack are you using? Can you give the exact name, something like "64bit Amazon Linux 2014.03 v1.0.9 running Ruby 2.1 (Puma)".
I think you will need to replace "/usr/local/bin/bundle" with the actual version of bundle that is used for the solution stack.
Can you just try using "bundle" instaed of "/usr/local/bin/bundle"?
Running:
knife bootstrap ec2-54-221-16-158.compute-1.amazonaws.com --sudo -x chef -P chef -N server --run-list 'role[inicial]'
My recipes/default.rb:
script "teste de script" do
interpreter "bash"
cwd "/home/ubuntu"
code <<-EOH
as-create-launch-config LcTiagoN --image-id ami-0521316c --instance-type t1.micro --key tiagov
EOH
end
My roles/inicial.rb:
name "inicial"
run_list "recipe[my_cookbook]"
The following error occurs below:
ShellOut::ShellCommandFailed←[
0m
------------------------------------←[
0m
Expected process to exit with [0], but
received '127'
---- Begin output of "bash" "/tmp/che
f-script20140501-8463-12uvvvl" ----
STDOUT:
STDERR: /tmp/chef-script20140501-8463-
12uvvvl: line 1: as-create-launch-config: command not found
---- End output of "bash" "/tmp/chef-
script20140501-8463-12uvvvl" ----
However when I run the same command (as-create-launch-config LcTiagoN --image-id ami-0521316c --instance-type t1.micro --key tiagov) directly logged in the Amazon instance, the command is executed successfully.
Any suggestions?
Sounds like a problem with the PATH environment. Did you login as "chef" when running the as-create-launch-config command manually?
Best advice I can offer is to include the full path to the command in the bash script. For example:
script "teste de script" do
..
code <<-EOH
/path/to/this/cmd/as-create-launch-config ...
EOH
end