From today only I started getting issue while running deployment scripts from my local VM. I am not sure if it's a known issue or some setup related thing which is missing on my VM .
Authentication failed while running deploy_one_off:
[vvaibhav#gld2vm40 debesys (topic/Add_FIX_IBDC_Session_Subtype_info_to_pub-DEB-107734)]$ /opt/virtualenv/devws/bin/python2 deploy/chef/scripts/deploy_one_off.py -s gla2vm178 --email vagesh.vaibhav#trade.tt -c dropcopyclientnode -r 5fff2fe00e5082b39fd5a978af7bf38770a95ef9 --request-build --run-chef --override-oneoff
Enter your INTAD username:vvaibhav
Enter password for INTAD user vvaibhav:
Checking if dropcopyclientnode cookbook has a build target...DONE
Oops: Failed to authenticate and retrieve AWS Keys from AWS, the package check will
fail. It is safe to try re-running, but if the problem persists please notify
Deployment and/or ELS. Exception:
Just FYI,
This issue was resolved as I worked with Tom today morning. It seems that some aws-keys file became stale and we had to delete those files.
$ ls -ltr ~/.aws-keys*
-rw-r--r-- 1 vvaibhav sysadmins 0 Jan 7 07:42 /home/vvaibhav/.aws-keys_deb_read
Related
the company i'm working with is developing a web application based on Laravel Framework, using Google Cloud Platform infrastructures. The frontend VM is a Centos8 OS with Apache webserver installed.
Seems that a developer ran a pretty massive "dnf upgrade" which included: kernel, openssl ,kerberos and others packages.
After the upgrade, seems that ldconfig has lost his mind:
[developer#webserver ~]$ sudo su - root
sudo: error in /etc/sudo.conf, line 19 while loading plugin "sudoers_policy"
sudo: unable to load /usr/libexec/sudo/sudoers.so: /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
sudo: fatal error, unable to load plugins
same for other commands like "dnf" or "rpm":
[developer#webserver ~]$ rpm
rpm: symbol lookup error: /lib64/librpmio.so.8: undefined symbol: EVP_md2, version OPENSSL_1_1_0
after a little bit of investigations, i found that the same commands, specifing the LD_LIBRARY_PATH variable, are working:
[developer#webserver ~]$ LD_LIBRARY_PATH=/lib64 rpm
RPM version 4.14.3
Copyright (C) 1998-2002 - Red Hat, Inc.
This program may be freely redistributed under the terms of the GNU GPL
...
...of course, i can't do the same trick with "sudo" command.
Important fact is that the VM is still running and it was never rebooted ( i'll exaplin later why i'm sayin this )
( and finally..at the point )
The major problem is that we can't use root account cause "sudo" is not working and, by default, Google use Public Key Authentication as deafult method (Local users has random passwords genereated by GCP). So actually, i can't even do a "dnf reinstall" to try fix the issues
I was afraid that, once rebooted, every services stops to work because of the incorrect dependecies library path, so instead of doing a reboot, i have created an image based on the VM and then a new VM based on that image.
As i was thinking: Once booted the new VM, every services stopped working. i was able to read the logs over the serial console of GCP web interface.
a snippet:
...
Oct 27 20:20:30 webserver google_oslogin_nss_cache[783]: /usr/bin/google_oslogin_nss_cache: /lib64/libjson-c.so.4: no version information available (required by /usr/bin/google_oslogin_nss_cache)
Oct 27 20:20:30 webserver NetworkManager[778]: /usr/sbin/NetworkManager: symbol lookup error: /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
Oct 27 20:20:30 webserver google_oslogin_nss_cache[783]: /usr/bin/google_oslogin_nss_cache: symbol lookup error: /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
Oct 27 20:20:30 webserver sssd[771]: ldb: unable to dlopen /usr/lib64/ldb/modules/ldb/ldap.so : /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
...
Using Google official documentation, i found the "startup-script" section of the VM properties that can be launched at every boot and that can be used to "change" user's passwords.
I know that, by default, all VMs has root access disabled, so i made this and added to vm's "automation" script:
#! /bin/bash
echo 'developer:PASSWORD' | chpasswd
echo 'root:PASSWORD' | chpasswd
Once rebooted, i've tried to login using the "serial console" option on the web interface, but with no luck. I've also used journalctl ( as normal user ) to find something in the logs... but nothing.
i suppose that is a consequence of that "google_oslogin_nss_cache" error
there's no way to run that script.
Searching on the internet, i've found some posts where someone was able to login directly as "root" using the "gcloud compute ssh" command. So i have tried to login as described using another VM of the same project, using both my google account user and root user...but also in this way ...no luck.
( i forgot to mention that my google account has "project owner" role, so actually i have all necessary permissions )
is there another way to reset "root" password without using "sudo" or i have to reinstall the VM from start?
I'm sorry for the long explanation....hope that everything is clear
Thanks
So... actually this question is divided by 2 different issues:
The only possible way for me to recover "root" account was to stop the VM, detach the boot disk, mount the boot disk on a new VM, mount the filesystem and modify the user. once boot disk is reattached to the original VM..you can use the modified account
second issue was made by upgrading openssl...so in the end the only way to avoid that error messages was to create a new file: /etc/ld.so.conf.d/libc.conf:
/usr/lib64
I've had a remote dashboard running fine for a couple of years (written for me by an external developer). It runs on an EC2 instance and is configured using OpsWorks.
Today it's not working, and I see in OpsWorks that the instance is showing as setup_failed.
According to the logs it fails here:
[2021-07-02T15:00:59+00:00] FATAL: Stacktrace dumped to /var/chef/runs/18bc4301-71c1-4393-bb26-eae958791d5a/local-mode-cache/cache/chef-stacktrace.out
[2021-07-02T15:00:59+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-07-02T15:00:59+00:00] ERROR: deploy_branch[/srv/api] (iparcelbox::deploy-api line 45) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
---- Begin output of git fetch origin ----
STDOUT:
STDERR: error: cannot open .git/FETCH_HEAD: Permission denied
---- End output of git fetch origin ----
Ran git fetch origin returned 255
I've checked the recipe file for iparcelbox::deploy-api and line 45 calls a deploy_branch:
deploy_branch server_path do
user userName
group groupName
repository node[:iparcelbox][:git_url]
revision node[:iparcelbox][:revision]
enable_submodules false
migrate false
shallow_clone true
git_ssh_wrapper "/tmp/api_git_wrapper.sh"
rollback_on_error false
keep_releases 5
symlink_before_migrate.clear
purge_before_symlink purge_dirs
create_dirs_before_symlink []
symlinks({})
action :deploy
end
So as I understand it, the deploy_branch is trying to fetch a git repo, and for some reason it's failing? I've checked my GitHub repository for the source files and I can see an ssh 'Deploy Key' which is showing as used within the last week.
If anyone could give me any suggestions as to what else to try, it would be much appreciated!
I found an answer to this - I thought the issue was permission denied accessing the git repository, but actually it was because the destination folder on my instance had modified ownership. Setting the ownership back to that specified in the Chef recipe using chown allowed the setup to complete successfully.
I am trying to install the redmine Git Hosting plugin for redmine 3.3.2.
I have followed the steps at (http://redmine-git-hosting.io/get_started/) but when i try the 9th step, which is to login to the git, I get asked for a password. There is no password on the GIT user, and just entering a blank password also doesn't work...
I can confirm that there's no password on the user:
[root#RedmineZonderDocker2 ~]# passwd -S git
git NP 2018-12-16 0 99999 7 -1 (Empty password.)
The log of the plugin does state this:
2018-12-16 20:16:47 +0000 [INFO] Testing if Redmine user 'redmine' can sudo to Gitolite user 'git'...
2018-12-16 20:16:47 +0000 [INFO] OK!
2018-12-16 20:16:47 +0000 [ERROR] Error while getting Gitolite infos, check
your SSH keys (path, permissions) or your Git user.
I clearly have no idea where the problem lays, and as you might notice, i'm quite new to all this. Any help would be appreciated.
If you need other config files or logs, please let me know and i'll gladly provide them.
Kind regards
Make sure you have completed all the settings described in step 9 and restarting Redmine after changing said settings, as some of them are put in cache.
The password is only asked if the SSH key is not found or not properly register. In both instances, double-check "Gitolite - ssh troubleshooting and tips".
I found the problem, and the plugin is functional now.
I am migrating a Django application from Openshift v2 to v3 (In case you don't know, RedHat is shutting down v2 on September 30th, see: https://blog.openshift.com/migrate-to-v3-v2-eol/)
So, I am following this blog post to help me: https://blog.openshift.com/migrating-django-applications-openshift-3/ . I am new to all these Docker / Kubernetes concepts the new version is build upon.
I was able to make some progress : I managed to get a successful build of my app. Yet it crashes at deployment time:
---> Running application from script (app.sh) ...
/usr/libexec/s2i/run: line 42: /opt/app-root/src/app.sh: Permission denied
Indeed, app.sh has lost its x permission. I log into the failing container as debug and see it:
> oc debug dc/<my app>
> (app-root)sh-4.2$ ls -l /opt/app-root/src/app.sh
-rw-rw-r--. 1 default root 127 Sep 6 21:20 /opt/app-root/src/app.sh
The blog posts states "Ensure that the app.sh file is executable by running chmod +x app.sh.", which I did on my local repo. Whatever, I want to do it again directly in the pod, but it doesn't work:
(app-root)sh-4.2$ chmod +x /opt/app-root/src/app.sh
chmod: changing permissions of ‘/opt/app-root/src/app.sh’: Operation not permitted
So, how can I set the x permission to app.sh ? Thank you
Without looking into more details, any S2I builder image will gladly use your custom supplied run script to start the application in an alternative way.
Create .s2i/bin/ (mind the dot) in your source code directory, place the run script into it and rebuild the app in OpenShift - it will automatically use your custom run script upon deployment.
This is the preferred way of starting applications using custom commands in OpenShift.
Regarding your immediate problem, there is a very simple reason why you can not change the permissions of the script: you were trying to modify the permissions in the deployed pod, and not the builder pod. Deployed pods run using different UIDs, usually somewhere in the range of 100000000, and definitely do not match the file ownership as generated by the build. Hence permission denied.
The root cause of your problem (app.sh losing executable permissions) must be in the way the build process installs those files, and indeed looking at the /usr/libexec/s2i/assemble script in the base image does seem to reveal the culprit. The last two lines are:
# set permissions for any installed artifacts
fix-permissions /opt/app-root
If you wanted to change this part of the build instead of using a custom run script, I suggest you then create .s2i/bin/assemble in your project's source code and make it look sort of like this:
#!/bin/bash
echo "Running stock build:"
${STI_SCRIPTS_PATH}/assemble
echo "Fixing the mess:"
chmod 755 /opt/app-root/src/app.sh
This will fix whatever the stock build process does to file permissions, and will do it using the same UID as the rest of the build, so file ownership shouldn't be an issue.
as I stumbled upon this issue myself I've found a way to resolve it.
You have to make your file app.sh executable and push it in your repo as such.
If git does not track this modification as it did for me, you have to use: git update-index --chmod=+x app.sh for it to work.
I have a rails 4 openshift application. I am trying to run a cron job. The script runs completely fine when I run it by itself. The script is:
#!/bin/bash
/bin/bash -l -c 'cd $OPENSHIFT_REPO_DIR && bundle exec bin/rails runner -e production "Payment.charge_customers_pay_experts"'
The problem is the log file gives me the following error
Wed Feb 3 22:57:05 EST 2016: START minutely cron run
__________________________________________________________________________
/var/lib/openshift/56a438107628e18b30000111/app-root/runtime/repo//.openshift/åcron/minutely/charge_customers_pay_experts:
Warning: You're using Rubygems 2.0.14 with Spring. Upgrade to at least Rubygems 2.1.0 and run `gem pristine --all` for better startup performance.
/var/lib/openshift/56a438107628e18b30000111/app-root/runtime/repo/vendor/bundle/ruby/gems/spring-1.6.2/lib/spring/sid.rb:39:in `getpgid': Permission denied (Errno::EACCES)
from /var/lib/openshift/56a438107628e18b30000111/app-root/runtime/repo/vendor/bundle/ruby/gems/spring-1.6.2/lib/spring/sid.rb:39:in `pgid'
from /var/lib/openshift/56a438107628e18b30000111/app-root/runtime/repo/vendor/bundle/ruby/gems/spring-1.6.2/lib/spring/server.rb:78:in `set_pgid'
from /var/lib/openshift/56a438107628e18b30000111/app-root/runtime/repo/vendor/bundle/ruby/gems/spring-1.6.2/lib/spring/server.rb:34:in `boot'
from /var/lib/openshift/56a438107628e18b30000111/app-root/runtime/repo/vendor/bundle/ruby/gems/spring-1.6.2/lib/spring/server.rb:14:in `boot'
from -e:1:in `<main>'
__________________________________________________________________________
Wed Feb 3 22:57:06 EST 2016: END minutely cron run - status=0
__________________________________________________________________________
I have made sure the script was executable. I'm not sure if I am missing something. Does anyone have any thoughts?
I don't know that the script being executable necessarily has anything to do with this. It looks like a permissions error more than anything. Does the system user that runs the cron job have the correct permissions to run? You can test this by logging into that user (or sudo su - <user>) and then execute the command in the script manually.
/bin/bash -l -c 'cd $OPENSHIFT_REPO_DIR && bundle exec bin/rails runner -e production "Payment.charge_customers_pay_experts"'
Be sure to replace your $OPENSHIFT_REPO_DIR variable with the correct path to your OpenShift repo directory.
You may just need to either add the user your cronjob runs as to the group that has permissions over the files, or perhaps run the cronjob as a more privileged user (privileged in that it has permissions over the required files).
BTW, I could only post this as an answer as Stack Overflow is telling me I need 50 reputation points to comment.
I fixed this by commenting out the 'spring' gem in my gemfile. But apparently this is a known issue. https://bugzilla.redhat.com/show_bug.cgi?id=1305544.
There is a workaround for the time being until this issue is resolved. You can edit the /usr/libexec/openshift/cartridges/cron/bin/cron_runjobs.sh to add setsid in front of timeout so that it runs setsid timeout ... as this allows for the timeout command to actually change the sid.