Below is what happened to one mail send from a drupal client.
$ grep 'B6693C0977' /var/log/maillog
Jan 19 14:12:30 instance-1 postfix/pickup[19329]: B6693C0977: uid=0 from=<admin#mailgun.domainA.com>
Jan 19 14:12:30 instance-1 postfix/cleanup[20035]: B6693C0977: message-id=<20170119141230.B6693C0977#mail.instance-1.c.tw-pilot.internal>
Jan 19 14:12:30 instance-1 postfix/qmgr[19330]: B6693C0977: from=<admin#mailgun.domainA.com>, size=5681, nrcpt=1 (queue active)
Jan 19 14:12:33 instance-1 postfix/smtp[20039]: B6693C0977:
to=<username#hotmail.com>, relay=smtp.mailgun.org[52.41.19.62]:2525, delay=2.4,
delays=0.02/0.05/1.8/0.53, dsn=5.7.1, status=bounced (host smtp.mailgun.org
[52.41.19.62] said: 550 5.7.1 **Relaying denied** (in reply to RCPT TO command))
Jan 19 14:12:33 instance-1 postfix/bounce[20050]: B6693C0977: sender non-delivery notification: ABB94C0976
Jan 19 14:12:33 instance-1 postfix/qmgr[19330]: B6693C0977: removed
Relevant excerpts from my /etc/postfix/main.cf are below
# RELAYHOST SETTINGS
smtp_tls_security_level = encrypt
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
sender_dependent_relayhost_maps = hash:/etc/postfix/relayhost_map
and from /etc/postfix/sasl_passwd is follows
#mailgun.domainA.com postmaster#mailgun.domainA.com:password
and from /etc/postfix/relayhost_map is follows
#mailgun.domainA.com [smtp.mailgun.org]:2525
The permissions of the db files are as follows
# ls -lZ /etc/postfix/relayhost_map.db
-rw-r-----. root postfix unconfined_u:object_r:postfix_etc_t:s0 /etc/postfix/relayhost_map.db
# ls -lZ /etc/postfix/sasl_passwd.db
-rw-r-----. root postfix unconfined_u:object_r:postfix_etc_t:s0 /etc/postfix/sasl_passwd.db
The problem is
Outbound mails are not going.
No logs are shown in mailgun console.
Any insight is appreciated
I know that this is an old question now but I've just had the same issue and wanted to post a response for anyone who comes across this article in future.
I believe your issue is in /etc/postfix/relayhost_map where you should have the following, note that there are no brackets, for me it was the inclusion of brackets that was causing the issue:
#mailgun.domainA.com smtp.mailgun.org:2525
For anyone who is not using /etc/postfix/relayhost_map and is doing it all in /etc/postfix/sasl_passwd directly the same applies:
smtp.mailgun.org:2525 postmaster#mailgun.domainA.com:password
Don't forget to regenerate the postfix sasl_passwd.db file and restart the service afterwards
sudo postmap /etc/postfix/sasl_passwd
sudo systemctl restart postfix
Or sudo service postfix restart if you're on an older system / not running systemd.
Usually this is realted to problems on their platform if everything was working ok previously just open a ticket and usually they fix it in a few hours (yes that its kind of hard a few hours)
Related
I created GCP VM instance and I can SSH.
miki#devsecops-cloud:~/kubernetes-devops-security/setup/jenkins-plugins$ ls -la
total 16
drwxrwxr-x 2 miki miki 4096 Sep 20 10:36 .
drwxrwxr-x 6 miki miki 4096 Sep 20 10:36 ..
-rw-rw-r-- 1 miki miki 1100 Sep 20 10:36 plugin-installer.sh
-rw-rw-r-- 1 miki miki 171 Sep 20 10:36 plugins.txt
Now the next step is to install Jenkins plugins from bash script.
Well it does not work at all.
I go this error
sudo bash plugin-installer.sh
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
http://localhost:8080
........Installing performance#3.18 ..
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Unauthorized</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized</h2>
<table>
<tr><th>URI:</th><td>/pluginManager/installNecessaryPlugins</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized</td></tr>
<tr><th>SERVLET:</th><td>Stapler</td></tr>
</table>
<hr/>Powered by Jetty:// 10.0.11<hr/>
may be problem is that I pointed to credentials at host machine
export GOOGLE_APPLICATION_CREDENTIALS=/home/miki/.config/gcloud/application_default_credentials.json
On the host,account list
gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* johnholmes#gmail.com
To set the active account, run:
$ gcloud config set account `ACCOUNT
On the VM
gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* 931909916149-compute#developer.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
On VM I tried to login
gcloud auth login
Your credentials may be visible to others with access to this
virtual machine. Are you sure you want to authenticate with
your personal account?
Do you want to continue (Y/n)? y
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32312940559.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=emNPFQkW7AHLJG97s6QGUqRSvvpCVV&prompt=consent&access_type=offline&code_challenge=LtH3n7LWTG8SZfOBT8jKZTrkb-fWECYekLVKNn2REBo&code_challenge_method=S256
Enter authorization code: 4/0ARtbsJrLZR41bqLkT2--0J9vOllonBoA2NU2l7NcI7nMGD0nbjFiRFSRkj2Cr7mjlADhHw
You are now logged in as [johnholmes#gmail.com].
But same problems are still here,401 error.
How to setup a proper credentials on VM machine?
When trying to SSH to GCE VMs using metadata-based SSH keys I get the following error:
ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
While troubleshooting I can see the keys in the instance metadata, but they are not being added to the user's authorized_keys file:
$ curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ssh-keys"
username:ssh-ed25519 AAAAC3NzaC....omitted....
admin:ssh-ed25519 AAAAC3NzaC....omitted....
$ sudo ls -hal /home/**/.ssh/
/home/ubuntu/.ssh/:
total 8.0K
drwx------ 2 ubuntu ubuntu 4.0K Aug 11 23:19 .
drwxr-xr-x 3 ubuntu ubuntu 4.0K Aug 11 23:19 ..
-rw------- 1 ubuntu ubuntu 0 Aug 11 23:19 authorized_keys
# Only result is the default zero-length file for ubuntu user
I also see the following errors in the ssh server auth log and Google Guest Environment services:
$ sudo less /var/log/auth.log
Aug 11 23:28:59 test-vm sshd[2197]: Invalid user admin from 1.2.3.4 port 34570
Aug 11 23:28:59 test-vm sshd[2197]: Connection closed by invalid user admin 1.2.3.4 port 34570 [preauth]
$ sudo journalctl -u google-guest-agent.service
Aug 11 22:24:42 test-vm oslogin_cache_refresh[907]: Refreshing passwd entry cache
Aug 11 22:24:42 test-vm oslogin_cache_refresh[907]: Refreshing group entry cache
Aug 11 22:24:42 test-vm oslogin_cache_refresh[907]: Failure getting groups, quitting
Aug 11 22:24:42 test-vm oslogin_cache_refresh[907]: Failed to get groups, not updating group cache file, removing /etc/oslogin_group.cache.bak.
# or
Aug 11 23:19:37 test-vm GCEGuestAgent[766]: 2022-08-11T23:19:37.6541Z GCEGuestAgent Info: Creating user admin.
Aug 11 23:19:37 test-vm useradd[885]: failed adding user 'admin', data deleted
Aug 11 23:19:37 test-vm GCEGuestAgent[766]: 2022-08-11T23:19:37.6869Z GCEGuestAgent Error non_windows_accounts.go:144:
Error creating user: useradd: group admin exists - if you want to add this user to that group, use -g.
Currently the latest cloud-init and guest-oslogin packages for Ubuntu 20.04.4 LTS (focal) seem to have an issue that causes google-guest-agent.service to exit before completing its task. The issue was fixed and committed but not yet released for focal (and likely other Ubuntu versions).
For now you can try disabling OS Login by setting instance or project metadata enable-oslogin=FALSE. After which you should see the expected results and be able to SSH using those keys:
$ sudo journalctl -u google-guest-agent.service
Aug 11 23:10:33 test-vm GCEGuestAgent[761]: 2022-08-11T23:10:33.0517Z GCEGuestAgent Info: Created google sudoers file
Aug 11 23:10:33 test-vm GCEGuestAgent[761]: 2022-08-11T23:10:33.0522Z GCEGuestAgent Info: Creating user username.
Aug 11 23:10:33 test-vm useradd[881]: new group: name=username, GID=1002
Aug 11 23:10:33 test-vm useradd[881]: new user: name=username, UID=1001, GID=1002, home=/home/username, shell=/bin/bash, from=none
Aug 11 23:10:33 test-vm gpasswd[895]: user username added by root to group ubuntu
Aug 11 23:10:33 test-vm gpasswd[904]: user username added by root to group adm
Aug 11 23:10:33 test-vm gpasswd[983]: user username added by root to group google-sudoers
Aug 11 23:10:33 test-vm GCEGuestAgent[761]: 2022-08-11T23:10:33.7615Z GCEGuestAgent Info: Updating keys for user username.
$ sudo ls -hal /home/username/.ssh/
/home/username/.ssh/:
total 12K
drwx------ 2 username username 4.0K Aug 11 23:19 .
drwxr-xr-x 4 username username 4.0K Aug 11 23:35 ..
-rw------- 1 username username 589 Aug 11 23:19 authorized_keys
The admin user however will not work, since it conflicts with an existing linux group. You should pick a username that does not conflict with any of the name:x:123: names listed at getent group
Hi i have been trying to do cpu pinning in my eks cluster. i have used amazon linux latest release, and my eks version is 1.22 . i have created a launch template where i have used this user data mentioned below.
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh $CLUSTER_NAME
sleep 2m
yum update -y
sudo rm /var/lib/kubelet/cpu_manager_state
sudo chmod 777 kubelet.service
sudo cat > /etc/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service iptables-restore.service
Requires=docker.service
[Service]
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT -w 5
ExecStart=/usr/bin/kubelet --cloud-provider aws \
--image-credential-provider-config /etc/eks/ecr-credential-provider/ecr-
credential-provider-config \
--image-credential-provider-bin-dir /etc/eks/ecr-credential-provider \
--cpu-manager-policy=static \
--kube-reserved=cpu=0.5,memory=1Gi,ephemeral-storage=0.5Gi \
--system-reserved=cpu=0.5,memory=1Gi,ephemeral-storage=0.5Gi \
--config /etc/kubernetes/kubelet/kubelet-config.json \
--kubeconfig /var/lib/kubelet/kubeconfig \
--container-runtime docker \
--network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS
Restart=always
RestartSec=5
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
sudo chmod 644 kubelet.service
sudo systemctl daemon-reload
sudo systemctl stop kubelet
sudo systemctl start kubelet
--//
after creating the template i have used it on the eks nodegroup creation. after waititng a while i am getting this error on the eks dashboard.
Health issues (1)
NodeCreationFailure Instances failed to join the kubernetes cluster .
and i have get into that ec2 instance and used the following command to view kubectl logs
$journalctl -f -u kubelet
the output is
[ec2-user#ip-10.100.11.111 kubelet]$ journalctl -f -u kubelet
-- Logs begin at Thu 2022-04-21 07:27:50 UTC. --
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.199868 12225 state_mem.go:80] "Updated desired CPUSet" podUID="3b513cfa-
441d-4e25-9441-093b4c2ed548" containerName="efs-plugin" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.244811 12225 state_mem.go:80] "Updated desired CPUSet" podUID="3b513cfa-
441d-4e25-9441-093b4c2ed548" containerName="csi-provisioner" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.305206 12225 state_mem.go:80] "Updated desired CPUSet" podUID="3b513cfa-
441d-4e25-9441-093b4c2ed548" containerName="liveness-probe" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.335744 12225 state_mem.go:80] "Updated desired CPUSet" podUID="de537700-
f5ac-4039-a151-110ddf27d140" containerName="efs-plugin" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.388843 12225 state_mem.go:80] "Updated desired CPUSet" podUID="de537700-
f5ac-4039-a151-110ddf27d140" containerName="csi-driver-registrar" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.464789 12225 state_mem.go:80] "Updated desired CPUSet" podUID="de537700-
f5ac-4039-a151-110ddf27d140" containerName="liveness-probe" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.545206 12225 state_mem.go:80] "Updated desired CPUSet" podUID="a2f09d0d-
69f5-4bb7-82bb-edfa86cb87e2" containerName="kube-controller" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.633078 12225 state_mem.go:80] "Updated desired CPUSet" podUID="3ec70fe1-
3680-4e3c-bcfa-81f80ebe20b0" containerName="kube-proxy" cpuSet="0-7"
Apr 21 07:31:21 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: I0421
07:31:21.696852 12225 state_mem.go:80] "Updated desired CPUSet" podUID="adbd9bef-
c4e0-4bd1-a6a6-52530ad4bea3" containerName="aws-node" cpuSet="0-7"
Apr 21 07:46:12 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: E0421
07:46:12.424801 12225 certificate_manager.go:488] kubernetes.io/kubelet-serving:
certificate request was not signed: timed out waiting for the condition
Apr 21 08:01:16 ip-10.100.11.111.us-west-2.compute.internal kubelet[12225]: E0421
08:01:16.810385 12225 certificate_manager.go:488] kubernetes.io/kubelet-serving:
certificate request was not signed: timed out waiting for the condition
this was the output..
But before using this method i have also tried another method, where i have created a node group and then i have created an ami from one of the nodes in that nodegroup.. then modified the kubelet.service file and removed the old cpu_manager_state file.. then the i have used this image to create the nodegroup. Then it worked fine But the problem was i am unable to get into the pods running in those nodes and also i am unable to get the logs of the pods running there. and strangely if i use
$kubectl get nodes -o wide
in the output i was not getting the internal and external both ip addresses.
so i moved on to using the userdata instead of this method.
kindly give me instructions to create a managed nodegroup with cpu_manager_state as static policy for eks cluster .
I had the same question. I added the following userdata script to my launch template
User data script
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
yum install -y jq
set -o xtrace
cp /etc/kubernetes/kubelet/kubelet-config.json /etc/kubernetes/kubelet/kubelet-config.json.back
jq '. += { "cpuManagerPolicy":"static"}' /etc/kubernetes/kubelet/kubelet-config.json.back > /etc/kubernetes/kubelet/kubelet-config.json
--==MYBOUNDARY==--
Verification
You can verify the change took effect using kubectl:
# start a k8s API proxy
$ kubectl proxy
# get the node name
$ kubectl get nodes
# get kubelet config
$ curl -sSL "http://localhost:8001/api/v1/nodes/<<node_name>>/proxy/configz"
I got the solution from this guide: https://aws.amazon.com/premiumsupport/knowledge-center/eks-worker-nodes-image-cache/. However, I could not make the sed command properly work so I used jq instead.
Logs
If you can ssh into the node, you can check the userdata logs in /var/log/cloud-init-output.log - See https://stackoverflow.com/a/32460849/4400704
CPU pinning
I have a pod with a status QoS Guarantee (CPU limit and requested = 2) and I can verify it has two CPU reserved
$ cat /sys/fs/cgroup/cpuset/cpuset.cpus
2,10
I experience a strange issue where my data is in UTC and the data picker selects time by a large offset. I'd like the UI to select UTC time only.
In the documentation it states Superset is build to run on UTC time only. I also found some threads one can change this by setting the Linux environment via ENV variable to other timezone:
ENV TZ Europe/Amsterdam
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
Wrong behaviour (UTC is approx 17h December 22nd):
(same behaviour in all dashboards)
Query behind this data (I used "view query in SQLLab")
SELECT toDateTime(intDiv(toUInt32(toDateTime(delivery_date)), 300)*300) AS __timestamp,
COUNT(*) AS count
FROM spearad_data.a_performance_view
WHERE delivery_date >= toDateTime('2020-12-21 23:00:00')
AND delivery_date < toDateTime('2020-12-22 00:00:00')
GROUP BY toDateTime(intDiv(toUInt32(toDateTime(delivery_date)), 300)*300)
ORDER BY count DESC
LIMIT 10000;
In My case I did check the ECS docker container and the EC2 instance the task (container) runs on.
EC2 machine:
[ec_user#1.2.3.4]$date
Tue Dec 22 16:31:07 UTC 2020
[ec2-user#1.2.3.4]$ echo $TZ
[ec2-user#ip-1-2-3-4]$ date +'%:z %Z'
+00:00 UTC
[ec2-user#ip-1-2-3-4]$ cat /etc/timezone
cat: /etc/timezone: No such file or directory
[ec2-user#ip-1-2-3-4]$ cat /etc/timezone
cat: /etc/timezone: No such file or directory
[ec2-user#ip-1-2-3-4]$ timedatectl
Local time: Tue 2020-12-22 16:40:47 UTC
Universal time: Tue 2020-12-22 16:40:47 UTC
RTC time: Tue 2020-12-22 16:40:42
Time zone: n/a (UTC, +0000)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
ECS container:
/ # date
Tue Dec 22 16:43:53 UTC 2020
/ # echo $TZ
/ # date +'%:z %Z'
/ # cat /etc/timezone
cat: can't open '/etc/timezone': No such file or directory
Clickhouse DB:
SELECT now();
2020-12-22T16:50:32
Superset MySQL (AWS RDS):
SELECT now();
2020-12-22T16:52:27
https://time.is/de/UTC
16:57:01
Save a query in SQLLab
created_on
2020-12-22T16:59:03
Data is UTC based as well. So where do I need to change this? It seems there's another setting or configuration missing.
I have the following setup:
Apache Superset 0.37 on AWS ECS
Superset ConfigDB: AWS RDS
Fact DB: Clickhouse DB 20.7
Driver: clickhouse-sqlalchemy (native mode)
I've been trying to change the ec2 instance timezone to IST but following the aws docs isn't helping at all.
ls /usr/share/zoneinfo/Asia
Aden Atyrau Brunei Damascus Hebron Jerusalem Kolkata Makassar Phnom_Penh Saigon Tashkent Ujung_Pandang Yangon
Almaty Baghdad Calcutta Dhaka Ho_Chi_Minh Kabul Krasnoyarsk Manila Pontianak Sakhalin Tbilisi Ulaanbaatar Yekaterinburg
Amman Bahrain Chita Dili Hong_Kong Kamchatka Kuala_Lumpur Muscat Pyongyang Samarkand Tehran Ulan_Bator Yerevan
Anadyr Baku Choibalsan Dubai Hovd Karachi Kuching Nicosia Qatar Seoul Tel_Aviv Urumqi
Aqtau Bangkok Chongqing Dushanbe Irkutsk Kashgar Kuwait Novokuznetsk Qostanay Shanghai Thimbu Ust-Nera
Aqtobe Barnaul Chungking Famagusta Istanbul Kathmandu Macao Novosibirsk Qyzylorda Singapore Thimphu Vientiane
Ashgabat Beirut Colombo Gaza Jakarta Katmandu Macau Omsk Rangoon Srednekolymsk Tokyo Vladivostok
Ashkhabad Bishkek Dacca Harbin Jayapura Khandyga Magadan Oral Riyadh Taipei Tomsk Yakutsk
sudo vi /etc/sysconfig/clock
ZONE="Asia/Calcutta"
UTC=true
I edited the file to the required timezone and linked it to local time
sudo ln -sf /usr/share/zoneinfo/Asia/Calcutta /etc/localtime
Rebooted the machine and check date only to see the below
Mon Sep 16 16:06:13 UTC 2019
Did this a few times with also changing the Zone to Kolkata, nothing changes. Any suggestions would be really helpful.
The simplest way to change EC2 timezone is to run the following command when logged in
$ sudo dpkg-reconfigure tzdata
this will open screen to select geographical areas, use enter to select and further select city and enter.
This will change the timezone for your currently logged in EC2 instance.
for instance setting for Europe Amsterdam timezone. (when run as root)
ln -sf /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime
echo -e "ZONE="Europe/Amsterdam"\nUTC=true”>/etc/sysconfig/clock
reboot
this worked for me , but it requires a reboot.