I launch an EC2 instance with security group that have an inbound SSH through port 22 from 0.0.0.0/0. In my local machine I can connect to this instance by using its public DNS like that:
ssh ec2-user#ec2-54-144-XX-XXX.compute-1.amazonaws.com -i KeyPair.pem
However, I cannot reach the same instance by using its public ip address
ssh -v ec2-user#54.144.XX.XXX -i KeyPair.pem
OpenSSH_8.0p1 Ubuntu-6ubuntu0.1, OpenSSL 1.1.1c 28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 54.144.XX.XXX [54.144.XX.XXX] port 22.
debug1: connect to address 54.144.XX.XXX port 22: Network is unreachable
ssh: connect to host 54.144.XX.XXX port 22: Network is unreachable
This is also my /etc/ssh/ssh_config file:
Host *
# ForwardAgent no
# ForwardX11 no
# ForwardX11Trusted yes
PasswordAuthentication yes
# HostbasedAuthentication no
# GSSAPIAuthentication no
# GSSAPIDelegateCredentials no
# GSSAPIKeyExchange no
# GSSAPITrustDNS no
# BatchMode no
# CheckHostIP yes
# AddressFamily any
# ConnectTimeout 0
# StrictHostKeyChecking ask
# IdentityFile ~/.ssh/id_rsa
# IdentityFile ~/.ssh/id_dsa
# IdentityFile ~/.ssh/id_ecdsa
# IdentityFile ~/.ssh/id_ed25519
# Port 22
# Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
# MACs hmac-md5,hmac-sha1,umac-64#openssh.com
# EscapeChar ~
# Tunnel no
# TunnelDevice any:any
# PermitLocalCommand no
# VisualHostKey no
# ProxyCommand ssh -q -W %h:%p gateway.example.com
# RekeyLimit 1G 1h
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
Related
I have a three node patroni postgres-11 cluster running.The patroni cluster settings are:
consul:
url: <consul URL>
bootstrap:
initdb:
encoding: UTF8
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
use_slots: true
remove_data_directory_on_diverged_timelines: false
remove_data_directory_on_rewind_failure: true
parameters:
archive_command: /bin/true
archive_mode: 'on'
archive_timeout: 1800s
checkpoint_completion_target: 0.9
default_statistics_target: 100
effective_cache_size: 24GB
effective_io_concurrency: 200
hot_standby: 'on'
log_directory: /var/log/postgresql
log_filename: postgresql-%a.log
logging_collector: 'on'
maintenance_work_mem: 2GB
max_connections: 2000
max_parallel_maintenance_workers: 4
max_parallel_workers: 8
max_parallel_workers_per_gather: 4
max_replication_slots: 10
max_wal_senders: 10
max_wal_size: 8GB
max_worker_processes: 8
min_wal_size: 2GB
random_page_cost: 1.1
shared_buffers: 8GB
ssl: 'on'
ssl_cert_file: /etc/ssl/certs/ssl-cert-snakeoil.pem
ssl_key_file: /etc/ssl/private/ssl-cert-snakeoil.key
track_commit_timestamp: 'on'
wal_buffers: 16MB
wal_keep_segments: 64
wal_level: logical
wal_log_hints: 'on'
work_mem: 5242kB
postgresql:
listen: 0.0.0.0:5432
connect_address: <IP Address of the node>:5432
data_dir: /var/lib/postgresql/11/<db name>
pgpass: /tmp/pgpass0
authentication:
superuser:
username: postgres
password: postgres
replication:
username: replicator
password: replicator
rewind:
username: rewind_user
password: rewind_password
pg_hba:
- local all all peer
- host all all 127.0.0.1/32 md5
- hostssl all postgres 0.0.0.0/0 md5
- host all postgres 0.0.0.0/0 reject
- host all all 0.0.0.0/0 md5
- hostssl all postgres ::0/0 md5
- host all postgres ::0/0 reject
- host all all ::0/0 md5
- local replication all peer
- host replication all 127.0.0.1/32 md5
- hostssl replication replicator 0.0.0.0/0 md5
- hostssl replication replicator ::0/0 md5
- host replication replicator 0.0.0.0/0 md5
- host replication replicator 127.0.0.1/32 trust
- local postgres postgres peer
- host all all 0.0.0.0/0 md5
create_replica_methods:
- basebackup
basebackup:
max-rate: '250M'
tags:
nofailover: false
noloadbalance: false
clonefrom: false
nosync: false
restapi:
connect_address: <IP Address of the node>:8008
listen: 0.0.0.0:8008
When I shutdown the primary, I was hoping the synchronous standby would become primary. But instead on both the standby nodes I see the following error:
INFO: following a different leader because i am not the healthiest node
Not sure why synchronous standby did not become primary.
I am using Patroni 1.6.1 on Ubuntu 18.04
I figured out the problem. For some reason the capital letters in the names of my nodes were converted to small letters so patroni failed to find the standby nodes. Like below:
{"leader":"Pa-db1","sync_standby":"pa-db2"}
The correct name of the node is Pa-db2 and similarly the second standby node name is Pa-db3. I changed all the nodes names to small letters i.e. pa-db1, pa-db2, pa-db3 and failover worked!
when I create amazon ubuntu instance from amazon web console and tries to log in to that instance using ssh from any remote computer I am able to log in but when I create ec2 instance using ansible aws.yml file and tries to do the same, I am unable to connect and got an error Permission denied (publickey) from every remote host except from that host in which I ran ansible script. Am I doing something wrong in my ansible file
Here is my ansiblle yml file
auth: {
auth_url: "",
# This should be your AWS Access Key ID
username: "AKIAJY32VWHYOFOR4J7Q",
# This should be your AWS Secret Access Key
# can be passed as part of cmd line when running the playbook
password: "{{ password | default(lookup('env', 'AWS_SECRET_KEY')) }}"
}
# These variable defines AWS cloud provision attributes
cluster: {
region_name: "us-east-1", #TODO Dynamic fetch
availability_zone: "", #TODO Dynamic fetch based on region
security_group: "Fabric",
target_os: "ubuntu",
image_name: "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*",
image_id: "ami-d15a75c7",
flavor_name: "t2.medium", # "m2.medium" is big enough for Fabric
ssh_user: "ubuntu",
validate_certs: True,
private_net_name: "demonet",
public_key_file: "/home/ubuntu/.ssh/fd.pub",
private_key_file: "/home/ubuntu/.ssh/fd",
ssh_key_name: "fabric",
# This variable indicate what IP should be used, only valid values are
# private_ip or public_ip
node_ip: "public_ip",
container_network: {
Network: "172.16.0.0/16",
SubnetLen: 24,
SubnetMin: "172.16.0.0",
SubnetMax: "172.16.255.0",
Backend: {
Type: "udp",
Port: 8285
}
},
service_ip_range: "172.15.0.0/24",
dns_service_ip: "172.15.0.4",
# the section defines preallocated IP addresses for each node, if there is no
# preallocated IPs, leave it blank
node_ips: [ ],
# fabric network node names expect to be using a clear pattern, this defines
# the prefix for the node names.
name_prefix: "fabric",
domain: "fabricnet",
# stack_size determines how many virtual or physical machines we will have
# each machine will be named ${name_prefix}001 to ${name_prefix}${stack_size}
stack_size: 3,
etcdnodes: ["fabric001", "fabric002", "fabric003"],
builders: ["fabric001"],
flannel_repo: "https://github.com/coreos/flannel/releases/download/v0.7.1/flannel-v0.7.1-linux-amd64.tar.gz",
etcd_repo: "https://github.com/coreos/etcd/releases/download/v3.2.0/etcd-v3.2.0-linux-amd64.tar.gz",
k8s_repo: "https://storage.googleapis.com/kubernetes-release/release/v1.7.0/bin/linux/amd64/",
go_ver: "1.8.3",
# If volume want to be used, specify a size in GB, make volume size 0 if wish
# not to use volume from your cloud
volume_size: 8,
# cloud block device name presented on virtual machines.
block_device_name: "/dev/vdb"
}
For Login:
For login using ssh I am doing these steps.
1- Download private key file.
2- chmod 600 private key.
3-ssh -vvv -i ~/.ssh/sshkeys.pem ubuntu#ec.compute-1.amazonaws.com .
I am getting error Permission denied (publickey)
You should be using the key that you created for connecting to AWS instance.
Got to EC2 dashboard and find instances and click on connect on the running instance that you need to ssh to.
It would be something like
ssh -i "XXX.pem" ubuntu#ec2-X-XXX-XX-XX.XX-XXX-2.compute.amazonaws.com
Save XXX.pem from security group to your machine.
Not the ssh keygen of your system
I have set up a list of ec2 instances to which I can log into using a ssh.cfg file that performs proxying & agent forwarding via the publicly accessible NAT instance as follows:
ssh -F ssh.cfg admin#private_instance_ip
ssh.cfg has the following structure:
Host 10.40.*
ProxyCommand ssh -W %h:%p ec2-user#nat_instance_ip -o StrictHostKeyChecking=no
Host *
ControlMaster auto
ControlPath ~/.ssh/mux-%r#%h:%p
ControlPersist 60m
ForwardAgent yes
My network is of course 10.40.*.
I want to test ansible connectivity to such hosts (defined in inventoryfile) as follows:
ansible -i inventoryfile -m ping all
Is there a way to do this?
edit: an example of how the ansible ping fails:
ansible -i inventoryfile -m ping 10.40.187.22
10.40.187.22 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Permission denied (publickey).\r\n",
"unreachable": true
}
However:
ssh -F ssh.cfg admin#10.40.187.22
Last login: Fri Oct 27 07:36:05 2017 from private_ip_of_NAT
admin#ip-10.40.187.22:~$ exit
Here is the ansible.cfg
[defaults]
nocows = 1
callback_whitelist = profile_tasks
host_key_checking = False
retry_files_enabled = False
gathering = explicit
forks=50
vault_password_file = .vault
[ssh_connection]
ssh_args = -F ssh.cfg
pipelining = True
edit2: when specifying user in the ping command, it still gives me an error:
ansible -i inventoryfile -m ping 10.40.187.22 -u admin
10.40.187.22 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Permission denied (publickey).\r\n",
"unreachable": true
}
It turns out that private IPs won't work in ansible ping when the later is proxied via e.g. a public nat instance.
The inventory file is dynamic and based on aws tags, i.e.
[ec2_hosts:children]
tag_Name_my_srv1
tag_Name_my_srv2
so the following command succeeds:
ansible -i inventoryfile -m ping tag_Name_my_srv1 -u admin
10.40.187.22 | SUCCESS => {
"changed": false,
"failed": false,
"ping": "pong"
}
I am trying to setup 3 Kafka brokers on ubuntu EC2 machines. But I am getting ConnectException while starting zookeeper. All the ports in the security group of my ec2 intsances are already open.
Below is the stack trace:
[2016-03-03 07:37:12,040] ERROR Exception while listening (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.BindException: Cannot assign requested address
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:330)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:507)
...
...
...
[2016-03-03 07:23:46,093] WARN Cannot open channel to 2 at election address /52.36.XXX.181:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
Below is the configuration of:
zookeeper.properties:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=52.37.XX.70:2888:3888
server.2=52.36.XXX.181:2888:3888
server.3=52.37.XX.42:2888:3888
initLimit=5
syncLimit=2
server.properties:
broker.id=1
port=9092
host.name=52.37.XX.70
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=52.37.XX.70:2181,52.36.XXX.181:2181,52.37.XX.42:2181
zookeeper.connection.timeout.ms=6000
I have added server's public IP in /etc/hosts of the instances. My modified /etc/hosts is as:
127.0.0.1 localhost localhost.localdomain ip-10-131-X-217
127.0.0.1 localhost localhost.localdomain 52.37.XX.70
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
The unique myid entries in /tmp/zookeper/myid are also entered correctly.
I have followed all the steps mentioned in: How to create a MultiNode - MultiBroker Cluster for Kafka on AWS
The issue was because I was using the Public IP of the server. Instead of that, use of Public DNS of ec2 instances fixed the issue.
How to remove all rules for a given port using the "aws ec2"?
aws ec2 revoke-security-group-ingress --group-name MySecurityGroup --protocol tcp --port 22 **--ALL-IP**
As per the documentation, This commands works on either --cidr or --source-group. so If you have multiple IP addresses then I would say the only option is to run the same command multiple times for the individual IP address (which would take the form of 1.1.1.1/32).
Or,
You can list all the ipadress in cidr format (1.1.1.1/32) in a file (each ip address on a new line) and then run a for loop over it running above command for each iteration. e.g.
for i in `cat ip_address_cidr.txt`; do aws ec2 revoke-security-group-ingress --group-name MySecurityGroup --protocol tcp --port 22 $i; done
I have not tested above command syntax but that should do it so that you can revoke the rules in a single one-liner command.
I think that this is what you are looking for: How to Close All Open SSH Ports in AWS Security Groups
Here a solution for a specific security group-id:
#!/bin/bash
sg = {security group}
# get the cidrs for the ingress rule
rules=$(aws ec2 describe-security-groups --group-ids $sg --output text --query 'SecurityGroups[*].IpPermissions')
# rules will contain something like:
# 22 tcp 22
# IPRANGES 108.42.177.53/32
# IPRANGES 10.0.0.0/16
# 80 tcp 80
# IPRANGES 0.0.0.0/0
# luckily, aws returns all ipranges per port grouped together
# flag for if we are reading ipranges
reading=0
# loop returned lines
while read -r line; do
# split the line up
rulebits=($line)
# check if if we are reading ssh port ipranges
if [ $reading -eq 0 ] ; then
# we are not reading ipranges
# check if '22 tcp 22'
if [ ${rulebits[0]} == "22" ] && [ ${rulebits[1]} == "tcp" ] && [ ${rulebits[2]} == "22" ] ; then
# found it
reading=1
fi
else
# we are reading ipranges
# check if first word is 'IPRANGES'
if [ ${rulebits[0]} == "IPRANGES" ] ; then
# found a cidr for open ssh port
cidr=${rulebits[1]}
echo -n found port 22 open cidr $cidr closing...
# close it
result=$(aws ec2 revoke-security-group-ingress --group-id $sg --protocol tcp --port 22 --cidr $cidr --output text)
if [ "$result" == "true" ] ; then
echo " OK"
else
echo " ERROR"
fi
else
# new port
reading=0
fi
fi
done
revoke-security-group-ingress in version 2 gives us to specify multiple IP CIDRS. See below solution written in PHP which I am trying to clean in multiple region and multiple prots.
To specify multiple rules in a single command use the --ip-permissions option https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/revoke-security-group-ingress.html
$cleanports = [22,5984];
$sgids = [["sgid"=>"sg1","region"=>"us-east-1"],["sgid"=>"sg1","region"=>"us-east-1"]];
foreach($sgids as $sgidDetail){
$iprules = json_decode(shell_exec("/usr/bin/aws ec2 describe-security-groups --group-ids {$sgidDetail['sgid']} --region {$sgidDetail['region']} --query 'SecurityGroups[*].IpPermissions'"), true)[0];
foreach ($iprules as $key => $ips) {
if(!empty($ips['FromPort']) && !empty($ips['ToPort']) && in_array($ips['FromPort'], $cleanports) && in_array($ips['ToPort'], $cleanports)){
echo "\n\n";
echo shell_exec("/usr/bin/aws ec2 revoke-security-group-ingress --group-id {$sgidDetail['sgid']} --region {$sgidDetail['region']} --ip-permissions '".json_encode($ips)."'");
}
}
}