I am trying to configure the puppet server and agent making my local laptop with ubuntu 18.04 as puppet server and aws ec2 instance as puppet agent. When trying to do so i am facing the issues related to hostname adding in /etc/hosts file and whether to use the public ip or private ip address and how to do the final configuration and make this work.
I have used the public ip and public dns of both the system to specify in the /etc/hosts file but when trying to run the puppet agent --test from the agent getting the error as temporary failure in name resolution and connecting to https://puppet:8140 failed. I am using this for a project and my setup needs to remain like this.
The connection is initiated from the Puppet agent to the PE server, so the agent is going to be looking for your laptop, even if you have the details of your laptop in the hosts file it probably has no route back to your laptop across the internet as the IP of your laptop was probably provided by your router at home.
Why not build your Puppet master on an ec2 instance and keep it all on the same network, edit code on your laptop, push to github/gitlab and then deploy the code from there to your PE server using code-manager.
Alternatively you may be able to use a VPN to get your laptop onto the AWS VPC directly in which case it'll appear as just another node on the network and everything should work.
The problem here is that the puppet server needs a public IP or an IP in the same network as your ec2 instance to which your puppet agent can connect to. However, there's one solution without using a VPN though it can't be permanent. You can tunnel your local port to the ec2 instance
ssh -i <pemfile-location> -R 8140:localhost:8140 username#ec2_ip -> This tunnels port 8140 on your ec2 instance to port 8140 in your localhost.
Then inside your ec2 instance you can modify your /etc/hosts file to add this:
127.0.0.1 puppet
Now run the puppet agent on your ec2 instance and everything should work as expected. Also note that if you close the ssh connection created above then the ssh tunnel will stop working.
If you want to keep the ssh tunnel open a bit more reliably then this answer might be helpful: https://superuser.com/questions/37738/how-to-reliably-keep-an-ssh-tunnel-open
Related
Launching an AWS EC2 instance seems quite straightforward although when it comes to connecting to the newly launched instance things get sticky. The process for connecting to an instance proposed by such a tech giant is very counter-intuitive.
As a short reminder I should add that an "instance" is technically a virtual machine running on Amazon's Elastic Compute Cloud (EC2), for more info one could have a look at this link.
The ec2 instance referred to in this discussion is Ubuntu Server 20.04 LTS (HVM).
The instruction for working with EC2 Linux instances is given here.
AWS EC2 proposes three different ways of connecting to an instance:
EC2 Instance connect (browser-based SSH connection),
Session Manager
SSH Client
Now with regard to connecting to the above-mentioned instance there are only certain connections that establish correctly and the rest of the proposed methods fail, here is the list of connection successes and failures :
Ubuntu instance, security group source "Custom=0.0.0.0/0", Connection establishes using both EC2 Instance Connect (browser-based SSH connection) and SSH client.
Ubuntu instance, security group source "My IP=$IP", Connection establishes only using SSH client (terminal on Ubuntu and PuTTY on windows) and not using EC2 instance connect.
Both above cases have been tried on Ubuntu 20.04 and Windows 10 as local machine and the problem remains similar on both machines. I went through most of the failure cases discussed in the troubleshooting documents proposed here and verified them on my instance. Yet the problem persists. I should also add that I never tried "session manager" connection method although opening its tab already would give some info about "not installed" agents and features.
Any idea regarding this problem? Somebody out there facing the same issue?
From Docs
(Amazon EC2 console browser-based client) We recommend that your instance allows inbound SSH traffic from the recommended IP block published for the service.
Reason for this -> EC2 Instance Connect works by making an HTTPS connection between your web browser and the backend EC2 Instance Connect service on aws. Then, EC2 Instance Connect establishes a "mostly normal" SSH connection to the target instance in other words the request is going from backend ec2 instance connect and not your browser that is why it needs IP address from accepted ranges of that region .
Browser based EC2 Instance Connect uses specific IP ranges for browser-based SSH connections to your instance. These IP ranges differ between AWS Regions. To find the AWS IP address range for EC2 Instance Connect in a specific Region, use the following( just replace your region with your region) ( for Linux required curl and jq as prerequisite)
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json| jq -r '.prefixes[] | select(.region=="Your region") | select(.service=="EC2_INSTANCE_CONNECT") | .ip_prefix'
whatever the value is returned just add up to your security rule and it will work.
Ubuntu instance, security group source "Custom=0.0.0.0/0", Connection establishes using both EC2 Instance Connect (browser-based SSH connection) and SSH client.
this works because 0.0.0.0/0 allows connection from all the IP ranges( which includes your region IP too).
for more details try reading this troubleshoot
I am trying to connect to Neptune DB in AWS Instance from my local machine in office, like connecting to RDS from office. Is it possible to connect Neptune db from local machine? Is Neptune db publicly available? Is there any way a developer can connect Neptune db from office?
Neptune does not support public endpoints (endpoints that are accessible from outside the VPC). However, there are few architectural options using which you can access your Neptune instance outside your VPC. All of them have the same theme: setup a proxy (EC2 machine, or ALB, or something similar, or a combination of these) that resides inside your VPC, and make that proxy accessible from outside your VPC.
It seems like you want to talk to your instance purely for development purposes. The easiest option for that would be to spin up an ALB, and create a target group that points to your instance's IP.
Brief Steps (These are intentionally not in detail, please refer to AWS Docs for detailed instructions):
dig +short <your cluster endpoint>
This would give you the current master's IP address.
Create an ALB (See AWS Docs on how to do this).
Make your ALB's target group point to the IP Address obtained for step #1. By the end of this step, you should have an ALB listening on PORT-A, that would forward requests to IP:PORT, where IP is your database IP (from Step 1) and PORT is your database port (default is 8182).
Create a security group that allows inbound traffic from everywhere. i.e. Inbound TCP rule for 0.0.0.0 on PORT-A.
Attach the security group to your ALB
Now from your developer boxes, you can connect to your ALB endpoint at PORT-A, which would internally forward the request to your Neptune instance.
Do checkout ALB docs for details around how you can create it and the concepts around it. If you need me to elaborate any of the steps, feel free to ask.
NOTE: This is not a recommended solution for a production setup. IP's used by Neptune instances are bound to change with failovers and host replacements. Use this solution only for testing purposes. If you want a similar setup for production, feel free to ask a question and we can discuss options.
As already mentioned you can't access directly outside your VPC.
The following link describes another solution using a SSH tunnel: connecting-to-aws-neptune-from-local-environment.
I find it much easier for testing and development purpose.
You can create the SSH tunnel with Putty as well.
Reference: https://github.com/M-Thirumal/aws-cloud-tutorial/blob/main/neptune/connect_from_local.md
Connect to AWS Neptune from the local system
There are many ways to connect to Amazon Neptune from outside of the VPC, such as setting up a load balancer or VPC peering.
Amazon Neptune DB clusters can only be created in an Amazon Virtual Private Cloud (VPC). One way to connect to Amazon Neptune from outside of the VPC is to set up an Amazon EC2 instance as a proxy server within the same VPC. With this approach, you will also want to set up an SSH tunnel to securely forward traffic to the VPC.
Part 1: Set up a EC2 proxy server.
Launch an Amazon EC2 instance located in the same region as your Neptune cluster. In terms of configuration, Ubuntu can be used. Since this is a proxy server, you can choose the lowest resource settings.
Make sure the EC2 instance is in the same VPC group as your Neptune cluster. To find the VPC group for your Neptune cluster, check the console under Neptune > Subnet groups. The instance's security group needs to be able to send and receive on port 22 for SSH and port 8182 for Neptune. See below for an example security group setup.
Lastly, make sure you save the key-pair file (.pem) and note the directory for use in the next step.
Part 2: Set up an SSH tunnel.
This step can vary depending on if you are running Windows or MacOS.
Modify your hosts file to map localhost to your Neptune endpoint.
Windows: Open the hosts file as an Administrator (C:\Windows\System32\drivers\etc\hosts)
MacOS: Open Terminal and type in the command: sudo nano /etc/hosts
Add the following line to the hosts file, replacing the text with your Neptune endpoint address.
127.0.0.1 localhost YourNeptuneEndpoint
Open Command Prompt as an Administrator for Windows or Terminal for MacOS and run the following command. For Windows, you may need to run SSH from C:\Users\YourUsername\
ssh -i path/to/keypairfilename.pem ec2-user#yourec2instanceendpoint -N -L 8182:YourNeptuneEndpoint:8182
The -N flag is set to prevent an interactive bash session with EC2 and to forward ports only. An initial successful connection will ask you if you want to continue connecting? Type yes and enter.
To test the success of your local graph-notebook connection to Amazon Neptune, open a browser and navigate to:
https://YourNeptuneEndpoint:8182/status
You should see a report, similar to the one below, indicating the status and details of your specific cluster:
{
"status": "healthy",
"startTime": "Wed Nov 04 23:24:44 UTC 2020",
"dbEngineVersion": "1.0.3.0.R1",
"role": "writer",
"gremlin": {
"version": "tinkerpop-3.4.3"
},
"sparql": {
"version": "sparql-1.1"
},
"labMode": {
"ObjectIndex": "disabled",
"DFEQueryEngine": "disabled",
"ReadWriteConflictDetection": "enabled"
}
}
Close Connection
When you're ready to close the connection, use Ctrl+D to exit.
Hi you can connect NeptuneDB by using gremlin console at your local machine.
USE THIS LINK to setup your local gremlin server, it works for me gremlin 3.3.2 version
Only you have to update the remote.yaml as per your url and port
I am new to setting up virtual machines. I created my first Ubuntu instance using AWS EC2. Everything seemed to check out until I tried connecting to it with ssh, as per instructions.
To provide some context, my app is called "smpapp". My computer is macOS High Sierra. Naturally, my smpapp.pem file saved to ~/Downloads. First, I opened up the Terminal and set my working directory to Downloads with cd ~/Downloads. Then I entered chmod 400 smpapp.pem, which didn't return any error, so I assume it was a success. Then, I entered ssh -i "smpapp.pem" ubuntu#ec2-XX-XX-XXX-XXX.us-east-2.compute.amazonaws.com (omitting public DNS numbers with Xs). It took awhile to process before spitting out, ssh: connect to host ec2-XX-XX-XXX-XXX.us-east-2.compute.amazonaws.com port 22: Operation timed out.
Can someone explain the general problem to me and how I can fix it (methodically and in layman's terms)?
Could be a few things:
Does your ec2 instance have a public ip? (if not, you might have to attach an elastic ip or put it in a public subnet)
Is the security group attached to the ec2 instance allowing connections to port 22?
Is the ACL on the subnet allowing public connections to the subnet?
Is your VPC configured to routetraffic through your IGW?
Amazon offers step by step instructions on determining the issue, it could be for any reason of the above not being configured properly. You can find step by step instructions on what do in the official amazon docs here.
Yesterday, I updated my Django website (on AWS EC2) to HTTPS by using lets-encrypt. Everything works well. The website has HTTPS green icon as expected.
Today when I try to connect my instance by using SSH. The connection keep hunging. Finally, It give some message like "ssh: connect to host ec2-34-202-93-189.compute-1.amazonaws.com port 22: Resource temporarily unavailable".
I thought it might be security group problem of this instance. So I double checked my security group setting of this instance, the SSH, HTTP and HTTPS port are all open correctly. I created another instance to test if there is any problem on my local. The new instance connected successfully. Then I apply the new instance to the security group that I made for the previous instance and It connected. Then I apply the previous instance to the new security that I made for the new instance, the connection got frozen again. I also tried to connect with putty and it was not working as well.
Now I am really confused. My local machine is Windows subsystem of Linux. My EC2 instance is Ubuntu 16. I am using Nginx as web server. My ssh command is "ssh -i blog_project.pem ubuntu#ec2-34-202-93-189.compute-1.amazonaws.com".
Here is my security group setup for the instance.
This is the result I command "ssh -vvv -i blog_project.pem ubuntu#ec2-34-202-93-189.compute-1.amazonaws.com"
BTW, Is there any way that I can login to my instance without ssh connection? Is there anything like console or shell inside the AWS that I can touch with my instance?
Check if the instance exists on AWS, maybe a new one was created with different Public DNS (xxxx.compute-1.amazonaws.com) than the one you are using in your command.
I am currently working on an AWS EC# LINUX AMI. I have a private IP. Is it possible to access that private IP with filezilla to transfer files. i am unable to do so.
For access an EC2 machine with private IP, you need to setup your own VPN server. If you already have VPN setup in your AWS cloud then you just need to install a VPN client and login with your credential and you will be able to access EC2 machine or transfer files using filezilla with private IP too. I am assuming that you haven't setup VPN server. you may use AMI of OPENVPN from AWS market place for setup VPN. Below is the good link for getting start.
https://docs.openvpn.net/how-to-tutorialsguides/virtual-platforms/amazon-ec2-appliance-ami-quick-start-guide/
After complete this you have to install OPENVPN in your machine and after Login with your credentials your will able to access your EC2 instance with private IP.
Below is the link for install OPENVPN in Ubuntu machine. For different operating system you can explore site.
https://docs.openvpn.net/getting-started/how-to-install-openvpn-as-software/
OPENVPN is one of the alternative, you can use other also as per your need.
Using 2 ways you can do this
Create a bastion host which will connect to the private instance
Using a port forwarding means tunnelling.
If you are using bastion host for connecting private ec2 instance then this steps will be useful
Using Filezilla to transfer files to a private ec2 instance through a bastion host:-
Note: Keep Pem file same of bastion host and private ec2 instance.
Open terminal or cmd(linux terminal i.e gitbash)
we are connecting to the AWS EC2 instance with one terminal command.
ssh -N -L 1234:<private_instance_ip or Private_DNS>:22 -i <Pem_File> #<Bastion_host_public_ip>
e.g.
ssh -N -L 1234: ip-171-12-21-208.us-east-1.compute.internal:22 -i app_prod.pem ubuntu#ec2-31-92-123-22.us-east-1.compute.amazonaws.com
Note: - For the first time when you enter this command it will ask for Are you sure you want to continue connecting - yes
3.Keep this terminal or cmd open.
If you close this session then the connection is broken
4.Open “FileZilla” application and on “Edit” section -> Click on “Settings”
5.On “Settings” page -> Click on “SFTP” and add PEM file of ec2 instance and click on “OK”
6.Add below entries:-
Host:- 127.0.0.1 or sftp://127.0.0.1
Username:- <your_user>
Password:- Keep empty
Port:- 1234
7.Click on Quick Connect.
Once the connection is established then you can easily transfer files from local to private instance.
See- scp-to-transfer-files-to-a-private-ec2-instance-through-a-bastion-host
https://www.davidbegin.com/using-scp-to-transfer-files-to-a-private-ec2-instance-through-a-bastion-host/