I'm running GitLab CE privately within an AWS VPC that I access via a VPN instance. I installed the latest AWS AMI of GitLab CE, then upgraded it to the latest version of GitLab. I've gotten everything working, except for one thing: Whenever I reboot the instance in EC2, my /etc/gitlab/gitlab.rb's external_url is reset to the IP address of my VPC's SNAT instance, almost as if GitLab is asking "what is my public IP?" and then changing the setting's value to that answer. I keep changing it back to the internal hostname provided by my VPC's Route 53 hosted zone, https://gitlab.corp.mydomain.com, but it's reset every time I reboot the instance. To be clear, this GitLab instance is not exposed to the internet, but it does have egress to the internet through the SNAT (e.g., to update OS packages).
How can I force my internal hostname to stick? I can still access GitLab through my browser at https://gitlab.corp.mydomain.com, so perhaps this doesn't matter?
After a quick search I have found this.
https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/2021
Summary of the content:
It seems like Gitlabs hostname detection is not working correctly if public IPs are deactivated in EC2. To enable usage in such cases Gitlab replaces the hostname with the assigned IP. Gitlab will return to the hostname if it can resolve it later, at least from version 10.1.3.
As it works for you, I would simply keep the configuration.
Related
I have a solution (AnzoGraph DB) deployed on my AWS Kubernetes Cluster (EC2 Instance), and was working totally fine.
Suddenly this solution stopped and i could not access it via the DNS anymore.
I tested the solution deployed on my cluster using kubectl port-forward command and they are working fine (the pods and services), thus i assume the problem is with AWS Loadbalancer.
To access the application we need to go through this path:
Request -> DNS -> AWS Load Balancer -> Services -> Pods.
The LoadBalancer is (classic) internal, so it's only accessible for me or the company using VPN.
Every time when I try to access the DNS , I got no response.
Any idea how i can fix it ? or where is the exact issue ? how can I troubleshoot this issue and follow the traffic on AWS ?
Thanks a lot for the help!
sorry I missed your post earlier.
lets start with a few questions...
You say you use k8s on AWS EC2, do you actually use EKS, or do you run a different k8s stack?
Also ... you mentioned that you access the LB from your (DB) client/ your software by DNS resolving the LB and then access AnzoGraph DB.
I want to make sure that the solution is actually DNS resolving the LB via DNS every time. if you have a long running service, and AWS changes the IP address of the LB, and your SW possibly had cached the IP, you would not be able to connect to the LB.
on the system you run your Software accessing AnzoGraph DB ... (I assume CentOS (7) )
make sure you have dig installed (yum install bind-utils)
dig {{ your DNS name of your LB }}
is that actually the IP address your SW is accessing?
has the IP address of the client changed? make sure the LB SG allows access
(https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-groups.html)
I assume you access the AnzoGraph DB frontend POD via 443?
as you write
"I tested the solution deployed on my cluster using kubectl port-forward command and they are working fine (the pods and services)"
we probably do not have to look for pod logs.
(if that was not the case, the LB would obviously block traffic as well.)
So I agree, that the most likely issue is (bad) DNS caching or SG due to different SRC IP being rejected by the classic LB SG.
also for completeness .. please tell us more about your env.
AnzoGraph DB image
EKS/k8s version
helm chart / AnzoGraph operator used.
Best - Frank
So, I've been working locally in a vagrant ubuntu box for the past month: I've spent a lot of time working on customizing it and installing exactly all the software I want on it. I started all of this through the normal vagrant tutorial (aka, nothing special). I packaged my local vagrant box into a package.box file. Now, I want to move my development environment (e.g. package.box file) to an Amazon EC2 instance on AWS. I know I'm not supposed to ask for software recommendations, but my question is basically: is this possible to do and, if it is, could you point me to some examples of people doing it? I've read that packer might be an option, but it looks to me (a very inexperienced perspective) that maybe I should have started with that instead of trying to use it now. Any help would be appreciated - I don't want to spend a couple weeks setting up a new environment when I have one locally set up.
Edit:
Progress! I followed #error2007s link and followed the tutorial. I'm at the point where I've uploaded the VMDK image to s3 and provisioned an instance using it (all done automatically with the ec2-import-instance command on the CLI). However, I don't see a Public IP to access the new instance after I start it up.
I think this is related to cloud-init somehow, but I'm not sure what that is really. I tried it with both the /etc/cloud/cloud.cfg file that came with the box as well as the one listed here and neither of the two boxes I uploaded gave me a Public IP to access.
Edit 2:
Here are some things I see in the Console (They all seem right to me, but a more experienced eye might see something wrong):
subnet info:
Auto-assign Public IP: yes
Network ACL:
VPC info:
DNS resolution: yes
DNS hostnames: yes
ClassicLink DNS Support: no
VPC CIDR: 172.31.0.0/16
DHCP Option Set:
Options: domain-name = ec2.internal domain-name-servers = AmazonProvidedDNS
From my perspective, those all look right, or am I missing something?
I assigned an Elastic IP per these instructions, but when I ssh ec2-user#<elastic-ip>, it says ssh: connect to host <elastic-ip> port 22: Connection refused. The security group assigned to the instance is set to allow all protocols on all ports. Also, this is the first time I encounter a Elastic IP and I'm unsure what exactly it is doing.
Amazon enables you to transfer your Vm to AWS as a EC2 instance. Check this tutorial this is more simple.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UsingVirtualMachinesinAmazonEC2.html
You want to use the Vagrant AWS provider found here:
https://github.com/mitchellh/vagrant-aws
This is a Vagrant 1.2+ plugin that adds an AWS provider to Vagrant,
allowing Vagrant to control and provision machines in EC2 and VPC.
This will allow you to provision your AWS instances using Vagrant, allowing you to migrate the same local development environment to an AWS EC2 instance.
There is a good tutorial here:
https://nurmrony.wordpress.com/2015/03/15/vagrant-deploy-and-provisioning-an-amazon-ec2-instance/
Hi I have found these articles but I have not yet tested them myself. Im still in the middle of organizing my personal notes and identifying my technology stack. I intend to have a Homestead vagrant box be replicated as an EC2 instance, so I wont have to configure the instance(s) manually.
https://nurmrony.wordpress.com/2015/03/15/vagrant-deploy-and-provisioning-an-amazon-ec2-instance/
https://www.tothenew.com/blog/using-vagrant-to-deploy-aws-ec2-instances/
https://foxutech.com/how-to-deploy-on-amazon-ec2-with-vagrant/
https://blog.scottlowe.org/2016/09/15/using-vagrant-with-aws/
https://devops.com/devops-primer-using-vagrant-with-aws/
I find their approaches similar. The only thing that I am worried at is the "vagrant add box" part.
I asked myselft, what if I had to do this setup again for familiarization purposes, what will happen since I already added a vagrant box (the dummy one, as instructed in the tutorials) previously.
I started a cluster in aws following the guides and then went about following the guestbook. The problem I have is accessing it externally. I set the PublicIP to the ec2 publicIP and then use the ip to access it in the browser with port 8000 as specified in the guide.
Nothing showed. To make sure it was actually the service that wasn't showing anything I then removed the service and set a host port to be 8000. When I went to the ec2 instance IP I could access it correctly. So it seems there is a problem with my setup or something. The one thing I can think of is, I am inside a VPC with an internet gateway. I didn't add any of my json files I used, because they are almost exactly the same as the guestbook example with a few changes to allow my ec2 PublicIP, and a few changes for the VPC.
On AWS you have to use your PRIVATE ip address with Kubernetes' services, since your instance is not aware of its public ip. The NAT-ing on amazon's side is done in such a way that your service will be accessible using this configuration.
Update: please note that the possibility to set the public IP of a service explicitly was removed in the v1 API, so this issue is not relevant anymore.
Please check the following documentation page for workarounds: https://kubernetes.io/docs/user-guide/services/
I'm trying to build a t2.mirco Ubuntu 12.04 EC2 environment running Airtime from Sourcefabric, however despite the installation going through OK I cannot access the login page via the address that the installer provided. I have change my security settings several times but I feel that it might have something to do with it. I have ran system checks to see if airtime is working and it returns a perfectly operating copy every time. The address that i'm trying to access the installation on is http://ip-172-31-5-46.us-west-2.compute.internal does anyone know why Amazon AWS is reacting this way?
The URL you just provided is EC2's internal DNS address (note the ".internal" at the end), if you want it to be accessibly publicly you'll need to assign an Elastic IP to the EC2 instance, or auto assign a public DNS on creation of the instance
Amazon docs for reference
I created a VM using Hyper-V on my local machine which I can RDP onto fine. I exported the VHDX to VHD and then uploaded it to AWS via the API.
This created a new EC2 instance for me but I cannot connect to it as it has no security keypair associated.
Thus I created a private AMI from the instance and re-instantiated a new EC2 instance from this AMI, setting the keypair in the process to ensure HTTP, RDP etc are permitted. The EC2 instance is running, but when I download teh RDP shortcut if fails to connect. I've tried rebooting the instance several times (as this was one of the suggested solutions on here when I searched) but it hasn't helped.
I also cannot ping any of these instances via the public DNS or private DNS.
Any idea what I'm doing wrong? I'm not even getting as far as the password prompt, its the connectivity is the problem.
Thanks
I had a similar problem - this is the tip from the AWS documentation that helped me solve it:
Verify that the route table for the subnet has a route that sends all traffic
destined outside the VPC (0.0.0.0/0) to the Internet gateway for the VPC
Additional debugging tips for this problem can be found here: Remote Desktop can't connect to the remote computer
The problem is probably the security group configuration attached to the instance. Make sure you have inbound TCP port 3389 permitted to be able to connect via RDP.
in username type:
.\Administrator
....and your decrypted password
See, this is local user. but you dont know real local hostname, that different from public DNS name.
Was helpful for me, hope for you too.
My guess for the reason you aren't getting anywhere has to do with the host certificates.
Be mindful that each computer connecting to the server via RDP needs to have an IAM user in AWS with an X.509 certificate uploaded to their account.
When importing or creating a windows AMI it is best to install and configure the EC2Config service.
The EC2 Config service does the following when you launch the instance:
At initial setup:
Sets the hostname to the private DNS name
Generates and sets a random password on the Administrator account
Initializes and formats the ephemeral disks
Generates and installs the host certificate used for Terminal
Services
Syncs the instance clock with a time server
After initial setup:
Writes the last three entries in the System event log to the Amazon
EC2 console so you can debug startup problems.
Prepares instances for bundling
Also, by default, security groups do not allow you to ping the instances. You must enable ICMP in the security group.
Hope that helps.
EDIT: Here is the link to the 64bit version of the EC2Config Installer
I also had the same problem. This is what I got from EC2.
Public DNS ec2-23-22-109-251.compute-1.amazonaws.com
User name Administrator
Password MyPassword
On the remote desktop, enter the domain and user name as
ec2-23-22-109-251\Administrator
If you paste the password is may not work, try to paste it but reenter the last 1 or 2 characters. Once you get the security certificate prompt accept/install it and your connection should open soon.
This is what worked for me:
Use your cell phone as a "hot spot" - which gives you a static IP address. That worked. Now, I'll need to contact my Internet provider to assign a static IP to my wireless router. Currently, it is spitting out DHCP IP addresses.
UPDATE:
12.7.2016
You need to go to the Security Group, select the Instance, then EDIT the RDP. If you click "MyIP" that should work for those of us working from home. If you are onsite, you may need to put in a custom CIDR.
After rebooting an EC2 instance, I found that I needed to reassociate the Elastic IP address associated with the instance. You click the button below then select the instance and private instance IP address when prompted.
RDP access worked after this step.
After patching Windows 2019 on my server, I couldn't access the instance anymore despite all my working settings didn't change. I used EC2Rescue to fix my issue
Please refer to the link below for details on how to use it: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/troubleshoot-connect-windows-instance.html#AWSSupport-ExecuteEC2Rescue