I want to allow password logins to my EC2 instances. I know which line it is that controls this in /etc/ssh/sshd_config and what it should be set to. Specifically:
PasswordAuthentication yes
However, even when I've set this on a master image that I keep, whenever I restore it to a new instance, the value on the line keeps getting reset to 'no'. Which means that every time I launch a new instance I have to yet again manually change this file. This has made the automation of my instances one step away from being fully automated.
What do I need to do to my master image so that every instance I create from it leaves my sshd_config file the way I like?
This is a Fedora 16 image fully configured with proprietary and other software.
If you used an old AMI as the basis for your images, that option used to be changed by the kickstart file, but as far as I know that option was removed some time ago.
These days the AMI is most likely configured by cloud-init and if that is the case you should find and change the ssh_pwauth option in /etc/cloud/cloud.cfg
Edit file /etc/cloud/cloud.cfg (needs root permission, e.g. sudo)
Look for the ssh_pwauth key
Change its value from 0 to true. Not 1, but true!
ssh_pwauth: true
Related
I'm trying to restore data in EFS from recovery points managed by AWS Backup. It seems AWS Backup does not support destructive restores and will always restore to a directory in the target EFS file system, even when creating a new one.
I would like to sync the data extracted from such a recovery point to another volume, but right now I can only do this manually as I need to lookup the directory name that is used by the start-restore-job operation (e.g. aws-backup-restore_2022-05-16T11-01-17-599Z), as stated in the docs:
You can restore those items to either a new or existing file system. Either way, AWS Backup creates a new Amazon EFS directory (aws-backup-restore_datetime) off of the root directory to contain the items.
Further looking through the documentation I can't find either of:
an option to set the name of the directory used
the value of directory name returned in any call (either start-restore-job or describe-restore-job)
I have also checked how the datetime portion of the directory name maps to the creationDate and completionDate of the restore job but it seems neither match (completionDate is very close, but it's not the exact same timestamp).
Is there any way for me to do one of these two things? Both of them missing make restoring a file system from a recovery point in an automated fashion very hard.
Is there any way for me to do one of these two things?
As it stands, no.
However, since we know that the directory will always be in the root, doing find . -type d -name "aws-backup-restore_*" should return the directory name to you. You could also further filter this down based on the year, month, day, hour & minute.
You could have something polling the job status on the machine that has the EFS file system mounted, finding the correct directory and then pushing that to AWS Systems Manager Parameter Store for later retrieval. If restoring to a new file system, this of course becomes more difficult but still doable in an automated fashion.
If you're not mounting this on an EC2 instance, for example, running a Lambda with the EFS file system mounted, will let you obtain the directory & then push it to Parameter Store for retrieval elsewhere. The Lambda service mounts EFS file systems when the execution environment is prepared - in other words, during the 'cold start' duration so there are no extra costs here for extra invocation time & as such, would be the cheapest option.
There's no built-in way via the APIs however to obtain the directory or configure it so you're stuck there.
It's an AWS failure that neither do they return the filename that they use in any way nor does any of the metadata returned - creationDate/completionData - exactly match the timestamp they use to name the file.
If you're an enterprise customer, suggest this as a missing feature to your TAM or SA.
In the AWS website https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html, it shows how to modify the instance user data of an existing EC2.
However, at the end it says the modified user data will not be executed (7). What's the point to modify the user data and not executed? Is it possible to execute the modified user data once?
To modify instance user data
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
In the navigation pane, choose Instances.
Select the instance and choose Instance state, Stop instance. If this
option is disabled, either the instance is already stopped or its root
device is an instance store volume.
When prompted for confirmation, choose Stop. It can take a few minutes
for the instance to stop.
With the instance still selected, choose Actions, Instance settings,
Edit user data.
Modify the user data as needed, and then choose Save.
Restart the instance. The new user data is visible on your instance
after you restart it; however, user data scripts are not executed.
The User Data field was created as a way to pass information to an instance. For example, providing a password to a database, or configuration information.
Then, the Ubuntu community came up with the clever idea of passing a script via the User Data field, and having some code on the instance execute the script when the system boots. This has enabled "self-configuring" systems, and is called cloud-init.
By default, the User Data script only executes once per instance, with the intention of installing software.
From Boot Stages — cloud-init documentation:
cloud-init ships a command for manually cleaning the cache: cloud-init clean
Running this command will 'forget' the previous runs, and will execute the User Data script on the next boot.
It is also possible to run a script on every boot by placing the script in /var/lib/cloud/scripts/per-boot/.
Simple step to step working Solution:
Stop the Instance
Go to Action > Instance Setting > Edit User Data
And make sure to choose As Text in Edit User Data Screen
Add commands on Textarea
Now start the Instance And check that it is on Running State
check your public IP (This has been changed after restart)
Now finally connect to the instance using ssh :
ssh -i #<IP_ADDRESS>
I have two EC2 instances and i am trying to sync a directory between the two of them.
I have set up the lsyncd service on one of the instances and was able to sync a directory to different directory on the same instance.
Now i am trying to sync the same directory with the second instance and it is not working.
The reason it is not working is that I am not able to put the key that was generated on the first instance using ssh-keygen -t rsa on the second instance in order to allow them access each other.
I have tried sudo ssh-copy-id -i /path/to/key ec2-user#ip-of-second-instance but it did not work.
I have also tried to manually copy the public part from the key.pub file of the first instance to the ~/.ssh/authorized_keys of the second instance but it did not work either.
That is my lsynd configuration settings:
settings = {
insist = true,
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"}
sync {
default.rsyncssh,
source = "/home/ec2-user/IntSrv/Sync",
host = "second-instance-ip",
target = "/home/ec2-user/GenSrv/Sync",
}
What am i doing wrong? How can i fix that issue?
Any help would be appreciated. Thank you.
You might want to start again with the keys.
You should really be generating your own keys for each user. Then, for each user you want to grant access to the instance, add their key to the .ssh/authorized_keys file, either for the ec2-user or preferably create a user account for them first and add it to their authorized_keys file.
The keys generated by Amazon EC2 should be used to gain initial access to your instances. Then, proper security practice is to remove that key and add your own keys. This way, you have each person accessing via their own keypair, which can be removed if you wish to rescind access.
While I'm not familiar with lsyncd, I suspect that if you get ssh working, then lsyncd will probably work fine, too.
So, quick summary:
Generate a key for YOU using ssh-keygen
Connect to the desired instances, and add your public keypair to authorized_keys within the desired user home directory
Use those keys instead of the ones generated by Amazon EC2
I am trying to setup Replica outside of AWS and master is running at AWS RDS. And I do not want any downtime at my master. So I setup my slave node and now I want to backup my current database which is at AWS.
mysqldump -h RDS ENDPOINT -u root -p --skip-lock-tables --single-transaction --flush-logs --hex-blob --master-data=2 --all-databases > /root/dump.sql
I tested it on my VM and it worked fine but when tying it with RDS it gives me error
mysqldump: Couldn't execute 'FLUSH TABLES WITH READ LOCK': Access denied for user 'root'#'%' (using password: YES) (1045)
Is it because i do not have super user privilege or how to I fix this problem? Please someone suggest me.
RDS does not allow even the master user the SUPER privilege, and this is required in order to execute FLUSH TABLES WITH READ LOCK. (This is an unfortunate limitation of RDS).
The failing statement is being generated by the --master-data option, which is, of course, necessary if you want to be able to learn the precise binlog coordinates where the backup begins. FLUSH TABLES WITH READ LOCK acquires a global read lock on all tables, which allows mysqldump to START TRANSACTION WITH CONSISTENT SNAPSHOT (as it does with --single-transaction) and then SHOW MASTER STATUS to obtain the binary log coordinates, after which it releases the global read lock because it has a transaction that will keep the visible data in a state consistent with that log position.
RDS breaks this mechanism by denying the SUPER privilege and providing no obvious workaround.
There are some hacky options available to properly work around this, none of which may be particularly attractive:
do the backup during a period of low traffic. If the binlog coordinates have not changed between the time you start the backup and after the backup has begin writing data to the output file or destination server (assuming you used --single-transaction) then this will work because you know the coordinates didn't change while the process was running.
observe the binlog position on the master right before starting the backup, and use these coordinates with CHANGE MASTER TO. If your master's binlog_format is set to ROW then this should work, though you will likely have to skip past a few initial errors, but should not have to subsequently have any errors. This works because row-based replication is very deterministic and will stop if it tries to insert something that's already there or delete something that's already gone. Once past the errors, you will be at the true binlog coordinates where the consistent snapshot actually started.
as in the previous item, but, after restoring the backup try to determine the correct position by using mysqlbinlog --base64-output=decode-rows --verbose to read the master's binlog at the coordinates you obtained, checking your new slave to see which of the events must have already been executed before the snapshot actually started, and using the coordinates determined this way to CHANGE MASTER TO.
use an external process to obtain a read lock on each and every table on the server, which will stop all writes; observe that the binlog position from SHOW MASTER STATUS has stopped incrementing, start the backup, and release those locks.
If you use any of these approaches other than perhaps the last one, it's especially critical that you do table comparisons to be certain your slave is identical to the master once it is running. If you hit subsequent replication errors... then it wasn't.
Probably the safest option -- but also maybe the most annoying since it seems like it should not be necessary -- is to begin by creating an RDS read replica of your RDS master. Once it is up and synchronized to the master, you can stop replication on the RDS read replica by executing an RDS-provided stored procedure, CALL mysql.rds_stop_replication which was introduced in RDS 5.6.13 and 5.5.33 which doesn't require the SUPER privilege.
With the RDS replica slave stopped, take your mysqldump from the RDS read replica, which will now have an unchanging data set on it as of a specific set of master coordinates. Restore this backup to your off-site slave and then use the RDS read replica's master coordinates from SHOW SLAVE STATUS Exec_Master_Log_Pos and Relay_Master_Log_File as your CHANGE MASTER TO coordinates.
The value shown in Exec_Master_Log_Pos on a slave is the start of the next transaction or event to be processed, and that's exactly where your new slave needs to start reading on the master.
Then you can decommission the RDS read replica once your external slave is up and running.
Thanks Michael, I think the most correct solution and the recommended by AWS is do the replication using a read replica as a source as explained here.
Having a RDS master, RDS read replica and an instance with MySQL ready, the steps to get an external slave are:
On master, increase binlog retention period.
mysql> CALL mysql.rds_set_configuration('binlog retention hours', 12);
On read replica stop replication to avoid changes during the backup.
mysql> CALL mysql.rds_stop_replication;
On read replica annotate the binlog status (Master_Log_File and Read_Master_Log_Pos)
mysql> SHOW SLAVE STATUS;
On server instance do a backup and import it (Using mydumper as suggested by Max can speed up the process).
mysqldump -h RDS_READ_REPLICA_IP -u root -p YOUR_DATABASE > backup.sql
mysql -u root -p YOUR_DATABASE < backup.sql
On server instance set it as slave of RDS master.
mysql> CHANGE MASTER TO MASTER_HOST='RDS_MASTER_IP',MASTER_USER='myrepladmin', MASTER_PASSWORD='pass', MASTER_LOG_FILE='mysql-bin-changelog.313534', MASTER_LOG_POS=1097;
Relace MASTER_LOG_FILE and MASTER_LOG_POS to the values of Master_Log_File Read_Master_Log_Pos you saved before, also you need an user in RDS master to be used by slave replication.
mysql> START SLAVE;
On server instance check if replication was success.
mysql> SHOW SLAVE STATUS;
On RDS read replica resume replication.
mysql> CALL mysql.rds_start_replication;
for RDS binlog position you can use mydumper with --lock-all-tables, it will use LOCK TABLES ... READ just to get the binlog coordinates and then realease it instead of FTWRL.
Michael's answer is extremely helpful and focuses on the main sticking point: you simply cannot GRANT the required SUPER privilege on RDS, and therefore you can't use the --master-data flag that would make things so much easier.
I read that it may be possible to work around this by creating or modifying a Database Parameter Group via the API, but I think using the RDS procedures is a better option.
The multi-tiered replication approach works well, though, and can include tiers outside RDS/VPC so it's possible to replicate from "Classic" EC2 to VPC using this method.
A lot of the necessary functionality is only in later releases of MySQL 5.5 and 5.6, and I strongly recommend you run the same version on all the DBs involved in the replication stack, so you may have to do an upgrade of your old DB before all of this, which means yet more tedium and replication and so on.
I had faced a similar problem a quick workaround to this is:
Create a EBS Volume to have an extra space or extend current EBS volume on EC2. (or if you have an extra space you can use that).
Use mysqldump command without --master-data or --flush-data directive to generate a complete (FULL) backup of db.
mysqldump -h hostname --routines -uadmin -p12344 test_db > filename.sql
admin is DB name and 12344 is the password
Above is for taking backup of one single DB, if required to take all DBs then specify --all-databases and also mention DB Names.
Create a Cron of this command to run once a day that will automatically generate the dump.
Please note that this will incur an extra cost if your DB Size is huge. as it creates a complete DB dump.
hope this helps
You need to
1- create a read replica on aws
2- make sure that this instance is catching up with the master
3- stop the replication and get the log_file and log_position parameters by
show slave status \G
4- dump the database and use the parameters logged in step 3 to start the replication on your own server.
5- start the slave.
Detailed instructions here
Either things have changed, since #Michael - sqlbot 's response, or there is a misunderstanding going on here (could be on my part),
You can use COPY to import a csv file into rds, at least on the postgres version, you just need to use FROM STDIN instead of directly naming the file,
which means you end up piping things like:
cat data.csv | psql postgresql://server:5432/mydb -U user -c "COPY \"mytable\" FROM STDIN DELIMITER ',' "
I have two databases one on my local machine and one on my amazon ec2 instance.Now what I
do is I run a python program on my local machine which makes changes to the databse on my local machine.I want these changes to be reflected onto the database on amazon ec2 instance,
periodically.I want to do this in python.A script that logs onto the amazon server establishes a connection with the database there and makes the changes.
I came across some modules like pexcept,fabric and paramiko.But I am struggling with the
key authentication.
The way I ssh from my terminal is ssh -i my_rsa_file.pem username#ip_address.There is no password.How do I go about this ??
Also I want to know whether simply using Popen in subprocess to execute the login command work ?
The Boto EC2 documentation here describes the EC2 instance object, of which "key_pair" is an attribute. Look about 3/4 of the way down, under "boto.ec2.instance".
http://boto.readthedocs.org/en/latest/ref/ec2.html
So, e.g., you could run some instances as follows, and then store the first instance as "inst":
reservation = conn.run_instances(...)
inst = reservation.instances[0]
To retrieve your key-pair name as a unicode string, just use:
kp_name = inst.key_name
You can then retrieve the corresponding Boto object using get_key_pair:
kp_obj = conn.get_key_pair(kp_name)
Of course, this is a silly example, since I would have needed my key pair name to run_instances in the first place. May you find a more fruitful application!