How could I drop an exisitng AlloyDB database programatically?
I'd like to run a test suite of our application on AlloyDB. I'm able to connect to the cluster using the proxy. However, when I try to drop database to cleanup the test environment, using code like:
echo "DROP DATABASE IF EXISTS application_test" | psql
I'm getting:
ERROR: syntax error at or near "DROP"
LINE 2: DROP DATABASE IF EXISTS application_test
I'm sure I can connect to the cluster correctly, because I run other queries before this one.
How could I remove an existing database from a script? I can't find a good way to do that in the docs.
To run psql from the CLI you'll want syntax like:
psql -d postgresql://\<user>:\<password>#\<AlloyDB IP>:5432/postgres \
-c "DROP DATABASE IF EXISTS \<dbname>"
Related
I am trying to restore my Postgresql database to AWS RDS. I think I am almost there. I can get a dump, and recreate the db locally, but I am missing the last step to restore it to AWS RDS.
Here is what I am doing:
I get my dump
$ pg_dump -h my_public dns -U myusername -f dump.sql myawsdb
I create a local db in my shell called test:
create database test;
I put the dump into my test db
$ psql -U myusername -d test -f dump.sql
so far so good.
I get an error: psql:dump.sql:2705: ERROR: role "rdsadmin" does not exist, but I think I can ignore it, because my db is there with all the content. (I checked with \list and \connect test).
Now I want to restore this dump/test to my AWS RDS.
Following this https://gist.github.com/syafiqfaiz/5273cd41df6f08fdedeb96e12af70e3b
I now should do:
pg_restore -h <host> -U <username> -c -d <database name> <filename to be restored>
But what is my filename and what is my database name?
I tried:
pg_restore -h mydns -U myusername -c -d myawsdbname test
pg_restore -h mydns -U myusername -c -d myawsdbname dump.sql
and a couple of more options that I don't recall.
Most of the times it tells me something like: pg_restore: [archiver] could not open input file "test.dump": No such file or directory
Or, for the second: input file appears to be a text format dump. Please use psql.
Can somone point me into the right direction? Help is very much appreciated!
EDIT: So I created a .dump file using $ pg_dump -Fc mydb > db.dump
Using this file I think it works. Now I get the error [archiver (db)] could not execute query: ERROR: role "myuser" does not exist
Command was: ALTER TABLE public.users_user_user_permissions_id_seq OWNER TO micromegas;
Can I ingore that?
EDIT2: I got rid of the error adding the flags--no-owner --role=mypguser --no-privileges --no-owner
Ok, since this is apparently useful to some I will post - to the best of what I remember - an answer to this. I will answer this more broadly and not too AWS-specific because a) I don't use this instance anymore and b) I also don't remember perfectly how I did this.
But I gained experience with PostreSQL and since AWS RDS was also just a postgres instance the steps should work quite similar.
Here are my recommended steps when restoring a postgreSQL DB instance:
Pull the backup in a .dump-format and not in .sql-format. Why? The file-size will be smaller and it is easier to restore. Do this with the following command:
pg_dump -h <your_public_dns_ending_with.rds.amazonaws.com> -U <username_for_your_db> -Fc <name_of_your_db> > name_for_your_backup.dump
Now you can restore the backup easily to any postgreSQL instance. In general I'd recommend to set up a fresh DB instance with a new username and new databasename. Let's say you have a DB that is called testname with superuser testuser. Then you can just do:
pg_restore --no-owner --no-privileges --role=testuser -d testname <your_backup_file.dump>
And that should restore your instance.
When restoring to AWS or to any remote postgreSQL instance you will have to specify the host with the -h-flag. So this might be something like:
pg_restore -h <your_public_dns_ending_with.rds.amazonaws.com> -p 5432 --no-owner --no-privileges --role=testuser -d testname <your_backup_file.dump>
If you have a DB-instance running on a remote linux server, the host will be be your remote IP-address (-h <ip_od_server>) and the rest will be the same.
I hope this helps. Any questions please comment and I'll try my best to help more.
I'm trying to import sql file in postgresql but unable to do. I tried pgAdmin 4 to restore sql file. I get the error (pg_restore: [archiver] input file does not appear to be a valid archive).
I've also tried to do this with the command prompt but unable to do.
If I do
D:\Program Files (x86)\PostgreSQL\9.1\bin>psql -h 127.0.0.1 -U postgres gorkha < D:/gorkha.sql
It returns
SET
SET
SET
SET
SET
CREATE EXTENSION
COMMENT
REVOKE
REVOKE
GRANT
GRANT
Create a DB like (Employees)
Lets say you have a employees.sql file which you want to restore
postgres is your username
AT command prompt you are in the same folder with employees.sql
Code sample is:
psql -U postgres -d Employees < employees.sql
Follow This Steps:
Open pgAdmin and open Servers -> PostgreSQL 14 -> Databases,
And Choose your DB and right click and click on the PSQL Tool
If you not set your binary path and you get this alert
and click File option in top of pgadmin & open preference
Click The file icon and set the following path and Save
Final Step Open your Sql File in VSCode or note pad and copy then paste it on the PSQL Tool in pgAdmin
Its Work For Me
If any Error and Convert SQL To PGSQL (more online tools are available)
and the copy paste in to PSQL Tool in pgAdmin
use this command to restore an archived backup using pg_restore
pg_restore -U <username> -d <dbname> -1 <filename>.sql
for text/sql based backup you can try
psql -U <username> -d <dbname> -1 -f <filename>.sql
Is there any native table level replication(Pratial replication) in postgres?
If no, I want to replicate only some tables from my primary to standby using slony
Could anyone help me in doing this?
I am referring this link:"http://www.slony.info/documentation/1.2/firstdb.html"
But they are saying there will be a pgbench tool in contrib module in postgres
server.I am unable to find any, and my current server version is postgres-9.5.
pgbench is a command in postgres that is included in the bin folder.
If the DB doesn't exist you have skipped steps in the documentation link you provided.
createdb -O $PGBENCHUSER -h $MASTERHOST $MASTERDBNAME
createdb -O $PGBENCHUSER -h $SLAVEHOST $SLAVEDBNAME
pgbench -i -s 1 -U $PGBENCHUSER -h $MASTERHOST $MASTERDBNAME
I could not get this tutorial to fully work myself, but the tables were connected and replicated initially.
I have a Django postgres db (v9.3.10) running on digital ocean and am trying to migrate it over to Amazon RDS (postgres v 9.4.5). The RDS is a db.m3.xlarge instance with 300GB. I've dumped the Digital Ocean db with:
sudo -u postgres pg_dump -Fc -o -f /home/<user>/db.sql <dbname>
And now I'm trying to migrate it over with:
pg_restore -h <RDS endpoint> --clean -Fc -v -d <dbname> -U <RDS master user> /home/<user>/db.sql
The only error I see is:
pg_restore: [archiver (db)] Error from TOC entry 2516; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension plpgsql
Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
Apart from that everything seems to be going fine and then it just grinds to a halt. The dumped file is ~550MB and there are a few tables with multiple indices, otherwise pretty standard.
The Read and Write IOPS on the AWS interface are near 0, as is the CPU, memory, and storage. I'm very new to AWS and know that the parameter groups might need tweaking to do this better. Can anyone advise on this or a better way to migrate a Django db over to RDS?
Edit:
Looking at the db users the DO db looks like:
Role Name Attr Member Of
<user> Superuser {}
postgres Superuser, Create role, Create DB, Replication {}
And the RDS one looks like:
Role Name Attr Member Of
<user> Create role, Create DB {rds_superuser}
rds_superuser Cannot login {}
rdsadmin ... ...
So it doesn't look like it's a permissions issue to me as <user> has superuser permissions in each case.
Solution for anyone looking:
I finally got this working using:
cat <db.sql> | sed -e '/^COMMENT ON EXTENSION plpgsql IS/d' > edited.dump
psql -h <RDS endpoint> -U <user> -e <dname> < edited.dump
It's not ideal for a reliable backup/restore mechanism but given it is only a comment I guess I can do without. My only other observation is that running psql/pg_restore to a remote host is slow. Hopefully the new database migration service will add something.
Considering your dumped DB file is of ~550MB, I think using the Amazon guide for doing this is the way out. I hope it helps.
Importing Data into PostgreSQL on Amazon RDS
I think it did not halt. It was just recreating indexes, foreign keys etc. Use pg_restore -v to see what's going on during the restore. Check the logs or redirect output to a file to check for any errors after import, as this is verbose.
Also I'd recommend using directory format (pg_dump -v -Fd) as it allows for parallel restore (pg_restore -v -j4).
You can ignore this ERROR: must be owner of extension plpgsql. This is only setting a comment on extension, which is installed by default anyway. This is caused by a peculiarity in RDS flavor of PostgreSQL, which does not allow to restore a database while connecting as postgres user.
I have a django 1.4 app with a populated postgres 9.1 database in development server locally. After successful deployment, I wanted to move the data from local to online database, so I used:
pg_dump -f dump.sql -Ox database
and then restored on the server with:
psql -1 -f dump.sql database
Now trying to login online to the website admin throws a "permission denied for relation django_session" exception. I've tried to dump the data with/without -Ox switch and all its combinations but without success. I am also dropping the database and recreating it from scratch on the server with the correct owner as set in settings.py.
If I run a normal syndb without a restore then everything works well.
Am I missing something here?
It turns out that you should grant explicit ownership of all objects in the database to the owner after restore. The owner is not a superuser. It's not enough to only set the owner at database creation time. The final solution for migration goes like this:
on the client:
pg_dump -f dump.sql -Ox database
on the server:
su postgres
dropdb database
createdb database -O user
psql database -f dump.sql
and then to set the privileges:
psql database -c "GRANT ALL ON ALL TABLES IN SCHEMA public to user;"
psql database -c "GRANT ALL ON ALL SEQUENCES IN SCHEMA public to user;"
psql database -c "GRANT ALL ON ALL FUNCTIONS IN SCHEMA public to user;"
Note that we could've run the sql command in psql console but this form is easily embeddable in scripts and such.
Try to do this from postgres user:
sudo su - postgres
pg_dump -f dump.sql -Ox database
Or just pass -U flag:
pg_dump -f dump.sql -Ox database -U postgres
Here's how I fixed mine. I saved myself a ton of a headache by simply changing the user to match the current logged in user of the destination server where the import will happen.
In my case, the imported db had a user of x (x was also the username for the machine it was running on), and the destination machine had a username of y, and a postgres user of y too.
Therefore, I simply changed the Database User and Password in my Django settings to match the destination machine's y user details.
Then did this:
$ sudo -u postgres psql
psql > GRANT ALL PRIVILEGES DATABASE ON mydb TO y;
Sipping some kool-aid now!