How to apply Superset Migrations when running in docker mode? - apache-superset

Currently I'm running Superset in Docker mode. No native installation. The metadata database is an external(non-docker) Postgres DB which has lots of Dashboards, Charts etc.
Current installation is running on git tag 1.0.0. I want to upgrade to v1.1.0. I can do this by switching the repo to git tag 1.1.0 and restarting docker containers.
However as per UPDATING.md notes, v1.1.0 has a DB migration .
In native installation, the way to migrate DB is superset db upgrade
What's the proper method to apply these migration scripts to an existing external database in docker installation?

Lauch you stack if done with compose it will run automaticly the db upgrade command.
if not docker exec -it <supersetcontainerID> /bin/bash
Just ensure to have the correct sqlalchemy chain on the superset config file.
And then fire the superset db upgrade
You're done.

First, chekout your container id ,then use below command to backup superset.db
docker cp 1263b3cdf7e7:/root/.superset/superset.db
Then after the upgrade,you can just simple cp and replace superset.db back into your new version

Related

How to change the setting of apache superset from sqlite to MySQL

Can anyone provide a steps to Change the setting of apache superset from sqlite to MySQL?
I have create superset_config.py to override the configuration
after adding the property i am able to enabled swagger url
I have added SQLALCHEMY_DATABASE_URI = 'mysql://root:xxxxxx#127.0.0.1:3306/superset' property in superset_config.py file
but still it is connecting with SQLlite.
I wrote article that can help you.
I had problem using docker compose and native installation . Port is closed can be due to networking problem or problem with packages. Host.docker.internal doesn’t worked for me on Ubuntu 22. I would like to recommend to not follow official doc and use better approach with single docker image to start. Instead of running 5 containers by compose, run everything in one. Use official docker image, here image. Than modify docker file as follows to install custom db driver:
FROM apache/superset
USER root
RUN pip install mysqlclient
RUN pip install sqlalchemy-redshift
USER superset
Second step is to build new image based on docker file description. To avoid networking problems start both containers on same network (superset, your db) easier is to use host network. I used this on Google cloud example as follow:
docker run -d --network host --name superset supers
The same command to start container with your database. —network host. This solved my problems. More about in whole step to step tutorial: medium or here blog

docker context create ecs myecs - requires exactly one argument

I'm trying to create a Docker context that will automatically integrate with AWS's ECS.
I'm following this tutorial
The author just does:
docker context create ecs myecs and gets a "pick an integration" prompt, whereas I get an error saying it needs exactly 1 argument.
docker context create" requires exactly 1 argument.
See 'docker context create --help'.
Usage: docker context create [OPTIONS] CONTEXT
Create a context
You need to install the Docker Compose CLI preview
The below curl is from here: Docker docs
curl -L https://raw.githubusercontent.com/docker/compose-cli/main/scripts/install/install_linux.sh | sh
sudo docker context create ecs myecs
It didn't work without sudo for me for some reason.
After the script finished I had some weird errors:
cp: cannot stat '/tmp/tmp.d4QjhW8T6k/docker-compose': No such file or directory and docker context create ecs myecs didn't work at first, but once I tried with sudo it worked fine.
EDIT: . ~/.zshrc (or just close your terminal and open a new one) made it possible for me to run docker context create ecs myecs without sudo.
Author of the blog/tutorial here. It looks like you don't have the pre-requsite installed. In the blog I call out the pre-req in pieces like this.
....In July, Docker released a beta for Docker Desktop that embedded these functionalities and, on September 15th, Docker released an updated experience in their Docker Desktop stable channel....
and then
...For now the only thing you need is Docker Desktop and an AWS account. For this test , I am using Docker Desktop (stable) version 2.5.0.1....
and finally
The core of this integration is built around a new tool dubbed Compose CLI (this is not to be confused with the original docker-compose CLI). This new CLI surfaces to the user as new functionalities in the docker command. While in Docker Desktop all this plumbing is completely hidden and available out of the box, if you are using a Linux machine you can set it up using either a script or a manual install. This new CLI is, essentially, a new version of the docker binary.
Eager to understand more how we could make it more clear / front and center that there were stuff to install and/or minimum software versions you had to use.
Thanks for trying it out!
If you're on Linux and you're running the docker context create ecs myecscontext command from the docs then try enabling experimental features in docker:
Edit /etc/docker/daemon.json
Set contents to
{
"experimental": true
}
Restart docker service sudo systemctl restart docker
Exit your terminal and open a new one so that the changes take effect.
Source1
Source2
I had same issue but after installing Docker Desktop version problem resolved.
Server side version doesn't have such kind of functionality.

Running postgres 9.5 and django in one container for CI (Bamboo)

I am trying to configure a CI job on Bamboo for a Django app, the tests to be run rely on a database (postgres 9.5). It seems that a prudent way to go about is it run the whole test in a docker container, as I do not control the agent environment so I cannot install Postgres there.
Most guides I found recommend running postgres and django in two separate containers and using docker-compose to easily manage them. In this scenario each docker image runs just one service, started with CMD. In Bamboo I cannot use docker-compose however, I need to use just one image, so I am trying to get Postgres and Django to run nicely together in one container but with little success so far.
My problem is that I see no easy way to start Postgres as a service inside docker but NOT as a docker CMD command, official postgre image uses an entrypoint.sh approach, also described in the official docker docs
But it is not clear to me how to implement that. I would appreciate your help!
Well, basically you would start postgres as a background process in the docker-entrypoint shell script that does otherwise start your django application.
The only trick here is that you need to put a 'trap' command in it so that you can send a shutdown/kill to the background process when your master process stops.
Although I have done that a thousand times, I know that it is a good source for programming errors. In general I do just use my docker-systemctl-replacement which takes care of running multiple applications as services, just as if the container is a virtual machine hosting multiple applications.
Your only other option is to add in a startup script in your Dockerfile, or kick it off as part of your docker run ... commands. We don't generally use the "Docker" tasks, as I find them ... distasteful (also why I usually just fall back to running a "Script" task, and directly calling docker run in that script task)
Anyway, you'd have to have your Docker container execute a script that would:
Start up Postgres (like a sudo systemctl start postgresql)
Execute your tests.
Your Dockerfile will have to install Postgresql and do some minor setup work I imagine (like create relevant users and databases with the proper owner). Since we're all good citizens, we remember to never run your containers as root, right?
Note - you can always hack around getting two containers to talk to each other without using docker-compose. It's a bit less convenient, but you could do something like:
docker run --detach --cidfile=db_cidfile --name ci_db postgresql_image
...
docker run --link ci_db testing_image
Make sure that you EXPOSE the right ports on the postgresql image to the testing_image container.
EDIT: I'm looking more at my specific case - we just install Postgresql into a base CentOS host rather than use the postgresql default image (using yum install http://yum.postgresql.org/..../pgdg-centos...rpm and then just install postgresql-server and postgresql-contrib packages from there). There is a CMD [ "/usr/pgsql-ver/bin/postgres", "-D", "/var/lib/pgsql/ver/data"] in our Dockerfile, too. We don't do anything fancy with the docker container, though. NOTE: we don't use this in production at all, this is strictly for local and CI testing.

How do I update Django on Openshift?

I'm learning to deploy Django on Openshift.
Right now I have a python-2.7 cartridge up and running with Django 1.6
The git repo cloned in the cartridge is,
git://github.com/rancavil/django-openshift-quickstart.git (Github)
How can I update the Django version of a running webapp?
I've looked at this question that just explain about updating a cartridge, while I'm asking about updating the packages inside a cartridge while keeping the cartridge same as python-2.7.
The easiest way to achieve this is to change the setup dependencies (install_requires parameter for setup ()) in setup.py. Instead of
packages = ['Django<=1.6',]
as in the cartridge default you could write
packages = ['Django>=1.7,<1.8',]
to get the latest version of Django 1.7. More details of how to specify values can be found in the Python Packaging User Guide.
With your next git push this file will be executed and the packages get updated, if required.
Warnings!
make sure new version is ok for your app. Django 1.7 brought DB migrations feature, which might break your compatibility. (We had some issues as we used South before that.)
before applying upgrade backup the app instance snapshot (takes time)
Actually git push takes some time while your application will be down.
If you want to shorten the time, you can follow this approach:
ssh into your app openshift server
pip install --upgrade Django==<new version>
That will upgrade django immediately. However the running web process still keeps the older version. So you need to restart python cartridge.
From you local command line:
rhc cartridge restart -a <your app> -c python
Now its running with the new django and the downtime is minimal.
Make sure to update setup.py as mentioned in the other answer in order to be aligned with the next git push.

Installing Django, PostgreSQL on Google Compute Engine Debian 7 Instance

I am trying to deploy a Django application on Google Compute Engine. I'm using a Debian 7 image and want to set up Django with Nginx, Gunicorn, virtualenv, supervisor and PostgreSQL. I have everything running fine on my development machine which is running Ubuntu 14.04 with Django installed and PostgreSQL as the backend.
I'm using the tutorial located at http://datacommunitydc.org/blog/2013/12/a-tutorial-for-deploying-a-django-application-that-uses-numpy-and-scipy-to-google-compute-engine-using-apache2-and-modwsgi/. I'm also using the tutorial located at http://michal.karzynski.pl/blog/2013/06/09/django-nginx-gunicorn-virtualenv-supervisor/ as it's specific to virtualenv and PostgreSQL which I'm using on my development machine. I've setup my GCE instance, instaled and updated aptitude. I've installed PostgreSQL however when I attempt to create a database user and a new database for the app I get an error and nothing is created.
Following the tutorial I've run:
$ sudo su - postgres
postgres#django:~$ createuser -P
Enter name of role to add: hello_django
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
When it attempts to create the new user role I receive the following error:
createuser: could not connect to database postgres: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
When I run the shell script ls /etc/init.d it says that postgresql is running, but I still can't add the new role. Can someone tell me what I'm doing wrong?
Regards.
I wasn't able to reproduce the issue on my end, but it appears to be an issue with PostgreSQL and its dependencies. You can try removing all installed PostgreSQL components and dependencies and then reinstalling PostgreSQL:
sudo apt-get remove --purge postgresql-9.1*
sudo apt-get install postgresql-9.1
If it's still unable to connect to the database, the issue might be originating from your $PATH, in which case you'll need to point it to /usr/local/bin/psql.
I have just had the same problem.
This is most likely cause the postgres cluster has not been initialised yet.
And the reason that this didn't install automatically is because you have set up the locale of the box yet. This is something that has to be done on Amazon EC2 instances as well.
You need to run:
sudo apt-get install locales
And then:
sudo dpkg-reconfigure locales
I had to choose which locales I wanted to setup, I chose en_AU UTF-8.
After this I rebooted, then I could run this to initialise the cluster:
sudo pg_createcluster 9.1 main --start
This started the service and created the pg_hba.conf files etc.
I faced a similar problem a while back. It can resolved using a few simple steps:
As postgres user run : initdb --locale en_US.UTF-8 -E UTF8 -D 'var/lib/postgres/data'. Note depending on the distro postgres in the command can be pgsql. You can easily check if the directory exists with an ls command.
systemctl start postgresql (if you have systemd) or just a /etc/init.d/postgresql start should do. These commands must be rub as the superuser.
All this is from the ArchWiki.