I am trying to fetch data from a big query table using BigQueryHook and I am facing this error
File "/home/abdul/etl-pipelines/venv/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2061, in run_query
raise ValueError("The project_id should be set")
ValueError: The project_id should be set
I have tried exporting the AIRFLOW_CON_BIGQUERY_DEFAULT environment variable still it didn't help. The way I did it.
export AIRFLOW_CONN_BIGQUERY_DEFAULT="google-cloud-platform://?extra__google_cloud_platform__project=<gcp_project_id>"
Here is my function
def get_data_from_bq(**kwargs):
hook = BigQueryHook(delegate_to=None, use_legacy_sql=False)
hook.run_query('SELECT max(created_date) FROM `dataset.table`')
Although I tried to put project_id as an argument in BigQueryHook it gave me Unexpected argument error
Tried the second way using the BigQueryHook object
hook.project_id = project_id
Got the same ValueError.
I went through every piece of documentation and still couldn't find any solution maybe I am missing something?
Airflow BigQueryHook Docs
I have a Django project I have been working on offline and now I have hosted it on Heroku and it works well on Heroku but fails on my local machine with this error.
File "/usr/lib/python3.9/os.py", line 679, in __getitem__
raise KeyError(key) from None
KeyError: 'DEBUG'
and I think it is because I used environment variables like this.
from boto.s3.connection import S3Connection
import os
DEBUG = S3Connection(os.environ['DEBUG'], os.environ['DEBUG'])
I also have a .env file in my root(project folder) with the environment variables like this.
export JWT_SECRET_KEY = "dfge..."
export DEBUG = 1
What is the right way to store the environment variables on my local machine?
I have local file secret.py added to .gitignore with all keys, env values needed:
#secret.py
DEBUG = 1
Then in settings.py:
# settings.py
try:
import secret
DEBUG = secret.DEBUG
except ModuleNotFoundError:
DEBUG = S3Connection(os.environ['DEBUG'], os.environ['DEBUG'])
I created environment variables for my django project within my pipenv virtual envronment bin/activate (linux) or scripts\activate(windows) file , i made necessary changes in settings file as well as exiting and re activating the virtual environment but im still getting a keyerror (I'm working on a windows machine)
variables in settings.py
SECRET_KEY = os.environ['SECRET_KEY']
EMAIL_HOST_PASSWORD = os.environ['EMAIL_HOST_PASSWORD']
evnvironment variables in virtualenv\scripts\activate file
export SECRET_KEY= "mysecretkey"
export EMAIL_HOST_PASSWORD= "mypassword"
error
File "C:\Users\Dell\.virtualenvs\team-272-SMES-Server-dSgdZ4Ig\lib\os.py", line 673, in __getitem__
raise KeyError(key) from None
KeyError: 'SECRET_KEY'
Make sure you have "SECRET_KEY" in your os.environ
Use this code to check if you have "SECRET_KEY" there:
import os
import pprint
# Get the list of user's
# environment variables
env_var = os.environ
# Print the list of user's
# environment variables
print("User's Environment variable:")
pprint.pprint(dict(env_var), width = 1)
You are probably missing "SECRET_KEY" in the environment variable list. You can add a variable:
# importing os module
import os
# Add a new environment variable
os.environ['GeeksForGeeks'] = 'www.geeksforgeeks.org'
source
On a Windows server, I recommend creating a JSON (or YAML) file with all your database and app secrets. I personally prefer JSON, so an example of one is
{
"SECRET_KEY": "...",
"MYSQL_DBUSER": "jon"
"MYSQL_PW": "..."
...
}
Then in your settings.py you should add something like
import json
with open("config.json") as config:
config = json.load(config)
Then to simply load in your project's secrets, index them by the variable name like
SECRET_KEY = config['SECRET_KEY']
On upgrading to Django 1.7 I'm getting the following error message from ./manage.py
$ ./manage.py
Traceback (most recent call last):
File "./manage.py", line 16, in <module>
execute_from_command_line(sys.argv)
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 427, in execute_from_command_line
utility.execute()
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 391, in execute
django.setup()
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/__init__.py", line 21, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/johnc/.virtualenvs/myproj-django1.7/local/lib/python2.7/site-packages/django/apps/registry.py", line 89, in populate
"duplicates: %s" % app_config.label)
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: foo
What's the problem and how do I resolve it?
The problem is that with the changes to apps in Django 1.7, apps are required to have a unique label.
By default the app label is the package name, so if you've got a package with the same name as one of your app modules (foo in this case), you'll hit this error.
The solution is to override the default label for your app, and force this config to be loaded by adding it to __init__.py.
# foo/apps.py
from django.apps import AppConfig
class FooConfig(AppConfig):
name = 'full.python.path.to.your.app.foo'
label = 'my.foo' # <-- this is the important line - change it to anything other than the default, which is the module name ('foo' in this case)
and
# foo/__init__.py
default_app_config = 'full.python.path.to.your.app.foo.apps.FooConfig'
See https://docs.djangoproject.com/en/1.7/ref/applications/#for-application-authors
I found simple solution for this. In my case following line is added twice under INSTALLED_APPS,
'django.contrib.foo',
Removed one line fixes the issue for me.
I had the same error - try this:
In INSTALLED_APPS, if you are including 'foo.apps.FooConfig', then Django already knows to include the foo app in the application, there is therefore no need to also include 'foo'.
Having both 'foo' and 'foo.apps.FooConfig' under INSTALLED_APPS could be the source of your problem.
Well, I created a auth app, and included it in INSTALLED_APP like src.auth (because it's in src folder) and got this error, because there is django.contrib.auth app also. So I renamed it like authentication and problem solved!
I got the same problem.
Here my app name was chat and in the settings.py , under installed apps i have written chat.apps.ChatConfig while i have already included the app name chat at the bottom. When i removed the chat.apps.ChatConfig mine problem was solved while migrations. This error may be due to the same instance that you might have defined you app name foo twice in the settings.py. I hope this works out!!
please check if anything is duplicated in INSTALLED_APPS of settings.py
This exception may also be raised if the name of the AppConfig class itself matches the name of another class in the project. For example:
class MessagesConfig(AppConfig):
name = 'mysite.messages'
and
class MessagesConfig(AppConfig):
name = 'django.contrib.messages'
will also clash even though the name attributes are different for each configuration.
In previous answer 'django.contrib.foo', was mentioned, but basically adding any app twice can cause this error just delete one (Django 3.0)
for me it was in settings.py
INSTALLED_APPS = [
...
'accounts.apps.AccountsConfig',
'accounts.apps.AccountsConfig',
...
]
just delete one of them
Basically this problem has been created due to duplication of name of installed app in the settings:
This is how I resolved the problem. In settings.py file:
Check the install app in the setting.py if the install app are duplicate
Error shown due to duplication of app name
Remove the duplicate name in the install file
After problem is resolved, you will see interface in your screen
For this I have created application name as polls instead of foo
As therefromhere said this is a new Django 1.7 feature which adds a kind of “app registry” where applications must be determined uniquely (and not only having different python paths).
The name attribute is the python path (unique), but the label also should be unique. For example if you have an app named 'admin', then you have to define the name (name='python.path') and a label which must be also unique (label='my admin' or as said put the full python path which is always unique).
Had same issue, read through the settings.py of the root folder, removed any INSTALLED APPS causing conflict... works fine. Will have to rename the apps names
Need to check in two file
1- apps.py
code something like
from django.apps import AppConfig
from django.utils.translation import ugettext_lazy as _
class ModuleConfig(AppConfig):
name = "ocb.module_name"
verbose_name = _("Module Name")
2 - init.py
code something like
default_app_config = "ocb.users.apps.ModuleConfig"
default_app_config is pointed to your apps.py's class name
in my case, in mysite settings.py , in INSTALLED_APPS array variable I put the name of the app twice by mistake.
I had almost the same issue.
```File "/Users/apples/.local/share/virtualenvs/ecommerce-pOPGWC06/lib/python3.7/site-packages/django/apps/registry.py", line 95, in populate
"duplicates: %s" % app_config.label)
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: auth```
I had installed Django.contrib.auth twice. I removed one and it worked well.
From my experience, this exception was masking the real error. To see the real error (which in my case was an uninstalled python package) comment out the following in django/apps/registry.py:
if app_config.label in self.app_configs:
# raise ImproperlyConfigured(
# "Application labels aren't unique, "
# "duplicates: %s" % app_config.label)
pass
Check for duplicates in INSTALLED_APPS inside the settings.py...If so remove one of it and rerun the command
I had Django==3.2.9 when tried to test my existing Django app on a new environment. I had this exact issue and fixed it by downgrading to Django==3.1.13.
There seems to be an update to applications, check the Django 3.2 documentation for detailed information.
This error occurs because of duplication in your INSTALLED_APPS in settings.py file which is inside your project.
For me, the problem was that I had copy-pasted entire app instead of creating it using command line. So, the app name in the apps.py file was same for 2 apps. After I corrected it, the problem was gone.
In case if you have added your app name in settings.py
example as shown in figure than IN settings.py Remove it and Try this worked for me.
give it a try .
This Worked Because settings.py assumes installing it twice and does not allow for migration
If you want to back older version, command
pip install django==1.6.7
When trying to use loaddata on my local machine (win/sqlite):
python django-admin.py loaddata dumpdata.json
I get the following error:
raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE) ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
I am using djangoconfig app if that helps:
"""
Django-config settings loader.
"""
import os
CONFIG_IDENTIFIER = os.getenv("CONFIG_IDENTIFIER")
if CONFIG_IDENTIFIER is None:
CONFIG_IDENTIFIER = 'local'
# Import defaults
from config.base import *
# Import overrides
overrides = __import__(
"config." + CONFIG_IDENTIFIER,
globals(),
locals(),
["config"]
)
for attribute in dir(overrides):
if attribute.isupper():
globals()[attribute] = getattr(overrides, attribute)
projects>python manage.py loaddata dumpdata.json --settings=config.base
WARNING: Forced to run environment with LOCAL configuration.
Problem installing fixture 'dumpdata.json': Traceback
(most recent call last):
File "loaddata.py", line 174, in handle
obj.save(using=using)
...
File "C:\Python26\lib\site-packages\django\db\backends\sqlite3\base.py", line
234, in execute
return Database.Cursor.execute(self, query, params)
IntegrityError: columns app_label, model are not unique
Don't use django-admin.py for anything other than setting up an initial project. Once you have a project, use manage.py instead - it sets up the reference to your settings file.
syncdb will load content_types, you need to clear that table before loading data. Something like this:
c:\> sqlite3 classifier.db
sqlite> delete from django_content_type;
sqlite> ^Z
c:\> python django-admin.py loaddata dumpdata.json
Also, make sure you do not create a superuser, or any user, when you syncdb, as those are likely to also collide with your data fixture ...
There are two standard ways to provide your settings to Django.
Using set (or export on Unix) set DJANGO_SETTINGS_MODULE=mysite.settings
Alternatively as an option with django-admin.py --settings=mysite.settings
Django-config does things differently because it allows you to have multiple settings files. Django-config works with manage.py to specify which to use. You should use manage.py whenever possible; it sets up the environment. In your case try this where --settings points to the specific .py file you want to use from django-config's config folder.
django-admin.py loaddata dumpdata.json --settings=<config/settings.py>
Actually --settings wants python package syntax so maybe <mysite>.config.<your settings>.py