Related
I have an onpremise CDAP data fusion instance with multiple namespaces. How to trigger the pipeline using airflow operators? I have tried exploring the airflow available operators and this page but not very helpful https://cloud.google.com/data-fusion/docs/reference/cdap-reference#start_a_batch_pipeline
Assuming you already deployed the pipeline and you have the location, instance name and pipeline name of the pipeline you want to run. See CloudDataFusionStartPipelineOperator() for the parameters that it accepts.
Using the quickstart pipeline, I triggered the pipeline using CloudDataFusionStartPipelineOperator(). See operator usage below:
import airflow
from airflow.providers.google.cloud.operators.datafusion import CloudDataFusionStartPipelineOperator
YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)
default_args = {
'owner': 'Composer Example',
'depends_on_past': False,
'email': [''],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': datetime.timedelta(minutes=5),
'start_date': YESTERDAY,
}
with airflow.DAG(
'trigger_df',
'catchup=False',
default_args=default_args,
schedule_interval=datetime.timedelta(days=1)) as dag:
start_pipeline = CloudDataFusionStartPipelineOperator(
location='us-central1',
pipeline_name='DataFusionQuickstart',
instance_name='test',
task_id="start_pipeline",
)
start_pipeline
Success "Graph View":
Logs:
I'm trying to send an email using SenDGrid but the DAG is stuck on running.
I did the following:
set the environment variable SENDGRID_MAIL_FROM as my email
set the environment variable SENDGRID_API_KEY as the api I've generated from Sendgrid after confirming my personal email (same as sender email).
No spam im my email inbox.
Nothing found in the Activity section on SendGrid page and nothing is sent.
Can someone maybe point out what am I doing wrong?
My code:
from airflow.models import (DAG, Variable)
import os
from airflow.operators.email import EmailOperator
from datetime import datetime,timedelta
default_args = {
'start_date': datetime(2020, 1, 1),
'owner': 'Airflow',
"email_on_failure" : False,
"email_on_retry" : False,
"emails" : ['my#myemail.com']
}
PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "bigquery_default")
PROJECT_ID_GCP = os.environ.get("GCP_PROJECT_ID", "my_progect")
with DAG(
'retries_test',
schedule_interval=None,
catchup=False,
default_args=default_args
) as dag:
send_email_notification = EmailOperator(
task_id = "send_email_notification",
to = "test#sendgrid.com",
subject = "test",
html_content = "<h3>Hello</h3>"
)
send_email_notification
I have a Document model with a FileField for uploading images, pdfs, etc., and ImageFields for eight different images sizes created from the single uploaded file. I convert the uploaded image into 8 different sizes using Pillow. I allow multiple file uploads in the DocumentAdmin, and in save_model I create a thumbnail and large image for display in the DocumentAdmin page, and then use a celery task (with redis) to create the other different image sizes in the background and save them to the Document model.
My code creates the different images correctly, but I seem to have a strange situation in the celery task. The celery task uses the same code to create the different image sizes as the code used in save_model to create the thumb and large image.
The relevant part of the save_model code:
def save_model(self, request, obj, form, change):
if form.is_valid():
if not change:
# Uploading one or more images
files = request.FILES.getlist('storage_file_name')
if files:
for f in files:
with transaction.atomic():
original_file_name, extension = os.path.splitext(f.name)
new_file_name = original_file_name + "." + settings.DEFAULT_IMAGE_EXTENSION
obj2 = Document()
# save the original file
if extension.lower() == ".pdf":
obj2.pdf_file = f
else:
obj2.storage_file_name = f
save_document_images(obj2, new_file_name, image_file=f, rot_angle=0, thumbnail=True, xsmall=False, small=False, medium=False, large=True, xlarge=False, xxlarge=False, xxxlarge=False)
obj2.save()
logger.debug("Starting transaction for document-id=%s, new_file_name=%s" % (obj2.document_id, new_file_name))
transaction.on_commit(lambda: tasks.create_additional_images_task.apply_async(args=[obj2.document_id, new_file_name,], kwargs={'rot_angle':0, 'thumbnail': False, 'xsmall':True, 'small':True, 'medium':True, 'large':False, 'xlarge':True, 'xxlarge':True, 'xxxlarge':True}))
And the code for the celery task:
#app.task(bind=True)
def create_additional_images_task(self, *args, **kwargs):
from memorabilia.models import Document
obj2 = Document.objects.get(document_id=args[0])
image_file = obj2.storage_file_name
clogger.debug("document_id=%s, obj2=%s, new_file_name=%s, image_file=%s" % (args[0], obj2, args[1], image_file.name))
for key in kwargs:
clogger.debug("kwargs: key=%s, value=%s" % (key, kwargs[key]))
save_document_images(obj2, args[1], image_file, kwargs['rot_angle'], kwargs['thumbnail'], kwargs['xsmall'], kwargs['small'], kwargs['medium'], kwargs['large'], kwargs['xlarge'], kwargs['xxlarge'], kwargs['xxxlarge'])
For example, I upload three different images. When I look at the Django logs, I see the celery task is called three times with three different document_ids and image names:
[2020-04-19 17:10:18] DEBUG [memorabilia.admin.save_model:1739] Starting transaction for document-id=110, new_file_name=image-1.jpg
[2020-04-19 17:10:19] DEBUG [memorabilia.admin.save_model:1739] Starting transaction for document-id=111, new_file_name=image-2.jpg
[2020-04-19 17:10:20] DEBUG [memorabilia.admin.save_model:1739] Starting transaction for document-id=112, new_file_name=image-3.jpg
When I look at the celery log, I see the following:
[2020-04-19 17:10:20,665: INFO/MainProcess] Received task: memorabilia.tasks.create_additional_images_task[ec7699d8-7b73-4004-bb78-9371bcf0e1d6]
[2020-04-19 17:10:20,666: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f85ce09ae18> (args:('memorabilia.tasks.create_additional_images_task', 'ec7699d8-7b73-4004-bb78-9371bcf0e1d6', {'lang': 'py', 'task': 'memorabilia.tasks.create_additional_images_task', 'id': 'ec7699d8-7b73-4004-bb78-9371bcf0e1d6', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ec7699d8-7b73-4004-bb78-9371bcf0e1d6', 'parent_id': None, 'argsrepr': "[112, 'image-3.jpg']", 'kwargsrepr': "{'rot_angle': 0, 'thumbnail': False, 'xsmall': True, 'small': True, 'medium': True, 'large': False, 'xlarge': True, 'xxlarge': True, 'xxxlarge': True}", 'origin': 'gen18286#tsunami', 'reply_to': '9dc88644-e203-30ff-a643-a14314f2572a', 'correlation_id': 'ec7699d8-7b73-4004-bb78-9371bcf0e1d6', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[112, "image-3.jpg"], {"rot_angle": 0, "thumbnail": false, "xsmall": true, "small": true, "medium": true, "large": false, "xlarge": true, "xxlarge": true, "xxxlarge": true}, {"callbacks": null,... kwargs:{})
[2020-04-19 17:10:20,667: INFO/MainProcess] Received task: memorabilia.tasks.create_additional_images_task[eeb2ac99-ae1f-4530-9e41-fe923cfba912]
[2020-04-19 17:10:20,667: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f85ce09ae18> (args:('memorabilia.tasks.create_additional_images_task', 'eeb2ac99-ae1f-4530-9e41-fe923cfba912', {'lang': 'py', 'task': 'memorabilia.tasks.create_additional_images_task', 'id': 'eeb2ac99-ae1f-4530-9e41-fe923cfba912', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'eeb2ac99-ae1f-4530-9e41-fe923cfba912', 'parent_id': None, 'argsrepr': "[112, 'image-3.jpg']", 'kwargsrepr': "{'rot_angle': 0, 'thumbnail': False, 'xsmall': True, 'small': True, 'medium': True, 'large': False, 'xlarge': True, 'xxlarge': True, 'xxxlarge': True}", 'origin': 'gen18286#tsunami', 'reply_to': '9dc88644-e203-30ff-a643-a14314f2572a', 'correlation_id': 'eeb2ac99-ae1f-4530-9e41-fe923cfba912', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[112, "image-3.jpg"], {"rot_angle": 0, "thumbnail": false, "xsmall": true, "small": true, "medium": true, "large": false, "xlarge": true, "xxlarge": true, "xxxlarge": true}, {"callbacks": null,... kwargs:{})
[2020-04-19 17:10:20,668: INFO/MainProcess] Received task: memorabilia.tasks.create_additional_images_task[04cb7f54-285d-4e0f-86e0-aaa2995d1cb4]
[2020-04-19 17:10:20,669: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f85ce09ae18> (args:('memorabilia.tasks.create_additional_images_task', '04cb7f54-285d-4e0f-86e0-aaa2995d1cb4', {'lang': 'py', 'task': 'memorabilia.tasks.create_additional_images_task', 'id': '04cb7f54-285d-4e0f-86e0-aaa2995d1cb4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '04cb7f54-285d-4e0f-86e0-aaa2995d1cb4', 'parent_id': None, 'argsrepr': "[112, 'image-3.jpg']", 'kwargsrepr': "{'rot_angle': 0, 'thumbnail': False, 'xsmall': True, 'small': True, 'medium': True, 'large': False, 'xlarge': True, 'xxlarge': True, 'xxxlarge': True}", 'origin': 'gen18286#tsunami', 'reply_to': '9dc88644-e203-30ff-a643-a14314f2572a', 'correlation_id': '04cb7f54-285d-4e0f-86e0-aaa2995d1cb4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[112, "image-3.jpg"], {"rot_angle": 0, "thumbnail": false, "xsmall": true, "small": true, "medium": true, "large": false, "xlarge": true, "xxlarge": true, "xxxlarge": true}, {"callbacks": null,... kwargs:{})
Three tasks are created (1d6, 912, cb4 - to use the last three digits of the task id). However, all three tasks have as their arguments, [112, "image-3.jpg"] (document_id=112, file_name=image-3jpg) When I look on the file system, I see that the images the celery task should have made for document_ids 110 and 111 have not been made, but the images for document_id 112 has been made. The thumbnail and large image for all three document_ids have been created, so the code for making the images appears to be working.
What am I doing wrong in my code? I must confess, this is my first foray into using celery, so I am a bit lost.
Thanks!
Mark
1) I'd say you're using on_commit improperly. According to documentation it should have been like
for f in files:
with transaction.atomic():
transaction.on_commit(...)
Otherwise how should Django know which transaction those on_commit are addressed? So you added three images, save them, and that is the one transaction with three listeners.
2) Please do not use same import in for statements. Just move it higher before for
I have a strange issue with django ldap backend.
this is my configuration:
AUTHENTICATION_BACKENDS = (
'django_auth_ldap.backend.LDAPBackend',
'django.contrib.auth.backends.ModelBackend'
)
AUTH_LDAP_SERVER_URI = "ldap://XXXXXXx"
AUTH_LDAP_BIND_DN = "CN=Pti Release,OU=Service-Accounts,OU=DD,DC=ad,DC=como,DC=com"
AUTH_LDAP_BIND_PASSWORD = "XXXXXXXX"
AUTH_LDAP_USER_SEARCH = LDAPSearch("OU=DD,DC=ad,DC=como,DC=com", ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)")
AUTH_LDAP_GLOBAL_OPTIONS = {
ldap.OPT_X_TLS_REQUIRE_CERT: False,
}
AUTH_LDAP_GROUP_SEARCH = LDAPSearch("OU=DD,DC=ad,DC=como,DC=com", ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)")
AUTH_LDAP_USER_FLAGS_BY_GROUP = {
"is_superuser" : "CN=airflow-super-users_GLOBAL,OU=Global,OU=Security-Groups,OU=DD,DC=ad,DC=como,DC=com",
"is_active" : "CN=airflow-data-profilers_GLOBAL,OU=Global,OU=Security-Groups,OU=DD,DC=ad,DC=como,DC=com",
}
AUTH_LDAP_GROUP_TYPE = GroupOfNamesType()
AUTH_PROFILE_MODULE = 'main.UserProfile'
AUTH_LDAP_USER_ATTR_MAP = {
"first_name": "givenName",
"last_name": "sn",
"email": "mail",
"country": "c"
}
AUTH_LDAP_ALWAYS_UPDATE_USER = True
AUTH_LDAP_CACHE_GROUPS = True
AUTH_LDAP_GROUP_CACHE_TIMEOUT = 3600
When I attempt to login, the django auth_user is populated correctly but django shows me this form:
Please enter the correct username and password for a staff account. Note that both fields may be case-sensitive.
And no way to login.
Any ideas?
I'm adding here more information which can help to debug this. I know yo will notice VariableDoesNotExist: Failed lookup for key [is_popup] but it seems not be an error Getting error with is_popup variable in django 1.9. I'm using these packages have been installed into the django virtualenv:
appdirs==1.4.3
coreapi==2.3.0
coreschema==0.0.4
Django==1.11
django-auth-ldap==1.2.11
django-crispy-forms==1.6.1
django-enumchoicefield==0.8.0
django-enumfields==0.9.0
django-filter==1.0.2
django-filters==0.2.1
django-mysql==1.1.1
djangorestframework==3.6.2
drf-enum-field==0.9.1
enum==0.4.6
enum34==1.1.6
itypes==1.1.0
Jinja2==2.9.6
MarkupSafe==1.0
MySQL-python==1.2.5
packaging==16.8
pyparsing==2.2.0
python-ldap==2.4.38
pytz==2017.2
requests==2.13.0
six==1.10.0
uritemplate==3.0.0
Database before the login attempt
mysql> select * from auth_user\G
*************************** 1. row ***************************
id: 1
password: pbkdf2_sha256$36000$giZoAk1hhuFG$aSJ1w6nOs05VjecVnFVFVYTK2h84mqHqDTKg5St6BBE=
last_login: 2017-05-04 08:28:05.580156
is_superuser: 1
username: root
first_name:
last_name:
email:
is_staff: 1
is_active: 1
date_joined: 2017-04-10 13:27:46.202097
django logs ( DEBUG=True, output to console )
DEBUG search_s('OU=DD,DC=ad,DC=como,DC=com', 2, '(sAMAccountName=%(user)s)') returned 1 objects: cn=alessio palma,ou=hr,ou=florence,ou=italy,ou=dd,dc=ad,dc=como,dc=com
DEBUG (0.000) SELECT ##SQL_AUTO_IS_NULL; args=None
DEBUG (0.000) SELECT VERSION(); args=None
DEBUG (0.002) SELECT `auth_user`.`id`, `auth_user`.`password`, `auth_user`....
DEBUG (0.001) INSERT INTO `auth_user` (`password`, `last_login`, `is_superuser`...
DEBUG (0.001) INSERT INTO `authtoken_token` (`key`, `user_id`, `created`) VALUES ('91567e9aece75cd7fc97809b8685267fc53bc983', 7, '2017-05-05 06:21:15.359687'); args=(u'91567e9aece75cd7fc97809b8685267fc53bc983', 7, u'2017-05-05 06:21:15.359687')
DEBUG Created Django user alessio.palma
DEBUG Populating Django user alessio.palma
DEBUG cn=alessio palma,ou=hr,ou=florence,ou=italy,ou=dd,dc=ad,dc=como,dc=com is a member of cn=airflow-super-users_global,ou=global,ou=security-groups,ou=dd,dc=ad,dc=como,dc=com
DEBUG cn=alessio palma,ou=hr,ou=florence,ou=italy,ou=dd,dc=ad,dc=como,dc=com is a member of cn=airflow-data-profilers_global,ou=global,ou=security-groups,ou=dd,dc=ad,dc=como,dc=com
DEBUG (0.001) UPDATE `auth_user` SET `password` = '!QBPYrozr5C3wMRneLcBJeiWmaE4YiTizNgJURGTQ', `last_login` = NULL, `is_superuser` = 1, `username` = 'alessio.palma', `first_name` = 'Alessio', `last_name` = 'Palma', `email` = 'xxxxx#como.com', `is_staff` = 0, `is_active` = 1, `date_joined` = '2017-05-05 06:21:15.357377' WHERE `auth_user`.`id` = 7; args=(u'!QBPYrozr5C3wMRneLcBJeiWmaE4YiTizNgJURGTQ', True, u'alessio.palma', u'Alessio', u'Palma', u'alessio.palma#docomodigital.com', False, True, u'2017-05-05 06:21:15.357377', 7)
DEBUG Exception while resolving variable 'is_popup' in template 'admin/login.html'.
Traceback (most recent call last):
File "/home/room/.virtualenvs/django/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
(bit, current)) # missing attribute
VariableDoesNotExist: Failed lookup for key [is_popup] in u"[{'False': False, 'None': None, 'True': True}, {u'csrf_token': <SimpleLazyObject: <function _get_val at 0x7f5af0051d70>>, 'user': <SimpleLazyObject: <django.contrib.auth.models.AnonymousUser object at 0x7f5af83a7cd0>>, 'perms': <django.contrib.auth.context_processors.PermWrapper object at 0x7f5af005e690>, 'DEFAULT_MESSAGE_LEVELS': {'DEBUG': 10, 'INFO': 20, 'WARNING': 30, 'SUCCESS': 25, 'ERROR': 40}, 'messages': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x7f5af8484c10>, u'request': <WSGIRequest: POST '/admin/login/?next=/admin/'>}, {}, {'username': u'', 'app_path': u'/admin/login/?next=/admin/', 'available_apps': [], 'site_name': '172.17.0.2:8000', 'form': <AdminAuthenticationForm bound=True, valid=False, fields=(username;password)>, 'title': u'Log in', 'site_header': u'Django administration', 'site': <django.contrib.sites.requests.RequestSite object at 0x7f5af84b6d50>, 'next': u'/admin/', 'site_title': u'Django site admin', u'LANGUAGE_CODE': 'en-us', 'has_permission': False, 'site_url': '/', u'LANGUAGE_BIDI': False, u'view': <django.contrib.auth.views.LoginView object at 0x7f5af83a7e10>}]"
DEBUG Exception while resolving variable 'is_popup' in template 'admin/login.html'.
Traceback (most recent call last):
File "/home/room/.virtualenvs/django/lib/python2.7/site-packages/django/template/base.py", line 903, in _resolve_lookup
(bit, current)) # missing attribute
VariableDoesNotExist: Failed lookup for key [is_popup] in u"[{'False': False, 'None': None, 'True': True}, {u'csrf_token': <SimpleLazyObject: <function _get_val at 0x7f5af0051d70>>, 'user': <SimpleLazyObject: <django.contrib.auth.models.AnonymousUser object at 0x7f5af83a7cd0>>, 'perms': <django.contrib.auth.context_processors.PermWrapper object at 0x7f5af005e690>, 'DEFAULT_MESSAGE_LEVELS': {'DEBUG': 10, 'INFO': 20, 'WARNING': 30, 'SUCCESS': 25, 'ERROR': 40}, 'messages': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x7f5af8484c10>, u'request': <WSGIRequest: POST '/admin/login/?next=/admin/'>}, {}, {'username': u'', 'app_path': u'/admin/login/?next=/admin/', 'available_apps': [], 'site_name': '172.17.0.2:8000', 'form': <AdminAuthenticationForm bound=True, valid=False, fields=(username;password)>, 'title': u'Log in', 'site_header': u'Django administration', 'site': <django.contrib.sites.requests.RequestSite object at 0x7f5af84b6d50>, 'next': u'/admin/', 'site_title': u'Django site admin', u'LANGUAGE_CODE': 'en-us', 'has_permission': False, 'site_url': '/', u'LANGUAGE_BIDI': False, u'view': <django.contrib.auth.views.LoginView object at 0x7f5af83a7e10>}]"
INFO "POST /admin/login/?next=/admin/ HTTP/1.1" 200 1819
Database after the login attempt
*************************** 1. row ***************************
id: 1
password: pbkdf2_sha256$36000$giZoAk1hhuFG$aSJ1w6nOs05VjecVnFVFVYTK2h84mqHqDTKg5St6BBE=
last_login: 2017-05-04 08:28:05.580156
is_superuser: 1
username: root
first_name:
last_name:
email:
is_staff: 1
is_active: 1
date_joined: 2017-04-10 13:27:46.202097
*************************** 2. row ***************************
id: 7
password: !QBPYrozr5C3wMRneLcBJhiWmaE4YiTizNgJURGTQ
last_login: NULL
is_superuser: 1
username: alessio.palma
first_name: Alessio
last_name: Palma
email: xxxxxx#como.com
is_staff: 0
is_active: 1
date_joined: 2017-05-05 06:21:15.357377
2 rows in set (0,00 sec)
HELP! I think all this problem is the usual comma out of place.
I found the problem.
When you set the ldap authentication, users are loaded into the django's auth_user table and is_staff is set to 0. So you can't login also if you are a valid ldap user.
Next step is to login as admin or update the auth_user table setting is_staff to 1 to every user which has to login.
After this update, you can access django using you ldap account.
AUTH_LDAP_USER_SEARCH simply didn't find a matching user. If it's Active Directory you should be able to use Direct Bind to get rid of the explicit user search. Otherwise you could enable logging to gain more insight.
i want to test user registration but i can't test image here is my test:
test.py
response = self.client.post('/api/v1/signup/',
content_type='application/json',
data=json.dumps({"username": "casino", "email": "casinoluxwin#gmail.com",
"password1": "android12", "password2": "android12", "photo": {
'real_filename': "u'banner3.jpg'",
'path': "u'C:/Users/Dima/Desktop'"}
}))
self.assertEqual(response.status_code, 200)
i get code 400(bad request), but without photo my test pass
service/users.py
#validate_input({
'username': {'min_length': 3, 'max_length': 50},
'email': {'validation_type': "email", 'max_length': 50},
'password1': {'min_length': 8, 'max_length': 50},
'password2': {'min_length': 8, 'equal_to': 'password1',
'messages': {'equal_to': _(u"Пароли не совпадают")}},
'photo': {'required': True}
})
#transaction.atomic
def signup(self, data):
user_data = {
'username': data['username'],
'email': data['email'],
'password': data['password1'],
'coins_amount': 0
}
user = self._create_user(user_data)
if data.get("photo"):
self._attach_photo(user, data["photo"])
obj, created = VerificationCode.objects.get_or_create(user=user, code_type="registration")
obj.create_expiration_date()
obj.create_code()
obj.save()
return user.id
So i want to test user photo anything else works fine. Thanks for any help
Problem probably resides in either Users._attach_photo() or your user model. Not enough information here to decipher it entirely. There are a couple of things to try.
I'd write a normal unittest that does not use the client. It'll give you a more helpful traceback than just the HTTP status code from the running server. Something like:
def test_user_add_method(self):
x = Users.signup(json.dumps({"username": "casino", "email": "casinoluxwin#gmail.com",
"password1": "android12", "password2": "android12", "photo": {
'real_filename': "u'banner3.jpg'",
'path': "u'C:/Users/Dima/Desktop'"})
Users.get(pk=x) #will fail if user was not created.
Second, try commenting out your validator. 400 bad request could easily be kicked off by that. It is possible that your validator isn't playing nice with the image and you'll have to mess around with that.