Related
I'm going to backup of my Cognito users with Lambda but I can't get Cognito users list in JSON-format with boto3. I do:
import boto3
import os
import json
from botocore.exceptions import ClientError
COGNITO_POOL_ID = os.getenv('POOL_ID')
S3_BUCKET = os.getenv('BACKUP_BUCKET')
ENV_NAME = os.getenv('ENV_NAME')
filename = ENV_NAME + "-cognito-backup.json"
REGION = os.getenv('REGION')
cognito = boto3.client('cognito-idp', region_name=REGION)
s3 = boto3.resource('s3')
def lambda_handler (event,context):
try:
response = (cognito.list_users(UserPoolId=COGNITO_POOL_ID,AttributesToGet=['email_verified','email']))['Users']
data = json.dumps(str(response)).encode('UTF-8')
s3object = s3.Object(S3_BUCKET, filename)
s3object.put(Body=(bytes(data)))
except ClientError as error:
print(error)
But get one string and I'm not sure that is JSON at all:
[{'Username': 'user1', 'Attributes': [{'Name': 'email_verified', 'Value': 'true'}, {'Name': 'email', 'Value': 'user1#xxxx.com'}], 'UserCreateDate': datetime.datetime(2020, 2, 10, 13, 13, 34, 457000, tzinfo=tzlocal()), 'UserLastModifiedDate': datetime.datetime(2020, 2, 10, 13, 13, 34, 457000, tzinfo=tzlocal()), 'Enabled': True, 'UserStatus': 'FORCE_CHANGE_PASSWORD'}]
I need something like this:
[
{
"Username": "user1",
"Attributes": [
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "email",
"Value": "user1#xxxx.com"
}
],
"Enabled": "true",
"UserStatus": "CONFIRMED"
}
]
Try this:
import ast
import json
print(ast.literal_eval(json.dumps(response)))
For the dict response from the SDK?
Edit: Just realized since the list_users SDK also UserCreateDate object, json.dumps will complain about the transformation due to the datatime value of the UserCreateDate key. If you get that off, this will work without the ast module -
import json
data = {'Username': 'Google_11761250', 'Attributes': [{'Name': 'email', 'Value': 'abc#gmail.com'}],'Enabled': True, 'UserStatus': 'EXTERNAL_PROVIDER'}
print((json.dumps(data)))
> {"Username": "Google_1176125910", "Attributes": [{"Name": "email", "Value": "123#gmail.com"}], "Enabled": true, "UserStatus": "EXTERNAL_PROVIDER"}
You can check the output type by using
type(output)
I guess that it can be list type, so you can convert it into JSON and prettyprint by using:
print(json.dumps(output, indent=4))
I've a deployment with one container having postStart hook as shown below
containers:
- name: openvas
image: my-image:test
lifecycle:
postStart:
exec:
command:
- /usr/local/tools/is_service_ready.sh
I'm watching for the events for pods using python's kubernetes library.
when the pod gets deployed, container comes up and postStart script will be executed until postStart script exits successfully. I want to get the event from kubernetes using pythons kubernetes library when CONTAINER comes up.
I tried watching the event, I get the event with status as 'containersReady' only when postStart completes and the POD comes up,it can be seen below.
'status': {'conditions': [{'last_probe_time': None,
'last_transition_time': datetime.datetime(2019, 4, 18, 16, 25, 3, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Initialized'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2019, 4, 18, 16, 26, 51, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Ready'},
{'last_probe_time': None,
'last_transition_time': None,
'message': None,
'reason': None,
'status': 'True',
'type': 'ContainersReady'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2019, 4, 18, 16, 25, 3, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'PodScheduled'}],
'container_statuses': [{'container_id': 'docker://1c39e13dc777a34c38d4194edc23c3668697223746b60276acffe3d62f9f0c44',
'image': 'my-image:test',
'image_id': 'docker://sha256:9903437699d871c1f3af7958a7294fe419ed7b1076cdb8e839687e67501b301b',
'last_state': {'running': None,
'terminated': None,
'waiting': None},
'name': 'samplename',
'ready': True,
'restart_count': 0,
'state': {'running': {'started_at': datetime.datetime(2019, 4, 18, 16, 25, 14, tzinfo=tzlocal())},
'terminated': None,
'waiting': None}}],
and before this I get status 'podScheduled' as 'True'
'status': {'conditions': [{'last_probe_time': None,
'last_transition_time': datetime.datetime(2019, 4, 18, 16, 25, 3, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Initialized'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2019, 4, 18, 16, 25, 3, tzinfo=tzlocal()),
'message': 'containers with unready status: [openvas]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'Ready'},
{'last_probe_time': None,
'last_transition_time': None,
'message': 'containers with unready status: [openvas]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'ContainersReady'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2019, 4, 18, 16, 25, 3, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'PodScheduled'}],
'container_statuses': [{'container_id': None,
'image': 'ns-openvas:test',
'image_id': '',
'last_state': {'running': None,
'terminated': None,
'waiting': None},
'name': 'openvas',
'ready': False,
'restart_count': 0,
'state': {'running': None,
'terminated': None,
'waiting': {'message': None,
'reason': 'ContainerCreating'}}}],
Anything I can try to get the event when the CONTAINER comes up.
Obviously, with current approach you will never get it working, because, as describe here :
The postStart handler runs asynchronously relative to the Container’s
code, but Kubernetes’ management of the container blocks until the
postStart handler completes. The Container’s status is not set to
RUNNING until the postStart handler completes.
Maybe you should create another pod with is_service_ready.sh script, which will be watching events of the main pod.
I'm using Django 1.4 and PostgreSQL.
In settings.py I have:
USE_TZ = True
Creation of dates always makes offset-aware timezones:
>>> from django.utils import timezone
>>> timezone.now()
datetime.datetime(2014, 9, 1, 7, 48, 12, 636318, tzinfo=<UTC>)
Models from my own app return timezone aware dates.
>>> from fundedbyme.campaign.models import *
>>> e = EquityCampaign.objects.all()[0].created
datetime.datetime(2013, 9, 19, 11, 29, 57, 844642, tzinfo=<UTC>)
From some other third party apps I return offset aware dates.
>>> from paypaladaptive.models import Payment
>>> Payment.objects.all()[0].created_date
datetime.datetime(2013, 5, 14, 11, 58, 47, 713878, tzinfo=<UTC>)
However the User model does not have a timezone aware date.
>>> u = User.objects.all()[0]
>>> u.date_joined
datetime.datetime(2014, 5, 13, 16, 20, 37, 709550)
Why would only the django user model not return offset aware datetimes? This is causing me a serious bug when I use django-registration. In the line linked to below, a native and aware date get compared.
https://bitbucket.org/ubernostrum/django-registration/src/8f242e35ef7c004e035e54b4bb093c32bf77c29f/registration/models.py?at=default#cl-218 it
I've looked in the auth_user table in postgres and the date_joined field is a "timestamp with time zone" field.
Why would I get native dates returned only on the User model?
UPDATE: The problem wasn't with the previous transaction, it was happening because the database wasn't synced / migrated properly.
I've recently switched my local database to Postgres, and I'm now getting an InternalError.
I understand from this question that the problem likely originates from a previous transaction not executing properly:
"This is what postgres does when a query produces an error and you try to run another query without first rolling back the transaction."
However, from the logs below, it seems like the first query, DEBUG (0.0001), executes fine (I also tested this exact query though the Django DB Shell):
DEBUG (0.001) SELECT "django_site"."id", "django_site"."domain", "django_site"."name"
FROM "django_site" WHERE "django_site"."id" = 12 ; args=(12,)
Full SQL Logs
DEBUG (0.001) SELECT "django_site"."id", "django_site"."domain", "django_site"."name" FROM "django_site" WHERE "django_site"."id" = 12 ; args=(12,)
DEBUG (0.003) INSERT INTO "application_app" ("applied_date", "fname", "lname",
"email_address", "phone_number", "skype_id", "applied_track", "college1",
"field_of_study1", "graduation_month1", "graduation_year1", "degree1",
"degree_other1", "working_during_program", "explain_working_during_program",
"sib_goal", "twitter_link", "linkedin_link", "other_social_link1",
"other_social_link2", "other_social_link3", "applied_class", "applied_location",
"referral", "colossal_failure", "next_week_year_10year", "you_created",
"your_inspiration", "dev_years_of_exp", "dev_fav_lang", "dev_fav_lang_why",
"dev_link_youve_built", "dev_link_github", "dev_fav_resource", "prod_cool_prod",
"prod_fav_designer", "prod_portfolio", "prod_bad_design", "prod_link_dribble",
"mark_ind_trend", "mark_email_to_coworkers", "mark_keep_em_happy",
"mark_article_or_blog", "sales_why_you", "sales_convince_restaurant",
"sales_hardest_door", "sales_sale_within_the_year", "housing_needed",
"program_payment", "any_last_requests") VALUES ('2013-04-20 13:22:06.565691+00:00',
'Brian', 'Dant', 'test#gmail.com', '', '', 'MAR', '', '', 1, NULL, '', '', false,
'', 'GN', '', 'http://linkedin.com/', '', '', '', 'NYCSUM13', '', 'test', 'test',
'test', 'test', 'test', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '',
'', '', '', '', false, 'UF', 'test') RETURNING "application_app"."id"; args=(u'2013-04-20 13:22:06.565691+00:00',
u'Brian', u'Dant', u'test#gmail.com', u'', u'', u'MAR', u'', u'', 1, None, '', u'',
False, u'', u'GN', u'', u'http://linkedin.com/', u'', u'', '', u'NYCSUM13', '',
u'test', u'test', u'test', u'test', u'test', '', '', u'', u'', u'', u'', u'', '',
u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', False, u'UF', u'test')
ERROR Internal Server Error: /
Traceback (most recent call last):
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/Users/chaz/dev/projects/startupinstitute.com/apps/application/views.py", line 22, in application
new_app = f.save()
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/forms/models.py", line 364, in save
fail_message, commit, construct=False)
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/forms/models.py", line 86, in save_instance
instance.save()
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/db/models/query.py", line 1593, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 912, in execute_sql
cursor.execute(sql, params)
File "/Users/chaz/dev/envs/startupinstitute.com/lib/python2.7/site-packages/debug_toolbar/utils/tracking/db.py", line 153, in execute
'iso_level': conn.isolation_level,
InternalError: current transaction is aborted, commands ignored until end of transaction block
[20/Apr/2013 08:22:06] "POST / HTTP/1.1" 500 413131
DEBUG (0.002) SELECT "django_site"."id", "django_site"."domain", "django_site"."name" FROM "django_site" WHERE "django_site"."id" = 12 ; args=(12,)
WARNING Not Found: /favicon.ico
views.py:
def application(request):
if request.method == 'POST':
f = forms.AppForm(request.POST)
selected_track = request.POST['applied_track']
if f.is_valid():
new_app = f.save()
new_app.save()
The problem was that my database wasn't migrated properly. Apparently this error can come from (at least) both a) the previous transaction, or b) the database not being synced properly by south.
This might be a bit of a trivial question, but can't I get django now() to be in the time zone defined in settings.TIME_ZONE?
This is what is actually happening:
>>> from django.utils import timezone
>>> timezone.now()
datetime.datetime(2012, 5, 30, 16, 30, 0, 782087, tzinfo=<UTC>)
>>> timezone.get_default_timezone()
<DstTzInfo 'Asia/Singapore' SMT+6:55:00 STD>
Django's source code (as displayed in the chosen answer) explains the concept of timezone.now():
datetime.now() yields the current time (in your active timezone!) without timezone information ("naive datetime"), whereas ...
timezone.now() always yields the current time in UTC (!) with timezone information.
This is irritating at first sight, yes. They could have decided to yield the current time of the active timezone, but they didn't. You can still use timezone.localtime(timezone.now()) to get what you want:
from django.utils import timezone
from datetime import datetime
timezone.get_current_timezone()
# <DstTzInfo 'Antarctica/McMurdo' LMT+11:39:00 STD>
datetime.now()
# datetime.datetime(2014, 8, 19, 20, 8, 8, 440959)
timezone.localtime(timezone.now())
# datetime.datetime(2014, 8, 19, 20, 8, 14, 889429, tzinfo=<DstTzInfo 'Antarctica/McMurdo' NZST+12:00:00 STD>)
timezone.now()
# datetime.datetime(2014, 8, 19, 8, 8, 22, 273529, tzinfo=<UTC>)
datetime.utcnow()
# datetime.datetime(2014, 8, 19, 8, 8, 29, 769312)
For newcomers and ordinary users timezone.localtime(timezone.now()) is probably the most intuitive. A local time which still retains timezone information.
EDIT: The standard library equivalent for a timezone-aware local time is datetime.now().astimezone(). astimezone applies the system local timezone by default, which allows you to correctly convert any timezone-aware datetime to your local time. To set an arbitrary timezone using timezone names you need the pytz package.
from datetime import datetime, timezone
import pytz
datetime.now()
# datetime.datetime(2022, 9, 14, 5, 26, 35, 146551)
datetime.now().astimezone()
# datetime.datetime(2022, 9, 14, 5, 26, 44, 19645, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200), 'CEST'))
datetime.now(tz=timezone.utc)
# datetime.datetime(2022, 9, 14, 3, 26, 51, 203917, tzinfo=datetime.timezone.utc)
datetime.now(tz=timezone.utc).astimezone()
# datetime.datetime(2022, 9, 14, 5, 26, 58, 546724, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200), 'CEST'))
datetime.now(tz=pytz.timezone('Europe/Kiev'))
# datetime.datetime(2022, 9, 14, 6, 27, 38, 714633, tzinfo=<DstTzInfo 'Europe/Kiev' EEST+3:00:00 DST>)
Or I could just read the source:
def now():
"""
Returns an aware or naive datetime.datetime, depending on settings.USE_TZ.
"""
if settings.USE_TZ:
# timeit shows that datetime.now(tz=utc) is 24% slower
return datetime.utcnow().replace(tzinfo=utc)
else:
return datetime.now()
Answer is nope, I have to adjust it myself.
It depends
now()
Returns an aware or naive datetime that represents the current point in time when USE_TZ is True or False respectively.
https://docs.djangoproject.com/en/dev/ref/utils/#django-utils-timezone
So all would indicate that USE_TZ is false in your case, and it's not taking the TZ in consideration.