I'm trying to do the following request with django :
I tried the following code but it doesn't work :
data = {'username': admin,
'password': 123,
'grant_type': 'password',
'client_id': 'xxxx',
'client_secret': 'xxxx'}
headers = {'content-type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=data, headers=headers)
Thanks for your help !
It is form-encoded by default.
Typically, you want to send some form-encoded data — much like an HTML
form. To do this, simply pass a dictionary to the data argument. Your
dictionary of data will automatically be form-encoded when the request
is made.
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> print r.text
{
"origin": "179.13.100.4",
"files": {},
"form": {
"key2": "value2",
"key1": "value1"
},
"url": "http://httpbin.org/post",
"args": {},
"headers": {
"Content-Length": "23",
"Accept-Encoding": "identity, deflate, compress, gzip",
"Accept": "*/*",
"User-Agent": "python-requests/0.8.0",
"Host": "127.0.0.1:7077",
"Content-Type": "application/x-www-form-urlencoded"
},
"data": ""
}
http://docs.python-requests.org/en/v0.10.7/user/quickstart/#make-a-post-request
Related
I have 2 lambda functions, to upload and downlaod a file from an S3 bucket. Im using Postman and configuring the POST and GET requests to have either a filename sent via json (GET Json payload: {"thefile" : "test_upload.txt"} ) and have set a form-data key of "thefile" and the value with the test file selected from my working directory on the computer.
The issue comes when sending the API requests via postman. Its giving me 'Internal Server Error'
The code for my lambdas is below:
**UPLOAD **
import json
import boto3
import os
def lambda_handler(event, context):
s3 = boto3.client('s3')
data = json.loads(event["body"])
file = data["thefile"]
print(file)
try:
s3.upload_file(file, os.environ['BUCKET_NAME'], file)
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': os.environ['BUCKET_NAME'],
'Key': file
},
ExpiresIn=24 * 3600
)
print("Upload Successful", url)
return {
"headers": { "Content-Type": "application/json" },
"statusCode": 200,
"isBase64Encoded": False,
"body": str(url)
}
except FileNotFoundError:
print("The file was not found")
return {
"headers": { "Content-Type": "application/json" },
"statusCode": 404,
"isBase64Encoded": False,
"body": "File not found"
}
**DOWNLOAD **
import json
import boto3
import botocore
import base64
from botocore.vendored import requests
def lambda_handler(event, context):
s3 = boto3.client('s3')
data = json.loads(event["body"])
file = data["thefile"]
try:
response = s3.get_object(Bucket=BUCKET_NAME, Key=file,)
download_file = response["Body"].read()
filereturn = base64.b64encode(download_file).decode("utf-8")
return {
"headers": { "Content-Type": "application/json" },
"statusCode": 200,
"body": json.dumps(filereturn),
"isBase64Encoded": True,
"File Downloaded": "File Downloaded successfully"
}
except Exception as e:
return {
"headers": { "Content-Type": "application/json" },
"statusCode": 404,
"body": "Error: File not found!"
}
Can anyone tell me what I'm doing wrong? The Lambdas have full access to S3 in their policy, I've even switched off authorisation and made the bucket public in case it was a permissions error but still nothing. It must be something stupid im either forgetting or have mis-coded but I cant for the life of me figure it out!
When I use the Superset's API to import a dashboard the response shows me a login page.
I am doing the request using Python.
import requests
headers = {
'accept': 'application/json',
'Authorization': f'Bearer {jwt_token}',
'X-CSRFToken': csrf_token,
'Referer': url
}
files = {
'formData': (
dashboard_path,
open(dashboard_path, 'rb'),
'application/json'
)
}
response = requests.get(url, files=files, headers=headers)
Does anyone know how to solve this problem?
I had some trouble with the Superset API myself, mostly because I did not handle the CSRF Token correctly:
It seems to be important that the retrieval of the JWT Token, the CSRF Token and the actual request happen in the same session.
If I don't do that, I can reproduce your error and are also sent to the login page (also you use a GET request in this example, but it should be POST).
Here an example from my local test-setup:
import requests
session = requests.session()
jwt_token = session.post(
url='http://localhost:8088/api/v1/security/login',
json={
"username": "admin",
"password": "admin",
"refresh": False,
"provider": "db"
}
).json()["access_token"]
csrf_token = session.get(
url='http://localhost:8088/api/v1/security/csrf_token/',
headers={
'Authorization': f'Bearer {jwt_token}',
}
).json()["result"]
headers = {
'accept': 'application/json',
'Authorization': f'Bearer {jwt_token}',
'X-CSRFToken': csrf_token,
}
response = requests.post(
'http://localhost:8088/api/v1/dashboard/import',
headers=headers,
files={
"formData": ('dashboards.json', open("dashboards.json", "rb"), 'application/json')
},
)
session.close()
I'm going to backup of my Cognito users with Lambda but I can't get Cognito users list in JSON-format with boto3. I do:
import boto3
import os
import json
from botocore.exceptions import ClientError
COGNITO_POOL_ID = os.getenv('POOL_ID')
S3_BUCKET = os.getenv('BACKUP_BUCKET')
ENV_NAME = os.getenv('ENV_NAME')
filename = ENV_NAME + "-cognito-backup.json"
REGION = os.getenv('REGION')
cognito = boto3.client('cognito-idp', region_name=REGION)
s3 = boto3.resource('s3')
def lambda_handler (event,context):
try:
response = (cognito.list_users(UserPoolId=COGNITO_POOL_ID,AttributesToGet=['email_verified','email']))['Users']
data = json.dumps(str(response)).encode('UTF-8')
s3object = s3.Object(S3_BUCKET, filename)
s3object.put(Body=(bytes(data)))
except ClientError as error:
print(error)
But get one string and I'm not sure that is JSON at all:
[{'Username': 'user1', 'Attributes': [{'Name': 'email_verified', 'Value': 'true'}, {'Name': 'email', 'Value': 'user1#xxxx.com'}], 'UserCreateDate': datetime.datetime(2020, 2, 10, 13, 13, 34, 457000, tzinfo=tzlocal()), 'UserLastModifiedDate': datetime.datetime(2020, 2, 10, 13, 13, 34, 457000, tzinfo=tzlocal()), 'Enabled': True, 'UserStatus': 'FORCE_CHANGE_PASSWORD'}]
I need something like this:
[
{
"Username": "user1",
"Attributes": [
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "email",
"Value": "user1#xxxx.com"
}
],
"Enabled": "true",
"UserStatus": "CONFIRMED"
}
]
Try this:
import ast
import json
print(ast.literal_eval(json.dumps(response)))
For the dict response from the SDK?
Edit: Just realized since the list_users SDK also UserCreateDate object, json.dumps will complain about the transformation due to the datatime value of the UserCreateDate key. If you get that off, this will work without the ast module -
import json
data = {'Username': 'Google_11761250', 'Attributes': [{'Name': 'email', 'Value': 'abc#gmail.com'}],'Enabled': True, 'UserStatus': 'EXTERNAL_PROVIDER'}
print((json.dumps(data)))
> {"Username": "Google_1176125910", "Attributes": [{"Name": "email", "Value": "123#gmail.com"}], "Enabled": true, "UserStatus": "EXTERNAL_PROVIDER"}
You can check the output type by using
type(output)
I guess that it can be list type, so you can convert it into JSON and prettyprint by using:
print(json.dumps(output, indent=4))
I want to create a data catalog externally in Amazon Glue. Is there any way?
AWS Glue Data Catalog consists of meta information about various data sources within AWS, e.g. S3, DynamoDB etc.
Instead of using Crawlers or AWS Console, you can populate data catalog directly with
AWS Glue API
related to different structures, like Database, Table etc. AWS provides several SDKs for different languages, e.g.
boto3 for python with easy to
use object-oriented API. So as long as you know how your data structure, you can use methods
create_database()
create_table()
create_partition()
batch_create_partition()
Create Database definition:
from pprint import pprint
import boto3
client = boto3.client('glue')
response = client.create_database(
DatabaseInput={
'Name': 'my_database', # Required
'Description': 'Database created with boto3 API',
'Parameters': {
'my_param_1': 'my_param_value_1'
},
}
)
pprint(response)
# Output
{
'ResponseMetadata': {
'HTTPHeaders': {
'connection': 'keep-alive',
'content-length': '2',
'content-type': 'application/x-amz-json-1.1',
'date': 'Fri, 11 Oct 2019 12:37:12 GMT',
'x-amzn-requestid': '12345-67890'
},
'HTTPStatusCode': 200,
'RequestId': '12345-67890',
'RetryAttempts': 0
}
}
Create Table definition:
response = client.create_table(
DatabaseName='my_database',
TableInput={
'Name': 'my_table',
'Description': 'Table created with boto3 API',
'StorageDescriptor': {
'Columns': [
{
'Name': 'my_column_1',
'Type': 'string',
'Comment': 'This is very useful column',
},
{
'Name': 'my_column_2',
'Type': 'string',
'Comment': 'This is not as useful',
},
],
'Location': 's3://some/location/on/s3',
},
'Parameters': {
'classification': 'json',
'typeOfData': 'file',
}
}
)
pprint(response)
# Output
{
'ResponseMetadata': {
'HTTPHeaders': {
'connection': 'keep-alive',
'content-length': '2',
'content-type': 'application/x-amz-json-1.1',
'date': 'Fri, 11 Oct 2019 12:38:57 GMT',
'x-amzn-requestid': '67890-12345'
},
'HTTPStatusCode': 200,
'RequestId': '67890-12345',
'RetryAttempts': 0
}
}
After some hours of reading manuals and other helps, finally i got my nginx+uwsgi1.9+django1.6+python3.3 server.
But now I have problems (mb problem in my understanding something) - how to get POST data from request ? I mean how to get it correctly.
The code in django view:
def info(request):
print(request)
return HttpResponse(request)
request to server:
http://127.0.0.1:8000/info/
{
"test":"test"
}
and interesting part - output in uwsgi log (POST and GET dicts):
<WSGIRequest
path:/info/,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '24',
'CONTENT_TYPE': 'application/json',
'CSRF_COOKIE': 'upJxA8TWO0nhKACr0dfU46Qyu0DzzUTR',
'DOCUMENT_ROOT': '/usr/share/nginx/html',
'HTTP_ACCEPT': '*/*',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
'HTTP_ACCEPT_LANGUAGE': 'ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
'HTTP_CONNECTION': 'keep-alive',
'HTTP_CONTENT_LENGTH': '24',
'HTTP_CONTENT_TYPE': 'application/json',
'HTTP_COOKIE': '',
'HTTP_HOST': '127.0.0.1:8000',
'HTTP_ORIGIN': 'chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo',
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36',
'PATH_INFO': '/info/',
'QUERY_STRING': '',
'REMOTE_ADDR': '127.0.0.1',
'REMOTE_PORT': '53315',
'REQUEST_METHOD': 'POST',
'REQUEST_URI': '/info/',
'SCRIPT_NAME': '',
'SERVER_NAME': 'django',
'SERVER_PORT': '8000',
'SERVER_PROTOCOL': 'HTTP/1.1',
'uwsgi.node': b'',
'uwsgi.version': b'1.9.18.2',
'wsgi.errors': <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'>,
'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
'wsgi.input': <uwsgi._Input object at 0x7fa4ec6b8ee8>,
'wsgi.multiprocess': True,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 0)}>
and response in browser (same as request to server):
{
"test":"test"
}
What I'm doing wrong ?
request.POST
and
request.POSt.dict()
returns empty dicts.
So questions is how to get POST data in code and why it looks so different when I use environment ?
UPD:
return HttpResponse(str(request)) - returns WSGIRequest object instead of POST data to me. But still dont know how to get POST data in code.
UPD2:
uwsgi config:
[uwsgi]
module = mysite.wsgi
master = true
processes = 5
socket = :8001
daemonize = /var/log/uwsgi/mysite.log
touch-reload = /tmp/uwsgi-touch
post-buffering = 1
UPD3:
versions of software:
Python 3.3.2
Django 1.6.2
uWSGI 1.9.18.2
UPD4:
Final code:
if request.method == "POST":
if request.META["CONTENT_TYPE"] == "application/json":
result = json.loads(request.body.decode())
else: result = request.POST.dict()
return HttpResponse(json.dumps(result), content_type="application/json")
Thats what I exactly want.
When I send data to server with
POST
Content-Type: application/x-www-form-urlencoded
a=1
or:
POST
Content-Type: application/json
{"a":"1"}
I see same response (and in code variables):
Content-Type: application/json
{"a": "1"}
Looks like you a POSTing JSON data, not HTML form data.
In that case you are looking for the raw post data, which is accessed like this:
request.body
See HttpRequest.body in the documentation.
If you want to parse that JSON string, use this:
import json
data = json.loads(request.body)
The request.POST dictionary is only populated when the request contains form data. This is when the Content-Type header is application/x-www-form-urlencoded or multipart/form-data.