I have stumbled on a strange issue that I can't resolve:
In my Django app there is a method which gets hit by a POST from a java applet, which sends it a JSON object. Django method parses it like so:
req = json.loads(request.raw_post_data)
and based on the results returns a value. I haven't written this code, but yesterday I was sent to investigate an error triggered in this method. It was saying there was "ValueError: Expecting property name: line 1 column 1 (char 1)".
What I discovered is that my raw post data looks like this:
{#012#011"ImmutableMachineFactors": #012#011{#012#011#011"machineName": "lukka",#012#011#011"osName": "MacOS"}}
The type of it was string, however, my attempts to replace these weird characters with spaces or nothing failed. It would just ignore the sub() command. I know that raw_post_data returns a bytestring, but when I tried to convert it to a regular string using:
mystring.decode('utf-8')
it did add the u'' notation, but didn't remove those weird characters. Stranger still, in many cases (on my personal machine), Django would happily convert this kind of data into JSON, it only fails sometimes, which led me to believe that the JSON which triggered the error was malformed, but when I would strip out all the #011 and #012 characters, it parsed perfectly.
My questions are:
1) What are those crazy things? (#011, #012). I tried to google around, but these are very common things to find in a search, so I couldn't find anything relevant.
2) How can I turn this bytestring into a regular string so that I can replace those characters? Or is it the wring way to approach this problem?
Thanks!
Luka
This may be way too late to help, but since QueryDict instances (request.POST or request.DATA) are immutable, it's reasonable to expect that request.raw_post_data is also immutable. You'd have to make a copy before changing it.
Related
I'm building a multi-exchange bot in C++ and I'm having one small problem. KuCoin exchange has proven to be frustrating to say the least, one of the headers is a signature header where you encode the string with HMAC sha256 and then encode THAT with base64. However I'm not concerned with the encoding, I can do all that. What is stumping me is the string KuCoins API is expecting, I've scoured their documentation 100 times over and I still can't get it right, here are the instructions
For the header of KC-API-KEY:
Use API-Secret to encrypt the prehash string {timestamp+method+endpoint+body} with sha256 HMAC. The request body is a JSON string and need to be the same with the parameters passed by the API.
After that, use base64-encode to encrypt the result in step 1 again.
I've attempted to craft this string in every way possible, and the documentation provides no examples on what a good string should look like, here are the strings I've crafted BEFORE encoding that DO NOT work.
EXAMPLE 1: 1616096476134POST/api/v1/orders?clientOid=55be5&side=BUY&symbol=BTC-USDT&type=MARKET&funds=0.005000
EXAMPLE 2: 1616099932367POST/api/v1/orders{"clientOid":"55be5","side":"BUY","symbol":"BTC-USDT","type":"MARKET","funds":"0"}
As you can see, in the 2nd example I tried to make the body a JSON string with all the correct parameters, but still, I'm getting a bad signature error returned. Literally all I need is to know what the string is supposed to look like so I can craft it properly. Thanks.
I take the assumption that your code works for a private request without parameters (like getting the balance for instance).
I also struggled a bit for POST requests, but managed to get it right after a few attempts. You need to dump the parameters as a JSON string without spaces, exactly like in your example 2.
Since that time, have you managed to solve it ? I have a code on my application that works if you are interested.
Also, don't forget to add in the headers of the curl request:
Content-Type: application/json
Solved with Kucoin support and pythone example.
The "body" also must be included in POST request.
Was:
reply = netman->post(req, "");
Become:
tradereply = trademan->post(req, data);
I am currently receiving 400 errors when invoking the list_builds() method (seen here: https://googleapis.dev/python/cloudbuild/latest/gapic/v1/api.html?highlight=list_builds#google.cloud.devtools.cloudbuild_v1.CloudBuildClient.list_builds)
The following command works using gcloud:
gcloud builds list --filter="status=FAILURE"
However, the following API call returns google.api_core.exceptions.InvalidArgument: 400 Error processing filter expression
for element in client.list_builds("REDACTED", filter_="status=FAILURE"):
# process element
pass
I'm guessing I'm missing something very obvious and simple here but I can't exactly figure out what I'm doing wrong
The correct way to pass in the filter string to the API call includes using double apostrophes around the actual text like so:
filter_='status="FAILURE"'
Unsure of whether or not this will be changed in the future, but this is the same behavior for passing it in via the REST API here: https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.builds/list
e.g specifiying status=FAILURE will fail, but status="FAILURE" returns a 200 response.
welcome to Stack O! So I'm just spitballing here, but the docs say that the Filter string is "The raw filter text to constrain the results."
When you filter a Cloud Build in the console, the "raw filter text" in the UI says Status : Failed - you could try:
--filter="Status : Failed"
--filter="Status: Failed"
--filter="Status:Failed"
Alternately, it might be the string in the URL, the formatting of which is too bananapants for Stack O's robots to handle, so I can't paste it here, but it starts with f and ends with Failed:
The key here is that you have an equal sign in your string, as well as Failure instead of Failed - changing one or both of those might do the trick.
Hope this helps!
I am trying to save an array of text containing category types for a hotel system which looks something like this ['category-1', category-2', category-3, category-4] . I am using category_type = ArrayField(models.CharField(max_length=200),null=True) in my models.py
The error i get is
malformed array literal: "" LINE 1: ..., '{category-1,
category-2}'::varchar(200)[], ''::varcha...
^ DETAIL: Array value must start with "{" or dimension information.
The error persist even after processing python list from ['category-1', category-2', category-3, category-4] to {category-1, category-2, category-3, category-4}.
I have gone through postgresql documentation and have found very limited help,
https://pganalyze.com/docs/log-insights/app-errors/U114 this is something similar posted to what i am facing problem with.
Could someone please tell me what am i doing wrong? Any help would be appreciated.
EDIT:
Following is in my View.py
hotel_category=categoryTable(category_type=categorytype)
hotel_category.save()
and i am using categorytype=request.POST.getlist('category-type') in my Views.py to get it from the POST request after user submits the form. This returns a Python list that i have mentioned above, i have manipulated this list to match PostgreSQL ArrayField with '{','}' but i still have this error. If there is anything else you would like me to add, please let me know. :)
This is an update/answer to my question for anyone who faces this issue in the future. After struggling to find help from different resources, i decided to use JSON string to store my python list.
I am using :
categorytype = json.dumps(request.POST.getlist('category-type'))
to encode and using JSONDecoder() to fetch from database and decode. I have no idea how would this impact my further development but for now it seems a decent approach since personally i think ArrayFields are not well supported and documented in Django.
I will keep this post updated as i progress further on how this approach has impacted my development.
Have a nice day.
I'm trying to POST some JSON and a binary file from an iPhone to a Django server running django-piston using ASIHTTPRequest
I know how to get it to work if I am ONLY sending JSON strings, and I know how to make it work if I am ONLY sending a file, but doing both is tricky.
So we'll start with ASIHTTPRequest code
ASIFormDataRequest *request = [[ASIFormDataRequest alloc] initWithURL:url];
[request setRequestMethod:#"POST"];
[request setPostFormat:ASIMultipartFormDataPostFormat];
[request appendPostData:[#"{\"save\":{\"name\":\"iostest\"}}" dataUsingEncoding:NSUTF8StringEncoding]];
[request addData:UIImageJPEGRepresentation([UIImage imageNamed:#"test.jpg"], 1.0f)
withFileName:#"test.jpg"
andContentType:#"image/jpeg"
forKey:#"data"];
[request setDelegate:self];
[request startAsynchronous];
My best idea here is that adding raw string data directly to the POST body and then adding a file just doesn't work.
But if I instead try
[request setPostValue:#"{\"name\":\"iostest\"}" forKey:#"save"];
Then the piston data dictionary will store ['save'] as a string instead of a deserialized object, so it will literally deliver the string
"{\"name\":\"iostest\"}"
Here's my Piston handler code
def create(self, request):
data = request.data
print(data['save']) #{\"name\":\"iostest\"}"
print("Files: " + request.FILES['data'].name) #test.jpg
print("Data Save Name: " + data['save']['name']) #crash, interprets this as a string indeces lookup
Ideas are welcome.
I have basically hacked my way around this.
The basic problem is that the request format in which Django expects files to be submitted to the server is one which django-piston literally just drops the ball on.
When it encounters multipart requests, it simply doesn't try to parse the data.
The solution to this problem is to manually call the parsing engine, which, in the case of JSON, is straight out of django.utils (which is kind of disappointing).
You achieve this by using ASIHTTPRequest (or the request module of your choice) to set a standard post value by key, and then access it the old fashioned way.
from django.utils import simplejson
data = simplejson.loads(request.POST['save'])
Which basically just reduces this handler method at this point to nothing more than a regular old Django view in terms of the steps you have to take to get it going.
So clearly, django-piston is not built to deal with files apparently?
My best idea here is that adding raw
string data directly to the POST body
and then adding a file just doesn't
work.
That wouldn't work, no. If you're POSTing form data using 'application/x-www-form-urlencoded' format, or 'multipart/form-data' you're not going to be able to just tack some extra data on the end - it needs to go in as part of the form data. Something like this I guess...
[request setPostValue:#"{\"save\":{\"name\":\"iostest\"}}" forKey:#"data"];
But if I remove the string data and only post the file it still doesn't work.
Is more problematic...
or if it's Piston erroneously misreading the data.
I probably wouldn't look in that direction first - piston doesn't really mess with the request object, so it seems more likely that the ASI request isn't quite right.
I think the place to start would be to inspect the incoming request and check that it really is a valid formPOST request:
Check that request["CONTENT_TYPE"] is set to 'multipart/form-data'
Inspect the request.raw_post_data and make sure that it is valid form data as specified in http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.2 - check that the key names are as you expected and that the file content is present. (Obviously you'll want to use a small text file when you're testing this!)
Check which keys actually are present in request.FILES, if any, in case it's as simple as something like a misnamed field.
Failing all that I'd try to narrow down if it's a problem on the client or server side by trying to write a plain python client and seeing if you have the same issue then. Looking around, something like this: http://atlee.ca/software/poster/ might be useful.
I have a Django app that runs a tool and displays the results from the tool back to the user using a Django template. Sometimes Django does not display the results. It doesn't complain about anything, it just doesn't display the results. I'm guessing this is something to do with one or more of the characters in the results being illegal as far as Django is concerned. How can I get more information about what it is that Django doesn't like? Also, is there some method I can use to filter out "bad" characters? The results are normally just lots of text. They contain company confidential stuff, so I can't give an example unfortunately. I have DEBUG set to True and TEMPLATE_DEBUG set to DEBUG.
UPDATE:
I added some code to filter out all chars with a decimal value greater than 127 and it now works.
If you are using the development server, put in a breakpoint with pdb and see what is going on. Or print out the string that you think has "bad" characters. If you aren't using the development server you could use the Python logging module to log the string you are getting from the tool.
You might be leaping to conclusions about the data containing bad characters. It may be something else, and without debugging further it is hard to speculate.
you could try using the built in django encoding methods to remove illegal characters.
from django.utils.encoding import smart_str
smart_str(your_string)