Django body encoding vs slack-api secret - django

I am following the instruction from this page. I am building a slack slash command handling server and I can't rebuild the signature to validate slash request authenticity.
here is the code snippet from my django application (the view uses the django rest-framework APIView):
#property
def x_slack_req_ts(self):
if self.xsrts is not None:
return self.xsrts
self.xsrts = str(self.request.META['HTTP_X_SLACK_REQUEST_TIMESTAMP'])
return self.xsrts
#property
def x_slack_signature(self):
if self.xss is not None:
return self.xss
self.xss = self.request.META['HTTP_X_SLACK_SIGNATURE']
return self.xss
#property
def base_message(self):
if self.bs is not None:
return self.bs
self.bs = ':'.join(["v0", self.x_slack_req_ts, self.raw.decode('utf-8')])
return self.bs
#property
def encoded_secret(self):
return self.app.signing_secret.encode('utf-8')
#property
def signed(self):
if self.non_base is not None:
return self.non_base
hashed = hmac.new(self.encoded_secret, self.base_message.encode('utf-8'), hashlib.sha256)
self.non_base = "v0=" + hashed.hexdigest()
return self.non_base
This is within a class where self.raw = request.body the django request and self.app.signing_secret is a string with the appropriate slack secret string. It doesn't work as the self.non_base yield an innaccurate value.
Now if I open an interactive python repl and do the following:
>>> import hmac
>>> import hashlib
>>> secret = "8f742231b10e8888abcd99yyyzzz85a5"
>>> ts = "1531420618"
>>> msg = "token=xyzz0WbapA4vBCDEFasx0q6G&team_id=T1DC2JH3J&team_domain=testteamnow&channel_id=G8PSS9T3V&channel_name=foobar&user_id=U2CERLKJA&user_name=roadrunner&command=%2Fwebhook-collect&text=&response_url=https%3A%2F%2Fhooks.slack.com%2Fcommands%2FT1DC2JH3J%2F397700885554%2F96rGlfmibIGlgcZRskXaIFfN&trigger_id=398738663015.47445629121.803a0bc887a14d10d2c447fce8b6703c"
>>> ref_signature = "v0=a2114d57b48eac39b9ad189dd8316235a7b4a8d21a10bd27519666489c69b503"
>>> base = ":".join(["v0", ts, msg])
>>> hashed = hmac.new(secret.encode(), base.encode(), hashlib.sha256)
>>> hashed.hexdigest()
>>> 'a2114d57b48eac39b9ad189dd8316235a7b4a8d21a10bd27519666489c69b503'
You will recognise the referenced link example. If I use the values from my django app with one of MY examples, it works within the repl but doesn't within the django app.
MY QUESTION: I believe this is caused by the self.raw.decode() encoding not being consistent with the printout I extracted to copy/paste in the repl. Has anyone encountered that issue and what is the fix? I tried a few random things with the urllib.parse library... How can I make sure that the request.body encoding is consistent with the example from flask with get_data() (as suggested by the doc in the link)?
UPDATE: I defined a custom parser:
class SlashParser(BaseParser):
"""
Parser for form data.
"""
media_type = 'application/x-www-form-urlencoded'
def parse(self, stream, media_type=None, parser_context=None):
"""
Parses the incoming bytestream as a URL encoded form,
and returns the resulting QueryDict.
"""
parser_context = parser_context or {}
request = parser_context.get('request')
raw_data = stream.read()
data = QueryDict(raw_data, encoding='utf-8')
setattr(data, 'raw_body', raw_data) # setting a 'body' alike custom attr with raw POST content
return data
To test based on this question and the raw_body in the custom parser generates the exact same hashed signature as the normal "body" but again, copy pasting in the repl to test outside the DRF works. Pretty sure it's an encoding problem but completely at loss...

I found the problem which is very frustrating.
It turns out that the signing secret was stored in too short a str array and were missing trailing characters which obviously, resulted in bad hashing of the message.

Related

json serialisation of dates on flask restful

I have the following resource:
class Image(Resource):
def get(self, db_name, col_name, image_id):
col = mongo_client[db_name][col_name]
image = col.find_one({'_id':ObjectId(image_id)})
try:
image['_id'] = str(image['_id'])
except TypeError:
return {'image': 'notFound'}
return {'image':image}
linked to a certain endpoint.
However, image contains certain datetime objects inside. I could wrap this around with `json.dumps(..., default=str), but I see that there is a way of enforcing this on flask-restful. It's just not clear to me what exactly needs to be done.
In particular, I read:
It is possible to configure how the default Flask-RESTful JSON
representation will format JSON by providing a RESTFUL_JSON
attribute on the application configuration.
This setting is a dictionary with keys that
correspond to the keyword arguments of json.dumps().
class MyConfig(object):
RESTFUL_JSON = {'separators': (', ', ': '),
'indent': 2,
'cls': MyCustomEncoder}
But it's not clear to me where exactly this needs to be placed. Tried a few things and it didn't work.
EDIT:
I finally solved with this:
Right after
api = Api(app)
I added:
class CustomEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
#return int(obj.strftime('%s'))
return str(obj)
elif isinstance(obj, datetime.date):
#return int(obj.strftime('%s'))
return str(obj)
return json.JSONEncoder.default(self, obj)
def custom_json_output(data, code, headers=None):
dumped = json.dumps(data, cls=CustomEncoder)
resp = make_response(dumped, code)
resp.headers.extend(headers or {})
return resp
api = Api(app)
api.representations.update({
'application/json': custom_json_output
})
Just cleared this out, you just have to do the following:
app = Flask(__name__)
api = Api(app)
app.config['RESTFUL_JSON'] = {'cls':MyCustomEncoder}
This works both for plain Flask and Flask-RESTful.
NOTES:
1) Apparently the following part of the documentation is not that clear:
It is possible to configure how the default Flask-RESTful JSON
representation will format JSON by providing a RESTFUL_JSON attribute
on the application configuration. This setting is a dictionary with
keys that correspond to the keyword arguments of json.dumps().
class MyConfig(object):
RESTFUL_JSON = {'separators': (', ', ': '),
'indent': 2,
'cls': MyCustomEncoder}
2)Apart from the 'cls' argument you can actually overwrite any keyword argument of the json.dumps function.
Having created a Flask app, e.g. like so:
root_app = Flask(__name__)
place MyConfig in some module e.g. config.py and then configure root_app like:
root_app.config.from_object('config.MyConfig')

django-nocaptcha-recaptcha always shows additional verification box

I installed django-nocaptcha-recaptcha and integrated it into my form:
from nocaptcha_recaptcha.fields import NoReCaptchaField
class ClientForm(forms.ModelForm):
captcha = NoReCaptchaField()
It shows up fine on the form, but whenever I click on it an additional dialog pops up asking to enter some text and verify. It happens every time. I tested it from another computer on another network and it still asks for additional verification after clicking the box.
This is what it looks like: additional verification dialog box
Here's how I'm handling the form:
#xframe_options_exempt
def registration(request):
if request.method == 'POST':
clientform = ClientForm(request.POST)
# check whether it's valid:
if clientform.is_valid():
new_client = clientform.save()
...
What am I doing wrong? Is it a problem with django-nocaptcha-recaptcha? Should I use something else?
P.S. I'm using django 1.7.1 with python 3.4
Another alternative: Minimalist and non framework dependant.
This is the code, in case you want to rewrite it.
'''
NO-CAPTCHA VERSION: 1.0
PYTHON VERSION: 3.x
'''
import json
from urllib.request import Request, urlopen
from urllib.parse import urlencode
VERIFY_SERVER = "www.google.com"
class RecaptchaResponse(object):
def __init__(self, is_valid, error_code=None):
self.is_valid = is_valid
self.error_code = error_code
def __repr__(self):
return "Recaptcha response: %s %s" % (
self.is_valid, self.error_code)
def __str__(self):
return self.__repr__()
def displayhtml(site_key, language=''):
"""Gets the HTML to display for reCAPTCHA
site_key -- The site key
language -- The language code for the widget.
"""
return """<script src="https://www.google.com/recaptcha/api.js?hl=%(LanguageCode)s" async="async" defer="defer"></script>
<div class="g-recaptcha" data-sitekey="%(SiteKey)s"></div>
""" % {
'LanguageCode': language,
'SiteKey': site_key,
}
def submit(response,
secret_key,
remote_ip,
verify_server=VERIFY_SERVER):
"""
Submits a reCAPTCHA request for verification. Returns RecaptchaResponse
for the request
response -- The value of response from the form
secret_key -- your reCAPTCHA secret key
remote_ip -- the user's ip address
"""
if not(response and len(response)):
return RecaptchaResponse(is_valid=False, error_code='incorrect-captcha-sol')
def encode_if_necessary(s):
if isinstance(s, str):
return s.encode('utf-8')
return s
params = urlencode({
'secret': encode_if_necessary(secret_key),
'remoteip': encode_if_necessary(remote_ip),
'response': encode_if_necessary(response),
})
params = params.encode('utf-8')
request = Request(
url="https://%s/recaptcha/api/siteverify" % verify_server,
data=params,
headers={
"Content-type": "application/x-www-form-urlencoded",
"User-agent": "reCAPTCHA Python"
}
)
httpresp = urlopen(request)
return_values = json.loads(httpresp.read().decode('utf-8'))
httpresp.close()
return_code = return_values['success']
if return_code:
return RecaptchaResponse(is_valid=True)
else:
return RecaptchaResponse(is_valid=False, error_code=return_values['error-codes'])
Restart the server and don't forget to clear your browser's cache. Hope this helps.

ModelForm clean_xxxx() works for CharField, not for URLField. Django 1.5

How can I remove whitespace, prior to validation of a URLField?
Using "clean_[fieldname]()" would seem to be the documented way from https://docs.djangoproject.com/en/dev/ref/forms/validation/ , but it does not work for the URLField. I've reduced it to a basic test case which can be run in the django shell:
class XXXTestModel(models.Model):
url = models.URLField('URL',null=True,blank=True)
name = models.CharField(max_length=200)
class XXXTestForm(ModelForm):
def clean_url(self):
return self.cleaned_data['url'].strip()
def clean_name(self):
return self.cleaned_data['name'].strip()
class Meta:
model = XXXTestModel
fields = (
'url',
)
Tested from the Django shell with:
>>> django.VERSION
(1, 5, 1, 'final', 0)
>>> from xxx import XXXTestForm,XXXTestModel
>>> data = dict(url=' http://www.example.com/ ',name=' example ')
>>> f=XXXTestForm(data)
>>> f.is_valid();f.errors
False
{'url': [u'Enter a valid URL.']}
>>> f.cleaned_data
{'name': example'}
There are a number of close dupes of this question on stack overflow, but none of the answers guide toward a solution.
The issue here is how the django.forms.URLField works.
django.forms.Field.clean is defined as:
def clean(self, value):
"""
Validates the given value and returns its "cleaned" value as an
appropriate Python object.
Raises ValidationError for any errors.
"""
value = self.to_python(value)
self.validate(value)
self.run_validators(value)
return value
Note that to_python is performed before any validation. This is the issue here - django.forms.URLField can't understand the value you're giving it, so the value it produces fails the set of validators already defined as part of django.forms.URLField (namely, django.core.validators.URLValidator).
The reason it fails is django tries to "normalize" the URL. This includes things such as adding "http://" where needed. When given your example url, " http://www.example.com ", django uses urlparse.urlsplit to get it "parts" of the url. The leading space, however, messes it up and the entire value becomes part of the path. As such, django finds no scheme, and reconstitutes the URL as "http:// http://www.example.com ". This is then given to django.core.validators.URLValidator, which obviously fails.
To avoid this, we'll need to define our own URLField for our form
from django import forms
class StrippedURLField(forms.URLField):
def to_python(self, value):
return super(StrippedURLField, self).to_python(value and value.strip())
Using this ensures the process will all go as expected, and we wont need a clean_url method. (note: you should use clean_* where possible, but here it is not)
class XXXTestForm(forms.ModelForm):
url = StrippedURLField(blank=True, null=True)

Can't authenticate with custom PASSWORD_HASHERS

I am working on the migration of one website with php to Django framework.
There is used to a specific hash passwords algorithm, so I had to write:
#settings.py
PASSWORD_HASHERS = (
'django.contrib.auth.hashers.PBKDF2PasswordHasher',
'project.hashers.SHA1ProjPasswordHasher', # that's mine
'django.contrib.auth.hashers.PBKDF2SHA1PasswordHasher',
...
)
and:
#hashers.py
import hashlib
from django.contrib.auth.hashers import (BasePasswordHasher, mask_hash)
from django.utils.datastructures import SortedDict
from django.utils.encoding import force_bytes
from django.utils.crypto import constant_time_compare
from django.utils.translation import ugettext_noop as _
class SHA1ProjPasswordHasher(BasePasswordHasher):
"""
Special snowflake algorithm from the first version.
php code: $pass=substr(sha1(trim($_POST['password'])),0,8);
"""
algorithm = "unsalted_and_trimmed_sha1"
def salt(self):
return ''
def encode(self, password, salt):
return hashlib.sha1(force_bytes(salt + password)).hexdigest()[:8]
def verify(self, password, encoded):
encoded_2 = self.encode(password, '')
return constant_time_compare(encoded, encoded_2)
def safe_summary(self, encoded):
return SortedDict([
(_('algorithm'), self.algorithm),
(_('hash'), mask_hash(encoded, show=3)),
])
It's works well when PBKDF2PasswordHasher is first:
>>> from django.contrib.auth import authenticate
>>> u = authenticate(username='root', password='test')
>>> u.password
u'pbkdf2_sha256$10000$EX8BcgPFjygx$HvB6NmZ7uX1rWOOPbHRKd8GLYD3cAsQtlprXUq1KGMk='
>>> exit()
Then I put my SHA1ProjPasswordHasher on the first place, first authentication works great. The hash was changed.:
>>> from django.contrib.auth import authenticate
>>> u = authenticate(username='root', password='test')
>>> u.password
'a94a8fe5'
>>> exit()
Second authentication is failed. Can't authenticate with new hash.
>>> from django.contrib.auth import authenticate
>>> u = authenticate(username='root', password='test')
>>> u.password
Traceback (most recent call last):
File "<console>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'password'
What could be the problem? Thanks.
UPDATE: Ok, the problem became more clear. When I remove slice from here:
return hashlib.sha1(force_bytes(salt + password)).hexdigest()[:8]
everything working fine. I can't get why..
Although years have passed, i' m putting my solution here for future reference
Vlad is partially right; The following method from django.contrib.auth.hashers seems to be forcing you to use a hash format that includes the dollar $ sign to mark the algorithm used for django to decide which hasher to use
def identify_hasher(encoded):
"""
Returns an instance of a loaded password hasher.
Identifies hasher algorithm by examining encoded hash, and calls
get_hasher() to return hasher. Raises ValueError if
algorithm cannot be identified, or if hasher is not loaded.
"""
# Ancient versions of Django created plain MD5 passwords and accepted
# MD5 passwords with an empty salt.
if ((len(encoded) == 32 and '$' not in encoded) or
(len(encoded) == 37 and encoded.startswith('md5$$'))):
algorithm = 'unsalted_md5'
# Ancient versions of Django accepted SHA1 passwords with an empty salt.
elif len(encoded) == 46 and encoded.startswith('sha1$$'):
algorithm = 'unsalted_sha1'
else:
algorithm = encoded.split('$', 1)[0]
return get_hasher(algorithm)
There is a way though to "trick" django without hacking your django installation. You 'll have to create an authentication backend to use for your authentication. There you'll override django's check_password method. I had a db where hashes were {SSHA512}hash and i couldn't change this because i had to be able to communicate with dovecot. So i put the following in my backends.py class:
def check_password(self, raw_password, user):
"""
Returns a boolean of whether the raw_password was correct. Handles
hashing formats behind the scenes.
"""
def setter(raw_password):
user.set_password(raw_password)
user.save(update_fields=["password"])
return check_password(raw_password, "SSHA512$" + user.password, setter)
That way when django has to check if a password is correct it'll do the following:
-Get the hash from the db {SSHA512}hash
-Append it temporarily a SSHA512$ string on the beginning and then check
So while you'll be having {SSHA512}hash in your database, when django is using this backend it'll see SSHA512${SSHA512}hash.
This way in your hashers.py you can set in your class algorithm = "SSHA512" which will hint django to use this hasher for that case.
Your def encode(self, password, salt, iterations=None) method in your hashers.py will save hashes the way dovecot needs {SSHA512}hash (you don't have to do anything weird in your encode method).
Your def verify(self, password, encoded) method though will have to strip the SSHA512$ "trick" from the encoded string that it is passed to compare it with the one that encode will create.
So there you have it! Django will use your hasher for checking hashes that don't contain a dollar $ sign and you don't have to break anything inside django :)
Only unsalted md5 hashes can not include a dollar sign:
# django/contrib/auth/hashers.py
def identify_hasher(encoded):
"""
Returns an instance of a loaded password hasher.
Identifies hasher algorithm by examining encoded hash, and calls
get_hasher() to return hasher. Raises ValueError if
algorithm cannot be identified, or if hasher is not loaded.
"""
if len(encoded) == 32 and '$' not in encoded:
algorithm = 'unsalted_md5'
else:
algorithm = encoded.split('$', 1)[0]
return get_hasher(algorithm)
So the best way is convert the current password hashes to the format: alg$salt$hash
class SHA1ProjPasswordHasher(BasePasswordHasher):
"""
Special snowflake algorithm from the first version.
php code: $pass=substr(sha1(trim($_POST['password'])),0,8);
"""
algorithm = "unsalted_and_trimmed_sha1"
def salt(self):
return ''
def encode(self, password, salt):
assert password
assert '$' not in salt
hash = hashlib.sha1(force_bytes(salt + password)).hexdigest()[:8]
return "%s$%s$%s" % (self.algorithm, salt, hash)
def verify(self, password, encoded):
algorithm, salt, hash = encoded.split('$', 2)
assert algorithm == self.algorithm
encoded_2 = self.encode(password, salt)
return constant_time_compare(encoded, encoded_2)
def safe_summary(self, encoded):
algorithm, salt, hash = encoded.split('$', 2)
assert algorithm == self.algorithm
return SortedDict([
(_('algorithm'), algorithm),
(_('salt'), mask_hash(salt, show=2)),
(_('hash'), mask_hash(hash)),
])
.
>>> from django.contrib.auth import authenticate
>>> x = authenticate(username='root', password='test')
>>> x
<User: root>
>>> x.password
u'unsalted_and_trimmed_sha1$$a94a8fe5'

How to serialize cleaned_data if it contains models?

I'm trying to serialize some form data so that I can stuff it into a hidden field until the user is ready to submit the whole form (think of a wizard).
I'm trying this:
print simplejson.dumps(vehicle_form.cleaned_data)
But I keep getting errors like this:
<VehicleMake: Honda> is not JSON serializable
Really I just need it to output the PK for "Honda".
This doesn't work either:
print serializers.serialize('json', vehicle_form.cleaned_data)
Gives:
'str' object has no attribute '_meta'
Presumably because it's iterating over the keys, which are all strings, whereas I think it expects a queryset, which I don't have.
So how do I do this?
Okay, so far I've come up with this:
from django.utils.simplejson import JSONEncoder, dumps, loads
from django.utils.functional import curry
from django.db.models import Model
from django.db.models.query import QuerySet
from django.core.serializers import serialize
class DjangoJSONEncoder(JSONEncoder):
def default(self, obj):
if isinstance(obj, Model):
return obj.pk
elif isinstance(obj, QuerySet):
return loads(serialize('json', obj, ensure_ascii=False))
return JSONEncoder.default(self, obj)
json_encode = curry(dumps, cls=DjangoJSONEncoder)
json_decode = loads
Based on the answers I found [here][1]. Now I'm trying this:
json = json_encode(vehicle_form.cleaned_data)
data = json_decode(json)
vehicle = Vehicle(**data)
The first 2 lines work perfectly, but the 3rd results in an exception:
Cannot assign "3": "Vehicle.model" must be a "VehicleModel" instance.
Getting close! Not sure how to deal with this one though...
This is a bit of a hack, but I don't know of a better way:
try:
vehicle_data = simplejson.loads(request.POST['vehicle_data'])
except ValueError:
vehicle_data = []
vehicle_data.append(vehicle_form.raw_data())
request.POST['vehicle_data'] = simplejson.dumps(vehicle_data)
It grabs the JSON data from hidden field in your form and decodes it into a Python dict. If it doesn't exist, it starts a new list. Then it appends the new raw/uncleaned data and re-encodes it and dumps it into the hidden field.
For this to work you need to either make a copy of the POST data (request.POST.copy()) so that it becomes mutable, or hack it like I did: request.POST._mutable = True
In my form template I put this:
<input type="hidden" name="vehicle_data" value="{{request.POST.vehicle_data}}" />
And lastly, to access the raw data for a form I added these methods:
from django.forms import *
def _raw_data(self):
return dict((k,self.data[self.add_prefix(k)]) for k in self.fields.iterkeys())
def _raw_value(self, key, value=None):
if value is None:
return self.data[self.add_prefix(key)]
self.data[self.add_prefix(key)] = value
Form.raw_data = _raw_data
Form.raw_value = _raw_value
ModelForm.raw_data = _raw_data
ModelForm.raw_value = _raw_value
Since .data returns too much data (in fact, it just returns the POST data you initially passed in), plus it's got the prefixes, which I didn't want.