I'am trying to update version of my spider, i wrote:
curl http://localhost:6800/addversion.json -d project=comicvn -d spider=comicvn2 -d version= 141667324 -d egg=14116674324.egg
It made error : {"status"": error,"message": "ValuesError: Unkow or corrupt egg"
What's egg paramerter ? and what should I set values for it ? Thank so much !
I created '14116674324.egg' by ussing command bdist_egg. After that, it made a 14116674324.egg file in path :C:\Python27\Scripts\eggs\comicvn2 , so i copy this file to root folder of my project like C:\Python27\Scripts\comicvn2.
curl http://localhost:6800/addversion.json -d project=comicvn -d spider=comicvn2 -d version= 141667324 -d egg=#14116674324.egg
you must be add '#'
see scrapy.cfg format at scrapy deploy doc page. change version entry to update the spider version
[deploy]
url = http://scrapyd.mydomain.com
username = john
password = secret
project = project1
version = 123
Related
sql file that I backedup a long time ago of my website database and now I'm trying to use it but everytime I use:
psql -U <username> -d --password <dbname> -1 -f <filename>.sql
it gives me this error
peer authentication failed for user "USER"
I tried to edit the pg_hba.conf but that did nothing.
how can I do it ?
Several mistakes:
You didn't edit pg_hba.conf properly or you forgot to reload PostgreSQL.
Add your pg_hba.conf to the question for more details.
Your command line is syntactically incorrect; --password will be interpreted as argument of -d. Try
psql -U <username> -d <dbname> --password -1 -f <filename>.sql
This is the query I am trying to use in order to connect to elasticsearch, which is in (172.21.150.230) in order to pull out information in a csv format:
es2csv -u http://xxx.xx.xxx.xxx:5601/ -f _all -d doc -i test2 -r -q '{"query": {"match": {"NAME": "xxx"}}}' -o database.csv
However, I get SyntaxError: invalid syntax
Thanks
Which version are you using?
This error raised when your json query is not valid or when you forget to add -r argument.
I've a few APIs I'd like to test with cURL. I tried doing a GET as follows:
curl --user username:password --request GET http://my_domain/get_result/52d6428f3ea9a008358ad2d8/
On the server, it showed a '302' (which means redirection, right?). I'm guessing it redirected to the 'login/' page.
What is the proper way of getting this done?
Edit: I tried:
curl -c cookies.txt -b cookies.txt -L -d #login_form.txt http://my_domain/login/
where login_form.txt contains "username=username&password=password&this_is_the_login_form=1". Doesn't work. No cookies.txt files generated. And no login happening. Can you tell me how you achieve login to Django using cURL?
Here is a fully coded answer. The idea of the solution is:
you have to first visit the login page with GET to get the cookies file generated,
then parse the CSRF token out of the cookies file
and do the login using a POST request, passing the data with -d.
Afterwards you can perform any request always using that CSRF token in the data ($DJANGO_TOKEN) or with a custom X-CSRFToken header. To log out simply delete the cookies file.
Note that you need a referer (-e) to make Django's CSRF checks happy.
LOGIN_URL=https://yourdjangowebsite.com/login/
YOUR_USER='username'
YOUR_PASS='password'
COOKIES=cookies.txt
CURL_BIN="curl -s -c $COOKIES -b $COOKIES -e $LOGIN_URL"
echo -n "Django Auth: get csrftoken ..."
$CURL_BIN $LOGIN_URL > /dev/null
DJANGO_TOKEN="csrfmiddlewaretoken=$(grep csrftoken $COOKIES | sed 's/^.*csrftoken\s*//')"
echo -n " perform login ..."
$CURL_BIN \
-d "$DJANGO_TOKEN&username=$YOUR_USER&password=$YOUR_PASS" \
-X POST $LOGIN_URL
echo -n " do something while logged in ..."
$CURL_BIN \
-d "$DJANGO_TOKEN&..." \
-X POST https://yourdjangowebsite.com/whatever/
echo " logout"
rm $COOKIES
I have a slightly more secure version of this code, which uses a file for submitting the POST data, as a Gist on GitHub: django-csrftoken-login-demo.bash
Interesting background reading on Django's CSRF token is on docs.djangoproject.com.
Passing username:password in a curl request is only good for HTTP Authentication, which isn't how most websites do auth these days. Instead, you'll have to post to the login page, get the cookie, then pass it back when requesting your desired page.
Actually #Paterino answer is correct but it will not work on every implementation of sed. Instead sed 's/^.*csrftoken\s*//') we can use sed 's/^.*csrftoken[[:blank:]]*//') which is more old fashioned. MacOSXs curl doesn't use escaping, so \n\t\s don't work at all.
To use the token with a get request, use
$CURL_BIN \
-H "$DJANGO_TOKEN" \
-X GET https://yourdjangowebsite.com/whatever/
I tried using -d with -X GET, however it resulted in weird socket behaviour on the server side (Heruko H18 errors).
I'm using Django 4.1.2 and trying the #Paterino method found a couple of changes to make it work (but i have not enogh reputation to comment so wrote another answer).
Firstly, if the generated cookies.txt file is empty you have to ensure than csrf cookie is generated. I achieved this using django.views.decorators.csrf.ensure_csrf_cookie in django.contrib.auth.views.LoginView
Now, after login cookies.txt changes, so you have to recalculate DJANGO_TOKEN variable in the same way:
DJANGO_TOKEN="csrfmiddlewaretoken=$(grep csrftoken $COOKIES | sed 's/^.*csrftoken\s*//')"
From here the method doesn't change.
the accepted answer, until now(2022-12-19), has 2 issues:
misses updating DJANGO_TOKEN after login (since a new csrftoken cookie is returned after login)
doesn't include an example with a POST request (moving the csrftoken to a header) where -d already contains some payload
here is my version dealing with both:
# user and password from `./manage.py createsuperuser`
YOUR_USER='user'
YOUR_PASS='pass'
COOKIES=cookies.txt
LOGIN_URL=http://localhost:8000/admin/login/
# stores csrftoken cookie on cookies.txt
curl -s -c $COOKIES $LOGIN_URL > /dev/null
TOKEN_VALUE="$(grep -oP '(?<=csrftoken[[:space:]]).*' cookies.txt)" # https://stackoverflow.com/a/10358949/3026886 https://stackoverflow.com/a/4233691/3026886
# logs in, updating csrftoken and adding sessionid cookies
curl -b $COOKIES -c $COOKIES -d "csrfmiddlewaretoken=$TOKEN_VALUE&username=$YOUR_USER&password=$YOUR_PASS" $LOGIN_URL
# updates var env with new cookie
TOKEN_VALUE="$(grep -oP '(?<=csrftoken[[:space:]]).*' cookies.txt)"
# here comes the real request
curl -s -X POST -b $COOKIES -d "{\"a\":1}" -H "X-CSRFToken: $TOKEN_VALUE" http://localhost:8000/yourViewReceivingJsonPayload/ > /dev/null
rm cookies.txt
I'd like to use the existing Active Directory system for authentication of users, but there is a problem.
This works:
ldapsearch -D "LOGIN#dom.comp.local" -x -w PASSWORD -b "OU=users,OU=otdel,DC=dom,DC=comp,DC=local" -h x.x.x.x
Where should I enter the username to connect to the AD?
AUTH_LDAP_SERVER_URI = "ldap://ldap.example.com"
AUTH_LDAP_BIND_DN = ???
AUTH_LDAP_BIND_PASSWORD = "PASSWORD"
AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,ou=otdel,dc=dom,dc=comp,dc=local", ldap.SCOPE_SUBTREE, "(cn=%(user)s)")
I'm on Python 3 exclusively these days, and the only package I've been able to get working with Active Directory is django-python3-ldap. While configured for OpenLDAP by default, it is a cinch to configure it for Active Directory, and even documented:
https://github.com/etianen/django-python3-ldap
Give it a look, and good luck.
In your ldapsearch string, the -D argument in the binddn. So, assuming LOGIN#dom.comp.local works, you should be able to set AUTH_LDAP_BIND_DN to that same value.
I've a few APIs I'd like to test with cURL. I tried doing a GET as follows:
curl --user username:password --request GET http://my_domain/get_result/52d6428f3ea9a008358ad2d8/
On the server, it showed a '302' (which means redirection, right?). I'm guessing it redirected to the 'login/' page.
What is the proper way of getting this done?
Edit: I tried:
curl -c cookies.txt -b cookies.txt -L -d #login_form.txt http://my_domain/login/
where login_form.txt contains "username=username&password=password&this_is_the_login_form=1". Doesn't work. No cookies.txt files generated. And no login happening. Can you tell me how you achieve login to Django using cURL?
Here is a fully coded answer. The idea of the solution is:
you have to first visit the login page with GET to get the cookies file generated,
then parse the CSRF token out of the cookies file
and do the login using a POST request, passing the data with -d.
Afterwards you can perform any request always using that CSRF token in the data ($DJANGO_TOKEN) or with a custom X-CSRFToken header. To log out simply delete the cookies file.
Note that you need a referer (-e) to make Django's CSRF checks happy.
LOGIN_URL=https://yourdjangowebsite.com/login/
YOUR_USER='username'
YOUR_PASS='password'
COOKIES=cookies.txt
CURL_BIN="curl -s -c $COOKIES -b $COOKIES -e $LOGIN_URL"
echo -n "Django Auth: get csrftoken ..."
$CURL_BIN $LOGIN_URL > /dev/null
DJANGO_TOKEN="csrfmiddlewaretoken=$(grep csrftoken $COOKIES | sed 's/^.*csrftoken\s*//')"
echo -n " perform login ..."
$CURL_BIN \
-d "$DJANGO_TOKEN&username=$YOUR_USER&password=$YOUR_PASS" \
-X POST $LOGIN_URL
echo -n " do something while logged in ..."
$CURL_BIN \
-d "$DJANGO_TOKEN&..." \
-X POST https://yourdjangowebsite.com/whatever/
echo " logout"
rm $COOKIES
I have a slightly more secure version of this code, which uses a file for submitting the POST data, as a Gist on GitHub: django-csrftoken-login-demo.bash
Interesting background reading on Django's CSRF token is on docs.djangoproject.com.
Passing username:password in a curl request is only good for HTTP Authentication, which isn't how most websites do auth these days. Instead, you'll have to post to the login page, get the cookie, then pass it back when requesting your desired page.
Actually #Paterino answer is correct but it will not work on every implementation of sed. Instead sed 's/^.*csrftoken\s*//') we can use sed 's/^.*csrftoken[[:blank:]]*//') which is more old fashioned. MacOSXs curl doesn't use escaping, so \n\t\s don't work at all.
To use the token with a get request, use
$CURL_BIN \
-H "$DJANGO_TOKEN" \
-X GET https://yourdjangowebsite.com/whatever/
I tried using -d with -X GET, however it resulted in weird socket behaviour on the server side (Heruko H18 errors).
I'm using Django 4.1.2 and trying the #Paterino method found a couple of changes to make it work (but i have not enogh reputation to comment so wrote another answer).
Firstly, if the generated cookies.txt file is empty you have to ensure than csrf cookie is generated. I achieved this using django.views.decorators.csrf.ensure_csrf_cookie in django.contrib.auth.views.LoginView
Now, after login cookies.txt changes, so you have to recalculate DJANGO_TOKEN variable in the same way:
DJANGO_TOKEN="csrfmiddlewaretoken=$(grep csrftoken $COOKIES | sed 's/^.*csrftoken\s*//')"
From here the method doesn't change.
the accepted answer, until now(2022-12-19), has 2 issues:
misses updating DJANGO_TOKEN after login (since a new csrftoken cookie is returned after login)
doesn't include an example with a POST request (moving the csrftoken to a header) where -d already contains some payload
here is my version dealing with both:
# user and password from `./manage.py createsuperuser`
YOUR_USER='user'
YOUR_PASS='pass'
COOKIES=cookies.txt
LOGIN_URL=http://localhost:8000/admin/login/
# stores csrftoken cookie on cookies.txt
curl -s -c $COOKIES $LOGIN_URL > /dev/null
TOKEN_VALUE="$(grep -oP '(?<=csrftoken[[:space:]]).*' cookies.txt)" # https://stackoverflow.com/a/10358949/3026886 https://stackoverflow.com/a/4233691/3026886
# logs in, updating csrftoken and adding sessionid cookies
curl -b $COOKIES -c $COOKIES -d "csrfmiddlewaretoken=$TOKEN_VALUE&username=$YOUR_USER&password=$YOUR_PASS" $LOGIN_URL
# updates var env with new cookie
TOKEN_VALUE="$(grep -oP '(?<=csrftoken[[:space:]]).*' cookies.txt)"
# here comes the real request
curl -s -X POST -b $COOKIES -d "{\"a\":1}" -H "X-CSRFToken: $TOKEN_VALUE" http://localhost:8000/yourViewReceivingJsonPayload/ > /dev/null
rm cookies.txt