How would one go about implementing a live feedback on what the spider is doing ?
I have the logs that tell me each second pages per second and items, but i would like to access that via a webservice.
Maybe I'm wrong, but I found in the scrapy docs that there is something that should do this http://localhost:6080/stats/spider_stats but it returns empty.
Any ideas are welcome.
I would like to be able to do something like curl http://localhost:6800/stats.json -d project=default -d spider=somespider
and it would return {"pages_crawled": "650","pages_per_min":"342","items_scraped":"286","items_per_min":"156"}
Thanks,
Related
I have not utilized this part of Django before, but I have an endpoint which is giving me a 403 error and is telling me that my request needs a csrf token. I was trying to figure out how best to get this since I was attempting to set up a bunch of curl requests to handle some simple queries to the endpoint. Likewise, I was thinking to also use POSTman, but I was not sure where documentation is to handle these request.
I have seen the cookie csrftoken, but when I was attempting to curl with it, it was still giving me a 403. thought it would looking something like this:
curl -d #profilepicturev2.png -b "csrftoken=Ebfn2OlfhSwFjAEQdoQon7wUjbynFoJqrtHMNPla3cy7ZfCMT9cxZ3OQHsbaedam" http://127.0.0.1:8000/api/files/uploader
Maybe I am mistaken? I am trying to send a photo to the server, so i was thinking that this would be correct and wasnt sure if i needed to add additional params in order to append additional data information.
i need to see your code, but i think you need to install "pillow" to send pictures in django !
I have a django app built with graphene and I have a problem running a simple POST query for the GraphQL endpoint, it keeps returning a 400 Bad request syntax.
but it should work since I don't have any problems running the query from the endpoint http://localhost:8000/graphql-dev
and I can't see any issues in the way I send the postman request.
I looked online for suitable solutions but couldn't find any that would help.
Any help/tips would be greatly appreciated.
EDIT:
I still didn't manage to see why I'm having this issue with postman, but here are some observations:
first, i changed the request to GET (since in graphql, query is for GET and mutation for POST - sorry, I missed that)
I tried the same request with postman (which didn't work) and with insomnia (which did)
with postman
with insomnia
What's weird is that if i check my django console the requests look the same.
EDIT2: okay, I figured it out...removing the Content-Type application/json did the trick. Now it works with postman as well.
I have been struggling with Jenkins lately, and I'm stuck because I wanna send some parameters through HTTP Post, and I know how to do it, but the thing is that I am saving a Http request response to a file in my workspace, and then I want to use that file, read it and send the text I saved previously to a new HTTP Request, does anyone have any idea how can I achieve this?
Thanks in advance!!!
Install copy artifacts from another project plugin ( copy artifacts) add in build steps store the file in your workspace then you can run a shell script to read the desired content from that file .
if curl would work, that would be a simple way to send a file's contents as your POST body. see this answer.
Jenkins can work with Jmeter and Jmeter is great tool for handling request and response see tutorial
I have just started to use Siege to do load/stressing test on a new web server. I am trying to test on my most resource/performance heavy script, but the script requires cookies. What is the proper format for using the --header option in siege?
I have tried this with no luck:
siege --header="Set-Cookie: PHPSESSID=--COOKIE--; iptoken=--COOKIE--" http://www.myurl.com/script.php,
There is no documentation on this that I could find, so any ideas/suggestions would be appreciated.
The answer is to use --header="Cookie: --COOKIE_DATA--" (ref. wiki.wsmoak.net/cgi-bin/wiki.pl?Siege).
I read that REST WS can be called from a command line. I know that they can be invoked from program code, but how directly from command line? I looked around the internet to know how this can be done, but didn't find any information on this. Can anyone please tell me how this can be done? I have developed a rest ws app in grails.
You can use curl from the commandline of any unix-like os.
For example, you can test some sample Facebook API like this:
curl https://graph.facebook.com/19292868552
or to POST:
curl -X POST -d "param1=value1¶m2=value2" http://example.com/resource.cgi
You could use curl to ask a URL. I guess your web service can be queried using HTTP?
http://curl.haxx.se/
I believe you want to use cURL to do that.