How to get started with Scrapy Web Service? [closed] - web-services

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I have been using Scrapy for a long time now and I must say I am in love with it. Recently, I got to know about Scrapy Web Service. But I am unable to figure out how it works. Or how can I use it to monitor my current spiders.
Looking for a documentation or getting started guide.

Answer 1

So, that document you've linked is the closest thing to a guide there is. It doesn't do much hand-holding but should be enough to get you going -- it even comes with an example client script.
In short, the web service is enabled by default, and it answers to HTTP requests using the JSON RPC protocol for remote calls.
You can try it out after starting scrapy shell inside a Scrapy project, point your browser to: http://localhost:6080. You should see a page showing the available resources:
{"resources": ["enginestatus", "stats", "crawler"]}
From this point onwards, you will advance knowing a bit more of JSON RPC Version 2, to construct the proper calls -- here is an example using requests:
>>> import requests, json
>>> jsonrpc_call = {'jsonrpc': '2.0', 'method': 'list', 'params': [], 'id': 1}
>>> resource = 'http://localhost:6080/crawler/spiders'
>>> requests.post(resource, data=json.dumps(jsonrpc_call)).json()
{u'id': 1, u'jsonrpc': u'2.0', u'result': [u'spider1', u'spider2']}
IMPORTANT: You should know that this JSON RPC service has been extracted out from Scrapy for the next release, and now lives as a separate project: https://github.com/scrapy/scrapy-jsonrpc. So, be prepared when upgrading to Scrapy 1.0. ;)