Run CrawlerProcess in Scrapy with Splash - python-2.7

Question

I have a scrapy+splash file to crawl data. Now I want to run my scrapy file by script so I use CrawlerProcess. My file is like this:
import scrapy
from scrapy_splash import SplashRequest
from scrapy.crawler import CrawlerProcess
class ProvinceSpider(scrapy.Spider):
name = 'province'
def start_requests(self):
url = "https://e.vnexpress.net/covid-19/vaccine"
yield SplashRequest(url=url,callback=self.parse)
def parse(self, response):
provinces = response.xpath("//div[#id='total_vaccine_province']/ul[#data-weight]")
for province in provinces:
yield{
'province_name':province.xpath(".//li[1]/text()").get(),
'province_population':province.xpath(".//li[2]/text()").get(),
'province_expected_distribution':province.xpath(".//li[3]/text()").get(),
'province_actual_distribution':province.xpath(".//li[4]/text()").get(),
'province_distribution_percentage':province.xpath(".//li[5]/div/div/span/text()").get(),
}
process = CrawlerProcess(settings={
"FEEDS": {
"province.json": {"format": "json"},
},
})
process.crawl(ProvinceSpider)
process.start() # the script will block here until the crawling is finished
But when I run
python3 province.py
It doesn't connect to splash server thus can't crawl data. Any idea about which part I do wrong? Tks in advance

Answer 1

Turns out the issue you actually experienced has been covered by the following answer here: Answer
A quick break-down (if you're not interested in the details):
Go to settings.py and add a USER-AGENT, in my case I left it as:
USER_AGENT = 'testit (http://www.yourdomain.com)'
Then run your crawler and it should work. Why? your scrapy is being blocked by the site.
Output:
2021-12-26 13:15:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://e.vnexpress.net/covid-19/vaccine>
{'province_name': 'HCMC', 'province_population': '7.2M', 'province_expected_distribution': '13.8M', 'province_actual_distribution': '14.6M', 'province_distribution_percentage': '100%'}
2021-12-26 13:15:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://e.vnexpress.net/covid-19/vaccine>
{'province_name': 'Hanoi', 'province_population': '6.2M', 'province_expected_distribution': '11.4M', 'province_actual_distribution': '12.3M', 'province_distribution_percentage': '99,2%'}
2021-12-26 13:15:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://e.vnexpress.net/covid-19/vaccine>
{'province_name': 'Dong Nai', 'province_population': '2.4M', 'province_expected_distribution': '4.3M', 'province_actual_distribution': '5M', 'province_distribution_percentage': '100%'}
...
...
Here's my custom settings:
BOT_NAME = 'testing'
SPIDER_MODULES = ['testing.spiders']
NEWSPIDER_MODULE = 'testing.spiders'
SPLASH_URL = 'http://localhost:8050'
USER_AGENT = 'testing (http://www.yourdomain.com)'
ROBOTSTXT_OBEY = False
DEFAULT_REQUEST_HEADERS = {
'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Safari/605.1.15'
}
SPIDER_MIDDLEWARES = {
'testing.middlewares.TestingSpiderMiddleware': 100,
}
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'