I'm trying to scrape the page which loads after the print popup is gone(canceled).
testing the xpath to the product name and ID (as shown in the screenshot) with any possible combinations so far return empty and I suspect that print popup js is the reason.
Any tips about how to bypass the print popup would be appreciated.
Thanks :)
Here is the screenshot from the DOM:
Here's an example spider for getting the text you've highlighted on the screenshot:
from scrapy.item import Item, Field
from scrapy.selector import Selector
from scrapy.spider import BaseSpider
class MarketItem(Item):
name = Field()
class MarketSpider(BaseSpider):
name = "market"
allowed_domains = ["mymarket.ge"]
start_urls = ["http://www.mymarket.ge/classified_details_print.php?product_id=5827165"]
def parse(self, response):
contacts = Selector(response)
item = MarketItem()
item['name'] = contacts.xpath('//td[#class="product_info_details_text"]/b/text()').extract()[0].strip()
return item
this gets an item:
{'name': u'Nokia asha 515 dual sim'}
Hope that helps.
Related
I'm a newbie of Scrapy & Python. I try to get the comment from the following URL but the result always null : http://vnexpress.net/tin-tuc/oto-xe-may/toyota-camry-2016-dinh-loi-tui-khi-khong-bung-3386676.html
Here is my code :
from scrapy.spiders import Spider
from scrapy.selector import Selector
from tutorial.items import TutorialItem
import logging
class TutorialSpider(Spider):
name = "vnexpress"
allowed_domains = ["vnexpress.net"]
start_urls = [
"http://vnexpress.net/tin-tuc/oto-xe-may/toyota-camry-2016-dinh-loi-tui-khi-khong-bung-3386676.html"
]
def parse(self, response):
sel = Selector(response)
commentList = sel.xpath('//div[#class="comment_item"]')
items = []
id = 0;
logging.log(logging.INFO, "TOTAL COMMENT : " + str(len(commentList)))
for comment in commentList:
item = TutorialItem()
id = id + 1
item['id'] = id
item['mainId'] = 0
item['user'] = comment.xpath('//span[#class="left txt_666 txt_11"]/b').extract()
item['time'] = 'N/A'
item['content'] = comment.xpath('//p[#class="full_content"]').extract()
item['like'] = comment.xpath('//span[#class="txt_666 txt_11 right block_like_web"]/a[#class="txt_666 txt_11 total_like"]').extract()
items.append(item)
return items
Thanks for reading
Looks like the comments are loaded into the page with some JavaScript code.
Scrapy does not execute JavaScript on a page, it only downloads HTML pages. Try opening the page with JavaScript disabled in your browser, and you should see the page as Scrapy sees it.
You have a handful of options:
reverse-engineer how the comments are loaded into the page, using your browser's developer tools panel, in "network" tab (it could be some XHR call loading HTML or JSON data)
use a (headless)browser to render the page (selenium, casper.js, splash...);
e.g. you may want to try this page with Splash (one of the JavaScript rendering options for web scraping). This is the HTML you get back from Splash (it contains the comments): http://pastebin.com/njgCsM9w
I was working with ma project XYZ
and I got stuck in extracting text in from the source
gifts
I want to extrack the href as content
I tried this
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from XYZ.items import XYZ
class MySpider(BaseSpider):
name = "main"
allowed_domains = ["XYZ"]
start_urls = ["XYZ"]
def parse(self, response):
hxs = HtmlXPathSelector(response)
titles = hxs.select("//a[#data-tracking-id='mdd-heading']")
items = []
for titles in titles:
item = XYZ()
item ["title"] = titles.select("text()").extract()
item ["link"] = titles.select("#href").extract()
items.append(item)
print "www.xyz.com"+str(item["link"])
return items
and the output was
www.xyz.com[u'/gifts']
I was expecting output as
www.xyz.com/gifts
What i did wrong.... ?
According to the documentation for Selector's extract():
extract()
Serialize and return the matched nodes as a list of unicode
strings. Percent encoded content is unquoted.
So, extract() returns a list and you need the first item from it. Use item['link'][0].
Also, there are other problems in the code:
for titles in titles loop doesn't make sense, you need a separate loop variable
HtmlXPathSelector is deprecated, use Selector
use urljoin() to join the parts of a url
Here's the complete code with fixes and other improvements:
from urlparse import urljoin
from scrapy.spider import BaseSpider
from scrapy.selector import Selector
from XYZ.items import XYZ
class MySpider(BaseSpider):
name = "main"
allowed_domains = ["XYZ"]
start_urls = ["XYZ"]
def parse(self, response):
titles = response.xpath("//a[#data-tracking-id='mdd-heading']")
for title in titles:
item = XYZ()
item ["title"] = title.xpath("text()").extract()[0]
item ["link"] = title.xpath("#href").extract()[0]
print urljoin("www.xyz.com", item["link"])
yield item
I'm scraping few items from this site, but it grabs items only from the first product and doesn't loop further. I know I'm doing simple stupid mistake, but if you can just point out where I got this wrong, I'll appreciate it.
Here is the spider:
from scrapy.item import Item, Field
from scrapy.spider import BaseSpider
from scrapy.selector import Selector
import re
from zoomer.items import ZoomerItem
class ZoomSpider(BaseSpider):
name = "zoomSp"
allowed_domains = ["zoomer.ge"]
start_urls = [
"http://zoomer.ge/index.php?cid=35&act=search&category=1&search_type=mobile"
]
def parse(self, response):
sel = Selector(response)
titles = sel.xpath('//div[#class="productContainer"]/div[5]')
items = []
for t in titles:
item = ZoomerItem()
item["brand"] = t.xpath('//div[#class="productListContainer"]/div[3]/text()').re('^([\w, ]+)')
item["price"] = t.xpath('//div[#class="productListContainer"]/div[4]/text()').extract()[0].strip()
item["model"] = t.xpath('//div[#class="productListContainer"]/div[3]/text()').re('\s+(.*)$')[0].strip()
items.append(item)
return(items)
P.S. Also can't get regex for "brand" string to get only the first word "Blackberry" from the string:
"BlackBerry P9981 Porsche Design".
The <div/> element with the class productContainer is just a container and only appears one time, thus it is not repeating. The repeating element which you want to iterate over is the one with the class productListContainer.
def parse(self, response):
sel = Selector(response)
titles = sel.xpath('//div[#class="productContainer"]/div[5]/div[#class="productListContainer"]')
items = []
for t in titles:
item = ZoomerItem()
item["brand"] = t.xpath('div[3]/text()').re('^([\w\-]+)')
item["price"] = t.xpath('div[#class="productListPrice"]/div/text()').extract()
item["model"] = t.xpath('div[3]/text()').re('\s+(.*)$')[0].strip()
items.append(item)
items.append(item)
return(items)
This function is untested as I am not a python guy, so you might have to fiddle around a bit.
As a beginner I'm having a hard time, so I'm here to ask for help.
I'm trying to extract prices from the html page, which are nested deeply:
second price location:
from scrapy.spider import Spider
from scrapy.selector import Selector
from mymarket.items import MymarketItem
class MySpider(Spider):
name = "mymarket"
allowed_domains = ["url"]
start_urls = [
"http://url"
]
def parse(self, response):
sel = Selector(response)
titles = sel.xpath('//table[#class="tab_product_list"]//tr')
items = []
for t in titles:
item = MymarketItem()
item["price"] = t.xpath('//tr//span[2]/text()').extract()
items.append(item)
return items
I'm trying to export scraped prices to csv. they do export but are being populated like this:
And I want them to be sorted like this in .csv:
etc.
Can anybody point out where is the faulty part of the xpath or how I can make prices be sorted "properly" ?
It's difficult to say what's wrong with the path. Install firepath extension for Firefox to test your xpath queries. One note for now:
titles = sel.xpath('//table[#class="tab_product_list"]//tr')
In your screenshot you have nested tables, so //tr will give trs from nested tables too.
def parse(self, response):
sel = Selector(response)
titles = sel.xpath('//table[#class="tab_product_list"]/tr') # or with tbody
items = []
for t in titles:
item = MymarketItem()
item["price"] = t.xpath('.//span[#style="color:red;"]/text()').extract()[0]
items.append(item)
return items
.extract() returns a list, even if just one argument found, take the first element of the list .extract()[0]
I have just started using Scrapy:
Here is an example of a website that I want to crawl :
http://www.thefreedictionary.com/shame
The code for my Spider :
from scrapy.spider import BaseSpider
from scrapy.http import Request
from scrapy.selector import HtmlXPathSelector
from dic_crawler.items import DicCrawlerItem
from urlBuilder import *
class Dic_crawler(BaseSpider):
name = "dic"
allowed_domains = ["www.thefreedictionary.com"]
start_urls = listmaker()[:]
print start_urls
def parse(self, response):
hxs = HtmlXPathSelector(response)
sites = hxs.select('//*[#id="MainTxt"]/table/tbody')
print 'SITES:\n',sites
item = DicCrawlerItem()
item["meanings"] = sites.select('//*[#id="MainTxt"]/table/tbody/tr/td/div[1]/div[1]/div[1]/text()').extract()
print item
return item
The listmaker() returns a list of urls to scrap.
My problem is that the sites variable comes up empty if I select till 'tbody' in the xpath and returns an empty sites variable, Whereas if I select only table I get the part of the site I want.
I am not able to retrieve the meaning for a word as a result of this into item["meanings"] since the part after tbody is does not select beyond tbody.
Also while at it, the site gives multiple meanings which I would like to extract but I only know how to extract a single method.
Thanks
Here's a spider skeleton to get you started:
from scrapy.spider import BaseSpider
from scrapy.http import Request
from scrapy.selector import HtmlXPathSelector
class Dic_crawler(BaseSpider):
name = "thefreedictionary"
allowed_domains = ["www.thefreedictionary.com"]
start_urls = ['http://www.thefreedictionary.com/shame']
def parse(self, response):
hxs = HtmlXPathSelector(response)
# loop on each "noun" or "verb" or something... section
for category in hxs.select('id("MainTxt")//div[#class="pseg"]'):
# this is simply to get what's in the <i> tag
category_name = u''.join(category.select('./i/text()').extract())
self.log("category: %s" % category_name)
# for each category, a term can have multiple definition
# category from .select() is a selector
# so you can call .select() on it also,
# here with a relative XPath expression selecting all definitions
for definition in category.select('div[#class="ds-list"]'):
definition_text = u'\n'.join(
definition.select('.//text()').extract())
self.log(" - definition: %s" % definition_text)