Issue in scraping data from a website using beautiful soup - python-2.7

Question

I am trying to scrape list of 41 items & their prices from a website. But my output csv is missing some 2-3 items which come at the end of the page. Reason for this being, some devices have their price mentioned in different class than rest of the devices.
Recursion in my code is running for name and price together and for items where price is mentioned under different class, it is taking the price value from the next device. Hence, it is skipping last 2-3 items as prices for those devices already entered in recursion for previous devices.
Below is the referred code:
# -*- coding: cp1252 -*-
import csv
import urllib2
import sys
import time
from bs4 import BeautifulSoup
page = urllib2.urlopen('http://www.att.com/shop/wireless/devices/smartphones.deviceListGridView.xhr.flowtype-NEW.deviceGroupType-Cellphone.paymentType-postpaid.packageType-undefined.html?taxoStyle=SMARTPHONES&showMoreListSize=1000').read()
soup = BeautifulSoup(page)
soup.prettify()
with open('AT&T_2012-12-28.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',')
spamwriter.writerow(["Date","Month","Day of Week","Device Name","Price"])
items = soup.findAll('a', {"class": "clickStreamSingleItem"},text=True)
prices = soup.findAll('div', {"class": "listGrid-price"})
for item, price in zip(items, prices):
textcontent = u' '.join(price.stripped_strings)
if textcontent:
spamwriter.writerow([time.strftime("%Y-%m-%d"),time.strftime("%B"),time.strftime("%A") ,unicode(item.string).encode('utf8').replace('™','').replace('®','').strip(),textcontent])
Price are usually mentioned under listGrid-price but for some 2-3 items which are outofstock at the moment price is under listGrid-price-outOfStock I need to include this also in my recursion so that right price comes before the item and loop runs for all the devices.
Please pardon my ignorance as I am new to programming

Answer 1

You can use a comparator function, to make custom comparison and pass it to your findAll().
So if you modify your line with prices assignment to:
prices = soup.findAll('div', class_=match_both)
and define the function as:
def match_both(arg):
if arg == "listGrid-price" or arg == "listGrid-price-outOfStock":
return True
return False
(function can be made much more concise, verbosity here just to give you an idea of how it works)
it will thus compare to both and return a match in any of the cases.
More info can be found in documentation. (The has_six_characters variant)
Now, since you also asked how to exclude particular text.
text argument to findAll() can also have custom comparators.
So in this case, you don't want text saying Write a review to match and cause a shift in price vs text.
Hence your edited script to exclude review part:
# -*- coding: cp1252 -*-
import csv
import urllib2
import sys
import time
from bs4 import BeautifulSoup
def match_both(arg):
if arg == "listGrid-price" or arg == "listGrid-price-outOfStock":
return True
return False
def not_review(arg):
if not arg:
return arg
return "Write a review" not in arg
page = urllib2.urlopen('http://www.att.com/shop/wireless/devices/smartphones.deviceListGridView.xhr.flowtype-NEW.deviceGroupType-Cellphone.paymentType-postpaid.packageType-undefined.html?taxoStyle=SMARTPHONES&showMoreListSize=1000').read()
soup = BeautifulSoup(page)
soup.prettify()
with open('AT&T_2012-12-28.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',')
spamwriter.writerow(["Date","Month","Day of Week","Device Name","Price"])
items = soup.findAll('a', {"class": "clickStreamSingleItem"},text=not_review)
prices = soup.findAll('div', class_=match_both)
for item, price in zip(items, prices):
textcontent = u' '.join(price.stripped_strings)
if textcontent:
spamwriter.writerow([time.strftime("%Y-%m-%d"),time.strftime("%B"),time.strftime("%A") ,unicode(item.string).encode('utf8').replace('™','').replace('®','').strip(),textcontent])