The below code returns the text including the html code. However, I need to retrieve the text only so that it can be loaded nicely into a pd.DataFrame. How do I 'strip' the text?
#importing packages
from bs4 import BeautifulSoup
import requests
#url
url = "https://example.com/this_is_just_an_example"
#request to get text from url
r = requests.get(url).text
#create soup version of the text
soup = BeautifulSoup(r, features="lxml")
#create a list to store the text
MyHeadlines= []
#appended the text to list Names
for i in soup.find_all('h3', {'class': 'headline'}):
MyHeadlines.append(str(i))
You can easily do this with some simple regex:
import re
CLEAN_TEXT = re.sub('<[^<]+?>', '', YOUR_TEXT)
Enjoy!
It's my first time using Python and BeautifulSoup. The thing is I'm doing a migration of all articles within a blog from one website to another, and to perform this, I'm extracting certain information from a xml file; the last part of my code tells to extract only the text between the position 0 and 164 from the meta tag, so this way it can appear on google SERP as they want to appear.
The problem here is some articles from the blog has img tags on the first lines inside the tag and I want to remove them, including the src attributes so the code can grab just the text after those img tags.
I tried to solve it in many ways but I did not succeed.
Here is my code:
from bs4 import BeautifulSoup
from urllib2 import urlopen
import csv
import sys
import re
reload(sys)
sys.setdefaultencoding('utf8')
base_url = ("http://pimacleanpro.com/blog?rss=true")
soup = BeautifulSoup(urlopen(base_url).read(),"xml")
titles = soup("title")
slugs = soup("link")
bodies = soup("description")
with open("blog-data.csv", "w") as f:
fieldnames = ("title", "content", "slug", "seo_title", "seo_description","site_id", "page_path", "category")
output = csv.writer(f, delimiter=",")
output.writerow(fieldnames)
for i in xrange(len(titles)):
output.writerow([titles[i].encode_contents(),bodies[i].encode_contents(formatter=None),slugs[i].get_text(),titles[i].encode_contents(),bodies[i].encode_contents(formatter=None)[4:164]])
print "Done writing file"
any help will be appreciated.
Here's a Python 2.7 example that I think does what you want:
from bs4 import BeautifulSoup
from urllib2 import urlopen
from xml.sax.saxutils import unescape
base_url = ("http://pimacleanpro.com/blog?rss=true")
# Unescape to allow BS to parse the <img> tags
soup = BeautifulSoup(unescape(urlopen(base_url).read()))
titles = soup("title")
slugs = soup("link")
bodies = soup("description")
print bodies[2].encode_contents(formatter=None)[4:164]
# Remove all 'img' tags in all the 'description' tags in bodies
for body in bodies:
for img in body("img"):
img.decompose()
print bodies[2].encode_contents(formatter=None)[4:164]
# Proceed to writing to CSV, etc.
The first print statement outputs the following:
<img src='"http://ekblog.s3.amazonaws.com/contentp/wp-content/uploads/2018/09/03082910/decoration-design-detail-691710-300x221.jpg"'><br>
<em>Whether you are up
While the second one after removing the <img> tags is as follows:
<em>Whether you are upgrading just one room or giving your home a complete renovation, it’s likely that your first thought is to choose carpet for all of
Of course you could just remove all image tags in the soup object before creating titles, slugs, or bodies if they're not of interest to you:
for tag in soup("img"):
tag.decompose()
from bs4 import BeautifulSoup
import re
import HTMLParser
import urllib
url = raw_input('enter - ')
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)
scripts=soup.find_all('script')
for tag in scripts:
try:
Script = tag["src"]
print Script
except:
print "No source"
using this code I m not getting all the java script from html document.
I have checked your code and it seems that you are getting all the javascript. At least you check for all the tags. Of course some of the javascript may be directly embedded into the html and thereby won't have a src attribute. Merely the actual javascript between the <script>...</script> tags. You can get the javscript between these embedded tags using tag.contents in your loop.
Furthermore, I would advise to specify a parser. By default bs4 uses html.parser. Other parsers may perform better/differently. Check out: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser
from bs4 import BeautifulSoup
import urllib2
r = urllib2.urlopen('<your url>').read()
soup = BeautifulSoup(r, 'html.parser')
for s in soup.findAll('script'):
print s.get('src')
Common piece of code:
# -*- coding: cp1252 -*-
import csv
import urllib2
import sys
import time
from bs4 import BeautifulSoup
from itertools import islice
page = urllib2.urlopen('http://www.vodafone.de/privat/tarife/red-smartphone-tarife.html').read()
soup = BeautifulSoup(page)
prices = soup.findAll('div', {"class": "price"})
After this I am trying following codes to get data:
Code 1:
for price in prices:
print unicode(price.string).encode('utf8')
Output1: No Output, code runs without any error and prints nothing.
Code 2:
for price in prices:
textcontent3= u' '.join(price.stripped_strings)
if textcontent3:
print textcontent3
Output2: No output again, same situation as in Output1.
Code 3:
for price in prices:
fonttag = price.find('div')
if fonttag is not None:
print unicode(fonttag.string).encode('utf8').strip()
Output3: No output, same as in Output1
After this I tried printing the concerned part of the html:
Code 4:
print prices
Output4:
</span></div>, <div class="price">
<span id="price"><br/>
</span></div>, <div class="price">
<span id="price"><br/>
</span></div>]
As it can be seen from the Output4, no price value is coming in the html beautiful soup is scraping for me. While on webpage this html structure looks like this:
<div class="price"><span id="price">49,90 €</span><br>einmalig</div>
Beautiful soup is not extracting the price values as mentioned in the html page, thus I am not able to scrape data for the price.
Please help me in solving this issue & pardon my ignorance as I am new to programming.
The page uses a large JavaScript structure to load the prices. You can load just that structure:
scripts = soup.find_all('script')
script = next(s.text for s in scripts if s.string and 'window.rates' in s.string)
datastring = script.split('phones=')[1].split(';window.')[0]
This results in a large JavaScript structure, starting with:
{sku844082:{name:"Samsung Galaxy SII",image:"/images/m677391_300468.jpg",deliveryTime:"Vorauss. verfügbar ab Anfang Januar",sku1444291:{p:"prod954312",e:"19.90"},sku1444286:{p:"prod954312",e:"19.90"},sku1444283:{p:"prod954312",e:"39.90"},sku1444275:{p:"prod954312",e:"59.90"},sku1104261:{p:"prod954312",e:"99.90"}},sku894279:{name:"BlackBerry Torch 9810",image:"/images/m727477_300464.jpg",deliveryTime:"Lieferbar innerhalb 48 Stunden",sku1444275:{p:"prod1004495",e:"179.90"},sku1104261:{p:"prod1004495",e:"259.90"},sku1444291:{p:"prod1004495",e:"29.90"},sku1444286:{p:"prod1004495",e:"29.90"},sku1444283:{p:"prod1004495",e:"49.90"}},sku864221:{name:"BlackBerry Bold 9900",image:"/images/m707491_300465.jpg",deliveryTime:"Lieferbar innerhalb 48 Stunden",sku1444275:{p:"prod974431",e:"129.90"},sku1104261:{p:"prod974431",e:"169.90"},sku1444291:{p:"prod974431",e:"49.90"},sku1444286:{p:"prod974431",e:"49.90"},sku1444283:{p:"prod974431",e:"89.90"}}
Unfortunately, that's not directly loadable with the json module; although valid JavaScript, without quoting around the keys it is not valid JSON. You'd need to use regular expressions to clean that up further, or grab the p:"someprice" information directly from that string.
Luckily the structure can be fixed with a small amount of regular expression magic:
import re
import json
datastring = re.sub(ur'([{,])([a-z]\w*):', ur'\1"\2":', datastring)
data = json.loads(datastring)
This gives you a large dictionary, with SKU keys and dictionaries with nested dicts as data, including nested SKUs with p product codes and e prices:
>>> from pprint import pprint
>>> pprint(data['sku864221'])
{u'deliveryTime': u'Lieferbar innerhalb 48 Stunden',
u'image': u'/images/m707491_300465.jpg',
u'name': u'BlackBerry Bold 9900',
u'sku1104261': {u'e': u'169.90', u'p': u'prod974431'},
u'sku1444275': {u'e': u'129.90', u'p': u'prod974431'},
u'sku1444283': {u'e': u'89.90', u'p': u'prod974431'},
u'sku1444286': {u'e': u'49.90', u'p': u'prod974431'},
u'sku1444291': {u'e': u'49.90', u'p': u'prod974431'}}
I am using Beautiful Soup4 with Python and now any way i have come upto the below. So now in what way I can get the values Dog,Cat, Horse and also the Ids. Please help!
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc)
soup.find_all('a')
# [<a class="Animal" href="http://example.com/elsie" id="link1">Dog</a>,
# <a class="Animal" href="http://example.com/lacie" id="link2">Cat</a>,
# <a class="Animal" href="http://example.com/tillie" id="link3">Horse</a>]
Documentation
for a in soup.find_all('a'):
id = a.get('id')
value = a.string # This is a NavigableString
unicode_value = unicode(value)