I'm trying to extract features from a text document. Here is my code:
import sklearn
from sklearn.datasets import load_files
from sklearn.feature_extraction.text import CountVectorizer
files = sklearn.datasets.load_files('/home/niyas/Documents/project/container', shuffle = False)
vectorizer = CountVectorizer(min_df=1)
X = vectorizer.fit_transform(files.data[1])
Y=vectorizer.get_feature_names()
I'm getting an error "ValueError: empty vocabulary; perhaps the documents only contain stop words". The code works fine when I pass a string with the exact same content of the text doc.
Help me. Thanks in advance.
Related
I've been trying to return list of unique words from file and sort them alphabetically by using NLTK but it didn't work although I've used several different approaches. Here is my code:
import nltk
from nltk import FreqDist
def get_vocabulary(self):
with open(self.path, "r") as file:
split = [line.split('\n') for line in file]
fdist1 = FreqDist(split)
unique_words = fdist1.hapaxes()
return sorted(set(unique_words))
And the error:
TypeError: unhashable type: 'list'
Other similar approaches that I've tried also threw similar errors. The solution doesn't have to include nltk but I'd appreciate if you could show me where I made mistake(s) on my own solution.
TL;DR
from collections import Counter
from nltk import word_tokenize
with open('filename.txt') as fin:
word_count = Counter(word_tokenize(fin.read()))
# Sorted by most common.
word_count.most_common()
# Sorted alphabetically
sorted(word_count.items())
# If you just need the words.
sorted(word_count)
I have the following code that I need to get absolute links from rather than relative links.
I believe I need to use urlparse and urljoin somewhere in here, but I'm just not sure where to use that.
The .csv from this code is also giving me rows like this: "/about.html" which is obviously not an link to another web page.
import urllib
import pandas as pd
from bs4 import BeautifulSoup
import numpy as np
import re
r = urllib.urlopen('https://www.census.gov/programs-surveys/popest.html')
soup = BeautifulSoup(r, "lxml")
links = []
for link in soup.findAll('a', attrs={'href': re.compile(r'(^http|.html)')}):
links.append(link.get('href'))
web_links_df = pd.DataFrame(links)
web_links_df.columns = ['web_link']
web_links_df['web_link'] = web_links_df['web_link'].apply(lambda x:
x.rstrip('/'))
url_tail = web_links_df['web_link'].apply(lambda x: x[-4:])
web_links = pd.DataFrame(web_links_df['web_link'].unique())
web_links.columns = ['web_link']
print web_links.head()
web_links.to_csv("D:/MLCV/web_links_1.csv")
Any help would be greatly appreciated. I have spent hours going through other examples on Stack but I am just not getting the correct results.
I am pulling PNG images from Jupyter Notebooks and manage to display with IPython.display.Image but not with matplotib.pyplot.plt. What am I missing? I use python 2.7.
I am using the following algorithm:
To open the notebook JSON content I do:
import nbformat
notebook_ = nbformat.read(file_notebook, 4)
After retrieving the relevant cell information I pull the png information from it using:
def cell_to_image(cell, out_value_item_number=1):
if "execution_count" in cell.keys(): # i.e version >=4
return cell["outputs"][out_value_item_number]['data']['image/png']
elif "prompt_number" in cell.keys(): # i.e version < 4
return cell["outputs"][out_value_item_number]['png']
return None
cell_image = cell_to_image(cell)
The first few characters of cell_image (which is unicode) looks like:
iVBORw0KGgoAAAANSUhEUgAAA64AAAFMCAYAAADLFeHSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n
AAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8jef/x/HXyTjZiYQkCGrU3ruR0tr9oq2qGtGo0dbe
\nm5pVlJpFUSMoVb6UoEZ/lCpatWuPUiNEEiMDmef3R75OexonJKUO3s/HI4/mXPd1X/d1f+LRR965
\n7/u6DSaTyYSIiIiIiIiIjbJ70hMQERERERERyYiCq4iIiIiIiNg0BVcRERERERGxaQquIiIiIiIi
\nYtMUXEVERERERMSmKbiKiIiIiIiITVNwFRGRxyIkJIRixYqxfv36+24/e/YsxYoVo3jx4v/yzGxb
\naGgoderUIS4uDoBdu3bRsmVLKlasyCuvvMKgQYOIjo622CcsLIyGDRtSunRp6tSpw8KFC62OW7p0
\naRo2bJju53Lnzh1GjRrFyy+/TNmyZWnRogW//fbbQ835q6++olGjRpQvX5769eszc+ZMkpOTzdtT
\nU1OZNGkSNWrUoHTp0jRp0oTdu3enGyc2NpZOn
I can easily plot in my Jupityer notebook using
from IPython.display import Image
Image(cell_image)
And now to my question:
How can I manipulate cell_image to be plt.subplot friendly?
(Assuming import matplotlib.pyplot as plt).
I realise that plt.imshow wouldn't work because this would require an array, which is not my case (which is a string, as far as I understand).
If you have your image string representation in a variable string_rep, the following code should work.
from io import BytesIO
import matplotlib.image as mpimage
import matplotlib.pyplot as plt
with BytesIO(string_rep.decode('base64')) as byte_rep:
image = mpimage.imread(byte_rep)
plt.imshow(image)
Update: Issue resolved. (see comment section below.) Ultimately, the following two lines were required to transform my .csv to unicode and utilize TextBlob: row = [cell.decode('utf-8') for cell in row], and text = ' '.join(row).
Original question:
I am trying to use a Python library called Textblob to analyze text from a .csv file. Error I receive when I call Textblob in my code is:
Traceback (most recent call last): File
"C:\Users\Marcus\Documents\Blog\Python\Scripts\Brooks\textblob_sentiment.py",
line 30, in
blob = TextBlob(row) File "C:\Python27\lib\site-packages\textblob\blob.py", line 344, in
init
'must be a string, not {0}'.format(type(text)))TypeError: The text argument passed to __init__(text) must be a string, not
My code is:
#from __future__ import division, unicode_literals #(This was recommended for Python 2.x, but didn't help in my case.)
#-*- coding: utf-8 -*-
import csv
from textblob import TextBlob
with open(u'items.csv', 'rb') as scrape_file:
reader = csv.reader(scrape_file, delimiter=',', quotechar='"')
for row in reader:
row = [unicode(cell, 'utf-8') for cell in row]
print row
blob = TextBlob(row)
print type(blob)
I have been working through UTF/unicode issues. I'd originally had a different subject which I posed to this thread. (Since my code and the error have changed, I'm posting to a new thread.) Print statements indicate that the variable "row" is of type=str, which I thought indicated that the reader object had been transformed as required by Textblob. The source .csv file is saved as UTF-8. Can anyone provide feedback as to how I can get unblocked on this, and the flaws in my code?
Thanks so much for the help.
So maybe you can make change as below:
row = str([cell.encode('utf-8') for cell in row])
I have written this code
import csv as csv
import numpy as np
csv_file_object=
csv.reader(open('C:\Users\hostname\Desktop\spyder\train.csv', 'rb'))
header = csv_file_object.next()
data=[]
for row in csv_file_object:
data.append(row)
data = np.array(data)
but Error ([Errno 22] invalid mode ('rb') or filename:) appears.
I'd suggest using numpy genfromtxt
import numpy as np
np.genfromtxt('C:\Users\hostname\Desktop\spyder\train.csv',delimiter=',',dtype=None)
You'll have to adjust the delimiter and dtype parameters based on your csv file.