Find top 5 word lengths in a text - list

I'm trying to write a program that takes two functions:
count_word_lengths which takes the argument text, a string of text, and returns a default dictionary that records the count for each word length. An example call to this function:
top5_lengths which takes the same argument text and returns a list of the top 5 word lengths.
Note: that in the event that
two lengths have the same frequency, they should be sorted in descending order. Also, if there are fewer than 5 word lengths it should return a shorter list of the sorted word lengths.
Example calls to count_word_lengths:
count_word_lengths("one one was a racehorse two two was one too"):
defaultdict(<class 'int'>, {1: 1, 3: 8, 9: 1})
Example calls to top5_lengths:
top5_lengths("one one was a racehorse two two was one too")
[3, 9, 1]
top5_lengths("feather feather feather chicken feather")
[7]
top5_lengths("the swift green fox jumped over a cool cat")
[3, 5, 4, 6, 1]
My current code is this, and seems to output all these calls, however it is failing a hidden test. What type of input am I not considering? Is my code actually behaving correctly? If not, how could I fix this?
from collections import defaultdict
length_tally = defaultdict(int)
final_list = []
def count_word_lengths(text):
words = text.split(' ')
for word in words:
length_tally[len(word)] += 1
return length_tally
def top5_word_lengths(text):
frequencies = count_word_lengths(text)
list_of_frequencies = frequencies.items()
flipped = [(t[1], t[0]) for t in list_of_frequencies]
sorted_flipped = sorted(flipped)
reversed_sorted_flipped = sorted_flipped[::-1]
for item in reversed_sorted_flipped:
final_list.append(item[1])
return final_list

One thing to note is that you do not account for an empty string. That would cause count() to return null/undefined. Also you can use iteritems() during list comprehension to get the key and value from a dict like for k,v in dict.iteritems():

I'm not a Python guy, but I can see a few things that might cause issues.
You keep referring to top5_lengths, but your code has a function called top5_word_lengths.
You use a function called count_lengths that isn't defined anywhere.
Fix these and see what happens!
Edit:
This shouldn't impact your code, but it's not great practice for your functions to update variables outside their scope. You probably want to move the variable assignments at the top to functions where they're used.

Not really an answer, but an alternative way of tracking words instead of just lengths:
from collections import defaultdict
def count_words_by_length(text):
words = [(len(word),word) for word in text.split(" ")]
d = defaultdict(list)
for k, v in words:
d[k].append(v)
return d
def top_words(dict, how_many):
return [{"word_length": length, "num_words": len(words)} for length, words in dict.items()[-how_many:]]
Use as follows:
my_dict = count_words_by_length('hello sir this is a beautiful day right')
my_top_words = num_top_words_by_length(my_dict, 5)
print(my_top_words)
print(my_dict)
Output:
[{'word_length': 9, 'num_words': 1}]
defaultdict(<type 'list'>, {1: ['a'], 2: ['is'], 3: ['sir', 'day'], 4: ['this'], 5: ['hello', 'right'], 9: ['beautiful']})

Related

For-loop error: list index out of range

So I am rather new to programming and just recently started with Classes and we are supposed to make a phonebook that can be loaded in seperate text files.
I however keep running into the problem in this section that when I get into the for-loop. It hits a brick wall on
if storage[2] == permaStorage[i].number:
And tells me "IndexError: list index out of range". I am almost certain it is due to permaStorage starts out empty, but even when I attempt to fill it with temporary instances of Phonebook it tells me it out of range. The main reason it is there is to check if a phone number already exists within the permaStorage.
Anyone got a good tip on how to solve this or work around it?
(Sorry if the text is badly written. Just joined this site and not sure on the style)
class Phonebook():
def __init__(self):
self.name = ''
self.number = ''
def Add(name1, number1):
y = Phonebook()
y.name = name1
y.number = number1
return y
def Main():
permaStorage = []
while True:
print " add name number\n lookup name\n alias name newname\n change name number\n save filename\n load filename\n quit\n"
choices = raw_input ("What would you like to do?: ")
storage = choices.split(" ")
if storage[0] == "add":
for i in range(0, len(permaStorage)+1):
if storage[2] == permaStorage[i].number:
print "This number already exists. No two people can have the same phonenumber!\n"
break
if i == len(permaStorage):
print "hej"
try:
tempbox = Add(storage[1], storage[2])
permaStorage.append(tempbox)
except:
raw_input ("Remember to write name and phonenumber! Press any key to continue \n")
I think problem is that permaStorage is empty list and then u try to:
for i in range(0, len(permaStorage)+1):
if storage[2] == permaStorage[i].number:
will cause an error because permaStorage has 0 items but u trying to get first (i=0, permaStorage[0]) item.
I think you should replace second if clause with first one:
for i in range(0, len(permaStorage)+1):
if i == len(permaStorage):
print "hej"
try:
tempbox = Add(storage[1], storage[2])
permaStorage.append(tempbox)
if storage[2] == permaStorage[i].number:
print "This number already exists. No two people can have the same phonenumber!\n"
break
So in this case if perStorage is blank you will append some value and next if clause will be ok.
Indexing starts at zero in python. Hence, a list of length 5 has the last element index as 4 starting from 0. Change range to range(0, len(permastorage))
You should iterate upto the last element of the list, not beyond.
Try -
for i in range(0, len(permaStorage)):
The list of numbers produced in range() is from the start, but not including the end, so range(3) == [0, 1, 2].
So if your list x has length 10, range(0, len(x)) will give you 0 through 9, which is the correct indices of the elements of your list.
Adding 1 to len(x) will produce the range 0 through 10, and when you try to access x[10], it will fail.

Can't merge two lists into a dictionary

I can't merge two lists into a dictionary.I tried the following :
Map two lists into a dictionary in Python
I tried all solutions and I still get an empty dictionary
from sklearn.feature_extraction import DictVectorizer
from itertools import izip
import itertools
text_file = open("/home/vesko_/evnt_classification/bag_of_words", "r")
text_fiel2 = open("/home/vesko_/evnt_classification/sdas", "r")
lines = text_file.read().split('\n')
words = text_fiel2.read().split('\n')
diction = dict(itertools.izip(words,lines))
new_dict = {k: v for k, v in zip(words, lines)}
print new_dict
I get the following :
{'word': ''}
['word=']
The two lists are not empty.
I'm using python2.7
EDIT :
Output from the two lists (I'm only showing a few because it's a vector with 11k features)
//lines
['change', 'I/O', 'fcnet2', 'ifconfig',....
//words
['word', 'word', 'word', .....
EDIT :
Now at least I have some output #DamianLattenero
{'word\n': 'XXAMSDB35:XXAMSDB35_NGCEAC_DAT_L_Drivei\n'}
['word\n=XXAMSDB35:XXAMSDB35_NGCEAC_DAT_L_Drivei\n']
I think the root of a lot of confusion is code in the example that is not relevant.
Try this:
text_file = open("/home/vesko_/evnt_classification/bag_of_words", "r")
text_fiel2 = open("/home/vesko_/evnt_classification/sdas", "r")
lines = text_file.read().split('\n')
words = text_fiel2.read().split('\n')
# to remove any extra newline or whitespace from what was read in
map(lambda line: line.rstrip(), lines)
map(lambda word: word.rstrip(), words)
new_dict = dict(zip(words,lines))
print new_dict
Python builtin zip() returns an iterable of tuples from each of the arguments. Giving this iterable of tuples to the dict() object constructor creates a dictionary where each of the items in words is the key and items in lines is the corresponding value.
Also note that if the words file has more items than lines then there will either keys with empty values. If lines has items then only the last one will be added with an None key.
I tryed this and worked for me, I created two files, added numbers 1 to 4, letters a to d, and the code creates the dictionary ok, I didn't need to import itertools, actually there is an extra line not needed:
lines = [1,2,3,4]
words = ["a","b","c","d"]
diction = dict(zip(words,lines))
# new_dict = {k: v for k, v in zip(words, lines)}
print(diction)
{'a': 1, 'b': 2, 'c': 3, 'd': 4}
If that worked, and not the other, you must have a problem in loading the list, try loading like this:
def create_list_from_file(file):
with open(file, "r") as ins:
my_list = []
for line in ins:
my_list.append(line)
return my_list
lines = create_list_from_file("/home/vesko_/evnt_classification/bag_of_words")
words = create_list_from_file("/home/vesko_/evnt_classification/sdas")
diction = dict(zip(words,lines))
# new_dict = {k: v for k, v in zip(words, lines)}
print(diction)
Observation:
If you files.txt looks like this:
1
2
3
4
and
a
b
c
d
the result will have for keys in the dictionary, one per line:
{'a\n': '1\n', 'b\n': '2\n', 'c\n': '3\n', 'd': '4'}
But if you file looks like:
1 2 3 4
and
a b c d
the result will be {'a b c d': '1 2 3 4'}, only one value

How to filter on pandas dataframe when column data type is a list

I am having some trouble filtering a pandas dataframe on a column (let's call it column_1) whose data type is a list. Specifically, I want to return only rows such that column_1 and the intersection of another predetermined list are not empty. However, when I try to put the logic inside the arguments of the .where, function, I always get errors. Below are my attempts, with the errors returned.
Attemping to test whether or not a single element is inside the list:
table[element in table['column_1']]
returns the error ...
KeyError: False
trying to compare a list to all of the lists in the rows of the dataframe:
table[[349569] == table.column_1] returns the error Arrays were different lengths: 23041 vs 1
I'm trying to get these two intermediate steps down before I test the intersection of the two lists.
Thanks for taking the time to read over my problem!
consider the pd.Series s
s = pd.Series([[1, 2, 3], list('abcd'), [9, 8, 3], ['a', 4]])
print(s)
0 [1, 2, 3]
1 [a, b, c, d]
2 [9, 8, 3]
3 [a, 4]
dtype: object
And a testing list test
test = ['b', 3, 4]
Apply a lambda function that converts each element of s to a set and intersection with test
print(s.apply(lambda x: list(set(x).intersection(test))))
0 [3]
1 [b]
2 [3]
3 [4]
dtype: object
To use it as a mask, use bool instead of list
s.apply(lambda x: bool(set(x).intersection(test)))
0 True
1 True
2 True
3 True
dtype: bool
Hi for long term use you can wrap the whole work flow in functions and apply the functions where you need. As you did not put any example dataset. I am taking an example data set and resolving it. Considering I have text database. First I will find the #tags into a list then I will search the only #tags I want and filter the data.
# find all the tags in the message
def find_hashtags(post_msg):
combo = r'#\w+'
rx = re.compile(combo)
hash_tags = rx.findall(post_msg)
return hash_tags
# find the requered match according to a tag list and return true or false
def match_tags(tag_list, htag_list):
matched_items = bool(set(tag_list).intersection(htag_list))
return matched_items
test_data = [{'text': 'Head nipid mõnusateks sõitudeks kitsastel tänavatel. #TipStop'},
{'text': 'Homses Rooli Võimus uus #Peugeot208!\nVaata kindlasti.'},
{'text': 'Soovitame ennast tulevikuks ette valmistada, electric car sest uus #PeugeotE208 on peagi kohal! ⚡️⚡️\n#UnboringTheFuture'},
{'text': "Aeg on täiesti uueks roadtrip'i kogemuseks! \nLase ennast üllatada - #Peugeot5008!"},
{'text': 'Tõeline ikoon, mille stiil avaldab muljet läbi eco car, electric cars generatsioonide #Peugeot504!'}
]
test_df = pd.DataFrame(test_data)
# find all the hashtags
test_df["hashtags"] = test_df["text"].apply(lambda x: find_hashtags(x))
# the only hashtags we are interested
tag_search = ["#TipStop", "#Peugeot208"]
# match the tags in our list
test_df["tag_exist"] = test_df["hashtags"].apply(lambda x: match_tags(x, tag_search))
# filter the data
main_df = test_df[test_df.tag_exist]

PYTHON 2.7 - Modifying List of Lists and Re-Assembling Without Mutating

I currently have a list of lists that looks like this:
My_List = [[This, Is, A, Sample, Text, Sentence] [This, too, is, a, sample, text] [finally, so, is, this, one]]
Now what I need to do is "tag" each of these words with one of 3, in this case arbitrary, tags such as "EE", "FF", or "GG" based on which list the word is in and then reassemble them into the same order they came in. My final code would need to look like:
GG_List = [This, Sentence]
FF_List = [Is, A, Text]
EE_List = [Sample]
My_List = [[(This, GG), (Is, FF), (A, FF), (Sample, "EE), (Text, FF), (Sentence, GG)] [*same with this sentence*] [*and this one*]]
I tried this by using for loops to turn each item into a dict but the dicts then got rearranged by their tags which sadly can't happen because of the nature of this thing... the experiment needs everything to stay in the same order because eventually I need to measure the proximity of tags relative to others but only in the same sentence (list).
I thought about doing this with NLTK (which I have little experience with) but it looks like that is much more sophisticated then what I need and the tags aren't easily customized by a novice like myself.
I think this could be done by iterating through each of these items, using an if statement as I have to determine what tag they should have, and then making a tuple out of the word and its associated tag so it doesn't shift around within its list.
I've devised this.. but I can't figure out how to rebuild my list-of-lists and keep them in order :(.
for i in My_List: #For each list in the list of lists
for h in i: #For each item in each list
if h in GG_List: # Check for the tag
MyDicts = {"GG":h for h in i} #Make Dict from tag + word
Thank you so much for your help!
Putting the tags in a dictionary would work:
My_List = [['This', 'Is', 'A', 'Sample', 'Text', 'Sentence'],
['This', 'too', 'is', 'a', 'sample', 'text'],
['finally', 'so', 'is', 'this', 'one']]
GG_List = ['This', 'Sentence']
FF_List = ['Is', 'A', 'Text']
EE_List = ['Sample']
zipped = zip((GG_List, FF_List, EE_List), ('GG', 'FF', 'EE'))
tags = {item: tag for tag_list, tag in zipped for item in tag_list}
res = [[(word, tags[word]) for word in entry if word in tags] for entry in My_List]
Now:
>>> res
[[('This', 'GG'),
('Is', 'FF'),
('A', 'FF'),
('Sample', 'EE'),
('Text', 'FF'),
('Sentence', 'GG')],
[('This', 'GG')],
[]]
Dictionary works by key-value pairs. Each key is assigned a value. To search the dictionary, you search the index by the key, e.g.
>>> d = {1:'a', 2:'b', 3:'c'}
>>> d[1]
'a'
In the above case, we always search the dictionary by its keys, i.e. the integers.
In the case that you want to assign the tag/label to each word, you are searching by the key word and finding the "value", i.e. the tag/label, so your dictionary would have to look something like this (assuming that the strings are words and numbers as tag/label):
>>> d = {'a':1, 'b':1, 'c':3}
>>> d['a']
1
>>> sent = 'a b c a b'.split()
>>> sent
['a', 'b', 'c', 'a', 'b']
>>> [d[word] for word in sent]
[1, 1, 3, 1, 1]
This way the order of the tags follows the order of the words when you use a list comprehension to iterate through the words and find the appropriate tags.
So the problem comes when you have the initial dictionary indexed with the wrong way, i.e. key -> labels, value -> words, e.g.:
>>> d = {1:['a', 'd'], 2:['b', 'h'], 3:['c', 'x']}
>>> [d[word] for word in sent]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'a'
Then you would have to reverse your dictionary, assuming that all elements in your value lists are unique, you can do this:
>>> from collections import ChainMap
>>> d = {1:['a', 'd'], 2:['b', 'h'], 3:['c', 'x']}
>>> d_inv = dict(ChainMap(*[{value:key for value in values} for key, values in d.items()]))
>>> d_inv
{'h': 2, 'c': 3, 'a': 1, 'x': 3, 'b': 2, 'd': 1}
But the caveat is that ChainMap is only available in Python3.5 (yet another reason to upgrade your Python ;P). For Python <3.5, solutions, see How do I merge a list of dicts into a single dict?.
So going back to the problem of assigning labels/tags to words, let's say we have these input:
>>> d = {1:['a', 'd'], 2:['b', 'h'], 3:['c', 'x']}
>>> sent = 'a b c a b'.split()
First, we invert the dictionary (assuming that there're one to one mapping for every word and its tag/label:
>>> d_inv = dict(ChainMap(*[{value:key for value in values} for key, values in d.items()]))
Then, we apply the tags to the words through a list comprehension:
>>> [d_inv[word] for word in sent]
[1, 2, 3, 1, 2]
And for multiple sentences:
>>> sentences = ['a b c'.split(), 'h a x'.split()]
>>> [[d_inv[word] for word in sent] for sent in sentences]
[[1, 2, 3], [2, 1, 3]]

Split Array with looping on Python 2.7.5.1

def split(self):
assert input_array >= 0
if input_array == 0:
return [0]
array=[]
while input_array> 0:
array.append(int(input_array%10))
input_array = input_array//10
print input_array
return input_array
else:
print "END"
is there any way to split input array with looping?
i tried using selection but it just doesn't work
Are you trying to get the individual digits from a number? Try converting it into a string, iterating over it, and converting back to int.
>>> x = 2342
>>> [int(digit) for digit in str(x)]
[2, 3, 4, 2]
I'm guessing what you want is a list of digits conforming a certain number (input_array in this case).
First the main issues.
You declare a variable called array and if you are a good observer
you will notice you never return it.
print "END" has no purpose here.
input_arry == 0 can be treated as any other number > 0.
Try to not modify input_array
The solution:
Since I see you're working with a class I will code a solution for you using a class as well.
class SomeClass:
def __init__(self, input_array):
""" Constructor """
self.input_array = input_array
def split(self):
array = []
number = self.input_array # Don't modify input_array.
while number > 0:
array.append(number%10)
number = number // 10
array.reverse() # This is easy ;)
return array
def split_1(self):
""" Kevin's solution. When you become more skilled this is the way to go. """
return [int(digit) for digit in str(x)]
>>> some_instance = SomeClass(12345)
>>> print(some_instance.split())
[1, 2, 3, 4, 5]