n_available_markets=[playlist['tracks']['items'][i]['track']['available_markets']
for i in range(len(playlist['tracks']['items']))]
for i in n_available_markets:
if i=="[]":
i==None
n_available_markets
I was working with the Spotify API and I found that some values when I created the list were empty [[],['NZ','ES']] so I was wondering how I could change this empty values into None and I tried the code attached above(the first 2 lines are just to retrieve the available markets data) but it did not make any change.
This should do the trick:
n_available_markets = [p['track']['available_markets'] if len(p['track']['available_markets'])>0 else None for p in playlist['tracks']['items']]
Related
I'm using scrapy to iteratively scrape some data, and the data is being output as two lists through each iteration. I want to combine the two lists into one list at each iteration, so that in the end I will have one big list with many sublists(each sublist being the combination of the two lists created from each iteration)
That may be confusing so I will show my current output and code:
using Scrapy I"m iterating in the following way,
for i in response.css(''tr.insider....."):
i.css(a.tab-link:text).extract() #creating the first list
i.css('td::text').extract() #creating the second list
So the current output is something like this
[A,B,C] #first iteration
[1,2,3]
[D,E,F] #second iteration
[4,5,6]
[G,H,I] #third iteration
[7,8,9]
Desired output is
[[A,B,C,1,2,3], [D,E,F,4,5,6],[G,H,I,7,8,9]]
I tried the following code but I'm getting a list of None.
x =[]
for i in response.css(''tr.insider....."):
x.append(i.css(a.tablink::text).extract().extend(i.css('td::text').extract()))
But the return is just
None
None
None
None
None.....
Thanks!
extend function returns None, so you always append None to x.
For your purpose, I this is what you want:
for i in response.css(''tr.insider....."):
i.css('a.tab-link:text, td::text').extract()
You can simply add two lists together and append them to your results list.
results = []
for i in response.css("tr.insider....."):
first = i.css(a.tab-link:text).extract()
second = i.css('td::text').extract()
# combine both and append to results
results.append(first + second)
print(results)
# e.g.: [[A,B,C,1,2,3], [D,E,F,4,5,6],[G,H,I,7,8,9]]
I'm trying to scrape a career search website by going through all the different pages and I keep running into a problem when I try to append the dictionaries into a list using a for loop. When I execute the code below in Python 3.4, the code will pull all the relevant data from each page into a dictionary (I've checked with print()) and append into "FullJobDetails", but at the end of the for loop I get a list that is full of dictionaries from the last page only. The number of dictionaries is exactly the same as the number of pages in the list "ListofJobs". "ListofJobs" is a list of html links to each page that I am scrapping.
I just started learning code, so I know that the code below is not in any shape, way, or form the most efficient or best way of doing it. Any suggestions would be appreciated. Thanks in advance!
FullJobDetails = []
browser = webdriver.Chrome()
dictionary = {}
for jobs in ListofJobs:
browser.get(jobs)
dictionary["Web Page"] = jobs
try:
dictionary["Views"] = browser.find_element_by_class_name('job-viewed-item-count').text
except NoSuchElementException:
dictionary["Views"] = 0
try:
dictionary['Applicants'] = browser.find_element_by_class_name('job-applied-item-count').text
except NoSuchElementException:
dictionary["Applicants"] = 0
try:
dictionary["Last Application"] = browser.find_element_by_class_name('last-application-time-digit').text
except NoSuchElementException:
dictionary["Last Application"] = "N/A"
try:
dictionary["Job Title"] = browser.find_element_by_class_name('title').text
except NoSuchElementException:
dictionary["Job Title"] = "N/A"
try:
dictionary['Company'] = browser.find_element_by_xpath('/html/body/div[3]/article/section[2]/div/ul/li[4]/span/span').text
except NoSuchElementException:
dictionary['Company'] = "Not found"
try:
dictionary['Summary'] = browser.find_element_by_class_name('summary').text
except NoSuchElementException:
dictionary['Summary'] = "Not found"
FullJobDetails.append(dictionary)
The problem is that you create only a single dictionary - dicitonaries are mutable objects - the same ditionary is appended over and over to your list, and on each pass of the for loop you update its contents. Therefore, atthe end, you will have multple copies of the same dicitonry, all showing the information on the last page fectched.
Just create a new dictionary object for each run of the for loop. That new dictionary will be saved on the list, and the variable name dictionary can hold your new object with no conflicts.
for jobs in ListofJobs:
dictionary = {}
browser.get(jobs)
...
I am trying to just do a basic INSERT operation to a PostgreSQL database through Python via the Psycopg2 module. I have read a great many of the questions already posted regarding this subject as well as the documentation but I seem to have done something uniquely wrong and none of the fixes seem to work for my code.
#API CALL + JSON decoding here
x = 0
for item in ulist:
idValue = list['members'][x]['name']
activeUsers.append(str(idValue))
x += 1
dbShell.executemany("""INSERT INTO slickusers (username) VALUES (%s)""", activeUsers
)
The loop creates a list of strings that looks like this when printed:
['b2ong', 'dune', 'drble', 'drars', 'feman', 'got', 'urbo']
I am just trying to have the code INSERT these strings as 1 row each into the table.
The error specified when running is:
TypeError: not all arguments converted during string formatting
I tried changing the INSERT to:
dbShell.executemany("INSERT INTO slackusers (username) VALUES (%s)", (activeUsers,) )
But that seems like it's merely treating the entire list as a single string as it yields:
psycopg2.DataError: value too long for type character varying(30)
What am I missing?
First in the code you pasted:
x = 0
for item in ulist:
idValue = list['members'][x]['name']
activeUsers.append(str(idValue))
x += 1
Is not the right way to accomplish what you are trying to do.
first list is a reserved word in python and you shouldn't use it as a variable name. I am assuming you meant ulist.
if you really need access to the index of an item in python you can use enumerate:
for x, item in enumerate(ulist):
but, the best way to do what you are trying to do is something like
for item in ulist: # or list['members'] Your example is kinda broken here
activeUsers.append(str(item['name']))
Your first try was:
['b2ong', 'dune', 'drble', 'drars', 'feman', 'got', 'urbo']
Your second attempt was:
(['b2ong', 'dune', 'drble', 'drars', 'feman', 'got', 'urbo'], )
What I think you want is:
[['b2ong'], ['dune'], ['drble'], ['drars'], ['feman'], ['got'], ['urbo']]
You could get this many ways:
dbShell.executemany("INSERT INTO slackusers (username) VALUES (%s)", [ [a] for a in activeUsers] )
or event better:
for item in ulist: # or list['members'] Your example is kinda broken here
activeUsers.append([str(item['name'])])
dbShell.executemany("""INSERT INTO slickusers (username) VALUES (%s)""", activeUsers)
I am having a Regex and I want the matches to be added to my previous List. The list (called "Items") has already some entries. (It got the entries from a listbox1 and added is now a datasource of the listbox1)
This is my source:
Dim Items As List(Of String)
<some other code here>
For Each Bam As Match In Treffer
If hashtagz = False Then
ListBox1.Items.Add(Bam.Groups(1).ToString)
ElseIf hashtagz = True And FirstHashtags = True Then
ListBox1.Items.Add(Bam.Groups(1).ToString)
ElseIf hashtagz = True And FirstHashtags = False Then
Items.Add(Bam.Groups(1).ToString)
Console.WriteLine(Bam.Groups(1).ToString)
End If
Next
The last part is important (the last if loop with hashtagz = true and Firsthashtags = False)
I also added the console.writeline to see what is going on. In the console, I get all the new scraped and correct information. In the Items list however, I just get a duplicate of what has been already stored in there instead of adding and updating my list with the new regex matches.
edit Additional information: The whole if condition is in a timer, so it runs again and again. At first it will add entries to a listbox. Then (now important!) it will do the Items.Add(Bam.Groups(1).ToString). It seems to add the new entries the FIRST time THAT part of the code gets executed, but after it is being run trhough again trhough a timer, it will just add the previous entries again and again
Dim Items As New List(Of String)
Dim hashtagz As Boolean
Dim FirstHashtags As Boolean
' Other code here
For Each Bam As Match In Treffer
Select Case True
Case Not hashtagz
ListBox1.Items.Add(Bam.Groups(1).ToString)
Case FirstHashtags
ListBox1.Items.Add(Bam.Groups(1).ToString)
Case Items.Contains(Bam.Groups(1).ToString)
' Already Exists
Case Else
Items.Add(Bam.Groups(1).ToString)
Console.WriteLine(Bam.Groups(1).ToString)
End Select
Next
After using more than 7 hours today (and a few hours yesterday, I am a newbie in VB) of testing, going through my code and building in some outputs to see where the problem is...
My concept is correct, there are no logical errors in the flow of my programm, as one can see from the console.writeline(Bam.Groups(1).ToString . I tested everything and ended up putting in a richtextbox and its like this now
For Each Bam As Match In Treffer
If hashtagz = False Then
ListBox1.Items.Add(Bam.Groups(1).ToString)
ElseIf hashtagz = True And FirstHashtags = True Then
ListBox1.Items.Add(Bam.Groups(1).ToString)
ElseIf hashtagz = True And FirstHashtags = False Then
Items.Add(Bam.Groups(1).ToString)
RichTextBox1.Text = RichTextBox1.Text + Bam.Groups(1).ToString & vbNewLine
End If
Next
The adding procedure seems to be looking correct with the richtextbox, but when I want to add to the Items list, it just gives out the duplicates of the previous entries instead of adding to it. The .add seems to work only the FIRST time running through that line of code, after the second time it just gives out the duplicate of the entries.
I am still trying to figure out WHY that is and I will trying to understand the posts above. Maybe someone else has also an explanation, jsut in case the others were wrong
Edit I have been experimenting still with all this, I have a function that grabs the last entry of my item list and prints out a part of it. When I do that, the print seems correct and updated. So it seems like the list IS actually being updated somehow since the "getting a part of the latest entry of list" is working and always updated. But when I want to show or save the content of the item list, that's when I get the duplicates and the updated and newest entries are not being shown!
Alright guys, I am back again after testing.
The code I posted above was correct. The real reason behind the mistake was in a few other parts of the code, which resulted in partially duplicated output of the entries of the list etc.
I lost track of the variables and the procedures my program was doing. Any hints or tips for me on how to keep track of the code on what it does and having everything in mind or how to properly structure code etc. so that I won't get lost again and again in the code?
This case can be closed, sorry for the inconvenience.
I'm trying to match values of a list to a regex pattern. If the particular value within the list matches, I'll append it to a different list of dicts. If the above mentioned value does not match, I want to remove the value from the list.
import subprocess
def list_installed():
rawlist = subprocess.check_output(['yum', 'list', 'installed']).splitlines()
#print rawlist
for each_item in rawlist:
if "[\w86]" or \
"noarch" in each_item:
print each_item #additional stuff here to append list of dicts
#i haven't done the appending part yet
#the list of dict's will be returned at end of this funct
else:
remove(each_item)
list_installed()
The end goal is to eventually be able to do something similar to:
nifty_module.tellme(installed_packages[3]['version'])
nifty_module.dosomething(installed_packages[6])
Note to gnu/linux users going wtf:
This will eventually grow into a larger sysadmin frontend.
Despite the lack of an actual question in your post, I'll make a couple of comments.
You have a problem here:
if "[\w86]" or "noarch" in each_item:
It's not interpreted the way you think of it and it always evaluates to True. You probably need
if "[\w86]" in each_item or "noarch" in each_item:
Also, I'm not sure what you are doing, but in case you expect that Python will do regex matching here: it won't. If you need that, look at re module.
remove(each_item)
I don't know how it's implemented, but it probably won't work if you expect it to remove the element from rawlist: remove won't be able to actually access the list defined inside list_installed. I'd advise to use rawlist.remove(each_item) instead, but not in this case, because you are iterating over rawlist. You need to re-think the procedure a little (create another list and append needed elements to it instead of removing, for example).