I am trying to write a simple dataframe to gbq. One of the field is array so I tried using REPEATED mode. But it seems nothing is written to the array
df = pd.DataFrame(
{
'my_string': ['a', 'b', 'c'],
'my_int64': [1, 2, 3],
'my_float64': [4.0, 5.0, 6.0],
'my_timestamp': [
pd.Timestamp("1998-09-04T16:03:14"),
pd.Timestamp("2010-09-13T12:03:45"),
pd.Timestamp("2015-10-02T16:00:00")
],
'my_array': [
[1,2,3],
[4,5,6],
[7,8,9]
]
}
)
# client = bigquery.Client()
table_id = 'junk.pshah2_new_table'
# Since string columns use the "object" dtype, pass in a (partial) schema
# to ensure the correct BigQuery data type.
job_config = bigquery.LoadJobConfig(schema=[
bigquery.SchemaField("my_string", "STRING"),
bigquery.SchemaField("my_array", "INTEGER","REPEATED"),
])
job = client.load_table_from_dataframe(
df, table_id, job_config=job_config
)
# Wait for the load job to complete.
job.result()
In GBQ I do not find my_array written at all
Am I doing something wrong here?
I'm using Python and trying to access some items in a list within a dictionary. I do not need to loop, just need the item value. Please see this:
<pre>
BROKER_INFO = {'AMERITRADE': {
'file_info': {
'broker_name' : 'TD Ameritrade',
'filename' : 'ameritrade',
'row_start' : '1',
'active' : 'Y',
'actions' : 'Bought,Sold',
'buy' : 'Bought',
'sell' : 'Sold',
'columns' : [
{'date': '0', 'action': '2', 'ticker': '4'}]
}
</pre>
I'm looking to get the value to "date", "action", "ticker' within "columns", which is within "file_info" and "AMERITRADE".
I tried this: BROKER_INFO[p_broker]['file_info']['columns']['action'], but no luck.
Many many thanks!
Try
BROKER_INFO[p_broker]['file_info']['columns'][0]['action']
I try to compare elements of nested list between each other. Let's say I have the following list:
list1 = [['v1', '1', '2'],['v1', '2', '2'], ['v2', '1'], ['v3'], ['v4', '1'], ['v4', '2']]
and I would like to reach to:
result = [['v1', '2'],['v1', '2'],['v2', '1'], ['v3'], ['v4'], ['v4']]
I've done a small code, but it does not look to work very well.
for i in range(1, len(list1) - 1):
previousone = list1[i-1]
currentone = list1[i]
nextone = list1[i+1]
lenprevious = len(previousone)
lencurrent = len(currentone)
lennext = len(nextone)
minlen = min(lenprevious,lencurrent,lennext) -1
common = ''
for j in range(minlen):
if j == 0:
if previousone[j] == currentone[j]:
common += str(previousone[j])
if previousone[j] != currentone[j]:
if currentone[j] == nextone[j]:
common += str(currentone[j])
else:
common += currentone
break
else:
if common != '':
if previousone[j] == currentone[j]:
common.join('_',str(nextone[j]))
else:
if currentone[j] == nextone[j]:
common.join('_',str(nextone[j]))
else:
break
else:
break
print common
result.append(common)
The idea, is to compare the 1st element of the sub-list vs the 1st element of the previous sub-list. If no match, then we compare with the next sub-list. If no match, we get in common the 1st element of the current sub-list.
Then, if it's matching, we do the same for the next element of the sub-list and so one until the last one. In the end, I want to have in common, a list of the common elements if any, if not I want the current sub-list.
Does anyone has any idea how to make it work? Thanks in advance!
EDIT ::
The logic would be:
Iteration 1 -> Previous : ['v1', '1', '2'] and Current : ['v1', '2', '2'] and Next : ['v2', '1']
We compare each element from each list.
First, we compare the Previous and the Current.
First element of those lists are 'v1', so we append 'v1' in result and we go to the next element, here '1' and '2'.
They are not the same so we pass until the next element, which are '2' and '2': identical.
We append in result to get Result :
[['v1', '2'], ['v1', '2'], [], [], [], []]
Iteration 2 -> Previous : ['v1', '2', '2'] and Current : ['v2', '1'] and Next : ['v3']
First we compare Previous and Current. 'v1' different from 'v2'.
So we compare Current and Next. 'v2' different from 'v3'.
So we append in result the current and we get:
[['v1', '2'], ['v1', '2'], ['v2', '1'], [], [], []]
Iteration 3 -> Previous : ['v2', '1'] and Current : ['v3'] and Next : ['v4', '1']
Same as above, 'v2' different from 'v3' and 'v3' different from 'v4' so we append the current and get:
[['v1', '2'], ['v1', '2'], ['v2', '1'], ['v3'], [], []]
Iteration 4 -> Previous : ['v3'] and Current : ['v4', '1'] and Next: ['v4', '2']
'v3' different from 'v4' so we compare Current and Next: 'v4' is common so we append 'v4':
[['v1', '2'], ['v1', '2'], ['v2', '1'], ['v3'], ['v4'], []]
Iteration 5 -> Previous : ['v4', '1'] and Current : ['v4', '2'] and Next : ??
'v4' is common so we append 'v4' and get the final result:
Result: [['v1', '2'], ['v1', '2'], ['v2', '1'], ['v3'], ['v4'], ['v4']]
But I do not know how to reach there..
Below is the implementation according to your desired result,
Core Logic
def intersection(lst1, lst2):
return list(set(lst1) & set(lst2))
list1 = [['v1', '1', '2'],['v1', '2', '2'], ['v2', '1'], ['v3'], ['v4', '1'], ['v4', '2']]
result = []
list_len = len(list1)
if list_len == 0:
pass
elif list_len == 1:
result.append(list1)
else:
for i in range(list_len):
if i == 0:
current = list1[i]
next = list1[i+1]
print("Iteration {} -> Previous : No previous available Current : {} Next: {}".format(i, current, next))
if current[0] == next[0]:
result.append(intersection(current, next))
else:
result.append(current)
elif i == list_len - 1:
previous = list1[i-1]
current = list1[i]
print("Iteration {} -> Previous : {} Current : {} Next: No next available".format(i, previous, current))
if current[0] != previous[0]:
result.append(current)
else:
result.append(intersection(current, previous))
else:
previous = list1[i-1]
current = list1[i]
next = list1[i+1]
print("Iteration {} -> Previous : {} Current : {} Next: {}".format(i, previous, current, next))
if current[0] == previous[0]:
result.append(intersection(current, previous))
else:
if current[0] == next[0]:
result.append(intersection(current, next))
else:
result.append(current)
print("Result : {}".format(result))
Output
Iteration 0 -> Previous : No previous available Current : ['v1', '1', '2'] Next: ['v1', '2', '2']
Iteration 1 -> Previous : ['v1', '1', '2'] Current : ['v1', '2', '2'] Next: ['v2', '1']
Iteration 2 -> Previous : ['v1', '2', '2'] Current : ['v2', '1'] Next: ['v3']
Iteration 3 -> Previous : ['v2', '1'] Current : ['v3'] Next: ['v4', '1']
Iteration 4 -> Previous : ['v3'] Current : ['v4', '1'] Next: ['v4', '2']
Iteration 5 -> Previous : ['v4', '1'] Current : ['v4', '2'] Next: No next available
Result : [['v1', '2'], ['v1', '2'], ['v2', '1'], ['v3'], ['v4'], ['v4']]
I am trying to iterate through a map and create a new map value. The below is the input
def map = [[name: 'hello', email: ['on', 'off'] ], [ name: 'bye', email: ['abc', 'xyz']]]
I want the resulting data to be like:
[hello: ['on', 'off'], bye: ['abc', 'xyz']]
The code I have right now -
result = [:]
map.each { key ->
result[random] = key.email.each {random ->
"$random"
}
}
return result
The above code returns
[hello: [on, off], bye: [abc, xyz]]
As you can see from above, the quotes from on, off and abc, xyz have disappeared, which is causing problems for me when i am trying to do checks on the list value [on, off]
It should not matter. If you see the result in Groovy console, they are still String.
Below should be sufficient:
map.collectEntries {
[ it.name, it.email ]
}
If you still need the single quotes to create a GString instead of a String, then below tweak would be required:
map.collectEntries {
[ it.name, it.email.collect { "'$it'" } ]
}
I personally do not see any reasoning behind doing the later way. BTW, map is not a Map, it is a List, you can rename it to avoid unnecessary confusions.
You could convert it to a json object and then everything will have quotes
This does it. There should/may be a groovier way though.
def listOfMaps = [[name: 'hello', email: ['on', 'off'] ], [ name: 'bye', email: ['abc', 'xyz']]]
def result = [:]
listOfMaps.each { map ->
def list = map.collect { k, v ->
v
}
result[list[0]] = ["'${list[1][0]}'", "'${list[1][1]}'"]
}
println result
I have a list:
txtlst = [
['000001', 'DOE', 'JOHN', 'COMSCI', '', 'MATH', '', 'ENGLISH\n'],
['000002', 'DOE', 'JANE', 'FRENCH', '', 'MUSIC', '', 'COMSCI\n']
]
And I want to put the elements in a dictionary so it looks likes this
mydict = {
'000001': ['000001', 'DOE', 'JOHN', 'COMSCI', '', 'MATH', '', 'ENGLISH\n'],
'000002': ['000002', 'DOE', 'JANE', 'FRENCH', '', 'MUSIC', '', 'COMSCI\n']
}
My problem here is, after I ran the code
for i in txtlst:
key = i[0]
value = i
mydict = {key:value}
The two sublists of txtlst are added to different dictionaries. How can I fix my code so they will be in the same dictionary as I mentioned above?
You can easily create a new dictionary with the first element of each list as key:
mydict = { i[0]: i for i in txtlst }
If you wish to do that in a loop like in your approach, you need to initialize a dictionary beforehand and update it in each iteration:
mydict = {}
for i in txtlst:
key = i[0]
value = i
mydict[key] = value