How to replace this nested loop? - list

The goal here is to get the text from a website and append it to the lists so I can then create a dataframe out of it. I managed to make this after lots of Google but now I understand its not the most effective way of doing it, been researching list comprehension but wasnt able to get a successful result.
containers = soup.find_all('td', class_=['TableRecords_EvenLine', 'TableRecords_OddLine'])
dateli = []
descli = []
amtli = []
for container in containers:
date = container.select('div[id*=wtDataMov]')
for element1 in date:
seci1 = element1.get_text()
dateli.append(seci1)
description = container.select('div[id*=wtDescricao]')
for element2 in description:
seci2 = element2.get_text()
descli.append(seci2)
amount = container.select('div[id*=wtValorEur]')
for element3 in amount:
seci3 = element3.get_text()
amtli.append(float(price_str(seci3)))
Ideas? Thanks for you time.

If you want it done using list comprehensions, it would look like this
containers = soup.find_all('td', class_=['TableRecords_EvenLine', 'TableRecords_OddLine'])
dateli = []
descli = []
amtli = []
for container in containers:
dateli += [e.get_text() for e in container.select('div[id*=wtDataMov]')]
descli += [e.get_text() for e in container.select('div[id*=wtDescricao]')]
amtli += [float(price_str(e.get_text())) for e in container.select('div[id*=wtValorEur]')]

Related

filtering random objects in django

How can i filter 12 random objects from a model in django .
I tried to do this but It does not work and It just returned me 1 object.
max = product.objects.aggregate(id = Max('id'))
max_p = int(max['id'])
l = []
for s in range(1 , 13):
l.append(random.randint(1 , max_p))
for i in l:
great_proposal = product.objects.filter(id=i)
products = product.objects.all().order_by('-id')[:50]
great_proposal1 = random.sample(list(products) , 12)
Hi . It worked with this code !
Try this:
product.objects.order_by('?')[:12]
The '?' will "sort" randomly and "[:12]" will get only 12 objects.
I'm pretty sure the code is correct, but maybe you did not realize that you're just using great_proposal as variable to save the output, which is not an array, and therefore only returns one output.
Try:
result_array = []
for i in l:
result_array.append(product.objects.filter(index=i))

In Power Query, how can I remove duplicates either side of a delimiter?

I wish to turn : into :
For example amazon:amazon becomes amazon:
This is doable by hand using the replace values function but I need a way to do it programatically.
Thanks!
You can try this Transform but if it doesn't work, provide detail as to the
nature of the failure
examples of data on which it doesn't work
any error messages and the line which returns the error
remDups = Table.TransformColumns(#"Changed Type",{"Column1", each
let
sep = ":",
splitList = Text.Split(_, " "),
sepString = List.FindText(splitList,sep){0},
sepStringPosition = List.PositionOf(splitList,sepString),
//rem if the same remove last
splitSep = Text.Split(sepString, sep),
replString = if splitSep{0} = splitSep{1} then splitSep{0} & sep else sepString,
//put the string backtogether
replList = List.ReplaceRange(splitList,sepStringPosition,1,{replString})
in
Text.Combine(replList," ")
})

Django Query Set Aggregation Doesn't Work Like I Think It Should

The following snippet of code was intended to form a basic query set from the database against which several calculations could be made:
basicQuerySet = Job.objects.filter(g_end_time__gte=epochTimes[0]) \
.filter(g_end_time__lt=epochTimes[12]) \
.order_by('g_user__g_affiliation') \
.values('g_job_id','g_user_id','g_project','g_machine','g_cpu_days', \
'g_user__g_affiliation','g_user__g_user','g_user__g_organization', \
'g_user__g_common_name')
# get all g_affiliations during the past twelve months and convert to a list
interimList = basicQuerySet.values('g_user__g_affiliation').distinct()
affiliationList = []
for i,u in enumerate(interimList):
if u['g_user__g_affiliation'] is not None:
affiliationList.append(u['g_user__g_affiliation'])
# For each affiliation, get data
affiliationUsageData = {}
for m in range(len(affiliationList)):
thisAffiliation = affiliationList[m]
thisAffiliationJobs = basicQuerySet.filter(g_user__g_affiliation=thisAffiliation).count()
thisAffiliationUsers = basicQuerySet.filter(g_user__g_affiliation=thisAffiliation).values('g_user_id').distinct().count()
interimNumber = basicQuerySet.filter(g_user__g_affiliation=thisAffiliation).aggregate(Sum('g_cpu_days'))
thisAffiliationDays = interimNumber['g_cpu_days__sum']
affiliationUsageData[thisAffiliation] = []
thisData = {'jobs':thisAffiliationJobs,'users':thisAffiliationUsers,'days':thisAffiliationDays}
affiliationUsageData[thisAffiliation].append(thisData)
The part that doesn't work is the aggregation to get the sum of 'g_cpu_days'.
The 'basicQuerySet' is indeed returned with the expected fields.
The count for 'thisAffiliationJobs' returns the correct number.
The count for 'thisAffiliationUsers' returns the correct number.
However, the aggregation returns
interimNumber = {}
an empty dictionary.
If I create a new query set doing the aggregation directly
interimNumber = Job.objects.filter(g_end_time__gte=epochTimes[0]) \
.filter(g_end_time__lt=epochTimes[12]).filter(g_user__g_affiliation=thisAffiliation).aggregate(Sum('g_cpu_days'))
thisAffiliationDays = interimNumber['g_cpu_days__sum']
then the expected sum is returned.
Obviously, I'm needing some education. Is it possible to do an aggregation(Sum) on a query set without having to hit the database again?
This is being run in DJango v1.6.
Thanks in advance for your help (and any other pointers on this code snippet as I'm quite new to Django)!
I'm not answering the question yet, but I figured everyone might benefit from some a bit of styling help:
basicQuerySet = Job.objects.
filter(g_end_time__gte=epochTimes[0]). \
filter(g_end_time__lt=epochTimes[12]). \
order_by('g_user__g_affiliation'). \
values(
'g_job_id',
'g_user_id',
'g_project',
'g_machine',
'g_cpu_days',
'g_user__g_affiliation',
'g_user__g_user',
'g_user__g_organization',
'g_user__g_common_name'
)
# get all g_affiliations during the past twelve months and convert to a list
interimList = basicQuerySet.values('g_user__g_affiliation').distinct()
affiliationList = []
for i,u in enumerate(interimList):
if u['g_user__g_affiliation'] is not None:
affiliationList.append(u['g_user__g_affiliation'])
# For each affiliation, get data
affiliationUsageData = {}
for m in range(len(affiliationList)):
thisAffiliation = affiliationList[m]
thisAffiliationJobs = basicQuerySet.
filter(g_user__g_affiliation=thisAffiliation).\
count()
thisAffiliationUsers = basicQuerySet.\
filter(g_user__g_affiliation=thisAffiliation).\
values('g_user_id').\
distinct().\
count()
interimNumber = basicQuerySet.\
filter(g_user__g_affiliation=thisAffiliation).\
aggregate(Sum('g_cpu_days'))
thisAffiliationDays = interimNumber['g_cpu_days__sum']
affiliationUsageData[thisAffiliation] = []
thisData = {
'jobs':thisAffiliationJobs,
'users':thisAffiliationUsers,
'days':thisAffiliationDays
}
affiliationUsageData[thisAffiliation].append(thisData)

Erlang record item list

For example i have erlang record:
-record(state, {clients
}).
Can i make from clients field list?
That I could keep in client filed as in normal list? And how can i add some values in this list?
Thank you.
Maybe you mean something like:
-module(reclist).
-export([empty_state/0, some_state/0,
add_client/1, del_client/1,
get_clients/1]).
-record(state,
{
clients = [] ::[pos_integer()],
dbname ::char()
}).
empty_state() ->
#state{}.
some_state() ->
#state{
clients = [1,2,3],
dbname = "QA"}.
del_client(Client) ->
S = some_state(),
C = S#state.clients,
S#state{clients = lists:delete(Client, C)}.
add_client(Client) ->
S = some_state(),
C = S#state.clients,
S#state{clients = [Client|C]}.
get_clients(#state{clients = C, dbname = _D}) ->
C.
Test:
1> reclist:empty_state().
{state,[],undefined}
2> reclist:some_state().
{state,[1,2,3],"QA"}
3> reclist:add_client(4).
{state,[4,1,2,3],"QA"}
4> reclist:del_client(2).
{state,[1,3],"QA"}
::[pos_integer()] means that the type of the field is a list of positive integer values, starting from 1; it's the hint for the analysis tool dialyzer, when it performs type checking.
Erlang also allows you use pattern matching on records:
5> reclist:get_clients(reclist:some_state()).
[1,2,3]
Further reading:
Records
Types and Function Specifications
dialyzer(1)
#JUST MY correct OPINION's answer made me remember that I love how Haskell goes about getting the values of the fields in the data type.
Here's a definition of a data type, stolen from Learn You a Haskell for Great Good!, which leverages record syntax:
data Car = Car {company :: String
,model :: String
,year :: Int
} deriving (Show)
It creates functions company, model and year, that lookup fields in the data type. We first make a new car:
ghci> Car "Toyota" "Supra" 2005
Car {company = "Toyota", model = "Supra", year = 2005}
Or, using record syntax (the order of fields doesn't matter):
ghci> Car {model = "Supra", year = 2005, company = "Toyota"}
Car {company = "Toyota", model = "Supra", year = 2005}
ghci> let supra = Car {model = "Supra", year = 2005, company = "Toyota"}
ghci> year supra
2005
We can even use pattern matching:
ghci> let (Car {company = c, model = m, year = y}) = supra
ghci> "This " ++ c ++ " " ++ m ++ " was made in " ++ show y
"This Toyota Supra was made in 2005"
I remember there were attempts to implement something similar to Haskell's record syntax in Erlang, but not sure if they were successful.
Some posts, concerning these attempts:
In Response to "What Sucks About Erlang"
Geeking out with Lisp Flavoured Erlang. However I would ignore parameterized modules here.
It seems that LFE uses macros, which are similar to what provides Scheme (Racket, for instance), when you want to create a new value of some structure:
> (define-struct car (company model year))
> (define supra (make-car "Toyota" "Supra" 2005))
> (car-model supra)
"Supra"
I hope we'll have something close to Haskell record syntax in the future, that would be really practically useful and handy.
Yasir's answer is the correct one, but I'm going to show you WHY it works the way it works so you can understand records a bit better.
Records in Erlang are a hack (and a pretty ugly one). Using the record definition from Yasir's answer...
-record(state,
{
clients = [] ::[pos_integer()],
dbname ::char()
}).
...when you instantiate this with #state{} (as Yasir did in empty_state/0 function), what you really get back is this:
{state, [], undefined}
That is to say your "record" is just a tuple tagged with the name of the record (state in this case) followed by the record's contents. Inside BEAM itself there is no record. It's just another tuple with Erlang data types contained within it. This is the key to understanding how things work (and the limitations of records to boot).
Now when Yasir did this...
add_client(Client) ->
S = some_state(),
C = S#state.clients,
S#state{clients = [Client|C]}.
...the S#state.clients bit translates into code internally that looks like element(2,S). You're using, in other words, standard tuple manipulation functions. S#state.clients is just a symbolic way of saying the same thing, but in a way that lets you know what element 2 actually is. It's syntactic saccharine that's an improvement over keeping track of individual fields in your tuples in an error-prone way.
Now for that last S#state{clients = [Client|C]} bit, I'm not absolutely positive as to what code is generated behind the scenes, but it is likely just straightforward stuff that does the equivalent of {state, [Client|C], element(3,S)}. It:
tags a new tuple with the name of the record (provided as #state),
copies the elements from S (dictated by the S# portion),
except for the clients piece overridden by {clients = [Client|C]}.
All of this magic is done via a preprocessing hack behind the scenes.
Understanding how records work behind the scenes is beneficial both for understanding code written using records as well as for understanding how to use them yourself (not to mention understanding why things that seem to "make sense" don't work with records -- because they don't actually exist down in the abstract machine...yet).
If you are only adding or removing single items from the clients list in the state you could cut down on typing with a macro.
-record(state, {clients = [] }).
-define(AddClientToState(Client,State),
State#state{clients = lists:append([Client], State#state.clients) } ).
-define(RemoveClientFromState(Client,State),
State#state{clients = lists:delete(Client, State#state.clients) } ).
Here is a test escript that demonstrates:
#!/usr/bin/env escript
-record(state, {clients = [] }).
-define(AddClientToState(Client,State),
State#state{clients = lists:append([Client], State#state.clients)} ).
-define(RemoveClientFromState(Client,State),
State#state{clients = lists:delete(Client, State#state.clients)} ).
main(_) ->
%Start with a state with a empty list of clients.
State0 = #state{},
io:format("Empty State: ~p~n",[State0]),
%Add foo to the list
State1 = ?AddClientToState(foo,State0),
io:format("State after adding foo: ~p~n",[State1]),
%Add bar to the list.
State2 = ?AddClientToState(bar,State1),
io:format("State after adding bar: ~p~n",[State2]),
%Add baz to the list.
State3 = ?AddClientToState(baz,State2),
io:format("State after adding baz: ~p~n",[State3]),
%Remove bar from the list.
State4 = ?RemoveClientFromState(bar,State3),
io:format("State after removing bar: ~p~n",[State4]).
Result:
Empty State: {state,[]}
State after adding foo: {state,[foo]}
State after adding bar: {state,[bar,foo]}
State after adding baz: {state,[baz,bar,foo]}
State after removing bar: {state,[baz,foo]}

Erlang: List Comprehension to an Existing List

I am trying to create a new list via a list comprehension but want those new values to be included in an existing list.
More specifically, I am try to create a string out of the date and will have some string formatting between the values ( a dash - ). The existing list will be a template if you will with the dash.
Here is what I have so far:
{Date, Time} = erlang:universaltime().
DateList = tuple_to_list(Date).
DateListString = [ integer_to_list(X) || X < DateList ].
DateListStringConcatenate = lists:flatten(DateListString).
The result should be something along
"20101121"
But, what I want is
"2010-11-21"
So I am thinking about the DateListString comprehension "comprehending" to an existing list with "-" after the first and second element.
Any suggestions accompanied with concrete code samples much appreciated.
1> {{Y,M,D},_} = erlang:universaltime().
{{2010,11,21},{16,42,56}}
2> lists:flatten(io_lib:format("~p-~p-~p", [Y,M,D])).
"2010-11-21"
If you really want it in a list comprehension then you could do the following:
{Date, Time} = erlang:universaltime().
DateList = tuple_to_list(Date).
DateListString = [ [$- | integer_to_list(X)] || X <- DateList ].
[_ | DateListStringConcatenate] = lists:flatten(DateListString).
Roberto's is a better/more efficient solution to this but in case you wondered how you might do it with a list comprehension this would be one way.
This is a possible solution, but I feel that it is not an elegant one. Also, it does not use list comprehension.
1> {Date, Time} = erlang:universaltime().
{{2010,11,21},{14,51,23}}
2> DateList = tuple_to_list(Date).
[2010,11,21]
3> DateListString = lists:zipwith(fun(X,Y) -> integer_to_list(X) ++ Y end, DateList, ["-","-",""]).
["2010-","11-","21"]
4> DateListStringConcatenate = lists:flatten(DateListString).
"2010-11-21"