So, i'm just a noob when it comes to programming especially python. I have a list holding all the variable names that i'm using in my program:
dList=['market_a','market_b','market_c','market_d','market_e','market_f','market_g']
What i want to do is remove all these objects from memory i.e., this is what i believe needs to be done:
del market_a,market_b,market_c,market_d,market_e,market_f,market_g
market_a=market_b=market_c=market_d=market_e=market_f=market_g= None
I was trying to del the objects by doing something like this:
for index in (len(dList):
del dList[index]
But i'm getting this error.
IndexError: list index out of range
Can somebody please help me with this? Also can somebody please tell me how i can do market_a=market_b=market_c=market_d=market_e=market_f=market_g= Nonefrom dList?
Thanks in advance.
I am not recommending that you do this (see below for a dictionary based solution), however, you can use the exec statement to assign None to the variables:
dList = ['market_a', 'market_b', 'market_c', 'market_d', 'market_e', 'market_f', 'market_g']
for var in dList:
exec '{} = None'.format(var)
You can also explicitly call del on the variable:
for var in dList:
exec 'del {}'.format(var)
# exec '{} = None'.format(var)
although this is not usually required because, if the object to which a variable name was bound has no other references to it, rebinding the variable name results in the original object being eligible for garbage collection and so it will eventually be removed from memory.
In general it makes little sense to store a list of variable names: if the variables are static, that is, known to your code at "compile" time, then you already know what they are called and can simply refer to them by name in your code.
If the variables are dynamic then you'd be better off using a dictionary to associate the names with values. Treat the entries in the dictionary as your variables.
markets = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7}
# to access a "variable"
>>> markets['a']
1
# to modify a "variable"
>>> markets['a'] += 100
>>> markets['a']
101
To delete them all you can simply delete the dictionary:
del markets
or rebind it to an empty dictionary:
markets = {}
Assuming that there is no other reference to the objects stored as values in the dictionary, this will make these objects available for garbage collection, effectively deleting them from memory.
Or you can delete specific keys:
# to delete a specific "variable"
del markets['a']
Unless it is required by your algorithms there's no need to assign None.
As you delete your array is getting smaller, therefore the index is no longer correct.
You can try this:
while dList:
del dList[0]
Just a note when the array is empty, it will evaluate to False.
As mentioned above as you remove items the array gets smaller so a for loop is probably not the best solution. To me it looks like you are trying to accomplish 2 tasks: shrink the array and physically delete the variables:
while dList:
var = dList.pop(0)
exec('del {0}'.format(var))
Consider using the pop() method which will shrink your array and then using the exec() method to delete the variable from memory by passing in the variable name and instead of deleting the string deleting the actual variable instance.
Related
I am new to python and my coding experience so far is with MATLAB.
I am trying to understand more about lists and dictionaries as i am using a library about DOEs that takes an dictionary as a passing argument.
But my trouble so far is that this dictionary assumes the form of ex.
DOE={'Elastic Modulus':[10,20,30], 'Density':[1,2,3], 'Thickness':[2,3,5]}
But i need this dictionary to be user defined, for example:
Have an input to define how many variables are needed (in this example are 3: Elastic Modulus','Density'and 'Thickness)
as the variables are defined, it should be able to store values in the dictionary over a for loop.
Is this possible using dictionaries?
Or is it better to use a list and convert in a dicionary later?
Thank you in advance
One can add keys and the corresponding values to a dict one at a time like so:
my_dict = {}
num_entries = int(input("How many entries "))
for _ in range(num_entries):
key = input("Enter the key: ")
value = input("Enter the value: ")
my_dict[key] = value
Presumably you would have a loop to do the entry of key and value for the number of values you wish to enter. Also if you are in python 2 it needs to be raw_input rather than input function. [Edit: Showing how to do the loop, since I noticed that was part of your question]
In searching around the web for usages of custom constructors I see things like this:
def some_constructor(loader, node):
value = loader.construct_mapping(node, deep=True)
return SomeClass(value)
What does the deep=True do? I don't see it in the pyyaml documentation.
It looks like I need it; I have a yaml file generated by a pyyaml representer and it includes node anchors and aliases (like &id003 and *id003); without deep=True I get a shallow map back for those objects containing anchors/aliases.
That you don't see deep=True in the documentation is because you don't normally need to use it as an end-user of the PyYAML package.
If you trace the use of methods in constructor.py that use deep= you come to construct_mapping() and construct_sequence() in class BaseConstructor() and both of these call BaseConstructor.construct_object().
The relevant code in that method to study is:
if tag_suffix is None:
data = constructor(self, node)
else:
data = constructor(self, tag_suffix, node)
if isinstance(data, types.GeneratorType):
generator = data
data = next(generator)
if self.deep_construct:
for dummy in generator:
pass
else:
self.state_generators.append(generator)
and in particular the for loop in there, which only gets executed if deep=True was passed in.
Rougly said if the data comes from a constructor is a generator, then it walks over that data (in the for loop) until the generator is exhausted. With that mechanism, those constructors can contain a yield to create a base object, of which the details can be filled out after the yield. Because of their being only one yield in such constructors, e.g. for mappings (constructed as Python dicts):
def construct_yaml_map(self, node):
data = {}
yield data
value = self.construct_mapping(node)
data.update(value)
I call this a two step process (one step to the yield the next to the end of the method.
In such two-step constructors the data to be yielded is constructed empty, yielded and then filled out. And that has to be so because of what you already noticed: recursion. If there is a self reference to data somewhere underneath, data cannot be constructed after all its children are constructed, because it would have to wait for itself to be constructed.
The deep parameter indirectly controls whether objects that are potentially generators are recursively being built or appended to the list self.state_generators to be resolved later on.
Constructing a YAML document then boils down to constructing the top-level objects and looping over the potentially recursive objects in self.state_generators until no generators are left (a process that might take more than one pass).
The deep argument controls how nested dictionaries are handled during this process.
When deep=True, the construct_mapping method will recursively call itself on any
nested dictionaries it encounters, and merge the resulting dictionaries together.
for example:
a:
b: 1
c: 2
d:
b: 3
When "deep=True"
{'a': {'b': 1, 'c': 2}, 'd': {'b': 3}}
When "deep=False"
{'a': {'c': 2}, 'd': {'b': 3}}
How can i duplicate a list of lists (or any other types) in a way that the resulting lists are new objects and not references to the old ones? As an example i have the following list of lists:
l=[[1,2],[3,4]]
what i want as result is:
l=[[1,2],[3,4],[1,2],[3,4]]
If i do l*=2 the new sub-lists are references to the old sub-lists.
Doing l[0].append("python") will result in
l=[[1,2,'python'],[3,4],[1,2,'python'],[3,4]]
Also creating a new list like:
l2=list(l)
or
l2=l[:]
doesn't solve the problem. I want to have new sub-lists which are independent of their origin and which upon changing have no impact on their old fellows. How can i do this i python?
In general, the best way to copy a nested data structure so that copies get made of all the references (not just the ones at the top level) is to use copy.deepcopy. In your nested list example, you can do:
l.extend(copy.deepcopy(l))
deepcopy will still work even if the data structure contains references to itself, or multiple references to the same object. It usually works for objects stored as attributes on an instances of custom classes too. You can define a __deepcopy__ method if you want to give a class special copying behavior (e.g. if some of its attributes are bookkeeping data that shouldn't be copied).
Here's a version of your nested list example code using instances of a linked list class rather than Python lists. copy.deepcopy does the right thing!
class linked_list(object):
def __init__(self, value, next=None):
self.value = value
self.next = next
def __repr__(self):
if self.next is not None:
return "({!r})->{!r}".format(self.value, self.next)
else:
return "({!r})".format self.value
lst = linked_list(linked_list(1, linked_list(2)),
linked_list(linked_list(3, linked_list(4))))
print(lst) # prints ((1)->(2))->((3)->(4))
lst.next.next = copy.deepcopy(lst)
print(lst) # prints ((1)->(2))->((3)->(4))->((1)->(2))->((3)->(4))
lst.value.value = 5
print(lst) # prints ((5)->(2))->((3)->(4))->((1)->(2))->((3)->(4))
Python 2.7 on Mint Cinnamon 17.3.
I have a bit of test code employing a list of dicts and despite many hours of frustration, I cannot seem to work out why it is not working as it should do.
blockagedict = {'location': None, 'timestamp': None, 'blocked': None}
blockedlist = [blockagedict]
blockagedict['location'] = 'A'
blockagedict['timestamp'] = '12-Apr-2016 01:01:08.702149'
blockagedict['blocked'] = True
blockagedict['location'] = 'B'
blockagedict['timestamp'] = '12-Apr-2016 01:01:09.312459'
blockagedict['blocked'] = False
blockedlist.append(blockagedict)
for test in blockedlist:
print test['location'], test['timestamp'], test['blocked']
This always produces the following output and I cannot work out why and cannot see if I have anything wrong with my code. It always prints out the last set of dict values but should print all, if I am not mistaken.
B 12-Apr-2016 01:01:09.312459 False
B 12-Apr-2016 01:01:09.312459 False
I would be happy for someone to show me the error of my ways and put me out of my misery.
It is because the line blockedlist = [blockagedict] actually stores a reference to the dict, not a copy, in the list. Your code effectively creates a list that has two references to the very same object.
If you care about performance and will have 1 million dictionaries in a list, all with the same keys, you will be better off using a NumPy structured array. Then you can have a single, efficient data structure which is basically a matrix of rows and named columns of appropriate types. You mentioned in a comment that you may know the number of rows in advance. Here's a rewrite of your example code using NumPy instead, which will be massively more efficient than a list of a million dicts.
import numpy as np
dtype = [('location', str, 1), ('timestamp', str, 27), ('blocked', bool)]
count = 2 # will be much larger in the real program
blockages = np.empty(count, dtype) # use zeros() instead if some data may never be populated
blockages[0]['location'] = 'A'
blockages[0]['timestamp'] = '12-Apr-2016 01:01:08.702149'
blockages[0]['blocked'] = True
blockages['location'][1] = 'B' # n.b. indexing works this way too
blockages['timestamp'][1] = '12-Apr-2016 01:01:09.312459'
blockages['blocked'][1] = False
for test in blockages:
print test['location'], test['timestamp'], test['blocked']
Note that the usage is almost identical. But the storage is in a fixed size, single allocation. This will reduce memory usage and compute time.
As a nice side effect, writing it as above completely sidesteps the issue you originally had, with multiple references to the same row. Now all the data is placed directly into the matrix with no object references at all.
Later in a comment you mention you cannot use NumPy because it may not be installed. Well, we can still avoid unnecessary dicts, like this:
from array import array
blockages = {'location': [], 'timestamp': [], 'blocked': array('B')}
blockages['location'].append('A')
blockages['timestamp'].append('12-Apr-2016 01:01:08.702149')
blockages['blocked'].append(True)
blockages['location'].append('B')
blockages['timestamp'].append('12-Apr-2016 01:01:09.312459')
blockages['blocked'].append(False)
for location, timestamp, blocked in zip(*blockages.values()):
print location, timestamp, blocked
Note I use array here for efficient storage of the fixed-size blocked values (this way each value takes exactly one byte).
You still end up with resizable lists that you could avoid, but at least you don't need to store a dict in every slot of the list. This should still be more efficient.
Ok, I have initialised the list of dicts right off the bat and this seems to work. Although I am tempted to write a class for this.
blockedlist = [{'location': None, 'timestamp': None, 'blocked': None} for k in range(2)]
blockedlist[0]['location'] = 'A'
blockedlist[0]['timestamp'] = '12-Apr-2016 01:01:08.702149'
blockedlist[0]['blocked'] = True
blockedlist[1]['location'] = 'B'
blockedlist[1]['timestamp'] = '12-Apr-2016 01:01:09.312459'
blockedlist[1]['blocked'] = False
for test in blockedlist:
print test['location'], test['timestamp'], test['blocked']
And this produces what I was looking for:
A 12-Apr-2016 01:01:08.702149 True
B 12-Apr-2016 01:01:09.312459 False
I will be reading from a text file with 1 to 2 million lines, so converting the code to iterate through the lines won't be a problem.
Or maybe I should say, ways to skip having to initialize at all.
I really hate that every time I want to do a simple count variable, I have to say, "hey python, this variable starts at 0." I want to be able to say count+=1and have it instantly know to start from 0 at the first iteration of the loop. Maybe there's some sort of function I can design to accomodate this? count(1) that adds 1 to a self-created internal count variable that sticks around between iterations of the loop.
I have the same dislike for editing strings/lists into a new string/list.
(Initializing new_string=""/new_list=[] before the loop).
I think list comprehensions may work for some lists.
Does anyone have some pointers for how to solve this problem? I am fairly new, I've only been programming off and on for half a year.
Disclaimer: I do not think that this will make initialization any cleaner. Also, in case you have a typo in some uses of your counter variable, you will not get a NameError but instead it will just silently create and increment a second counter. Remember the Zen of Python:
Explicit is better than implicit.
Having said that, you could create a special class that will automatically add missing attributes and use this class to create and auto-initialize all sorts of counters:
class Counter:
def __init__(self, default_func=int):
self.default = default_func
def __getattr__(self, name):
if name not in self.__dict__:
self.__dict__[name] = self.default()
return self.__dict__[name]
Now you can create a single instance of that class to create an arbitrary number of counters of the same type. Example usage:
>>> c = Counter()
>>> c.foo
0
>>> c.bar += 1
>>> c.bar += 2
>>> c.bar
3
>>> l = Counter(list)
>>> l.blub += [1,2,3]
>>> l.blub
[1, 2, 3]
In fact, this is similar to what collections.defaultdict does, except that you can use dot-notation for accessing the counters, i.e. c.foo instead of c['foo']. Come to think of it, you could even extend defaultdict, making the whole thing much simpler:
class Counter(collections.defaultdict):
def __getattr__(self, name):
return self[name]
If you are using a counter in a for loop you can use enumerate:
for counter, list_index in enumerate(list):
the counter is the first variable in the statement and 1 is added to it per iteration of the loop, the next variable is the value of that iteration in the list. I hope this answers your first question as for your second, the following code might help
list_a = ["this", "is"]
list_b = ["a", "test"]
list_a += list_b
print(list_a)
["this", "is", "a", "test"]
The += works for strings as well because they are essentially lists aw well. Hope this helps!