def laser_callback(self, laserMsg):
cloud = self.laser_projector.projectLaser(laserMsg)
gen = pc2.read_points(cloud, skip_nans=True, field_names=('x', 'y', 'z'))
self.xyz_generator = gen
print(gen)
I'm trying to convert the laser data into pointcloud2 data, and then display them using matplotlib.pyplot. I tried traversing individual points in the generator but it takes a long time. Instead I'd like to convert them into a numpy array and then plot it. How do I go about doing that?
Take a look at some of these other posts which seem to answer the basic question of "convert a generator to an array":
How do I build a numpy array from a generator?
How to construct an np.array with fromiter
How to fill a 2D Python numpy array with values from a generator?
numpy fromiter with generator of list
Without knowing exactly what your generator is returning, the best I can do is provide a somewhat generic (but not particularly efficient) example:
#!/usr/bin/env -p python
import numpy as np
# Sample generator of (x, y, z) tuples
def my_generator():
for i in range(10):
yield (i, i*2, i*2 + 1)
i += 1
def gen_to_numpy(gen):
return np.array([x for x in gen])
gen = my_generator()
array = gen_to_numpy(gen)
print(type(array))
print(array)
Output:
<class 'numpy.ndarray'>
[[ 0 0 1]
[ 1 2 3]
[ 2 4 5]
[ 3 6 7]
[ 4 8 9]
[ 5 10 11]
[ 6 12 13]
[ 7 14 15]
[ 8 16 17]
[ 9 18 19]]
Again though, I cannot comment on the efficiency of this. You mentioned that it takes a long time to plot by reading points directly from the generator, but converting to a Numpy array will still require going through the whole generator to get the data. It would probably be much more efficient if the laser to pointcloud implementation you are using could provide the data directly as an array, but that is a question for the ROS Answers forum (I notice you already asked this there).
I apologise in advance for how simple the answer probably is to this question, I am very new to netlogo and very out of my depth.
I am trying to read a water-temperature from a file and consequently get my turtles to die/breed depending on the temperature. I have got the file to read finally, and set water-temperature as a global variable, however I am now stuck on the comparison part. It won't let me compare the variable to a number because I think the variable is a list. The following error message comes up;
The > operator can only be used on two numbers, two strings, or two agents of the same type, but not on a list and a number.
error while turtle 7 running >
called by procedure REPRODUCE
called by procedure GO
called by Button 'go'
Code is below;
globals [ year
month
water-temperature ]
extensions [ csv ]
to setup
ca
load-data
create-turtles 50
[ set size 1
set color red
setxy random-xcor random-ycor ]
reset-ticks
end
to go
ask turtles [ move
reproduce ]
run-temperature
end
to load-data
file-close-all
file-open "C:\\Users\\Hannah\\Documents\\Summer research project\\test3.csv"
end
to run-temperature
file-close-all
file-open "C:\\Users\\Hannah\\Documents\\Summer research project\\test3.csv"
while [ not file-at-end? ] [
set water-temperature csv:from-row file-read-line
tick ]
file-close
end
to move
rt random 50
lt random 50
fd 1
end
to reproduce
if water-temperature > 35 [ die ]
if water-temperature > 30 and water-temperature < 34 [ hatch 1 rt random-float 360 fd 1 ]
if water-temperature > 25 and water-temperature < 29 [ hatch 2 rt random-float 360 fd 1 ]
if water-temperature > 20 and water-temperature < 24 [ hatch 3 rt random-float 360 fd 1 ]
end
I would be so grateful for any help!
Thanks :)
Hannah
welcome to Stack Overflow. Can you please provide an example of the first few lines of your "test3.csv" file? That will help get your question sorted- if you have a header or multiple columns that could be causing your problems- multiple columns might be getting read in as a list. As well, I think you want file-read instead of file-read-line.
A few other things- your load-data procedure is unnecessary as far as I can tell (you only need the loading to occur in run-temperature).
More importantly, your code right now says something like: "All turtles, move and reproduce. Now, read the whole temperature file line by line." The problem is that your while statement is saying "until you have yet reached the end of the file, read a line, tick, and move to the next one." Additionally, your model will tick once per line, without the turtles ever doing anything- it probably is simpler to just have your tick at the very end of your go procedure. It is likely better to avoid the use of while in your go procedure in this scenario, as it will loop until the while condition is satisfied.
It might be easier to just read your whole test.csv and store it in a variable for easier access- here is one example. Using this setup:
globals [
water-temperature
water-temperature-list
]
to setup
ca
crt 50 [
setxy random-xcor random-ycor
]
First, tell Netlogo water-temperature-list is a list using set and []. Then, do the same file close/open as before to prep your file for reading. Then, use a similar while loop to read your temperatures into water-temperature-list, using lput:
set water-temperature-list []
file-close-all
file-open "test3.csv"
while [ not file-at-end? ] [
set water-temperature-list lput file-read water-temperature-list
]
file-close-all
reset-ticks
end
Now your model more simply access those values, since they are stored in a model variable directly. You can easily use the ticks value with item as an index for that list- for example, on tick 0 the first element in the list will be accessed, on tick 1 the second element, and so on. For example:
to go
set water-temperature item ticks water-temperature-list
ask turtles [
if water-temperature > 30 [
die
]
if water-temperature <= 30 [
rt random 60
fd 1
]
]
tick
end
Note that with this setup once you get to the end of your temperatures, there will be an error telling you that Netlogo can't find the next list element- you'll have to put a stop condition somewhere to prevent that.
I know that is an alternative to your approach but I hope that it's helpful. For another similar but more complicated example, check out this model by Uri Wilensky.
I have been trying to work out how to replicate IDL's smooth function in Python and I just can't get anything like the same results. (Disclaimer: It is probably 10 years since I touched this kind of mathematical problem so it has been dumped to make way for information like where to find the cheapest local fuel). I am trying to code this:
smooth(b,w,/nan)
where b is a 2D float array containing NANs (zeros - missing data - have also been converted to NAN).
From the IDL documents, it appears smooth uses a boxcar, so from scipy.ndimage.filters I have tried:
bsmooth = uniform_filter(b, w)
I am aware that there are some fundamental differences here:
the default edge behaviour from IDL is "the end points are copied
from the original array to the result with no smoothing" whereas I
don't seem to have the option to do this with the uniform filter.
Treatment of the NaN elements. In IDL, the /nan keyword seems to
mean that where possible the NaN values will be filled by the result
of the other points in the window. If there are no valid points to
generate a result, by a MISSING keyword. I thought I could
approximate this behaviour following the smoothing using
scipy.interpolate's NearestNDInterpolator (thanks to the brilliant
explanation by Alex on here:
filling gaps on an image using numpy and scipy)
Here is my test array:
>>>b array([[ 0.97599638, 0.93114936, 0.87070072, 0.5379253 ],
[ 0.34873217, nan, 0.40985891, 0.22407863],
[ nan, nan, nan, 0.67532134],
[ nan, nan, 0.85441768, nan]])
My answers bore not the SLIGHTEST resemblance to IDL, whether I use the /nan keyword or not.
IDL> smooth(b,2,/nan)
0.97599638 0.93114936 0.87070072 0.53792530
0.34873217 0.70728749 0.60817236 0.22407863
NaN 0.53766960 0.54091913 0.67532134
NaN NaN 0.85441768 NaN
IDL> smooth(b,2)
0.97599638 0.93114936 0.87070072 0.53792530
0.34873217 -NaN -NaN 0.22407863
-NaN -NaN -NaN 0.67532134
-NaN -NaN 0.85441768 NaN
I confess I find the scipy documentation rather sparse on detail so I have no idea if I am really doing what I think I doing. The fact that the two python approaches which I believed would both smooth the image give different answers suggests that things are not what I understood them to be.
>>>uniform_filter(b, 2)
array([[ 0.97599638, 0.95357287, 0.90092504, 0.70431301],
[ 0.66236428, nan, nan, nan],
[ nan, nan, nan, nan],
[ nan, nan, nan, nan]])
I thought it was a bit odd it was so empty so I tried this with an array of 100 elements (still using a window of 2) and output the images. The results (first image is 'b' second is 'bsmooth') are not quite what I was hoping for:
Going back to the smaller array and following the examples in: http://scipy.github.io/old-wiki/pages/Cookbook/SignalSmooth which I thought would give the same output as uniform_filter, I tried:
>>> box = np.array([1,1,1,1])
>>> box = box.reshape(2,2)
>>> box
array([[1, 1],
[1, 1]])
>>> bsmooth = scipy.signal.convolve2d(b,box,mode='same')
>>> print bsmooth
[[ 0.97599638 1.90714574 1.80185008 1.40862602]
[ 1.32472855 nan nan 2.04256356]
[ nan nan nan nan]
[ nan nan nan nan]]
Obviously I have completely misunderstood the scipy functions, maybe even the IDL one. If anyone can help me to replicate the IDL smooth function as closely as possible, I would be extremely grateful. I am under considerable time pressure to get a solution for this that doesn't rely on IDL and I am tossing a coin to decide whether to code the function from scratch or develop a very contagious illness.
How can I perform the same smoothing in python?
First: Please use matplotlib.pyplot.imshow with interpolation="none" that's nicer to look at and maybe with greyscale.
So for your example: There is actually no convolution (filter) within scipy and numpy that treat's NaN as missing values (they propagate them within the convolution). At least I've found none so far and your boundary-treatement is also (to my knowledge) not implemented. But the boundary could be just replaced afterwards.
If you want to do convolution with NaN you can for example use astropy.convolution.convolve. There NaNs are interpolated using the kernel of your filter. But their convolution has some drawbacks as well: Border handling like you want isn't implemented there neither and your kernel must be of odd shape and the sum of your kernel must not be zero (or very close to it)
For example:
from astropy.convolution import convolve
import numpy as np
array = np.random.uniform(10,100, (4,4))
array[1,1] = np.nan
kernel = np.ones((3,3))
convolve(array, kernel)
as an example an initial array of
array([[ 97.19514587, 62.36979751, 93.54811286, 30.23567842],
[ 51.02184613, nan, 46.14769821, 60.08088041],
[ 20.86482452, 42.39661484, 36.96961278, 96.89180175],
[ 45.54453509, 76.61274347, 46.44485141, 25.40985372]])
will become:
array([[ 266.9009961 , 406.59680717, 348.69637399, 230.01236989],
[ 330.16243546, 506.82785931, 524.95440336, 363.87378443],
[ 292.75477064, 422.31693304, 487.26826319, 311.94469828],
[ 185.41871792, 268.83318211, 324.72547798, 205.71611967]])
if you want to "normalize" it, astropy offers the normalize_kernel parameter:
convolved = convolve(array, kernel, normalize_kernel=True)
array([[ 29.58753936, 42.09982189, 49.31793529, 33.00203873],
[ 49.87040638, 65.67695002, 66.10447436, 40.44026448],
[ 52.51126383, 63.03914444, 60.85474739, 35.88011742],
[ 39.40188443, 46.82350749, 40.1380926 , 22.46090152]])
If you want to replace the "edge" values with the ones from the original array just replace them:
convolved[0,:] = array[0,:]
convolved[-1,:] = array[-1,:]
convolved[:,0] = array[:,0]
convolved[:,-1] = array[:,-1]
So that's what the existing packages offer (as far as I know it). If you want to learn a bit of Cython or numba you can easily write your own convolutions that is not much slower (only a factor of 2-10) than the numpy/scipy ones but does EXACTLY what you want without messing around.
Since this is not something that is available in the python packages and because I saw the question asked several times during my research without satisfactory answers, here is how I solved the issue.
Provided is a test version of my function that I'm off to tidy up. I am sure there will be better ways to do the things I have done as I'm still fairly new to Python - please do recommend any appropriate changes.
Plots use autumn colourmap just because it allowed me to see the NaNs clearly.
My results:
IDL propagate
0.033369284 0.067915268 0.96602046 0.85623550
0.30435592 NaN NaN 100.00000
0.94065958 NaN NaN 0.90966976
0.018516513 0.044460904 0.051047217 NaN
python propagate
[[ 3.33692829e-02 6.79152655e-02 9.66020487e-01 8.56235492e-01]
[ 3.04355923e-01 nan nan 1.00000000e+02]
[ 9.40659566e-01 nan nan 9.09669768e-01]
[ 1.85165123e-02 4.44609040e-02 5.10472165e-02 nan]]
IDL replace
0.033369284 0.067915268 0.96602046 0.85623550
0.30435592 0.47452110 14.829881 100.00000
0.94065958 0.33833817 17.002417 0.90966976
0.018516513 0.044460904 0.051047217 NaN
python replace
[[ 3.33692829e-02 6.79152655e-02 9.66020487e-01 8.56235492e-01]
[ 3.04355923e-01 4.74521092e-01 1.48298812e+01 1.00000000e+02]
[ 9.40659566e-01 3.38338177e-01 1.70024175e+01 9.09669768e-01]
[ 1.85165123e-02 4.44609040e-02 5.10472165e-02 nan]]
My function:
#!/usr/bin/env python
# smooth.py
__version__ = 0.1
# Version 0.1 29 Feb 2016 ELH Test release
import numpy as np
import matplotlib.pyplot as mp
def Smooth(v1, w, nanopt):
# v1 is the input 2D numpy array.
# w is the width of the square window along one dimension
# nanopt can be replace or propagate
'''
v1 = np.array(
[[3.33692829e-02, 6.79152655e-02, 9.66020487e-01, 8.56235492e-01],
[3.04355923e-01, np.nan , 4.86013025e-01, 1.00000000e+02],
[9.40659566e-01, 5.23314093e-01, np.nan , 9.09669768e-01],
[1.85165123e-02, 4.44609040e-02, 5.10472165e-02, np.nan ]])
w = 2
'''
mp.imshow(v1, interpolation='None', cmap='autumn')
mp.show()
# make a copy of the array for the output:
vout=np.copy(v1)
# If w is even, add one
if w % 2 == 0:
w = w + 1
# get the size of each dim of the input:
r,c = v1.shape
# Assume that w, the width of the window is always square.
startrc = (w - 1)/2
stopr = r - ((w + 1)/2) + 1
stopc = c - ((w + 1)/2) + 1
# For all pixels within the border defined by the box size, calculate the average in the window.
# There are two options:
# Ignore NaNs and replace the value where possible.
# Propagate the NaNs
for col in range(startrc,stopc):
# Calculate the window start and stop columns
startwc = col - (w/2)
stopwc = col + (w/2) + 1
for row in range (startrc,stopr):
# Calculate the window start and stop rows
startwr = row - (w/2)
stopwr = row + (w/2) + 1
# Extract the window
window = v1[startwr:stopwr, startwc:stopwc]
if nanopt == 'replace':
# If we're replacing Nans, then select only the finite elements
window = window[np.isfinite(window)]
# Calculate the mean of the window
vout[row,col] = np.mean(window)
mp.imshow(vout, interpolation='None', cmap='autumn')
mp.show()
return vout
I am doing this python program where i have to access :
This is what i am trying to achieve with my code: Return a dict mapping doc_id to length, computed as sqrt(sum(w_i**2)), where w_i is the tf-idf weight for each term in the document.
E.g., in the sample index below, document 0 has two terms 'a' (with
tf-idf weight 3) and 'b' (with tf-idf weight 4). It's length is
therefore 5 = sqrt(9 + 16).
>>> lengths = Index().compute_doc_lengths({'a': [[0, 3]], 'b': [[0,4]]})
>>> lengths[0]
5.0
The code i have is this:
templist=[]
for iter in index.values():
templist.append(iter)
d = defaultdict(list)
for i,l in templist[1]:
d[i].append(l)
lent = defaultdict()
for m in d:
lo= math.sqrt(sum(lent[m]**2))
return lo
So, if I'm understanding you correctly, we have to transform the input dictionary:
ind = {'a':[ [1,3] ], 'b': [ [1,4 ] ] }
To the output dictionary:
{1:5}
Where the 5 is calculated as the euclidian distance for the value portion of the input dictionary (the vector [3,4] in this case), Correct?
Given that information, the answer becomes a bit more straight-forwards:
def calculate_length(ind):
# Frist, let's transform the dictionary into a list of doc_id, tl_idf pairs; [[doc_id_1,tl_idf_1],...]
data = [entry[0] for entry in ind.itervalues()] # use just ind.values() in python 3.X
# Next, let's split that list into two, one for doc_id's, one for tl_idfs
doc_ids, tl_idfs = zip(*data)
# We can just assume that all the doc_id's are the same. you could check that here if you wanted
doc_id = doc_ids[0]
# Next, we calculate the length as per our formula
length = sqrt(sum(tl_idfs**2 for tl_idfs in tl_idfs))
# Finally, we return the output dictionary
return {doc_id: length}
Example:
>> calculate_length({'a':[ [1,3] ], 'b': [ [1,4 ] ] })
{1:5.0}
There are a couple places in here where you could optimize this to remove the intermidary lists (this method can be two lines of operation and a return) but I'll leave that to you to find out since this is a homework assignment. I also hope you take the time to actually understand what this code does, rather than just copying it wholesale.
Also note that this answer makes the very large asumption that all doc_id values are the same, and there will only ever be a single doc_id,tl_idf list at each key in the dictionary! If that's not true, then your transform becomes more complicated. But you did not provide sample input nore textual explination indicating that's the case (though, based on the data structure, I'd think it quite likely).
Update
In fact, it's really bothering me because I definitely think that's the case. Here is a version that solves the more complex case:
from itertools import chain
from collections import defaultdict
def calculate_length(ind):
# We want to transform this first into a dict of {doc_id:[tl_idf_a,...]}
# First we transform it into a generator of ([doc_id,tl_idf],...)
tf_gen = chain.from_iterable(ind.itervalues())
# which we then use to generate our transformed dictionary
tf_dict = defaultdict(list)
for doc_id, tl_idf in tf_gen:
tf_dict[doc_id].append(tl_idf)
# Now we proceed mostly as before, but we can just do it in one line
return dict((doc_id, sqrt(sum(tl_idfs**2 for tl_idfs in tl_idfs))) for doc_id, tl_idfs in tf_dict.iteritems())
Example use:
>>> calculate_length({'a':[ [1,3] ], 'b': [ [1,4 ] ] })
{1: 5.0}
>>> calculate_length({'a':[ [1,3],[2,3] ], 'b': [ [1,4 ], [2,1] ] })
{1: 5.0, 2: 3.1622776601683795}
Ok, so - this is heavily related to my previous question Transforming an object between two coordinate spaces, but a lot more direct and it should have an obvious answer.
An objects local coordinate space, how do I "get a hold of it"? Say that I load an Orc into my game, how do I know programatically where it's head, left arm, right arm and origin (belly button?) is? And when I know where it is do I need to save it manually or is it something that magically exists in the DirectX API? In my other question one person said something about storing a vertice for X,Y,Z directions and and origin? How do I find these vertices? Do I pick them arbitrarily, assign them in the model before loading it into my game? etc.
This is not about transforming between coordinate spaces
Actually, it is about transforming between coordinate spaces.
Ok, you understand that you can have a Matrix that does Translation or Rotation. (Or scaling. Or skew. Etc.) That you can multiply such a Matrix by a point (V) to get the new (translated/rotated/etc) point. E.g.:
Mtranlate = [ 1 0 0 Tx ] * V = [ Vx ] = [ Vx + Tx ]
[ 0 1 0 Ty ] [ Vy ] [ Vy + Ty ]
[ 0 0 1 Tz ] [ Vz ] [ Vz + Tz ]
[ 0 0 0 1 ] [ 1 ] [ 1 ]
Or rotation about the X/Y/Z axis:
Mrotate_x = [ 1 0 0 0 ] * V = [ Vx ] = [ Vx ]
[ 0 cos -sin 0 ] [ Vy ] [ Vy*cos - Vz*sin ]
[ 0 sin cos 0 ] [ Vz ] [ Vy*sin + Vz*cos ]
[ 0 0 0 1 ] [ 1 ] [ 1 ]
That you can combine multiple operations by multiplying the matrices together. For example: Mnew = Mtranslate * Mrotate_x.
(And yes, the order does matter!)
So given a real-world base origin, you could translate to your (known/supplied) orc's xy location, deduce from the terrain at that point what your orc's feet's z location is and translate to that. Now you have your orc's "feet" in global coordinates. From there, you might want to rotate your orc so he faces a particular direction.
Then translate upwards (based on your orc's model) to find it's neck. From the neck, we can translate upwards (and perhaps rotate) to find its head or over to find its shoulder. From the shoulder, we can rotate and translate to find the elbow. From the elbow we can rotate and translate to find the hand.
Each component of your orc model has a local reference frame. Each is related to the reference frame where it is attached by one or more transforms (matrices). By multiplying through all the local matrices (transforms), we can eventually see how to go from the orc's feet to his hand.
(Mind you, I'd animate the orc and just use his feet x,y,z location. Otherwise it's too much work.)
You see a lot of this sort of thing in robotics which can provide a nice elementary introduction to forward kinematics. In robotics, at least with the simpler robots, you have a motor which rotates, and a limb-length to translate by. And then there's another motor. (In contrast, the human shoulder-joint can rotate about all three axis at once.)
You might look over the course HANDOUTS for CMU's Robotic Manipulation class (15-384).
Perhaps something like their Forward Kinematics Intro.