I am new in using python. My problem might seems easy but unfortunately I could not find a solution for it. I have a set of images in Geotiff format which are at the same size, their pixel values range between 0 to 5 and their non values are -9999. I would like to do kind of image stacking using Numpy and Gdal. I am looking for an stacking algorithm in which those pixels of each image that have a value between 0 to 5 are used and the no data values are not used in computing the average. For example if I have 30 images and for two of them the value at the index Image[20,20] are 2 & 3 respectively and for the rest of images it is -9999 at this index. I want the single band output image to be 2.5 at this index. I am wondering if anyone knows the way to do it?
Any suggestions or hints are highly appreciated.
Edit:
let me clarify it a bit more. Here is a sample :
import numpy as np
myArray = np.random.randint(5,size=(3,3,3))
myArray [1,1,1] = -9999
myArray
>> array([[[ 0, 2, 1],
[ 1, 4, 1],
[ 1, 1, 2]],
[[ 4, 2, 0],
[ 3, -9999, 0],
[ 1, 0, 3]],
[[ 2, 0, 3],
[ 1, 3, 4],
[ 2, 4, 3]]])
suppose that myArray is an ndarray which contains three images as follow:
Image_01 = myArray[0]
Image_02 = myArray[1]
Image_03 = myArray[2]
the final stacked image is :
stackedImage = myArray.mean(axis=0)
>> array([[ 2.00000000e+00, 1.33333333e+00, 1.33333333e+00],
[ 1.66666667e+00, -3.33066667e+03, 1.66666667e+00],
[ 1.33333333e+00, 1.66666667e+00, 2.66666667e+00]])
But I want it to be this :
array([[ 2.00000000e+00, 1.33333333e+00, 1.33333333e+00],
[ 1.66666667e+00, 3.5, 1.66666667e+00],
[ 1.33333333e+00, 1.66666667e+00, 2.66666667e+00]])
Masked arrays are a good way to deal with missing or invalid values. Masked arrays have a .data attribute, which contains the numerical value for each element, and a .mask attribute that specifies which values should be considered 'invalid' and ignored.
Here's a full example using your data:
import numpy as np
# your example data, with a bad value at [1, 1, 1]
M = np.array([[[ 0, 2, 1],
[ 1, 4, 1],
[ 1, 1, 2]],
[[ 4, 2, 0],
[ 3, -9999, 0],
[ 1, 0, 3]],
[[ 2, 0, 3],
[ 1, 3, 4],
[ 2, 4, 3]]])
# create a masked array where all of the values in `M` that are equal to
# -9999 are masked
masked_M = np.ma.masked_equal(M, -9999)
# take the mean over the first axis
masked_mean = masked_M.mean(0)
# `masked_mean` is another `np.ma.masked_array`, whose `.data` attribute
# contains the result you're looking for
print masked_mean.data
# [[ 2. 1.33333333 1.33333333]
# [ 1.66666667 3.5 1.66666667]
# [ 1.33333333 1.66666667 2.66666667]]
Related
I'm trying to implement the grad-camm algorithm:
https://arxiv.org/pdf/1610.02391.pdf
My arguments are:
activations: Tensor with shape torch.Size([1, 512, 14, 14])
alpha values : Tensor with shape torch.Size([512])
I want to multiply each activation (in dimension index 1 (sized 512)) in each corresponding alpha value: for example if the i'th index out of the 512 in the activation is 4 and the i'th alpha value is 5, then my new i'th activation would be 20.
The shape of the output should be torch.Size([1, 512, 14, 14])
Assuming the desired output is of shape (1, 512, 14, 14).
You can achieve this with torch.einsum:
torch.einsum('nchw,c->nchw', x, y)
Or with a simple dot product, but you will first need to add a couple of additional dimensions on y:
x*y[None, :, None, None]
Here's an example with x.shape = (1, 4, 2, 2) and y = (4,):
>>> x = torch.arange(16).reshape(1, 4, 2, 2)
tensor([[[[ 0, 1],
[ 2, 3]],
[[ 4, 5],
[ 6, 7]],
[[ 8, 9],
[10, 11]],
[[12, 13],
[14, 15]]]])
>>> y = torch.arange(1, 5)
tensor([1, 2, 3, 4])
>>> x*y[None, :, None, None]
tensor([[[[ 0, 1],
[ 2, 3]],
[[ 8, 10],
[12, 14]],
[[24, 27],
[30, 33]],
[[48, 52],
[56, 60]]]])
I have a large database with several columns, i need data from 2 of these.
The end result is to have 2 drop down menus where the first one sets "names" and the second one is the "numbers" values that has been merged into the name. I just need the data available so i can input it into another program.
So a list or dictionary that contains the Unique values of the "names" list, with the numbers from the numbers list appended to them.
# Just a list of random names and numbers for testing
names = [
"Cindi Brookins",
"Cumberband Hamberdund",
"Roger Ramsden",
"Cumberband Hamberdund",
"Lorean Dibble",
"Lorean Dibble",
"Coleen Snider",
"Rey Bains",
"Maxine Rader",
"Cindi Brookins",
"Catharine Vena",
"Lanny Mckennon",
"Berta Urban",
"Rey Bains",
"Roger Ramsden",
"Lanny Mckennon",
"Catharine Vena",
"Berta Urban",
"Maxine Rader",
"Coleen Snider"
]
numbers = [
6,
5,
7,
10,
3,
9,
1,
1,
2,
7,
4,
2,
8,
3,
8,
10,
4,
9,
6,
5
]
So in the above example "Berta Urban" would appear once, but still have the numbers 8 and 9 assigned, "Rey Bains" would have 1 and 3.
I have tried with
mergedlist = dict(zip(names, numbers))
But that only assigns the last of the numbers to the name.
I am not sure if i can make a dictionary with Unique "names" that holds multiple "numbers".
You only get the last number associated with each name because dictionary keys are unique (otherwise they wouldn't be much use). So if you do
mergedlist["Berta Urban"] = 8
and after that
mergedlist["Berta Urban"] = 9
the result will be
{'Berta Urban': 9}
Just as if you did:
berta_urban = 8
berta_urban = 9
In that case you would expect the value of berta_urban to be 9 and not [8,9].
So, as you can see, you need an append not an assignment to your dict entry.
from collections import defaultdict
mergedlist = defaultdict(list)
for (name,number) in zip(names, numbers): mergedlist[name].append(number)
This gives:
{'Coleen Snider': [1, 5],
'Cindi Brookins': [6, 7],
'Cumberband Hamberdund': [5, 10],
'Roger Ramsden': [7, 8],
'Lorean Dibble': [3, 9],
'Rey Bains': [1, 3],
'Maxine Rader': [2, 6],
'Catharine Vena': [4, 4],
'Lanny Mckennon': [2, 10],
'Berta Urban': [8, 9]
}
which is what I think you want. Note that you will get duplicates, as in 'Catharine Vena': [4, 4] and you will also get a list of numbers for each name, even if the list has only one number in it.
You cannot have multiple keys of the same name in a dict, but your dict keys can be unique while holding a list of matching numbers. Something like:
mergedlist = {}
for i, v in enumerate(names):
mergedlist[v] = mergedlist.get(v, []) + [numbers[i]]
print(mergedlist["Berta Urban"]) # prints [8, 9]
Not terribly efficient, tho. In dependence of the datatbase you're using, chances are that the database can get you the results in the form you prefer faster than you post-processing and reconstructing the data.
I am trying to vertically concatenate two Dask DataFrames
I have the following Dask DataFrame:
d = [
['A','B','C','D','E','F'],
[1, 4, 8, 1, 3, 5],
[6, 6, 2, 2, 0, 0],
[9, 4, 5, 0, 6, 35],
[0, 1, 7, 10, 9, 4],
[0, 7, 2, 6, 1, 2]
]
df = pd.DataFrame(d[1:], columns=d[0])
ddf = dd.from_pandas(df, npartitions=5)
Here is the data as a Pandas DataFrame
A B C D E F
0 1 4 8 1 3 5
1 6 6 2 2 0 0
2 9 4 5 0 6 35
3 0 1 7 10 9 4
4 0 7 2 6 1 2
Here is the Dask DataFrame
Dask DataFrame Structure:
A B C D E F
npartitions=4
0 int64 int64 int64 int64 int64 int64
1 ... ... ... ... ... ...
2 ... ... ... ... ... ...
3 ... ... ... ... ... ...
4 ... ... ... ... ... ...
Dask Name: from_pandas, 4 tasks
I am trying to concatenate 2 Dask DataFrames vertically:
ddf_i = ddf + 11.5
dd.concat([ddf,ddf_i],axis=0)
but I get this error:
Traceback (most recent call last):
...
File "...", line 572, in concat
raise ValueError('All inputs have known divisions which cannot '
ValueError: All inputs have known divisions which cannot be concatenated
in order. Specify interleave_partitions=True to ignore order
However, if I try:
dd.concat([ddf,ddf_i],axis=0,interleave_partitions=True)
then it appears to be working. Is there a problem with setting this to True (in terms of performance - speed)? Or is there another way to vertically 2 concatenate Dask DataFrames?
If you inspect the divisions of the dataframe ddf.divisions, you will find, assuming one partition, that it has the edges of the index there: (0, 4). This is useful to dask, as it knows when you do some operation on the data, not to use a partition not including required index values. This is also why some dask operations are much faster when the index is appropriate for the job.
When you concatenate, the second dataframe has the same index as the first. Concatenation would work without interleaving if the values of the index had different ranges in the two partitions.
mdurant's answer is correct and this answer elaborate with MCVE code snippets using Dask v2021.08.1. Examples make it easier to understand divisions and interleaving.
Vertically concatenating DataFrames
Create two DataFrames, concatenate them, and view the results.
df = pd.DataFrame(
{"nums": [1, 2, 3, 4, 5, 6], "letters": ["a", "b", "c", "d", "e", "f"]}
)
ddf1 = dd.from_pandas(df, npartitions=2)
df = pd.DataFrame({"nums": [88, 99], "letters": ["xx", "yy"]})
ddf2 = dd.from_pandas(df, npartitions=1)
ddf3 = dd.concat([ddf1, ddf2])
print(ddf3.compute())
nums letters
0 1 a
1 2 b
2 3 c
3 4 d
4 5 e
5 6 f
0 88 xx
1 99 yy
Divisions metadata when vertically concatenating
Create two DataFrames, concatenate them, and illustrate that sometimes this operation will cause divisions metadata to be lost.
def print_partitions(ddf):
for i in range(ddf.npartitions):
print(ddf.partitions[i].compute())
df = pd.DataFrame(
{"nums": [1, 2, 3, 4, 5, 6], "letters": ["a", "b", "c", "d", "e", "f"]}
)
ddf1 = dd.from_pandas(df, npartitions=2)
ddf1.divisions # (0, 3, 5)
df = pd.DataFrame({"nums": [88, 99], "letters": ["xx", "yy"]})
ddf2 = dd.from_pandas(df, npartitions=1)
ddf2.divisions # (0, 1)
ddf3 = dd.concat([ddf1, ddf2])
ddf3.divisions # (None, None, None, None)
Set interleave_partitions=True to avoid losing the divisions metadata.
ddf3_interleave = dd.concat([ddf1, ddf2], interleave_partitions=True)
ddf3_interleave.divisions # (0, 1, 3, 5)
When interleaving isn't necessary
Create two DataFrames without overlapping divisions, concatenate them, and confirm that the divisions metadata is not lost:
df = pd.DataFrame(
{"nums": [1, 2, 3, 4], "letters": ["a", "b", "c", "d"], "some_index": [4, 5, 6, 7]}
)
ddf1 = dd.from_pandas(df, npartitions=2)
ddf1 = ddf1.set_index("some_index")
df = pd.DataFrame({"nums": [88, 99], "letters": ["xx", "yy"], "some_index": [10, 20]})
ddf2 = dd.from_pandas(df, npartitions=1)
ddf2 = ddf2.set_index("some_index")
ddf3 = dd.concat([ddf1, ddf2])
ddf3.divisions # (4, 6, 10, 20)
I wrote a blog post to explain this in more detail. Let me know if you'd like the link.
When iterating on a 2d array, how can I get the current row index? For example:
x = [[ 1. 2. 3. 4.]
[ 5. 6. 7. 8.]
[ 9. 0. 3. 6.]]
Something like:
for rows in x:
print x current index (for example, when iterating on [ 5. 6. 7. 8.], return 1)
Enumerate is a built-in function of Python. It’s usefulness can not be summarized in a single line. Yet most of the newcomers and even some advanced programmers are unaware of it. It allows us to loop over something and have an automatic counter. Here is an example:
for counter, value in enumerate(some_list):
print(counter, value)
And there is more! enumerate also accepts an optional argument which makes it even more useful.
my_list = ['apple', 'banana', 'grapes', 'pear']
for c, value in enumerate(my_list, 1):
print(c, value)
.
# Output:
# 1 apple
# 2 banana
# 3 grapes
# 4 pear
The optional argument allows us to tell enumerate from where to start the index. You can also create tuples containing the index and list item using a list. Here is an example:
my_list = ['apple', 'banana', 'grapes', 'pear']
counter_list = list(enumerate(my_list, 1))
print(counter_list)
.
# Output: [(1, 'apple'), (2, 'banana'), (3, 'grapes'), (4, 'pear')]
enumerate:
In [42]: x = [[ 1, 2, 3, 4],
...: [ 5, 6, 7, 8],
...: [ 9, 0, 3, 6]]
In [43]: for index, rows in enumerate(x):
...: print('current index {}'.format(index))
...: print('current row {}'.format(rows))
...:
current index 0
current row [1, 2, 3, 4]
current index 1
current row [5, 6, 7, 8]
current index 2
current row [9, 0, 3, 6]
I have a list with 10 records, and each record has one or more elements with 3 categories like below:
list = [('0.4', 2, 'doc4.txt'),('0.04', 13, 'doc4.txt'), ('0.5', 4, 'doc4.txt')]
[('0.5', 6, 'doc3.txt'),('0.04', 13, 'doc3.txt'), ('0.5', 4, 'doc3.txt')]
[('0.6', 8, 'doc2.txt')]
[('0.4', 2, 'doc5.txt'), ('1.0', 7, 'doc5.txt')]
[('0.2', 2, 'doc6.txt'), ('0.4', 2, 'doc6.txt'),('0.8', 2, 'doc6.txt'), ('0.34', 5, 'doc6.txt'),('0.76', 4, 'doc6.txt'), ('0.5', 3, 'doc6.txt')]
[('0.3', 7, 'doc9.txt')]
[('0.1', 8, 'doc12.txt')]
[('0.3', 9, 'doc11.txt'),('1.0', 8, 'doc11.txt')]
[('0.9', 7, 'doc22.txt')]
[('0.3', 7, 'doc24.txt')]
You many notice the third category of every record has the same text for each record. There are 10 categories as the list consists of 10 records.
According to the structure of the list:
For example, [('0.6', 8, 'doc2.txt')]
First element, '0.6' represents X-axis value in the range of [0.1 -> 1.0]
Second element of an integer represents Y-axis value in graph
Third element, 'doc2.txt' represents the Category name in graph
The list should be plotted as the image below,
I've been trying with several approaches, but still couldn't figure that out
>>> plt.scatter(*zip(*list))
>>> plt.xlabel('X-Axis')
>>> plt.ylabel('Y-Axis')
>>> plt.show()
I think you can just keep the list as it is and iterate over it. You'd then produce a scatter plot for each sublist in the outer list, as the items from the sublist should share the same marker, color and legend label.
import matplotlib.pyplot as plt
#don't call a variable "list" or "print" or any other python command's name
liste=[[('0.4', 2, 'doc4.txt'),('0.04', 13, 'doc4.txt'), ('0.5', 4, 'doc4.txt')],
[('0.5', 6, 'doc3.txt'),('0.04', 13, 'doc3.txt'), ('0.5', 4, 'doc3.txt')],
[('0.6', 8, 'doc2.txt')],
[('0.4', 2, 'doc5.txt'), ('1.0', 7, 'doc5.txt')],
[('0.2', 2, 'doc6.txt'), ('0.4', 2, 'doc6.txt'),('0.8', 2, 'doc6.txt'), ('0.34', 5, 'doc6.txt'),('0.76', 4, 'doc6.txt'), ('0.5', 3, 'doc6.txt')],
[('0.3', 7, 'doc9.txt')],
[('0.1', 8, 'doc12.txt')],
[('0.3', 9, 'doc11.txt'),('1.0', 8, 'doc11.txt')],
[('0.9', 7, 'doc22.txt')],
[('0.3', 7, 'doc24.txt')]]
markers=[ur"$\u25A1$", ur"$\u25A0$", ur"$\u25B2$", ur"$\u25E9$"]
colors= ["k", "crimson", "#112b77"]
fig, ax = plt.subplots()
for i, l in enumerate(liste):
x,y,cat = zip(*l)
ax.scatter(list(map(float, x)),y, s=64,c=colors[(i//4)%3],
marker=markers[i%4], label=cat[0])
ax.legend(bbox_to_anchor=(1.01,1), borderaxespad=0)
plt.subplots_adjust(left=0.1,right=0.8)
plt.show()
There are multiple issues. You assignment of list makes no sense (presumably you forgot some parentheses). Also, you really shouldn't reuse built-in names like "list". You should not represent floats as strings (your x coordinates). You cannot simply unpack a list into plt.scatter and hope that magically all of these issues work themselves out.
Below some code how to properly pass your data to scatter (I use plot instead of scatter as you can pass plot proper colour names).
import numpy as np
import matplotlib.pyplot as plt
# 'list' is a bad name for a variable as it overwrites the list() built-in function
# -> rename to data
data = [
[('0.4', 2, 'doc4.txt'),('0.04', 13, 'doc4.txt'), ('0.5', 4, 'doc4.txt')],
[('0.5', 6, 'doc3.txt'),('0.04', 13, 'doc3.txt'), ('0.5', 4, 'doc3.txt')],
[('0.6', 8, 'doc2.txt')],
[('0.4', 2, 'doc5.txt'), ('1.0', 7, 'doc5.txt')],
[('0.2', 2, 'doc6.txt'), ('0.4', 2, 'doc6.txt'),('0.8', 2, 'doc6.txt'), ('0.34', 5, 'doc6.txt'),('0.76', 4, 'doc6.txt'), ('0.5', 3, 'doc6.txt')],
[('0.3', 7, 'doc9.txt')],
[('0.1', 8, 'doc12.txt')],
[('0.3', 9, 'doc11.txt'),('1.0', 8, 'doc11.txt')],
[('0.9', 7, 'doc22.txt')],
[('0.3', 7, 'doc24.txt')]
]
# flatten nested list
flat = [item for sublist in data for item in sublist]
# convert strings to numbers
numeric = [(float(x), y, label) for (x, y, label) in flat]
# create a dictionary that maps a label to a set of x,y coordinates
data = dict()
for (x, y, label) in numeric:
if label in data:
data[label].append((x,y))
else:
data[label] = [(x,y)]
# initialise figure
fig, ax = plt.subplots(1,1)
colors = ['blue', 'red', 'yellow', 'green', 'orange', 'brown', 'violet', 'magenta', 'white', 'black']
# populate figure
for color, (label, xy) in zip(colors, data.iteritems()):
x, y = np.array(xy).T
ax.plot(x, y, 'o', label=label, color=color)
ax.set_xlim(0, 1.1)
ax.set_ylim(0, 16)
ax.legend(numpoints=1)
plt.show()