I'm trying to select the densest (in terms contained elements) areas of a DXF layer in ezdxf.
These boxes are evenly spaced and shaped.
In other words: given a BoundingBox2d instance bbox, how to obtain number of entities (lines, poly lines, basically - everything) contained in ezdxf.layouts.layout.Modelspace instance msp within that bbox?
The msp variable contains modelspace with only one variable turned on.
Intuitively, it should be something like:
def count_elements_in_layer_and_bbox(msp: Modelspace, layer: str, bbox: BoundingBox2d) -> int
# turn off all layers but one
for l in msp.layers:
if layer == l.dxf.name:
l.on()
else:
l.off()
# get entities within the bbox - this is the line I don't know how to approach
entities = msp.intersect(bbox).dxf.entities
return len(enitites)
ezdxf is a DXF file format interface not a CAD application. So switching layers on/off has no influence on the content of the modelspace.
But ezdxf gives you other tools to select entities based on DXF attributes like the query() method. BTW a layer is just a DXF attribute and not a real container:
from ezdxf import bbox
def count_entities(msp, layer, selection_box):
count = 0
# select all entities of a certain layer
for e in msp.query(f"*[layer=='{layer}']"):
# put a single entity into a list to get its bounding box
entity_box = bbox.extents([e])
if selection_box.has_intersection(entity_box):
count += 1
return count
Read the docs for query() and also read the limitations of the bbox module.
Related
I have a model trained to classify rgb values into 1000 categories.
#Model architecture
model = Sequential()
model.add(Dense(512,input_shape=(3,),activation="relu"))
model.add(BatchNormalization())
model.add(Dense(512,activation="relu"))
model.add(BatchNormalization())
model.add(Dense(1000,activation="relu"))
model.add(Dense(1000,activation="softmax"))
I want to be able to extract the output before the softmax layer so I can conduct analyses on different samples of categories within the model. I want execute softmax for each sample, and conduct analyses using a function named getinfo().
Model
Initially, I enter X_train data into model.predict, to get a vector of 1000 probabilities for each input. I execute getinfo() on this array to get the desired result.
Pop1
I then use model.pop() to remove the softmax layer. I get new predictions for the popped model, and execute scipy.special.softmax. However, getinfo() produces an entirely different result on this array.
Pop2
I write my own softmax function to validate the 2nd result, and I receive an almost identical answer to Pop1.
Pop3
However, when I simply calculate getinfo() on the output of model.pop() with no softmax function, I get the same result as the initial Model.
data = np.loadtxt("allData.csv",delimiter=",")
model = load_model("model.h5")
def getinfo(data):
objects = scipy.stats.entropy(np.mean(data, axis=0), base=2)
print(('objects_mean',objects))
colours_entropy = []
for i in data:
e = scipy.stats.entropy(i, base=2)
colours_entropy.append(e)
colours = np.mean(np.array(colours_entropy))
print(('colours_mean',colours))
info = objects - colours
print(('objects-colours',info))
return info
def softmax_max(data):
# calculate softmax whilst subtracting the max values (axis=1)
sm = []
count = 0
for row in data:
max = np.argmax(row)
e = np.exp(row-data[count,max])
s = np.sum(e)
sm.append(e/s)
sm = np.asarray(sm)
return sm
#model
preds = model.predict(X_train)
getinfo(preds)
#pop1
model.pop()
preds1 = model.predict(X_train)
sm1 = scipy.special.softmax(preds1,axis=1)
getinfo(sm1)
#pop2
sm2 = softmax_max(preds1)
getinfo(sm2)
#pop3
getinfo(preds1)
I expect to get the same output from Model, Pop1 and Pop2, but a different answer to Pop3, as I did not compute softmax here. I wonder if the issue is with computing softmax after model.predict? And whether I am getting the same result in Model and Pop3 because softmax is constraining the values between 0-1, so for the purpose of the getinfo() function, the result is mathematically equivalent?
If this is the case, then how do I execute softmax before model.predict?
I've gone around in circles with this, so any help or insight would be much appreciated. Please let me know if anything is unclear. Thank you!
model.pop() does not immediately have an effect. You need to run model.compile() again to recompile the new model that doesn't include the last layer.
Without the recompile, you're essentially running model.predict() twice in a row on the exact same model, which explains why Model and Pop3 give the same result. Pop1 and Pop2 give weird results because they are calculating the softmax of a softmax.
In addition, your model does not have the softmax as a separate layer, so pop takes off the entire last Dense layer. To fix this, add the softmax as a separate layer like so:
model.add(Dense(1000)) # softmax removed from this layer...
model.add(Activation('softmax')) # ...and added to its own layer
I have raster data for built up areas around the globe with 40m resolution as vrt file, download data from here , and I am trying to crop the data by a mask and then extract color index value for each cell.
Note: another 2 files exist with the data: vrt.clr and vrt.ovr
Here is a sample of data:
view of vrt data in arcmap.
My question: why I am getting empty cells values when I crop by mask ?
I have tried the following:
extract by mask using arcmap toolbox
using gdal in python 2.7
import gdal
ds = gdal.Open('input.vrt')
ds = gdal.Translate('output.vrt', ds, projWin =
[80.439,5.341,81.048,4.686])
ds = None
I have also try to save the data as tif
Also, is there any way to read the color index value at given coordinates (x,y) after masking the data?
The data appears to be in the Pseudo Mercator projection (EPSG 3857). So therefore you should either specify the extent for projWin in that coordinate system, or add projWinSRS if you want to provide them in a different coordinate system.
Also, if you want gdal.Translate to output to a VRT file, you should add format='VRT. Because in your code snippet outputs to the default file format, which is GeoTIFF.
When i assume your coordinates are WGS84 (EPSG 4326), it defines a small region over the ocean south of Sri Lanka. That doesn't make much sense given the nature of the data.
If you want to read the array given by your coordinates you could use:
invrt = 'GHS_BUILT_LDSMT_GLOBE_R2015B_3857_38_v1_0.vrt'
outfile = '/vsimem/tmpfile'
ds = gdal.Translate(outfile, invrt, projWin=[80.439, 5.341, 81.048, 4.686], projWinSRS='EPSG:4326')
data = ds.ReadAsArray()
ds = None
gdal.Unlink(outfile)
The plotted array looks like:
I am trying to train a Custom Object Detector by using the HOG+SVM method on OpenCV.
I have managed to extract HOG features from my positive and negative samples using the below line of code:
import cv2
hog = cv2.HOGDescriptor()
def poshoggify():
for i in range(1,20):
image = cv2.imread("/Users/munirmalik/cvprojek/cod/pos/" + str(i)+ ".jpg")
(winW, winH) = (500, 500)
for resized in pyramid(image, scale=1.5):
# loop over the sliding window for each layer of the pyramid
for (x, y, window) in sliding_window(resized, stepSize=32, windowSize=(winW, winH)):
# if the window does not meet our desired window size, ignore it
if window.shape[0] != winH or window.shape[1] != winW:
continue
img_pos = hog.compute(image)
np.savetxt('posdata.txt',img_pos)
return img_pos
And the equivalent function for the negative samples.
How do I format the data in such a way that the SVM knows which is positive and which is negative?
Furthermore, how do I translate this training to the "test" of detecting the desired objects through my webcam?
How do I format the data in such a way that the SVM knows which is positive and which is negative?
You would now create another list called labels which would store the class value associated with a corresponding image. For example, if you have a training set of features that looks like this:
features = [pos_features1, pos_features2, neg_features1, neg_features2, neg_features3, neg_features4]
you would have a corresponding labels class like
labels = [1, 1, 0, 0, 0, 0]
You would then feed this to a classifier like so:
clf=LinearSVC(C=1.0, class_weight='balanced')
clf.fit(features,labels)
Furthermore, how do I translate this training to the "test" of detecting the desired objects through my webcam?
Before training, you should have split your labelled dataset (groundtruth) into training and testing datasets. You can do this using skilearns KFold module.
Im trying to finetune the existing models in Keras to classify my own dataset. Till now I have tried the following code (taken from Keras docs: https://keras.io/applications/) in which Inception V3 is fine-tuned on a new set of classes.
from keras.applications.inception_v3 import InceptionV3
from keras.preprocessing import image
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K
# create the base pre-trained model
base_model = InceptionV3(weights='imagenet', include_top=False)
# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# let's add a fully-connected layer
x = Dense(1024, activation='relu')(x)
# and a logistic layer -- let's say we have 200 classes
predictions = Dense(200, activation='softmax')(x)
# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
layer.trainable = False
# compile the model (should be done *after* setting layers to non-trainable)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
# train the model on the new data for a few epochs
model.fit_generator(...)
# at this point, the top layers are well trained and we can start fine-tuning
# convolutional layers from inception V3. We will freeze the bottom N layers
# and train the remaining top layers.
# let's visualize layer names and layer indices to see how many layers
# we should freeze:
for i, layer in enumerate(base_model.layers):
print(i, layer.name)
# we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 172 layers and unfreeze the rest:
for layer in model.layers[:172]:
layer.trainable = False
for layer in model.layers[172:]:
layer.trainable = True
# we need to recompile the model for these modifications to take effect
# we use SGD with a low learning rate
from keras.optimizers import SGD
model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy')
# we train our model again (this time fine-tuning the top 2 inception blocks
# alongside the top Dense layers
model.fit_generator(...)
Can anyone plz guide me what changes should I do in the above code so as to fine-tune ResNet50 model present in Keras.
Thanks in advance.
It is difficult to make out a specific question, have you tried anything more than just copying the code without any changes?
That said, there is an abundance of problems in the code: It is a simple copy/paste from keras.io, not functional as it is, and needs some adaption before working at all (regardless of using ResNet50 or InceptionV3):
1): You need to define the input_shape when loading InceptionV3, specifically replace base_model = InceptionV3(weights='imagenet', include_top=False) with base_model = InceptionV3(weights='imagenet', include_top=False, input_shape=(299,299,3))
2): Further, you need to adapt the number of the classes in the last added layer, e.g. if you have only 2 classes to: predictions = Dense(2, activation='softmax')(x)
3): Change the loss-function when compiling your model from categorical_crossentropy to sparse_categorical_crossentropy
4): Most importantly, you need to define the fit_generator before calling model.fit_generator() and add steps_per_epoch. If you have your training images in ./data/train with every category in a different subfolder, this can be done e.g. like this:
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory(
"./data/train",
target_size=(299, 299),
batch_size=50,
class_mode='binary')
model.fit_generator(train_generator, steps_per_epoch=100)
This of course only does basic training, you will for example need to define save calls to hold on to the trained weights. Only if you get the code working for InceptionV3 with the changes above I suggest to proceed to work on implementing this for ResNet50: As a start you can replace InceptionV3() with ResNet50() (of course only after from keras.applications.resnet50 import ResNet50), and change the input_shape to (224,224,3) and target_size to (224,244).
The above mentioned code-changes should work on Python 3.5.3 / Keras 2.0 / Tensorflow backend.
Beyond the important points mentioned in the above answer for ResNet50 (! if your images are shaped into similar format as in the original Keras code (224,224) - not of rectangular shape) you may substitute:
# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
by
x = base_model.output
x = Flatten(x)
EDIT: Please read #Yu-Yang comment bellow
I think I experienced the same issue. It appeared to be a complex problem, which has a decent thread on github(https://github.com/keras-team/keras/issues/9214). The problem is in Batch Normalization of unfreezed blocks of the net. You have two solutions:
Only change top layer(leaving the blocks as they are)
Add a patch from the github thread above.
I have a map with a lot of markers on it. And I have a two not intersecting polygons (Box). I want to get all markers which covered by these polygons.
qb_1 = Polygon.from_bbox((-35.19153, -5.84512, -35.24054, -5.78552))
qb_2 = Polygon.from_bbox((64.16016, 50.26125, 61.80359, 52.04911))
q_box = MultiPolygon(qb_1, qb_2)
test1 = Marker.objects.filter(point__contained=qb_1)
test2 = Marker.objects.filter(point__contained=qb_2)
test = Marker.objects.filter(point__contained=q_box)
print "Count of Polygon 1 = %s" % test1.count()
print "Count of Polygon 2 = %s" % test2.count()
print "Count of MultiPolygon = %s" % test.count()
But the results are:
Count of Polygon 1 = 4
Count of Polygon 2 = 12
Count of MultiPolygon = 237
Why Polygon 1 + Polygon 2 is not equal MultiPolygon ?
The secret lies in the words I have highlighted (from the geoqueryset documentation)
contained
Availability: PostGIS, MySQL, SpatiaLite
Tests if the geometry field’s bounding box is completely contained by the lookup geometry’s bounding box
The two polygons you have created happen to be have small areas, and the multipoligon you have created also has a small area, but the same cannot be said about it's bounding box.
qb_1.envelope.area # 0.0029209960000001417
qb_2.envelope.area # 4.213217240200014
qbox.envelope.area # 5754.726987961
as you will see the last one is huge in comparison an it covers a lot more points than the two polygons taken on their own. Thus the whole is greater than the sum of it's parts.
You should be able to get the actual points covered by the two polygons as follows:
from django.db.models import Q
Marker.objects.filter(Q(point__contained=qb_1) | Q(point__contained=qb_1))
But perhaps contains_properly is what you are really looking for? But that's available only in postgresql so contains is a good substitute.
contains
Availability: PostGIS, Oracle, MySQL, SpatiaLite
Tests if the geometry field spatially contains the lookup geometry.
Then your query becomes
Marker.objects.filter(Q(point__contains=qb_1) | Q(point__contains=qb_1))