I am trying to train and test for object detection using transfer learning.
I used resnet50 and imagenet weights. But I am facing trouble while saving the model using model.save() format. Here is my code, I haven't used any class. Please help me:
resnet_model = keras.Sequential() #layers added in sequence
pretrained_model= tf.keras.applications.ResNet50(
include_top=False,
weights="imagenet",
input_shape=(180,180,3),
pooling='avg',
classes=2
)
for layer in pretrained_model.layers:
layer.trainable=False #this prevents editting resnet weights and adds our custom training weights.
#adding pretrained resnet model to custom resnet model
resnet_model.add(pretrained_model)
resnet_model.add(Flatten())
resnet_model.add(Dense(120, activation= 'relu'))
resnet_model.add(Dense(2, activation='softmax'))
resnet_model.summary() #to see update
resnet_model.compile(optimizer=Adam(learning_rate=0.001),loss='categorical_crossentropy', metrics=['accuracy'])
history= resnet_model.fit(
train_ds,
validation_data=val_ds,
epochs=2
)
#Saving Model
saveloc=r'C:\Users\Prajnan Karmakar\OneDrive\Desktop\python programming\Keras Resnet\abc.h5'
resnet_model.save(saveloc)
Error Message:
NotImplementedError:
Layer ModuleWrapper has arguments ['module', 'method_name']
in __init__ and therefore must override get_config().
Example:
class CustomLayer(keras.layers.Layer):
def init(self, arg1, arg2):
super().init()
self.arg1 = arg1
self.arg2 = arg2
def get_config(self):
config = super().get_config()
config.update({
"arg1": self.arg1,
"arg2": self.arg2,
})
return config
I tried to save the Model using model.save() syntax. I defined the path to be saved. But I think for using transfer learning and adding custom layers is preventing the model to be saved in this format. What to do?
Related
Using Django 3.2 -- I will simplify the problem as much as I can.
I have three model classes:
# abstract base class
MyAbstractModel(models.Model)
# derived model classes
Person(MyAbstractModel)
LogoImage(MyAbstractModel)
Each Person has:
image = ForeignKey(LogoImage, db_index=True, related_name="person", null=True,
on_delete=models.PROTECT)
The MyAbstractModel defines a few model managers:
objects = CustomModelManager()
objects_all_states = models.Manager()
as well as a state field, that can be either active or inactive
CustomModelManager is defined as something that'll bring only records that have state == 'active':
class CustomModelManager(models.Manager):
def get_queryset(self):
return super().get_query().filter(self.model, using=self._db).filter(state='active')
In my database I have two objects in two tables:
Person ID 1 state = 'active'
Image ID 1 state = 'inactive'
Person ID 1 has a foreign key connection to Image ID 1 via the Person.image field.
------ NOW for the issue ----------------
# CORRECT: gives me the person object
person = Person.objects.get(id=1)
# INCORRECT: I get the image, but it should not work...
image = person.image
Why is that incorrect? because I queried for the person object using the objects model manager which is supposed to bring only those items with active status. It brought the Person which is fine, because Person (ID=1) is state==active -- but the object under person.image is state==inactive. Why am I getting it?
WORKAROND ATTEMPT:
added base_manager_name = "objects" to the MyAbstractModel class Meta: section
ATTEMPTING AGAIN:
# CORRECT: gives me the person object
person = Person.objects.get(id=1)
# CORRECT: gives me a "Does not Exist" exception.
image = person.image
However..... Now I try this:
# CORRECT: getting the person
person.objects_all_states.get(id=1)
# INCORRECT: throws a DoesNotExist, as it's trying to use the `objects` model manager I hard coded in the `MyAbstractModel` class meta.
image = person.image
Since I got the Person under the objects_all_states which does not care about state==active -- I expect I would also get the person.image in a similar way. But that doesn't work as expected.
THE ROOT ISSUE
How do I force the same model manager used to fetch the parent object (Person) -- in the fetching of every single ForeignKey object a Person has? I can't find the answer. I've been going in circles for days. There is simply no clear answer anywhere. Either I am missing something very fundamental, or Django has a design flaw (which of course I don't really believe) -- so, what am I missing here?
Why they don't play well together
Foreign key classes use separate instances of managers, so there's no shared state.
There's no information about the manager used on the parent instance either.
As per django.db.models.Model._base_manager, Django simply uses _base_manager:
return self.field.remote_field.model._base_manager.db_manager(hints=hints).all()
...where hints would be {'instance': <Person: Person object (1)>}.
Since we have a reference to the parent, in some scenarios, we could support this inference.
Fair warning
Django specifically mentions not to do this.
From django.db.models.Model._base_manager:
Don’t filter away any results in this type of manager subclass
This manager is used to access objects that are related to from some other model. In those situations, Django has to be able to see all the objects for the model it is fetching, so that anything which is referred to can be retrieved.
Therefore, you should not override get_queryset() to filter out any rows. If you do so, Django will return incomplete results.
1. How you could implement this inference
You could:
override get() to actively store some information on the instance (that will be passed as hint) about whether an instance of CustomModelManager was used to get it, and then
in get_queryset, check that and try to fallback on objects_all_states.
class CustomModelManager(models.Manager):
def get(self, *args, **kwargs):
instance = super().get(*args, **kwargs)
instance.hint_manager = self
return instance
def get_queryset(self):
hint = self._hints.get('instance')
if hint and isinstance(hint.__class__.objects, self.__class__):
hint_manager = getattr(hint, 'hint_manager', None)
if not hint_manager or not isinstance(hint_manager, self.__class__):
manager = getattr(self.model, 'objects_all_states', None)
if manager:
return manager.db_manager(hints=self._hints).get_queryset()
return super().get_queryset().filter(state='active')
Limitations
One of possibly many edge cases where this wouldn't work is if you queried person via Person.objects.filter(id=1).first().
2. Using explicit instance context
Usage:
person = Person.objects_all_states.get(id=1)
# image = person.image
with CustomModelManager.disable_for_instance(person):
image = person.image
Implementation:
class CustomModelManager(models.Manager):
_disabled_for_instances = set()
#classmethod
#contextmanager
def disable_for_instance(cls, instance):
is_already_in = instance in cls._disabled_for_instances
if not is_already_in:
cls._disabled_for_instances.add(instance)
yield
if not is_already_in:
cls._disabled_for_instances.remove(instance)
def get_queryset(self):
if self._hints.get('instance') in self._disabled_for_instances:
return super().get_queryset()
return super().get_queryset().filter(state='active')
3. Using explicit thread-local context
Usage:
# person = Person.objects_all_states.get(id=1)
# image = person.image
with CustomModelManager.disable():
person = Person.objects.get(id=1)
image = person.image
Implementation:
import threading
from contextlib import contextmanager
from django.db import models
from django.utils.functional import classproperty
class CustomModelManager(models.Manager):
_data = threading.local()
#classmethod
#contextmanager
def disable(cls):
is_disabled = cls._is_disabled
cls._data.is_disabled = True
yield
cls._data.is_disabled = is_disabled
#classproperty
def _is_disabled(cls):
return getattr(cls._data, 'is_disabled', None)
def get_queryset(self):
if self._is_disabled:
return super().get_queryset()
return super().get_queryset().filter(state='active')
Well, i must point out a few design flaws in your approach.
First - you should not override get_queryset method for manager. Instead, make a separate method to filter specific cases. Even better if you make a custom QuerySet class with those methods, since then you will able to chain them
class ActiveQuerySet(QuerySet):
def active(self):
return self.filter(state="active")
# in your model
objects = ActiveQueryset.as_manager()
Also, you should not place field state in every model and expect, that Django will handle this for you. It will be much easier to handle for you if you decide from domain perspective, which model is your root model and have state there. For example, if Person can be inactive, then probably all of his images are also inactive, so you may safely assume, that Persons status is shared by all related models.
I would specifically look for a way to avoid such issue from design perspective, instead of trying to brutforce Django to process such filtration cases
I want to extract the format and mode parameter from the uploaded image within the Serializer and dynamically update my fields. Here is the code ...
class ImageDataSerializer(serializers.ModelSerializer):
class Meta:
model = models.ImageData
exclude = ['height','width']
And in my view
serializer = serializer(data=request.data,partial=True)
serializer.is_valid(raise_exception=True)
obj = serializer.save(user=request.user,extension="PNG",image_type="RGB")
return Response(serializer.data)
This works perfectly. I am sending my InMemeoryUploadedFile instance as my data and the serializer does its job saving it to the database. However, I would like to determine the extension and image_type automatically using the Pillow library.
This is what I have tried so far ...
class ImageDataSerializer(serializers.ModelSerializer):
def __init__(self,*args,**kwargs):
super(ImageDataSerializer,self).__init__(*args,**kwargs)
myimage = self.initial_data['data']
with Image.open(myimage) as myimage:
self.fields['extension'].initial = myimage.format
self.fields['image_type'].initial = myimage.mode
# Update the extension and image_type initial values
class Meta:
model = models.ImageData
exclude = ['height','width']
What happens is my image file get corrupted and in the response I am getting the message "Upload a valid image. The file you uploaded was either not an image or a corrupted image."
I have also tried determining the extension and mode within the view and pass it to request.data dictionary but accessing the image file once using Pillow.Image.open() is later corrupting it.
Issue was the cursor.
After opening the file using Pillow once you need to reset the cursor to the beginning of the file for future usage by Django. Below is fully working code.
class ImageDataSerializer(serializers.ModelSerializer):
def __init__(self,*args,**kwargs):
data = kwargs.get("data",None)
img = data.get("data",None)
with Image.open(img) as myimage:
data['extension'] = myimage.format
data['image_type'] = myimage.mode
img.seek(0)
super(ImageDataSerializer,self).__init__(*args,**kwargs)
class Meta:
model = models.ImageData
exclude = ['height','width']
read_only_fields = ['user']
img.seek(0) Resets the cursor to the beginning of the file
I am using django-rest-framework to make an api of my data. I am making an app that takes into account user data and remove outliers from that data using Pandas. I am able to present my data on frontend using django templates but somehow I am not able to make an API containing the statistical data using django-rest-framework . Can someone explain it and please help me to rectify my errors and also provide the necessary code
Here is my code
class Data(models.Model):
Name = models.CharField(max_length=30,null=True,blank=True)
Age = models.IntegerField(null=True,blank=True)
Weight = models.FloatField(null=True,blank=True)
Height = models.FloatField(null=True,blank=True)
Sugar = models.FloatField(null=True,blank=True)
def __unicode__(self):
return self.Name
My Serializer Class
class DataSerializer(serializers.ModelSerializer):
class Meta:
model = Data
fields = '__all__'
my views.py
def my_view(request):
con = sqlite3.connect("db.sqlite3")
df = pd.read_sql_query("SELECT * from visualapp_health", con)
a = df.fillna(0)
a['x-Mean'] = abs(a['Age'] - a['Age'].mean())
a['1.96*std'] = 1.96*a['Age'].std()
a['Outlier'] = abs(a['Age'] - a['Age'].mean()) > 1.96*a['Age'].std()
con.close()
return render(request, 'visual.html', {'visual': a})
I am able to get all the Data when using Django templates but somehow I don't able to understand how to make an API of all the data using django-rest- framework.
I finally got it I used django-pandas library and it worked and there is no need to connect to the database just convert your django queryset to pandas dataframe.
I am currently writing an API using an OSM DB, a regular DB, DRF3 and django-rest-framework-gis.
The OSM DB is using SRID 900913, but I would like the API to only output coordinates according to SRID 4326. This is achievable by using the transform(srid=4326) on my geoquerysets, but unfortunately that is not an option in my case (explained why further down).
Of what I understand, I should be able to specify another SRID in the field definition of the model like so:
class Polygon(models.Model):
osm_id = models.AutoField(primary_key=True)
name = models.CharField(null=True)
way = models.PolygonField(null=True, srid=4326)
admin_level = models.CharField(default='2', null=True)
objects = models.GeoManager()
class Meta:
db_table = 'osm_poly_lu'
But the API keeps returning polygons with coordinates in the 900913 format.
Do I understand it correctly; should I be able to set the srid on my polygon field to 4326 for Django to automatically convert it from my DBs 900913? If so, I guess I must be missing something; any ideas of what may be wrong? I feel like a true newbie in this area - any help is very appreciated.
The reason why I can not use transform() on my querysets:
What I am doing is querying my member table, and the member model is a regular model. It does however have a relation to the OSM Polygon table (field is called osm_id_origin) and this is a geoDjango model. So when I serialize the member with a DRF3 Serializer, I have a nested Serializer in the member serializer that serializes the osm_id_origin field in the member model as a geoJSON. The nested serializer I am using for creating the geoJSON is a subclass of the django-rest-framework-gis serializer GeoFeatureModelSerializer. Since I query the member model, which is a regular queryset, I can not run the transform(srid=4326) on it.
This is a late answer, but I encounter the same issue and I created a dedicated serializer field to handle transformation. As you will see, this code stores the geometry in srid 31370, and transform the geometry to srid 3857 (which is quite the same as 900913) when serialized:
from rest_framework_gis.fields import GeometryField
class ProjectedGeometrySerializerField(GeometryField):
"""
Transform a geometry from SRID 3857 (representation data)
to 31370 (internal value)
"""
def to_representation(self, value):
if isinstance(value, dict) or value is None:
return value
cloned = value.clone()
cloned.transform(3857)
return super().to_representation(cloned)
def to_internal_value(self, value):
geom = super().to_internal_value(value)
geom.srid = 3857
geom.transform(31370)
return geom
I'm trying to do something that should be very common: add/edit a bunch of related models in a single form. For example:
Visitor Details:
Select destinations and activities:
Miami [] - swimming [], clubbing [], sunbathing[]
Cancun [] - swimming [], clubbing [], sunbathing[]
My models are Visitor, Destination and Activity, with Visitor having a ManyToMany field into Destination through an intermediary model, VisitorDestination, which has the details of the activities to be done on the destination (in itself a ManyToMany field into Activity).
Visitor ---->(M2M though VisitorDestination) -------------> Destination
|
activities ---->(M2M)---> Activity
Note that I don't want to enter new destination / activity values, just choose from those available in the db (but that's a perfectly legit use of M2M fields right?)
To me this looks like an extremely common situation (a many to many relation with additional details which are a FK or M2M field into some other model), and this looks like the most sensible modelling, but please correct me if I'm wrong.
I've spent a few days searching Django docs / SO / googling but haven't been able to work out how to deal with this. I tried several approaches:
Custom Model form for Visitor, where I add multiple choice fields for Destination and Activity. That works ok if Destination and Activity could be selected independently, but here they are correlated, ie I want to choose one or several activities for each destination
Using inlineformset_factory to generate the set of destination / activities forms, with inlineformset_factory(Destination, Visitor). This breaks, because Visitor has a M2M relation to Destination, rather than a FK.
Customizing a plain formset, using formset_factory, eg DestinationActivityFormSet = formset_factory(DestinationActivityForm, extra=2). But how to design DestinationActivityForm? I haven't explored this enough, but it doesn't look very promising: I don't want to type in the destination and a list of activities, I want a list of checkboxes with the labels set to the destination / activities I want to select, but the formset_factory would return a list of forms with identical labels.
I'm a complete newbie with django so maybe the solution is obvious, but I find that the documentation in this area is very weak - if anyone has some pointers to examples of use for forms / formsets that would be also helpful
thanks!
In the end I opted for processing multiple forms within the same view, a Visitor model form for the visitor details, then a list of custom forms for each of the destinations.
Processing multiple forms in the same view turned out to be simple enough (at least in this case, where there were no cross-field validation issues).
I'm still surprised there is no built-in support for many to many relationships with an intermediary model, and looking around in the web I found no direct reference to it. I'll post the code in case it helps anyone.
First the custom forms:
class VisitorForm(ModelForm):
class Meta:
model = Visitor
exclude = ['destinations']
class VisitorDestinationForm(Form):
visited = forms.BooleanField(required=False)
activities = forms.MultipleChoiceField(choices = [(obj.pk, obj.name) for obj in Activity.objects.all()], required=False,
widget = CheckboxSelectMultipleInline(attrs={'style' : 'display:inline'}))
def __init__(self, visitor, destination, visited, *args, **kwargs):
super(VisitorDestinationForm, self).__init__(*args, **kwargs)
self.destination = destination
self.fields['visited'].initial = visited
self.fields['visited'].label= destination.destination
# load initial choices for activities
activities_initial = []
try:
visitorDestination_entry = VisitorDestination.objects.get(visitor=visitor, destination=destination)
activities = visitorDestination_entry.activities.all()
for activity in Activity.objects.all():
if activity in activities:
activities_initial.append(activity.pk)
except VisitorDestination.DoesNotExist:
pass
self.fields['activities'].initial = activities_initial
I customize each form by passing a Visitor and Destination objects (and a 'visited' flag which is calculated outside for convenience)
I use a boolean field to allow the user to select each destination. The field is called 'visited', however I set the label to the destination so it gets nicely displayed.
The activities get handled by the usual MultipleChoiceField (I used I customized widget to get the checkboxes to display on a table, pretty simple but can post it if somebody needs that)
Then the view code:
def edit_visitor(request, pk):
visitor_obj = Visitor.objects.get(pk=pk)
visitorDestinations = visitor_obj.destinations.all()
if request.method == 'POST':
visitorForm = VisitorForm(request.POST, instance=visitor_obj)
# set up the visitor destination forms
destinationForms = []
for destination in Destination.objects.all():
visited = destination in visitorDestinations
destinationForms.append(VisitorDestinationForm(visitor_obj, destination, visited, request.POST, prefix=destination.destination))
if visitorForm.is_valid() and all([form.is_valid() for form in destinationForms]):
visitor_obj = visitorForm.save()
# clear any existing entries,
visitor_obj.destinations.clear()
for form in destinationForms:
if form.cleaned_data['visited']:
visitorDestination_entry = VisitorDestination(visitor = visitor_obj, destination=form.destination)
visitorDestination_entry.save()
for activity_pk in form.cleaned_data['activities']:
activity = Activity.objects.get(pk=activity_pk)
visitorDestination_entry.activities.add(activity)
print 'activities: %s' % visitorDestination_entry.activities.all()
visitorDestination_entry.save()
success_url = reverse('visitor_detail', kwargs={'pk' : visitor_obj.pk})
return HttpResponseRedirect(success_url)
else:
visitorForm = VisitorForm(instance=visitor_obj)
# set up the visitor destination forms
destinationForms = []
for destination in Destination.objects.all():
visited = destination in visitorDestinations
destinationForms.append(VisitorDestinationForm(visitor_obj, destination, visited, prefix=destination.destination))
return render_to_response('testapp/edit_visitor.html', {'form': visitorForm, 'destinationForms' : destinationForms, 'visitor' : visitor_obj}, context_instance= RequestContext(request))
I simply collect my destination forms in a list and pass this list to my template, so that it can iterate over them and display them. It works well as long as you don't forget to pass a different prefix for each one in the constructor
I'll leave the question open for a few days in case some one has a cleaner method.
Thanks!
So, as you've seen, one of the things about inlineformset_factory is that it expects two models - a parent, and child, which has a foreign key relationship to the parent. How do you pass extra data on the fly to the form, for extra data in the intermediary model?
How I do this is by using curry:
from django.utils.functional import curry
from my_app.models import ParentModel, ChildModel, SomeOtherModel
def some_view(request, child_id, extra_object_id):
instance = ChildModel.objects.get(pk=child_id)
some_extra_model = SomeOtherModel.objects.get(pk=extra_object_id)
MyFormset = inlineformset_factory(ParentModel, ChildModel, form=ChildModelForm)
#This is where the object "some_extra_model" gets passed to each form via the
#static method
MyFormset.form = staticmethod(curry(ChildModelForm,
some_extra_model=some_extra_model))
formset = MyFormset(request.POST or None, request.FILES or None,
queryset=SomeObject.objects.filter(something=something), instance=instance)
The form class "ChildModelForm" would need to have an init override that adds the "some_extra_model" object from the arguments:
def ChildModelForm(forms.ModelForm):
class Meta:
model = ChildModel
def __init__(self, some_extra_model, *args, **kwargs):
super(ChildModelForm, self).__init__(*args, **kwargs)
#do something with "some_extra_model" here
Hope that helps get you on the right track.
From django 1.9, there is a support for passing custom parameters to formset forms :
https://docs.djangoproject.com/en/1.9/topics/forms/formsets/#passing-custom-parameters-to-formset-forms
Just add form_kwargs to your FormSet init like this :
from my_app.models import ParentModel, ChildModel, SomeOtherModel
def some_view(request, child_id, extra_object_id):
instance = ChildModel.objects.get(pk=child_id)
some_extra_model = SomeOtherModel.objects.get(pk=extra_object_id)
MyFormset = inlineformset_factory(ParentModel, ChildModel, form=ChildModelForm)
formset = MyFormset(request.POST or None, request.FILES or None,
queryset=SomeObject.objects.filter(something=something), instance=instance,
form_kwargs={"some_extra_model": some_extra_model})