I have trained weight matrix, I would like to extract features at each end every layer and store them in a file. How could I do that? Thanks.
Have a look at the Keras FAQ
One simple way is to create a new Model that will output the layers
that you are interested in:
from keras.models import Model
model = ... # create the original model
layer_name = 'my_layer'
intermediate_layer_model = Model(inputs=model.input,
outputs=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model.predict(data)
Alternatively, you can build a Keras function that will return the
output of a certain layer given a certain input, for example:
from keras import backend as K
get_3rd_layer_output = K.function([model.layers[0].input],
[model.layers[3].output])
layer_output = get_3rd_layer_output([X])[0]
Similarly, you could build a Theano and TensorFlow function directly.
Note that if your model has a different behavior in training and
testing phase (e.g. if it uses Dropout, BatchNormalization, etc.),
you will need to pass the learning phase flag to your function:
get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
[model.layers[3].output])
# output in test mode = 0
layer_output = get_3rd_layer_output([X, 0])[0]
# output in train mode = 1
layer_output = get_3rd_layer_output([X, 1])[0]
Then you just need to store your predictions in a file using e.g. np.save('filename.npz',intermediate_output )
Related
I have just started with pyomo and Python, and trying to create a simple model but have a problem with adding a constraint.
I followed the following example from GitHub
https://github.com/brentertainer/pyomo-tutorials/blob/master/introduction/02-lp-pyomo.ipynb
import pandas as pd
import pyomo.environ as pe
import pyomo.opt as po
#DATA
T=3;
CH=2;
time = ['t{0}'.format(t+1) for t in range(T)]
CHP=['CHP{0}'.format(s+1) for s in range(CH)]
#Technical characteristic
heat_maxprod = {'CHP1': 250,'CHP2': 250} #Only for CHPS
#MODEL
seq=pe.ConcreteModel
### SETS
seq.CHP = pe.Set(initialize = CHP)
seq.T = pe.Set(initialize = time)
### PARAMETERS
seq.heat_maxprod = pe.Param(seq.CHP, initialize = heat_maxprod) #Max heat production
### VARIABLES
seq.q_DA=pe.Var(seq.CHP, seq.T, domain=pe.Reals)
### CONSTRAINTS
##Maximum and Minimum Heat Production
seq.Heat_DA1 = pe.ConstraintList()
for t in seq.T:
for s in seq.CHP:
seq.Heat_DA1.add( 0 <= seq.q_DA[s,t])
seq.Heat_DA2 = pe.ConstraintList()
for t in seq.T:
for s in seq.CHP:
seq.Heat_DA2.add( seq.q_DA[s,t] <= seq.heat_maxprod[s])
### OBJECTIVE
seq.obj=Objective(expr=sum( seq.C_fuel[s]*(seq.rho_heat[s]*seq.q_DA[s,t]) for t in seq.T for s in seq.CHP))
When I run the program I am getting the following error:
RuntimeError: Cannot iterate over AbstractOrderedScalarSet 'AbstractOrderedScalarSet' before it has been constructed (initialized): 'iter' is an attribute on an Abstract component and cannot be accessed until the component has been fully constructed (converted to a Concrete component) using AbstractModel.create_instance() or AbstractOrderedScalarSet.construct().
Can someone, please, help with an issue? Thanks!
P.S. I know that the resulting answer for the problem is zero, I just want to make it work in terms of correct syntaxis.
In this line of code:
seq=pe.ConcreteModel
You are missing parenthesis. So, I think you are just creating an alias for the function instead of calling it.
Try:
seq=pe.ConcreteModel()
For creating a custom prediction routine with a Keras (Tensorflow 2.1) model, I am having trouble figuring out what form the json inputs are coming in as, and how to read them in the predictor class for multiple inputs. All of the custom prediction routine examples in the documentation use simple flat single-input lists. If for example we send in our inputs as:
{"instances": [
{
"event_type_input": [1, 2, 20],
"event_dwelltime_input": [1.368, 0.017, 0.0],
"rf_input": [1.2, -2.8]},
{
"event_type_input": [14, 40, 20],
"event_dwelltime_input": [1.758, 13.392, 0.0],
"rf_input": [1.29, -2.87]}
]}
How should we ingest the incoming json in our predictor class?
class MyPredictor(object):
def __init__(self, model):
self.model = model
def predict(self, instances, **kwargs):
inputs = np.array(instances)
# The above example from the docs is wrong for multiple inputs
# What should our inputs be to get the inputs in the right shape
# for our keras model?
outputs = self.model.predict(inputs)
return outputs.tolist()
Our json inputs to google ai platform are a list of dictionaries. However, for a keras model, our inputs need to be in different shape, like the following:
inputs = {
"event_type_input": np.array([[1, 2, 20], [14, 40, 20]]),
"event_dwelltime_input": np.array([[1.368, 0.017, 0.0], [1.758, 13.392, 0.0]])
"rf_input": np.array([[1.2, -2.8], [1.29, -2.87]]}
model.predict(inputs)
Am I right that the thing to do then is just reshape the instances? The only confusion is that if using the tensorflow framework (instead of a custom prediction routine), it handles predicting on the json input fine, and I thought that all the tensorflow framework is doing is calling the .predict method on the instances (unless indeed there is some under-the-hood reshaping of the data. I couldn't find a source to find out what is exactly happening)
Main question: How should we write our predictor class to take in the instances such that we can run the model.predict method on it?
I would suggest creating a new Keras Model and exporting it.
Create a separate Input layer for each of the three inputs to the new Model (with the name of the Input being the name in your JSON struct). Then, in this Model, reshape the inputs, and borrow the weights/structure from your trained model, and export the new model. Something like this:
trained_model = keras.models.load_model(...) # trained model
input1 = keras.Input(..., name='event_type_input')
input2 = keras.Input(..., name='event_dwelltime_input')
input3 = keras.Input(..., name='rf_input')
export_inputs = keras.concatenate([input1, input2, input3])
reshaped_inputs = keras.layers.Lambda(...) # reshape to what the first hidden layer of your trained model expects
layer1 = trained_model.get_layer(index=1)(reshaped_inputs)
layer2 = trained_model.get_layer(index=2)(layer1) # etc. ...
...
exportModel = keras.Model(export_inputs, export_output)
exportModel.save(...)
I loaded resnet18 into my two models (model1 and model2), with pretrained weights.
I want to use them as feature extractors
For model1: I freezed the parameters except the last linear layer model1.fc, then train it. After training, I set model1.fc into torch.nn.Identity()
For model2: I directly set model2.fc into torch.nn.Identity()
Then these 2 models should be the same, but I get different forward result from the same inputs.
If the training of model1 is not done, they will have the same result, maybe something wrong with the freezing of parameter.
However, I checked their parameters after the training of model1 and setting the last layer of both models to identity layer, and they seems to be the same.
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms, models
# Load weights pretrained on ImageNet
def load_weights(model):
model_dir = "....."
model.load_state_dict(tor`enter code here`ch.utils.model_zoo.load_url("https://download.pytorch.org/models/resnet18-5c106cde.pth", model_dir=model_dir))
return model
model1=models.resnet18()
model1=load_weights(model1)
for param in model1.parameters():
param.requires_grad = False
model1.fc=nn.Linear(512, 2)
model1.cuda()
optimizer = optim.SGD(model1.fc.parameters(), lr=1e-2, momentum=0.9)
result_freeze = \
run_training(model1, optimizer, device, train_loader, val_loader,num_epochs=10)
model2=models.resnet18()
model2=load_weights(model2)
model2.fc=nn.Identity()
model2.cuda()
model1.fc=nn.Identity()
model1.cuda()
# checking forward results(extracting features)
# The batch size is one here
for batch_idx, (data, target) in enumerate(X_train):
data, target = data.to(device), target.to(device)
d=data
X_train_feature[batch_idx]=model1(data).cpu().detach().numpy()
y_train[batch_idx]=target.cpu().detach().numpy()
X_train2_feature[batch_idx]=model2(d).cpu().detach().numpy()
y_train2[batch_idx]=target.cpu().detach().numpy()
print(sum(X_train_feature[batch_idx]==X_train2_feature[batch_idx]))
print(sum(y_train[batch_idx]==y_train2[batch_idx]))
print(torch.sum(d==data))
for batch_idx, (data, target) in enumerate(X_test):
data, target = data.to(device), target.to(device)
d=data
X_test_feature[batch_idx]=model1(data).cpu().detach().numpy()
y_test[batch_idx]=target.cpu().detach().numpy()
X_test2_feature[batch_idx]=model2(d).cpu().detach().numpy()
y_test2[batch_idx]=target.cpu().detach().numpy()
print(sum(X_test_feature[batch_idx]==X_test2_feature[batch_idx]))
print(sum(y_test[batch_idx]==y_test2[batch_idx]))
print(torch.sum(d==data))
# checking parameters
for a,b in zip(model1.parameters(),model2.parameters()):
print(torch.sum(a!=b))
Expect to get the same forward results from model1 and model2, but they are different. And If they produce different forward results, why do they have exactly the same parameters?
Have you taken into account changes that might occur to BatchNorm layers?
Batch norm layers do not behave like normal layers - their internal parameters are modified by computing running mean and std of the data, and not by gradient descent.
Try setting model1.eval() before the finetune and then check.
I have deployed a the universal_sentence_encoder_large_3 to an aws sagemaker. When I am attempting to predict with the deployed model I get Failed precondition: Table not initialized. as an error. I have included the part where I save my model below:
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
def tfhub_to_savedmodel(model_name, export_path):
model_path = '{}/{}/00000001'.format(export_path, model_name)
tfhub_uri = 'http://tfhub.dev/google/universal-sentence-encoder-large/3'
with tf.Session() as sess:
module = hub.Module(tfhub_uri)
sess.run([tf.global_variables_initializer(), tf.tables_initializer()])
input_params = module.get_input_info_dict()
dtype = input_params['text'].dtype
shape = input_params['text'].get_shape()
# define the model inputs
inputs = {'text': tf.placeholder(dtype, shape, 'text')}
output = module(inputs['text'])
outputs = {
'vector': output,
}
# export the model
tf.saved_model.simple_save(
sess,
model_path,
inputs=inputs,
outputs=outputs)
return model_path
I have seen other people ask this problem but no solution has been ever posted. It seems to be a common problem with tensorflow_hub sentence encoders
I was running into this exact issue earlier this week while trying to modify this example Sagemaker notebook. Particularly the part where serving the model. That is, running predictor.predict() on the Sagemaker Tensorflow Estimator.
The solution outlined in the issue worked perfectly for me- https://github.com/awslabs/amazon-sagemaker-examples/issues/773#issuecomment-509433290
I think it's just because tf.tables_initializer() only runs for training but it needs to be specified through the legacy_init_op if you want to run it during prediction.
I want to implement NiN using keras but I could not found useful in net. I want to implement below image architecture. anybody can help??
Just look at the functional API of Keras (https://keras.io/models/model/) and do something like this:
def build_model(input_layer, idx):
# model code (logits = first_layer(parameters)(input_layer)
# could also load an already trained model.
return logits
input_layer = Input(...)
output = input_layer
for i in range(num_models):
output = build_model(output , i)
final_layer = Model(input_layer, output)