In the alpha release of CloudML's online prediction service, the format for exporting model was:
inputs = {"x": x, "y_bytes": y}
g.add_to_collection("inputs", json.dumps(inputs))
outputs = {"a": a, "b_bytes": b}
g.add_to_collection("outputs", json.dumps(outputs))
I would like to convert this to a SavedModel without retraining my model. How can I do that?
We can convert this to a SavedModel by importing the old model, creating the Signatures, and re-exporting it. This code is untested, but something like this should work:
import json
import tensorflow as tf
from tensorflow.contrib.session_bundle import session_bundle
# Import the "old" model
session, _ = session_bundle.load_session_bundle_from_path(export_dir)
# Define the inputs and the outputs for the SavedModel
old_inputs = json.loads(tf.get_collection('inputs'))
inputs = {name: tf.saved_model.utils.build_tensor_info(tensor)
for name, tensor in old_inputs}
old_outputs = json.loads(tf.get_collection('outputs'))
outputs = {name: tf.saved_model.utils.build_tensor_info(tensor)
for name, tensor in old_outputs}
signature = tf.saved_model.signature_def_utils.build_signature_def(
inputs=inputs,
outputs=outputs,
method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME
)
# Save out the converted model
b = builder.SavedModelBuilder(new_export_dir)
b.add_meta_graph_and_variables(session,
[tf.saved_model.tag_constants.SERVING],
signature_def_map={'serving_default': signature})
b.save()
Related
For creating a custom prediction routine with a Keras (Tensorflow 2.1) model, I am having trouble figuring out what form the json inputs are coming in as, and how to read them in the predictor class for multiple inputs. All of the custom prediction routine examples in the documentation use simple flat single-input lists. If for example we send in our inputs as:
{"instances": [
{
"event_type_input": [1, 2, 20],
"event_dwelltime_input": [1.368, 0.017, 0.0],
"rf_input": [1.2, -2.8]},
{
"event_type_input": [14, 40, 20],
"event_dwelltime_input": [1.758, 13.392, 0.0],
"rf_input": [1.29, -2.87]}
]}
How should we ingest the incoming json in our predictor class?
class MyPredictor(object):
def __init__(self, model):
self.model = model
def predict(self, instances, **kwargs):
inputs = np.array(instances)
# The above example from the docs is wrong for multiple inputs
# What should our inputs be to get the inputs in the right shape
# for our keras model?
outputs = self.model.predict(inputs)
return outputs.tolist()
Our json inputs to google ai platform are a list of dictionaries. However, for a keras model, our inputs need to be in different shape, like the following:
inputs = {
"event_type_input": np.array([[1, 2, 20], [14, 40, 20]]),
"event_dwelltime_input": np.array([[1.368, 0.017, 0.0], [1.758, 13.392, 0.0]])
"rf_input": np.array([[1.2, -2.8], [1.29, -2.87]]}
model.predict(inputs)
Am I right that the thing to do then is just reshape the instances? The only confusion is that if using the tensorflow framework (instead of a custom prediction routine), it handles predicting on the json input fine, and I thought that all the tensorflow framework is doing is calling the .predict method on the instances (unless indeed there is some under-the-hood reshaping of the data. I couldn't find a source to find out what is exactly happening)
Main question: How should we write our predictor class to take in the instances such that we can run the model.predict method on it?
I would suggest creating a new Keras Model and exporting it.
Create a separate Input layer for each of the three inputs to the new Model (with the name of the Input being the name in your JSON struct). Then, in this Model, reshape the inputs, and borrow the weights/structure from your trained model, and export the new model. Something like this:
trained_model = keras.models.load_model(...) # trained model
input1 = keras.Input(..., name='event_type_input')
input2 = keras.Input(..., name='event_dwelltime_input')
input3 = keras.Input(..., name='rf_input')
export_inputs = keras.concatenate([input1, input2, input3])
reshaped_inputs = keras.layers.Lambda(...) # reshape to what the first hidden layer of your trained model expects
layer1 = trained_model.get_layer(index=1)(reshaped_inputs)
layer2 = trained_model.get_layer(index=2)(layer1) # etc. ...
...
exportModel = keras.Model(export_inputs, export_output)
exportModel.save(...)
I have deployed a the universal_sentence_encoder_large_3 to an aws sagemaker. When I am attempting to predict with the deployed model I get Failed precondition: Table not initialized. as an error. I have included the part where I save my model below:
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
def tfhub_to_savedmodel(model_name, export_path):
model_path = '{}/{}/00000001'.format(export_path, model_name)
tfhub_uri = 'http://tfhub.dev/google/universal-sentence-encoder-large/3'
with tf.Session() as sess:
module = hub.Module(tfhub_uri)
sess.run([tf.global_variables_initializer(), tf.tables_initializer()])
input_params = module.get_input_info_dict()
dtype = input_params['text'].dtype
shape = input_params['text'].get_shape()
# define the model inputs
inputs = {'text': tf.placeholder(dtype, shape, 'text')}
output = module(inputs['text'])
outputs = {
'vector': output,
}
# export the model
tf.saved_model.simple_save(
sess,
model_path,
inputs=inputs,
outputs=outputs)
return model_path
I have seen other people ask this problem but no solution has been ever posted. It seems to be a common problem with tensorflow_hub sentence encoders
I was running into this exact issue earlier this week while trying to modify this example Sagemaker notebook. Particularly the part where serving the model. That is, running predictor.predict() on the Sagemaker Tensorflow Estimator.
The solution outlined in the issue worked perfectly for me- https://github.com/awslabs/amazon-sagemaker-examples/issues/773#issuecomment-509433290
I think it's just because tf.tables_initializer() only runs for training but it needs to be specified through the legacy_init_op if you want to run it during prediction.
I get the following error when trying to make an online prediction on my ML Engine model.
The key "values" is not correct. (See error on image.)
enter image description here
I already tested with RAW image data : {"image_bytes":{"b64": base64.b64encode(jpeg_data)}}
& Converted the data to a numpy array.
Currently I have the following code:
from googleapiclient import discovery
import base64
import os
from PIL import Image
import json
import numpy as np
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/Users/jacob/Desktop/******"
def predict_json(project, model, instances, version=None):
"""Send json data to a deployed model for prediction.
Args:
project (str): project where the Cloud ML Engine Model is deployed.
model (str): model name.
instances ([Mapping[str: Any]]): Keys should be the names of Tensors
your deployed model expects as inputs. Values should be datatypes
convertible to Tensors, or (potentially nested) lists of datatypes
convertible to tensors.
version: str, version of the model to target.
Returns:
Mapping[str: any]: dictionary of prediction results defined by the
model.
"""
# Create the ML Engine service object.
# To authenticate set the environment variable
# GOOGLE_APPLICATION_CREDENTIALS=<path_to_service_account_file>
service = discovery.build('ml', 'v1')
name = 'projects/{}/models/{}'.format(project, model)
if version is not None:
name += '/versions/{}'.format(version)
response = service.projects().predict(
name=name,
body={'instances': instances}
).execute()
if 'error' in response:
raise RuntimeError(response['error'])
return response['predictions']
savepath = 'upload/11277229_F.jpg'
img = Image.open('test/01011000/11277229_F.jpg')
test = img.resize((299, 299))
test.save(savepath)
img1 = open(savepath, "rb").read()
def load_image(filename):
with open(filename) as f:
return np.array(f.read())
predict_json('image-recognition-25***08', 'm500_200_waug', [{"values": str(base64.b64encode(img1).decode("utf-8")), "key": '87'}], 'v1')
The error message itself indicates (as you point out in the question), that the key "values" is not one of the inputs specified in the model. To inspect the model's input, use saved_model_cli show --all --dir=/path/to/model. That will show you a list of the names of the inputs. You'll need to use the correct name.
That said, it appears there is another issue. It's not clear from the question what type of input your model is expecting, though it's likely one of two things:
A matrix of integers or floats
A byte string with the raw image file
contents.
The exact solution will depend on which of the above your exported model is using. saved_model_cli will help here, based on the type and shape of the input. It will either be DT_FLOAT32 (or some other int/float type) and [NONE, 299, 299, CHANNELS] or DT_STRING and [NONE], respectively.
If your model is type (1), then you will need to send a matrix of ints/floats (which does not use base64 encoding):
predict_json('image-recognition-25***08', 'm500_200_waug', [{CORRECT_INPUT_NAME: load_image(savepath).tolist(), "key": '87'}], 'v1')
Note the use of tolist to convert the numpy array to a list of lists.
In the case of type (2), you need to tell the service you have some base64 data by adding in {"b64": ...}:
predict_json('image-recognition-25***08', 'm500_200_waug', [{CORRECT_INPUT_NAME: {"b64": str(base64.b64encode(img1).decode("utf-8"))}, "key": '87'}], 'v1')
All of this, of course, depends on using the correct name for CORRECT_INPUT_NAME.
One final note, I'm assuming your model actually does have key as an additional inputs since you included it in your request; again, that can all be verified against the output of saved_model_cli show.
I used to get this errors too. If anyone comes across this error, and using gcloud.
Tensors are automatically called csv_rows. For example this works for me now
"instances": [{
"csv_row": "STRING,7,4.02611534,9,14,0.66700000,0.17600000,0.00000000,0.00000000,1299.76500000,57",
"key": "0"
}]
I have trained weight matrix, I would like to extract features at each end every layer and store them in a file. How could I do that? Thanks.
Have a look at the Keras FAQ
One simple way is to create a new Model that will output the layers
that you are interested in:
from keras.models import Model
model = ... # create the original model
layer_name = 'my_layer'
intermediate_layer_model = Model(inputs=model.input,
outputs=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model.predict(data)
Alternatively, you can build a Keras function that will return the
output of a certain layer given a certain input, for example:
from keras import backend as K
get_3rd_layer_output = K.function([model.layers[0].input],
[model.layers[3].output])
layer_output = get_3rd_layer_output([X])[0]
Similarly, you could build a Theano and TensorFlow function directly.
Note that if your model has a different behavior in training and
testing phase (e.g. if it uses Dropout, BatchNormalization, etc.),
you will need to pass the learning phase flag to your function:
get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
[model.layers[3].output])
# output in test mode = 0
layer_output = get_3rd_layer_output([X, 0])[0]
# output in train mode = 1
layer_output = get_3rd_layer_output([X, 1])[0]
Then you just need to store your predictions in a file using e.g. np.save('filename.npz',intermediate_output )
I am building an API using flask-restful. I am also using flask-resfulplus for generating swagger documentation. I want to return a dictionary of items where the key is going to vary depending on the item. My model looks like this:
item = api.model('Item',{
'item':fields.Integer()}) <- This is wrong
ItemModel = api.model('ItemsList', {
'_header': fields.Nested(_header),
'items':fields.Nested(item)
})
Note that I tried a few variations of this, but nothing seems to work; this is just the latest interation.
And the response I am looking for is something like this.
{
'_header':{} <-This works fine
'items': {
'item1':5,
'item2':2
}
}
Item in the items dictionary will have a different key for the item and the count as the value.
I've tried setting the item field to field.Raw() and it works fine, but then it doesn't get displayed in the swagger documentation.
Note that I do not want to return a list of dictionaries, which I have working.
Thanks
Sometimes you have your own custom formatting needs. You can subclass the fields.Raw class and implement the format function. This is especially useful when an attribute stores multiple pieces of information. e.g. a bit-field whose individual bits represent distinct values. You can use fields to multiplex a single attribute to multiple output values.
If you don’t know the name(s) of the field(s) you want to unmarshall, you can use Wildcard.
>>> from flask_restplus import fields, marshal
>>> import json
>>>
>>> wild = fields.Wildcard(fields.String)
>>> wildcard_fields = {'*': wild}
>>> data = {'John': 12, 'bob': 42, 'Jane': '68'}
>>> json.dumps(marshal(data, wildcard_fields))
>>> '{"Jane": "68", "bob": "42", "John": "12"}'
The name you give to your Wildcard acts as a real glob as shown below.
>>> from flask_restplus import fields, marshal
>>> import json
>>>
>>> wild = fields.Wildcard(fields.String)
>>> wildcard_fields = {'j*': wild}
>>> data = {'John': 12, 'bob': 42, 'Jane': '68'}
>>> json.dumps(marshal(data, wildcard_fields))
>>> '{"Jane": "68", "John": "12"}'
Note
It is important you define your Wildcard outside your model (ie. you cannot use it like this: res_fields = {'': fields.Wildcard(fields.String)})* because it has to be stateful to keep a track of what fields it has already treated.