I'm training my model using TensorFlow in C++. Python is used only for constructing the graph. So is there a way to save and restore the graph and its state purely in C++? I know about the Python class tf.train.Saver but as far as I understand it does not exist in C++.
The tf.train.Saver class currently exists only in Python, but (i) it is built from TensorFlow ops that you can run from C++, and (ii) it exposes the Saver.as_saver_def() method that lets you get a SaverDef protocol buffer with the names of ops that you must run to save or restore a model.
In Python, you can get the names of the save and restore ops as follows:
saver = tf.train.Saver(...)
saver_def = saver.as_saver_def()
# The name of the tensor you must feed with a filename when saving/restoring.
print saver_def.filename_tensor_name
# The name of the target operation you must run when restoring.
print saver_def.restore_op_name
# The name of the target operation you must run when saving.
print saver_def.save_tensor_name
In C++ to restore from a checkpoint, you call Session::Run(), feeding in the name of the checkpoint file as saver_def.filename_tensor_name, with a target op of saver_def.restore_op_name. To save another checkpoint, you call Session::Run(), again feeding in the name of the checkpoint file as saver_def.filename_tensor_name, and fetching the value of saver_def.save_tensor_name.
The recent TensorFlow version includes some helper functions to do the same in C++ without Python. These are generate from the ProtoBuf in the pip-package (${HOME}/.local/lib/python2.7/site-packages/tensorflow/include/tensorflow/core/protobuf/saver.pb.h).
// save
tensorflow::Tensor checkpointPathTensor(tensorflow::DT_STRING, tensorflow::TensorShape());
checkpointPathTensor.scalar<std::string>()() = "some/path";
tensor_dict feed_dict = {{graph_def.saver_def().filename_tensor_name(), checkpointPathTensor}};
status = sess->Run(feed_dict, {}, {graph_def.saver_def().save_tensor_name()}, nullptr);
// restore
tensorflow::Tensor checkpointPathTensor(tensorflow::DT_STRING, tensorflow::TensorShape());
checkpointPathTensor.scalar<std::string>()() = "some/path";
tensor_dict feed_dict = {{graph_def.saver_def().filename_tensor_name(), checkpointPathTensor}};
status = sess->Run(feed_dict, {}, {graph_def.saver_def().restore_op_name()}, nullptr);
This is based on the undocumented python-way (more details) of restoring a model
def restore(sess, metaGraph, fn):
restore_op_name = metaGraph.as_saver_def().restore_op_name # u'save/restore_all'
restore_op = tf.get_default_graph().get_operation_by_name(restore_op_name)
filename_tensor_name = metaGraph.as_saver_def().filename_tensor_name # u'save/Const'
sess.run(restore_op, {filename_tensor_name: fn})
For a working and complete version see here.
Related
Hi I'm new to Tensorflow and I've been practicing with the tensorflow.estimator library. Basically I ran the inbuilt tf.estimator.DNNClassifier algorithm below
import tensorflow as tf
def train_input_fn(features, labels, batch_size):
"""An input function for training"""
# Convert the inputs to a Dataset.
dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
# Shuffle, repeat, and batch the examples.
return dataset.shuffle(1000).repeat().batch(batch_size)
# Feature columns describe how to use the input.
my_feature_columns = []
for key in landmark_features.keys():
my_feature_columns.append(tf.feature_column.numeric_column(key=key))
# Build a DNN with 2 hidden layers and 10 nodes in each hidden layer.
classifier = tf.estimator.DNNClassifier(feature_columns=my_feature_columns, hidden_units=[10, 10],n_classes=10)
dataset = train_input_fn(landmark_features, emotion_labels, batch_size = 1375 )
However I keep getting the following error:
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpc_tag0rc
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpc_tag0rc', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
Any idea on what I can do to fix my code ?
The following code is from the tensorflow serving API:
// Implementation of Predict using the SavedModel SignatureDef format.
Status SavedModelPredict(const RunOptions& run_options, ServerCore* core,
const PredictRequest& request,
PredictResponse* response) {
// Validate signatures.
ServableHandle<SavedModelBundle> bundle;
TF_RETURN_IF_ERROR(core->GetServableHandle(request.model_spec(), &bundle));
const string signature_name = request.model_spec().signature_name().empty()
? kDefaultServingSignatureDefKey
: request.model_spec().signature_name();
auto iter = bundle->meta_graph_def.signature_def().find(signature_name);
if (iter == bundle->meta_graph_def.signature_def().end()) {
return errors::FailedPrecondition(strings::StrCat(
"Serving signature key \"", signature_name, "\" not found."));
}
SignatureDef signature = iter->second;
MakeModelSpec(request.model_spec().name(), signature_name,
bundle.id().version, response->mutable_model_spec());
std::vector<std::pair<string, Tensor>> input_tensors;
std::vector<string> output_tensor_names;
std::vector<string> output_tensor_aliases;
TF_RETURN_IF_ERROR(PreProcessPrediction(signature, request, &input_tensors,
&output_tensor_names,
&output_tensor_aliases));
std::vector<Tensor> outputs;
RunMetadata run_metadata;
TF_RETURN_IF_ERROR(bundle->session->Run(run_options, input_tensors,
output_tensor_names, {}, &outputs,
&run_metadata));
return PostProcessPredictionResult(signature, output_tensor_aliases, outputs,
response);
}
This code runs a prediction with a stored model. In my case, this stored model is an RNN.
After the following line where the prediction is actually run:
TF_RETURN_IF_ERROR(bundle->session->Run(run_options, input_tensors,
output_tensor_names, {}, &outputs,
&run_metadata));
I want to save the states of the RNN to a file/memory so that I can access them at a later date after each prediction. I assume these states can be accessed via the variable:
bundle->meta_graph_def
but am unclear on exactly how to access specifically the state values of the RNN and then to save them to a file.
You have to get the values via the session, not after the fact. In your line
bundle->session->Run(run_options, input_tensors,
output_tensor_names, {}, &outputs,
&run_metadata)
You should not only request the result of the prediction, but also the state tensor. Note that the value of the state is not stored after the particular call to session->Run, only variables are stored, the state of the RNN is a computed value and dropped after the results you requested are returned.
I don't use the c++ interface, so forgive me for giving a code example in python, hopefully it's still of use (I bet this isn't the first time you've had to put up with reading a python example), in python this would look like:
prediction, state = sess.run([prediction_tensor, state_tensor], feed_dict=...)
I am rying to use tensorboard embeddings page in order to visualize the results of word2vec. After debugging, digging of lots of codes i came to a point that tensorboard successfully runs, reads the confguration file, reads the tsv files but now the embeddings page does not show data.
( the page is opened , i can see the menus , items etc) this is my config file:
embeddings {
tensor_name: 'word_embedding'
metadata_path: 'c:\data\metadata.tsv'
tensor_path: 'c:\data\tensors2.tsv'
}
What can be the problem?
The tensor file originally is 1gb. in size, if i try that file , the app crashes becasue of the memory. So i copy and paste 1 or 2 pages of the original file into tensor2.tsv and use this file. May be this is the problem. May be i need to create more data by copy/ paste.
thx
tolga
Try following code snippet to get visualized word embedding in tensorboard. Open tensorboard with logdir, check localhost:6006 for viewing your embedding.
tensorboard --logdir="visual/1"
# code
fname = "word2vec_model_1000"
model = gensim.models.keyedvectors.KeyedVectors.load(fname)
# project part of vocab, max of 100 dimension
max = 1000
w2v = np.zeros((max,100))
with open("prefix_metadata.tsv", 'w+') as file_metadata:
for i,word in enumerate(model.wv.index2word[:max]):
w2v[i] = model.wv[word]
file_metadata.write(word + '\n')
# define the model without training
sess = tf.InteractiveSession()
with tf.device("/cpu:0"):
embedding = tf.Variable(w2v, trainable=False, name='prefix_embedding')
tf.global_variables_initializer().run()
path = 'visual/1'
saver = tf.train.Saver()
writer = tf.summary.FileWriter(path, sess.graph)
# adding into projector
config = projector.ProjectorConfig()
embed= config.embeddings.add()
embed.tensor_name = 'prefix_embedding'
embed.metadata_path = 'prefix_metadata.tsv'
# Specify the width and height of a single thumbnail.
projector.visualize_embeddings(writer, config)
saver.save(sess, path+'/prefix_model.ckpt', global_step=max)
I recently upgraded my tesnorflow from Rev8 to Rev12. In Rev8 the default "state_is_tuple" flag in rnn_cell.LSTMCell is set to False, so I initialized my LSTM Cell with an list, see code below.
#model definition
lstm_cell = rnn_cell.LSTMCell(self.config.hidden_dim)
outputs, states = tf.nn.rnn(lstm_cell, data, initial_state=self.init_state)
#init_state place holder and feed_dict
def add_placeholders(self):
self.init_state = tf.placeholder("float", [None, self.cell_size])
def get_feed_dict(self, data, label):
feed_dict = {self.input_data: data,
self.input_label: reg_label,
self.init_state: np.zeros((self.config.batch_size, self.cell_size))}
return feed_dict
In Rev12, the default "state_is_tuple" flag is set to True, in order to make my old code work I had to explicitly turn the flag to False. However, now I got an warning from tensorflow saying:
"Using a concatenated state is slower and will soon be deprecated.
Use state_is_tuple=True"
I tried to initialize LSTM cell with a tuple by changing the placeholder definition for self.init_state to the following:
self.init_state = tf.placeholder("float", (None, self.cell_size))
but now I got an error message saying:
"'Tensor' object is not iterable"
Does anyone know how to make this work?
Feeding a "zero state" to an LSTM is much simpler now using cell.zero_state. You do not need to explicitely define the initial state as a placeholder. Define it as a tensor instead and feed it if required. This is how it works,
lstm_cell = rnn_cell.LSTMCell(self.config.hidden_dim)
self.initial_state = lstm_cell.zero_state(self.batch_size, dtype=tf.float32)
outputs, states = tf.nn.rnn(lstm_cell, data, initial_state=self.init_state)
If you wish to feed some other value as the initial state, Let's say next_state = states[-1] for instance, calculate it in your session and pass it in the feed_dict like -
feed_dict[self.initial_state] = next_state
In the context of your question, lstm_cell.zero_state() should suffice.
Unrelated, but remember that you can pass both Tensors and Placeholders in the feed dictionary! That's how self.initial_state is working in the example above. Have a look at the PTB Tutorial for a working example.
So I have an instance of Liquidsoap, which I am using to stream to an Icecast server.
I'd like to record any live broadcasts that take place automatically, which I am now doing and is working well.
What I'd like to do is use the metadata (specifically the songname) of the live show when creating the mp3 archive.
#!/usr/local/bin/liquidsoap
set("log.file",true)
set("log.file.path","/var/log/liquidsoap/radiostation.log")
set("log.stdout",true)
set("log.level",3)
#-------------------------------------
set("harbor.bind_addr","0.0.0.0")
#-------------------------------------
backup_playlist = playlist("/home/radio/playlists/playlist.pls",conservative=true,reload_mode="watch")
output.dummy(fallible=true,backup_playlist)
#-------------------------------------
live_dj = input.harbor(id="live",port=9000,password="XXX", "live")
date = '%m-%d-%Y'
time = '%H:%M:%S'
output.file(%mp3, "/var/www/recorded-shows/#{Title} - Recorded On #{date} At #{time}.mp3", live_dj, fallible=true)
#time_stamp = '%m-%d-%Y, %H:%M:%S'
#output.file(%mp3, "/var/www/recorded-shows/live_dj_#{time_stamp}.mp3", live_dj, fallible=true)
#-------------------------------------
on_fail = single("/home/radio/fallback/Enei -The Moment Feat DRS.mp3")
#-------------------------------------
source = fallback(track_sensitive=false,
[live_dj, backup_playlist, on_fail])
# We output the stream to icecast
output.icecast(%mp3,id="icecast",
mount="myradio.mp3",
host="localhost", password="XXX",
icy_metadata="true",description="cool radio",
url="http://myradio.fm",
source)
I have added #{title} where I would like my song title to appear, sadly though I am unable to get this populate.
My Dj's use BUTT and the show title is connected as part of their connection, so the data should be available pre recording.
Any advice is much appreciated!
This is far from being as easy as it seems.
The title metadata is dynamic, thus not available as a variable on script initialization
The filename argument of output.file is compiled when the script is initialized
A solution would consist in:
Defining a variable reference title to populate with live metadata
Output to a temporary file
Rename the file on close using on_close argument with output.file (in this case, we can just prepend the title)
This would give the following code (on a Linux box, change mv with ren on Windows):
date = '%m-%d-%Y'
time = '%H:%M:%S'
# Title is a reference
title = ref ""
# Populate with metadata
def get_title(m)
title := m['title']
end
live_dj = on_metadata(get_title,live_dj)
# Rename file on close
def on_close(filename)
# Generate new file name
new_filename = "#{path.dirname(filename)}/#{!title} - #{basename(filename)}"
# Rename file
system("mv '#{filename}' '#{new_filename}'")
end
output.file(on_close=on_close, %mp3, "/var/www/recorded-shows/Recorded On #{date} At #{time}.mp3", live_dj, fallible=true)
I tested a similar scenario and it works just well. Just beware that this will create a new file every time a DJ disconnects or updates the title. Also keep in mind that time stamps will be resolved by output.file.
This is based on the following example from a Liquidsoap dev: https://github.com/savonet/liquidsoap/issues/661#issuecomment-439935854)