I run a ML.NET sample app , there is the code in the Program.cs :
…
Console.WriteLine(“Training model…”);
var model = pipeline.Fit(trainTestData.TrainSet);
…
i.e. every time the model trained and created.
The question: Is it possible to save the model in a database after it once trained and created and then load and reuse the saved one?
It can be saved to zip file and loaded and reused. But how to save/load it to database?
The model can be saved to a MemoryStream, then converted to a byte array and stored in the database.
using var memoryStream = new MemoryStream();
_mlContext.Model.Save(trainedModel, trainData.Schema, memoryStream);
_dbContext.Models.Add(new Model
{
Data = stream.ToArray()
// ...metadata
});
Save (stream overload signature):
// Summary:
// Save a transformer model and the loader used to create its input data to the
// stream.
//
// Parameters:
// model:
// The trained model to be saved. Note that this can be null, as a shorthand for
// an empty transformer chain. Upon loading with Microsoft.ML.ModelOperationsCatalog.LoadWithDataLoader(System.IO.Stream,Microsoft.ML.IDataLoader{Microsoft.ML.Data.IMultiStreamSource}#)
// the returned value will be an empty Microsoft.ML.Data.TransformerChain`1.
//
// loader:
// The loader that was used to create data to train the model.
//
// stream:
// A writeable, seekable stream to save to.
public void Save<TSource>(ITransformer model, IDataLoader<TSource> loader, Stream stream)
Related
The following code will load a saved model and will load 'mlModel' as part of that process.
var mlContext = new MLContext();
ITransformer mlModel = mlContext.Model.Load(MLNetModelPath, out var _);
return mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(mlModel);
However, If I have just an existing trained mlContext, how do I get the 'ITransformer' from that existing mlContext?
You can load it from a stream instead of file:
See this documentation
I have a data with FloatLabel and FloatFeatureVector. FeatureVector size is variable. Once the model is trianed on traindata from a csv file,i want to run it on testdata that i generate on the fly.
Each csv file is simply the sensor data from each pump in the plant. so the featurevector is variable(since some pumps will not have all the sensors.) I can load the csv file, do feature selection and build a model on it and get the score values(regression).
Next, I want to run it on testdata in a matrix form of double()() that i am generating.
The feature size in the testdata is the same as traindata. I am using the mlcontext.data.Loadfromenumerable.
For the class that is required by the enumerable, i am using the one below.
private const int FeatureLength = 10;
public class FloatLabelFloatFeatureVectorSample
{
public float Label;
[VectorType(FeatureLength)]
public float[] Features;
}
it errors if the featurelength is not a constant.
Answers I saw elsewhere were vague/inconsistent
(see link: https://github.com/dotnet/machinelearning/issues/164)
I am using ml.net 0.11
I'm training my model using TensorFlow in C++. Python is used only for constructing the graph. So is there a way to save and restore the graph and its state purely in C++? I know about the Python class tf.train.Saver but as far as I understand it does not exist in C++.
The tf.train.Saver class currently exists only in Python, but (i) it is built from TensorFlow ops that you can run from C++, and (ii) it exposes the Saver.as_saver_def() method that lets you get a SaverDef protocol buffer with the names of ops that you must run to save or restore a model.
In Python, you can get the names of the save and restore ops as follows:
saver = tf.train.Saver(...)
saver_def = saver.as_saver_def()
# The name of the tensor you must feed with a filename when saving/restoring.
print saver_def.filename_tensor_name
# The name of the target operation you must run when restoring.
print saver_def.restore_op_name
# The name of the target operation you must run when saving.
print saver_def.save_tensor_name
In C++ to restore from a checkpoint, you call Session::Run(), feeding in the name of the checkpoint file as saver_def.filename_tensor_name, with a target op of saver_def.restore_op_name. To save another checkpoint, you call Session::Run(), again feeding in the name of the checkpoint file as saver_def.filename_tensor_name, and fetching the value of saver_def.save_tensor_name.
The recent TensorFlow version includes some helper functions to do the same in C++ without Python. These are generate from the ProtoBuf in the pip-package (${HOME}/.local/lib/python2.7/site-packages/tensorflow/include/tensorflow/core/protobuf/saver.pb.h).
// save
tensorflow::Tensor checkpointPathTensor(tensorflow::DT_STRING, tensorflow::TensorShape());
checkpointPathTensor.scalar<std::string>()() = "some/path";
tensor_dict feed_dict = {{graph_def.saver_def().filename_tensor_name(), checkpointPathTensor}};
status = sess->Run(feed_dict, {}, {graph_def.saver_def().save_tensor_name()}, nullptr);
// restore
tensorflow::Tensor checkpointPathTensor(tensorflow::DT_STRING, tensorflow::TensorShape());
checkpointPathTensor.scalar<std::string>()() = "some/path";
tensor_dict feed_dict = {{graph_def.saver_def().filename_tensor_name(), checkpointPathTensor}};
status = sess->Run(feed_dict, {}, {graph_def.saver_def().restore_op_name()}, nullptr);
This is based on the undocumented python-way (more details) of restoring a model
def restore(sess, metaGraph, fn):
restore_op_name = metaGraph.as_saver_def().restore_op_name # u'save/restore_all'
restore_op = tf.get_default_graph().get_operation_by_name(restore_op_name)
filename_tensor_name = metaGraph.as_saver_def().filename_tensor_name # u'save/Const'
sess.run(restore_op, {filename_tensor_name: fn})
For a working and complete version see here.
How can I determine the "store name" (not sure what the proper terminology is) for a given ED Model? Say I have App.Payment, is there a store method that let's me look up its corresponding name, i.e. payment (for example to use in find queries)?
For Ember Data 1.0 (and later)
modelName is a dasherized string. It stored as a class property, so if you have an instance of a model:
var model = SuperUser.create();
console.log(model.constructor.modelName); // 'super-user'
For Ember Data Pre 1.0
typeKey is the string name of the model. It gets stored as a class property of the model, so if you have an instance of a model:
var model = App.Name.create({});
console.log(model.constructor.typeKey); // 'name'
You might be looking for Ember's string dasherize method:
var fullClassName = "App.SomeKindOfPayment";
var className = fullClassName.replace(/.*\./, ""); // => "SomeKindOfPayment"
var dasherizedName = Ember.String.dasherize(className); // "some-kind-of-payment"
There might be a built-in way to do this in Ember, but I haven't found it after spending some time looking.
EDIT: Ember Data might also let you get away with passing "App.SomeKindOfPayment" when a model name is needed - it usually checks the format of the model name and updates it to the required format by itself.
store.find, store.createRecord, and other persistence methods, use the store.modelFor('myModel'). After some setup it call container.lookupFactory('model:' + key); where key is the 'myModel'. So any valid factory lookup syntax is applicable. For example:
Given a model called OrderItems you can use: order.items, order_items, order-items, orderItems.
It turns out there was no need to do this after all, and here's why:
I was trying to the the string representation of the model ("payment" for App.Payment) in order to call store.findAll("payment"). However, looking at the ED source for store, the findQuery function calls modelFor to look up the factory (App.Payment) from the string (payment), unless a factory is already provided. And the factory is easily accessible from the controller by calling this.get('model').type. There's no need to convert it to a string (and back).
Here's the relevant code from the Ember Data source.
modelFor: function(key) {
var factory;
if (typeof key === 'string') {
factory = this.container.lookupFactory('model:' + key);
Ember.assert("No model was found for '" + key + "'", factory);
factory.typeKey = key;
} else {
// A factory already supplied.
factory = key;
}
factory.store = this;
return factory;
},
In several data structures with my application I have an attribute named "IsPrimary". One issue I am having is that with for each User they can only have 1 primary address but have multiple address. What I am trying to figure out is with in cf9's implementation of hibernate how can I update any existing record that has the attribute IsPrimary as true to false if a new record or updated record is saved with IsPrimary being true.
This is my existing save method
public UserAddress function save(required UserAddress)
{
var userAcc = entityLoadByPK('UserAccount', arguments.UserAddress.getUserID());
arguments.UserAddress.setUserID(userAcc);
entitySave(arguments.UserAddress);
return arguments.UserAddress;
}
you may do it at preInsert() if you like.