I have a data with FloatLabel and FloatFeatureVector. FeatureVector size is variable. Once the model is trianed on traindata from a csv file,i want to run it on testdata that i generate on the fly.
Each csv file is simply the sensor data from each pump in the plant. so the featurevector is variable(since some pumps will not have all the sensors.) I can load the csv file, do feature selection and build a model on it and get the score values(regression).
Next, I want to run it on testdata in a matrix form of double()() that i am generating.
The feature size in the testdata is the same as traindata. I am using the mlcontext.data.Loadfromenumerable.
For the class that is required by the enumerable, i am using the one below.
private const int FeatureLength = 10;
public class FloatLabelFloatFeatureVectorSample
{
public float Label;
[VectorType(FeatureLength)]
public float[] Features;
}
it errors if the featurelength is not a constant.
Answers I saw elsewhere were vague/inconsistent
(see link: https://github.com/dotnet/machinelearning/issues/164)
I am using ml.net 0.11
Related
I have a custom C++ class named UHandsAnimInstance :
UCLASS(transient, Blueprintable, hideCategories = AnimInstance, BlueprintType)
class SENSORIALSDK_API UHandsAnimInstance : public UAnimInstance
{
GENERATED_BODY()
public:
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = Fingers)
bool isRight = true;
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = Fingers)
FRotator wrist;
}
It has some custom properties inside and it is the parent class of an animation blueprint asset as you can see in this picture:
Blueprint Animation Editor
And it is located in this location: "/Game/SensorialXR/Blueprints/Gestures/RightHand"
Asset Location in Content Manager
My main problem is that I want to select that object from Content Manager so first I try to put that as a property in another class (AActor class):
UPROPERTY(EditAnywhere, BlueprintReadWrite)
TArray<UHandsAnimInstance*> gestureData;
But I cannot select any object in the editor selector:
Editor Selector Example
So I tried to load all assets from a FString parameter that is the path of all UHandsAnimInstance at runtime using this code in the BeginPlay function inside the custom AActor class:
FString gesturesPath = "/Game/SensorialXR/Blueprints/Gestures/RightHand";
FAssetRegistryModule& AssetRegistryModule = FModuleManager::LoadModuleChecked<FAssetRegistryModule>("AssetRegistry");
TArray<FAssetData> AssetData;
AssetRegistryModule.Get().GetAssetsByPath(FName(*gesturesPath), AssetData);
for (int i = 0; i < AssetData.Num(); i++) {
UAnimBlueprint* animx = Cast<UAnimBlueprint>(AssetData[i].GetAsset());
if (animx) {
//Do Something (like add to the gestureData list)
}
}
The problem is that the class of the obtained asset is UAnimBlueprint (that is why I cast that asset to that type):
Main class of the obtained asset
But I cannot do anything neither get the UHandsAnimInstance object from that asset:
Data from the UAnimBlueprint obtained variable
So, what can I do to get the data I want of my custom C++ class from a blueprint object in the content manager?
The Assets within the Content Browser are not real "instances" of those Objects.
If you want to be able to select the Asset Type from the Content Browser and input it into a Property, you will be required to use a TSubclassOf<UHandsAnimInstance> type.
This will give you a UClass* to the Asset Type. Keep in mind this is not an instance. If you want to access default property data on that Type you can use its CDO (Class Default Object). The CDO can be accessed directly from the UClass*.
UHandsAnimInstance* MyHandsAnimCDO = Cast<UHandsAnimInstance>(MyClass->GetDefaultObject());
The above CDO will require casting to your particular type. Then you can access any Property Default values from there. Do not modify or call non-const functions on the CDO, it is not design to be mutable in the general sense as it is only the template Object from which real instances are generated with at runtime.
If you are looking for a specific instance of your UHandsAnimInstance you will need to find the associated Skeletal Mesh that it has been instantiated with.
From the example given by Google, I have managed to load CSV files into BigQuery(BQ) table following the guide(link and code below)
Now I want to add several files into BQ, and want to add a new column filename which contains the filename.
Is there a way to add column with default data?
https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv
// Import the Google Cloud client libraries
const {BigQuery} = require('#google-cloud/bigquery');
const {Storage} = require('#google-cloud/storage');
// Instantiate clients
const bigquery = new BigQuery();
const storage = new Storage();
/**
* This sample loads the CSV file at
* https://storage.googleapis.com/cloud-samples-data/bigquery/us-states/us-states.csv
*
* TODO(developer): Replace the following lines with the path to your file.
*/
const bucketName = 'cloud-samples-data';
const filename = 'bigquery/us-states/us-states.csv';
async function loadCSVFromGCS() {
// Imports a GCS file into a table with manually defined schema.
/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// const datasetId = 'my_dataset';
// const tableId = 'my_table';
// Configure the load job. For full list of options, see:
// https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationLoad
const metadata = {
sourceFormat: 'CSV',
skipLeadingRows: 1,
schema: {
fields: [
{name: 'name', type: 'STRING'},
{name: 'post_abbr', type: 'STRING'},
// {name: 'filemame', type: 'STRING', value=filename} // I WANT TO ADD COLUMN WITH FILE NAME HERE
],
},
location: 'US',
};
// Load data from a Google Cloud Storage file into the table
const [job] = await bigquery
.dataset(datasetId)
.table(tableId)
.load(storage.bucket(bucketName).file(filename), metadata);
// load() waits for the job to finish
console.log(`Job ${job.id} completed.`);
// Check the job's status for errors
const errors = job.status.errors;
if (errors && errors.length > 0) {
throw errors;
}
}
I would say you have a few choices.
Add a column to the CSV before uploading, e.g. with awk or preprocessing in JS.
Add the individual CSV files to separate tables. You can easily query across many tables as one in BigQuery. This way you can easily see what data comes from which file, and you can access table meta data for the file-name
Post process the data, by adding the column after the data is loaded with normal sql/api calls.
See also this possible duplicate How to add new column with metadata value to csv when loading it to bigquery
According to BigQuery’s documentation [1], there is no option to set a default value for columns. The closest option without any post-processing, would be to use a NULL value for nullable columns.
However, a possible postprocessing workaround for this would be to create a View of the raw table and add a script that maps the NULL value to any default value. Here’s some information about scripting in BigQuery [2].
In case it is possible to add a pre-processing code, adding the value to the source file would be easy to achieve using any scripting language.
I think that static and function-based values will be a good feature for BigQuery’s future scope.
[1] -
https://cloud.google.com/bigquery/docs
[2] -
https://cloud.google.com/bigquery/docs/reference/standard-sql/scripting
You have multiple options:
you could rebuild your CSV with the filename as a column data
you can load data into a temporary table, then moving to final table with a second step specifying the missing file name column
convert the example to be an external table where _FILE_NAME is a pseduocolumn, and later you can query and move to a final table. See more about this here.
I run a ML.NET sample app , there is the code in the Program.cs :
…
Console.WriteLine(“Training model…”);
var model = pipeline.Fit(trainTestData.TrainSet);
…
i.e. every time the model trained and created.
The question: Is it possible to save the model in a database after it once trained and created and then load and reuse the saved one?
It can be saved to zip file and loaded and reused. But how to save/load it to database?
The model can be saved to a MemoryStream, then converted to a byte array and stored in the database.
using var memoryStream = new MemoryStream();
_mlContext.Model.Save(trainedModel, trainData.Schema, memoryStream);
_dbContext.Models.Add(new Model
{
Data = stream.ToArray()
// ...metadata
});
Save (stream overload signature):
// Summary:
// Save a transformer model and the loader used to create its input data to the
// stream.
//
// Parameters:
// model:
// The trained model to be saved. Note that this can be null, as a shorthand for
// an empty transformer chain. Upon loading with Microsoft.ML.ModelOperationsCatalog.LoadWithDataLoader(System.IO.Stream,Microsoft.ML.IDataLoader{Microsoft.ML.Data.IMultiStreamSource}#)
// the returned value will be an empty Microsoft.ML.Data.TransformerChain`1.
//
// loader:
// The loader that was used to create data to train the model.
//
// stream:
// A writeable, seekable stream to save to.
public void Save<TSource>(ITransformer model, IDataLoader<TSource> loader, Stream stream)
I'm using Google's Flatbuffer. I've created a simple schema for C++ that just takes a name and ID as the fields. After creating the auto generated code and running the fields through the CreateDetails() function, how would I get the bytearray to pass into ActiveMQ? I've searched around but couldn't find much about the byte array.
My schema:
table details {
name:string;
id: int;
};
root_type details;
My .cpp application:
auto name = builder.CreateString("some text here");
auto id = 25;
auto detail = CreateDetails(builder, name, id);
builder.Finish(detail);
Now, from my understanding, the sample message should be serialized, but I'm not sure how to grab the serialized data as a byte array. I was able to access the root and just go down the tree and look at the data, but I want to grab the entire message as a bytearray.
Please and thank you!
builder.GetBufferPointer()
builder.GetSize()
I'd like to create a GUI table to display a given list of features of an EObject sub-class. To do this I have to get the display names of the features for the column header.
How do I get the feature display names in the best way?
One solution that seems a bit like a hack:
If I have an instance of the class then I can use the adaptor factory to get a IItemPropertySource that can do this:
SomeEntity e = ...
String displayName = adaptorFactory.adapt(e, IItemPropertySource.class)
.getPropertyDescriptor(null, feature).getDisplayName(null));
But when the table is empty there is no SomeEntity object handy to use to get the IItemPropertySource.
I can create a dummy object using the EFactory in this way:
EClass containingClass = feature.getEContainingClass();
SomeEntity dummy = containingClass.getEPackage().getEFactoryInstance()
.create(containingClass));
... and then use that object the get the IItemPropertySource. But this seem a bit like a hack. Is there no better solution?
If you know the class at compile time, you can create the ItemProviderAdapter yourself:
MyClassItemProvider provider = new MyClassItemProvider(adaptorFactory);
String name = provider.getPropertyDescriptor(null, property).getDisplayName(null);
If you do not know the class at compile time, but only have an EClass instance at runtime, things are more complicated, because the necessary methods are protected. You have to "make" them public first.
I would add respective methods to the generated MyPackageSwitch and MyPackageAdapterFactory classes (in myPackage.util).
In MyPackageAdapterFactory:
/**
* #generated NOT
*/
public MyPackageSwitch<Adapter> getModelSwitch() {
return modelSwitch;
}
In MyPackageSwitch:
/**
* generated NOT
*/
public T doPublicSwitch(EClass theEClass, EObject theEObject) {
return doSwitch(theEClass, theEObject);
}
Now you can create an ItemProviderAdapter for an EClass theEClass like this:
provider = (ItemProviderAdapter) adapterFactory.getModelSwitch()
.doPublicSwitch(theEClass, null);
EMF was obviously not made for this. Keep in mind that this all is only working if you do not have any custom provider implementations that uses the EObject values.