I am using PyTorch Lightning to train a model and am just wondering how I can return 2 learning rate schedulers for 1 optimizer in def configure_optimizers? Below is my code draft:
def configure_optimizers(self):
optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)
scheduler1 = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)
scheduler2 = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, factor=0.2,
patience=3,
mode=self.monitor_mode,
threshold=0.0001, cooldown=0, min_lr=0)
return (
{"optimizer": optimizer, "lr_scheduler": scheduler1},
{
"optimizer": optimizer,
"lr_scheduler": {
"scheduler": scheduler2,
"monitor": "val_loss",
},
},
)
It seems that the return format is not quite right, the code returned this error:opt_idx set inside scheduler config does not match with the index of the respective optimizer returned from configure_optimizers.
Could I know how the code needs to be modified to achieve this?
Thanks!
Related
Hi I'm new to Tensorflow and I've been practicing with the tensorflow.estimator library. Basically I ran the inbuilt tf.estimator.DNNClassifier algorithm below
import tensorflow as tf
def train_input_fn(features, labels, batch_size):
"""An input function for training"""
# Convert the inputs to a Dataset.
dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
# Shuffle, repeat, and batch the examples.
return dataset.shuffle(1000).repeat().batch(batch_size)
# Feature columns describe how to use the input.
my_feature_columns = []
for key in landmark_features.keys():
my_feature_columns.append(tf.feature_column.numeric_column(key=key))
# Build a DNN with 2 hidden layers and 10 nodes in each hidden layer.
classifier = tf.estimator.DNNClassifier(feature_columns=my_feature_columns, hidden_units=[10, 10],n_classes=10)
dataset = train_input_fn(landmark_features, emotion_labels, batch_size = 1375 )
However I keep getting the following error:
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpc_tag0rc
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpc_tag0rc', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
Any idea on what I can do to fix my code ?
I have a mongo DB query as below
db.getCollection('ABC_COLLECTION_01').aggregate
([{ "$group" : {
"_id" : 0,
"total" : { "$sum" : "$columA" },
"total_sub" : { "$sum" : {$cond:{ if: { $gte: [ "$columnB", new ISODate("2018-01-01T04:58:09.000+0100") ] }, then: "$columA", else: 0 }}}
}}])
This query is working fine. If I run this on ABC_COLLECTION_01, it will return result like (just for an example)
total = 250 & total_sub = 120
Now I have to write this query in C++ code using mongo::BSONArrayBuilder as below
//Calculate 2 sum
aBuilder.append(BSON("$group"<<BSON("_id"<<0
<<"total"<<BSON("$sum"<<'$' +columA)
<<"total_sub"<<BSON("$sum"<<'$cond'<<'if'<<'$' +columB<<"$gte"<<rmsmongo::utils::Adaptor::ToMongoDate(StartTime,true)<<'then'<<'$' +columA<<'else'<<0))));
mongo::BSONArray New_AggregationQuery = aBuilder.arr();
std::auto_ptr<dsclient::Cursor> Cursor = _MyCollection.aggregate(New_AggregationQuery, dsclient::Option_aggregate().retry(dsclient::Retry(dsclient::ExponentialRetry, 2)).maxTimeMS(200000));
If you see the $cond I have written for $total_sub is wrong in C++ code- it is not working.
Can you please help me to get it corrected?
Thanks in advance
Please note that the legacy C++ driver which you appear to be using here is end-of-life. I cannot recommend strongly enough that you immediately switch to the new mongocxx driver, found on the master branch of the same repo. It offers significant advantages, not the least of which is that it is actively maintained.
That said, if you look closely, you can see that the if expression in your working query has a subdocument, but in your C++ code, you aren't starting a new BSON subdocument. There are probably other errors.
Seriously though, I really recommend you stop working on this and get migrated to the new driver. Any bugs or non-conformances in the legacy driver (or are you even using 26compat) will never be fixed.
I have a google-cloud-ml model that I can run prediction by passing a 3 dimensional array of float32...
{ 'instances' [ { 'input' : '[ [ [ 0.0 ], [ 0.5 ], [ 0.8 ] ] ... ] ]' } ] }
However this is not an efficient format to transmit images, so I'd like to pass base64 encoded png or jpeg. This document talks about doing that, but what is not clear is what the entire json object looks like. Does the { 'b64' : 'x0welkja...' } go in place of the '[ [ [ 0.0 ], [ 0.5 ], [ 0.8 ] ] ... ] ]', leaving the enclosing 'instances' and 'input' the same? Or some other structure? Or does the tensorflow model have to be trained on base64?
The TensorFlow model does not have to be trained on base64 data. Leave your training graph as is. However, when exporting the model, you'll need to export a model that can accept PNG or jpeg (or possibly raw, if it's small) data. Then, when you export the model, you'll need to be sure to use a name for the output that ends in _bytes. This signals to CloudML Engine that you will be sending base64 encoded data. Putting it all together would like something like this:
from tensorflow.contrib.saved_model.python.saved_model import utils
# Shape of [None] means we can have a batch of images.
image = tf.placeholder(shape = [None], dtype = tf.string)
# Decode the image.
decoded = tf.image.decode_jpeg(image, channels=3)
# Do the rest of the processing.
scores = build_model(decoded)
# The input name needs to have "_bytes" suffix.
inputs = { 'image_bytes': image }
outputs = { 'scores': scores }
utils.simple_save(session, export_dir, inputs, outputs)
The request you send will look something like this:
{
"instances": [{
"b64": "x0welkja..."
}]
}
If you just want an efficient way to send images to a model (and not necessarily base-64 encode it), I would suggest uploading your images(s) to Google Cloud Storage and then having your model read off GCS. This way, you are not limited by image size and you can take advantage of multi-part, multithreaded, resumable uploads etc. that the GCS API provides.
TensorFlow's tf.read_file will directly off GCS. Here's an example of a serving input_fn that will do this. Your request to CMLE would send it an image URL (gs://bucket/some/path/to/image.jpg)
def read_and_preprocess(filename, augment=False):
# decode the image file starting from the filename
# end up with pixel values that are in the -1, 1 range
image_contents = tf.read_file(filename)
image = tf.image.decode_jpeg(image_contents, channels=NUM_CHANNELS)
image = tf.image.convert_image_dtype(image, dtype=tf.float32) # 0-1
image = tf.expand_dims(image, 0) # resize_bilinear needs batches
image = tf.image.resize_bilinear(image, [HEIGHT, WIDTH], align_corners=False)
#image = tf.image.per_image_whitening(image) # useful if mean not important
image = tf.subtract(image, 0.5)
image = tf.multiply(image, 2.0) # -1 to 1
return image
def serving_input_fn():
inputs = {'imageurl': tf.placeholder(tf.string, shape=())}
filename = tf.squeeze(inputs['imageurl']) # make it a scalar
image = read_and_preprocess(filename)
# make the outer dimension unknown (and not 1)
image = tf.placeholder_with_default(image, shape=[None, HEIGHT, WIDTH, NUM_CHANNELS])
features = {'image' : image}
return tf.estimator.export.ServingInputReceiver(features, inputs)
Your training code will train off actual images, just as in rhaertel80's suggestion above. See https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive/08_image/flowersmodel/trainer/task.py#L27 for what the training/evaluation input functions would look like.
I was trying to use #Lak's answer (thanks Lak) to get online predictions for multiple instances in one json file, but kept getting the following error (I had two instances in my test json, hence the shape [2]):
input filename tensor must be scalar but had shape [2]
The problem is that ML engine apparently batches all the instances together and passes them to the serving inpur receiver function, but #Lak's sample code assumes the input is a single instance (it indeed works fine if you have a single instance in your json). I altered the code so that it can process a batch of inputs. I hope it will help someone:
def read_and_preprocess(filename):
image_contents = tf.read_file(filename)
image = tf.image.decode_image(image_contents, channels=NUM_CHANNELS)
image = tf.image.convert_image_dtype(image, dtype=tf.float32) # 0-1
return image
def serving_input_fn():
inputs = {'imageurl': tf.placeholder(tf.string, shape=(None))}
filename = inputs['imageurl']
image = tf.map_fn(read_and_preprocess, filename, dtype=tf.float32)
# make the outer dimension unknown (and not 1)
image = tf.placeholder_with_default(image, shape=[None, HEIGHT, WIDTH, NUM_CHANNELS])
features = {'image': image}
return tf.estimator.export.ServingInputReceiver(features, inputs)
The key changes are that 1) you don't squeeze the input tensor (that would cause trouble in the special case when your json contains only one instance) and, 2) use tf.map_fn to apply the read_and_preprocess function to a batch of input image urls.
I am trying to figure out a way to use progress bar in gdal.Warp() to show how much of a job is done. For progress bar, I am using Tqdm and gdal.Warp() is used to crop image from remote URL
def getSubArea(url):
vsicurl_url = '/vsicurl/' + url
output_file = 'someID_subarea.tif'
gdal.SetConfigOption('GDAL_HTTP_UNSAFESSL', 'YES')
gdal.Warp(output_file, vsicurl_url, dstSRS='EPSG:4326', cutlineDSName='area.geojson', cropToCutline=True)
I know there is callback argument that reports progress from 0 to 1, but its only called after gdal.warp has finished downloading cropped image.
You may add a callback function for progress through the 'kwargs' parameter in 'gdal.Warp' (documentation: https://gdal.org/python/).
Code:
def getSubArea(url):
vsicurl_url = '/vsicurl/' + url
output_file = 'someID_subarea.tif'
# Data you want to pass to your callback (goes in to unknown parameter)
es_obj = { ... }
kwargs = {
'dstSRS': 'EPSG:4326',
'cutlineDSName': 'area.geojson',
'cropToCutline': True,
'callback': progress_callback,
'callback_data': es_obj
}
gdal.SetConfigOption('GDAL_HTTP_UNSAFESSL', 'YES')
gdal.Warp(output_file, vsicurl_url, **kwargs)
def progress_callback(self, complete, message, unknown):
# Calculate percent by integer values (1, 2, ..., 100)
percent = floor(complete * 100)
# Code for saving or using percent value
...
About progress callback: https://gdal.org/api/cpl.html#_CPPv416GDALProgressFunc
See answer here around native gdal callback function.
If you wanted to report the download progress of the raster, you'd likely need to split that out as a separate step using something like requests and wrapping that with a progress bar like tqdm or progressbar2.
I recently upgraded my tesnorflow from Rev8 to Rev12. In Rev8 the default "state_is_tuple" flag in rnn_cell.LSTMCell is set to False, so I initialized my LSTM Cell with an list, see code below.
#model definition
lstm_cell = rnn_cell.LSTMCell(self.config.hidden_dim)
outputs, states = tf.nn.rnn(lstm_cell, data, initial_state=self.init_state)
#init_state place holder and feed_dict
def add_placeholders(self):
self.init_state = tf.placeholder("float", [None, self.cell_size])
def get_feed_dict(self, data, label):
feed_dict = {self.input_data: data,
self.input_label: reg_label,
self.init_state: np.zeros((self.config.batch_size, self.cell_size))}
return feed_dict
In Rev12, the default "state_is_tuple" flag is set to True, in order to make my old code work I had to explicitly turn the flag to False. However, now I got an warning from tensorflow saying:
"Using a concatenated state is slower and will soon be deprecated.
Use state_is_tuple=True"
I tried to initialize LSTM cell with a tuple by changing the placeholder definition for self.init_state to the following:
self.init_state = tf.placeholder("float", (None, self.cell_size))
but now I got an error message saying:
"'Tensor' object is not iterable"
Does anyone know how to make this work?
Feeding a "zero state" to an LSTM is much simpler now using cell.zero_state. You do not need to explicitely define the initial state as a placeholder. Define it as a tensor instead and feed it if required. This is how it works,
lstm_cell = rnn_cell.LSTMCell(self.config.hidden_dim)
self.initial_state = lstm_cell.zero_state(self.batch_size, dtype=tf.float32)
outputs, states = tf.nn.rnn(lstm_cell, data, initial_state=self.init_state)
If you wish to feed some other value as the initial state, Let's say next_state = states[-1] for instance, calculate it in your session and pass it in the feed_dict like -
feed_dict[self.initial_state] = next_state
In the context of your question, lstm_cell.zero_state() should suffice.
Unrelated, but remember that you can pass both Tensors and Placeholders in the feed dictionary! That's how self.initial_state is working in the example above. Have a look at the PTB Tutorial for a working example.