I'm trying to run a prediction using a sagemaker endpoint. The input format is comma separated features and | separated observations.
However when I try to iterate over the input data and invoke the end point on every iteration like this :
ENDPOINT_NAME = "my_endpoint"
runtime= boto3.client('runtime.sagemaker')
results = []
for r in request_body.split('|'):
response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,
ContentType='text/csv',
Body=r)
result = json.loads(response['Body'].read().decode())
results.append(result)
I get the following error:
ValidationError: an error occurred (ValidationError) when calling the InvokeEndpoint operation: 1 validation error detected: Value at 'body' failed to satisfy constraint: Member must not be null
As a sanity check I ran :
for r in request_body.split('|'):
print(r)
And I get the result I expect to get:
3.0,0.0,4795.0,0.0,1.0,24.0,30.0,25.0,3.0
3.0,2.0,3818.0,0.0,3.0,10.0,22.0,11.0,11.0
5.0,0.0,3565.0,0.0,1.0,79.0,89.0,80.0,-66.0
5.0,-1.0,3227.7,0.0,0.0,16.0,17.0,17.0,1.0
5.0,0.0,3375.0,0.0,2.0,21.0,45.0,22.0,6.0...etc
Which leads me to believe that the logic in extracting the separate observations is sound, but somehow when I execute the call I get this null value error.
The idea is to get ordered predictions so that I can later map them to an id that is not part of the training features and hence not in the dataset.
Thank you in advance.
I had the same issue. Check if you are also passing to the endpoint an empty "r".
request_body.split('|') will generate a list with each of the rows of the dataframe, but it will also include one empty: ''
Related
I have found Doctrine\Common\Collections\Criteria to be a very useful concept, if they worked for me.
In a symfony controller, I am calling this code:
$criteria = Criteria::create()
->where(Criteria::expr()->gt('position', 0))
->orderBy(['riskPosition', Criteria::ASC]);
$positions= $this->getDoctrine()->getRepository(DataCategory::class)->matching($criteria);
dump($positions->count()); // dumps 1, correct!
dump($positions);
foreach($positions as $r)
dump($r); // -> Unrecognized field: 0
dump($positions) gives
LazyCriteriaCollection {#881 ▼
#entityPersister: JoinedSubclassPersister {#849 ▶}
#criteria: Criteria {#848 ▼
-expression: Comparison {#836 ▶}
-orderings: array:2 [▶]
-firstResult: null
-maxResults: null
}
-count: 1
#collection: null
#initialized: false
}
As soon as I access an element of the returned array, I get an error
ORMException::unrecognizedField(0)
in vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php (line 1193)
But as soon as I want to access the elements (e.g. loop and dump) I get some error like An exception has been thrown during the rendering of a template ("Unrecognized field: 0").
As far as I have studied the code, the problem is that the query results have not been fetched from the database. Only count() works. How can I trigger this fetch?
Does it matter that my entity has #ORM\InheritanceType("JOINED")?
This code (circumventing the use of Criteria) does give correct results, but I'd like to use Criteria:
$riskPositions = $this->getDoctrine()->getRepository(DataCategory::class)
->createQueryBuilder('p')
->where('p.position > 0')
->orderBy('p.position', 'ASC')
->getQuery()
->execute();
The issue is caused by line:
->orderBy(['riskPosition', Criteria::ASC]);
Doctrine\Common\Collections\Criteria `s orderBy accepts an array argument where
Keys are field and values are the order, being either ASC or DESC.
github link
Apparently, there is a mistake at doctrine s documentation.
So doctrine thinks that "0", which is the 1st key of the array argument, is the field to sort by, but cannot find it.
To solve, change the above line to:
->orderBy(['riskPosition' => Criteria::ASC]);
I am trying to just do a basic INSERT operation to a PostgreSQL database through Python via the Psycopg2 module. I have read a great many of the questions already posted regarding this subject as well as the documentation but I seem to have done something uniquely wrong and none of the fixes seem to work for my code.
#API CALL + JSON decoding here
x = 0
for item in ulist:
idValue = list['members'][x]['name']
activeUsers.append(str(idValue))
x += 1
dbShell.executemany("""INSERT INTO slickusers (username) VALUES (%s)""", activeUsers
)
The loop creates a list of strings that looks like this when printed:
['b2ong', 'dune', 'drble', 'drars', 'feman', 'got', 'urbo']
I am just trying to have the code INSERT these strings as 1 row each into the table.
The error specified when running is:
TypeError: not all arguments converted during string formatting
I tried changing the INSERT to:
dbShell.executemany("INSERT INTO slackusers (username) VALUES (%s)", (activeUsers,) )
But that seems like it's merely treating the entire list as a single string as it yields:
psycopg2.DataError: value too long for type character varying(30)
What am I missing?
First in the code you pasted:
x = 0
for item in ulist:
idValue = list['members'][x]['name']
activeUsers.append(str(idValue))
x += 1
Is not the right way to accomplish what you are trying to do.
first list is a reserved word in python and you shouldn't use it as a variable name. I am assuming you meant ulist.
if you really need access to the index of an item in python you can use enumerate:
for x, item in enumerate(ulist):
but, the best way to do what you are trying to do is something like
for item in ulist: # or list['members'] Your example is kinda broken here
activeUsers.append(str(item['name']))
Your first try was:
['b2ong', 'dune', 'drble', 'drars', 'feman', 'got', 'urbo']
Your second attempt was:
(['b2ong', 'dune', 'drble', 'drars', 'feman', 'got', 'urbo'], )
What I think you want is:
[['b2ong'], ['dune'], ['drble'], ['drars'], ['feman'], ['got'], ['urbo']]
You could get this many ways:
dbShell.executemany("INSERT INTO slackusers (username) VALUES (%s)", [ [a] for a in activeUsers] )
or event better:
for item in ulist: # or list['members'] Your example is kinda broken here
activeUsers.append([str(item['name'])])
dbShell.executemany("""INSERT INTO slickusers (username) VALUES (%s)""", activeUsers)
mymodel.rb
enum status: { posted: 1, failed: 2, suspended: 3 }
mycontroller.rb
def filter_params
params.fetch(:mymodel, {}).
permit(
:status => []
)
end
And i have params like mymodel[:status] => ["failed", "suspended"]
How can i get all results by status is failed and suspended
Something like: Mymodel.where(status: filter_params[:status])
Thanks a lot!
And when a call:
#mymodel = Mymodel.new(filter_params)
I got this error:
'["failed", "suspended"]' is not a valid status
When running a query, you need to supply the ordinal values for the enum attribute. So instead of strings like 'failed' or 'suspended', you need to query using their integer values.
Luckily, you can access a hash to easily map all the statuses to integers from your filter_params hash:
values = Mymodel.statuses.values_at(*Array(filter_params[:status]))
With that you can run your query to get all records which have any of the filtered statuses:
Mymodel.where(status: values)
You don't want to scatter that piece of code all over the place though, so I recommend you'd implement this as a scope in your model:
class Mymodel < ActiveRecord::Base
enum status: { posted: 1, failed: 2, suspended: 3 }
scope :for_statuses, ->(values) do
return all if values.blank?
where(status: statuses.values_at(*Array(values)))
end
end
Note that the return all if values.blank? line makes it possible to throw in nil or an empty array without breaking your query.
You can now easily query the records:
Mymodel.for_statuses(filter_params[:status])
Note that you cannot create a record which has multiple statuses. enum only restricts the values that can be assigned, but you can assign only one, otherwise you get the not a valid status error.
See the Rails documentation for more information about enum.
In Rails 5 you can now pass a string array to the query, e.g:
Mymodel.where(status: ['failed', 'suspended'])
For earlier versions, just convert your array values to symbols:
statuses = filter_params[:status].map(&:to_sym)
Mymodel.where(status: statuses)
I want to query a HDF5-file. I do
df.to_hdf(pfad,'df', format='table')
to write the dataframe on disc.
To read I use
hdf = pandas.HDFStore(pfad)
I have a list that contains numpy.datetime64 values called expirations and try to read the portion of the hd5 table into a dataframe, that has values between expirations[1] and expirations[0] in column "expiration". Column expiration entries have the format Timestamp('2002-05-18 00:00:00').
I use the following command:
df = hdf.select('df',
where=['expiration<expiration[1]','expiration>=expirations[0]'])
However, this fails and produces a value error:
ValueError: The passed where expression: [expiration=expirations[0]]
contains an invalid variable reference
all of the variable refrences must be a reference to
an axis (e.g. 'index' or 'columns'), or a data_column
The currently defined references are: index,columns
Can you try this code:
df = hdf.select('df', where='expiration < expirations[1] and expiration >= expirations[0]')
or, as a query:
df = hdf.query('expiration < #expirations[1] and expiration >= #expirations[0]')
Not sure which one fits best your case, I noticed you are trying to use 'where' to filter rows, without a string or a list, does it make sense ?
I am writing an API which has loads of optional parameters which are passed to a query.
Problem is that query is a list of all parameters and if one of them is None I get an error. Can the variables which are empty just to be completely skipped?
Some code:
val param01 = BSONDocument(/*Query parametters*/)
val param02 = BSONDocument()//Causes an error
val query = BSONDocument(BSONArray(param01, param02))
P.S. Matching all the prams to Some or None is not an option there are way to many of them