I'm trying to run the following GooleCould's BigQuery:
select REGEXP_REPLACE(SPLIT(site, "=")[OFFSET(1)], r'%\d+', ' ')
from some_db
where site = 'something'
and STARTS_WITH(site, 'XXX')
and during the execution I get the following error:
Array index 1 is out of bounds (overflow)
When I was working with AWS Athena, I used to solve such errors using try statements, but I could not find anything equivalent for BigQuery.
How should I handle exceptions?
You should use SAFE_OFFSET instead of OFFSET
select REGEXP_REPLACE(SPLIT(site, "=")[SAFE_OFFSET(1)], r'%\d+', ' ')
from some_db
where site = 'something'
and STARTS_WITH(site, 'XXX')
As of more generic try / catch question - BigQuery does not have one - but there is a SAFE prefix that can be used in most functions as SAFE.function_name() - https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#safe-prefix
Related
I have the following query in Athena, I am using S3 as data store. I have 2 query parameters in the lambda expression.
PREPARE my_select1 FROM
SELECT address.phonenumbers FROM \"nested-query-db\".\"data_with_json1\" where
cardinality(filter(address.phonenumbers,js->js.type = ? and js.number = ?)) >0 and
cardinality(filter(address.phonenumbers,js->js.type = 'city'and js.number = '4')) > 0 and
firstname ='Emily';
When I execute it using
EXECUTE my_select1 USING 'Home', '1';
It throws the following error.
java.lang.RuntimeException: Query Failed to run with Error Message: SYNTAX_ERROR: line 1:1:
Sample Data:
{"firstname":"Emily","address":{"streetaddress":"101","city":"abc","state":"","phonenumbers":[{"type":"home","number":"11"},{"type":"city","number":"4"}]}}
I think the problem can be caused by the following.
You are probably using Athena engine version 2, which is based on Presto, explicitly Presto version 0.217.
The PrestoDB Github repository identifies a very similar problem like the one you described in this issue.
The issue was addressed in PrestoDB by this pull request, and included in the release 0.256.
This means the fix is not included in Athena.
I am not sure if it will work, but to solve the issue you probably could try using Athena engine version 3, which is based on Trino instead.
#jccampanero explain lambda expression does not support in Athena 2.
But I have a workaround for you.
This query works well, according your needs.
PREPARE my_select1 FROM
SELECT
address.phonenumbers
FROM address
CROSS JOIN UNNEST(address.phonenumbers) as t(phone)
where
phone.type = ? and phone.number = ? and
cardinality(filter(address.phonenumbers, js -> js.type = 'city' and js.number = '4')) > 0 and
firstname = 'Emily';
EXECUTE my_select1 USING 'home', '11'
I am trying to query for all logs that meets a simple condition:
The jsonPayload of some DEFAULT log entries have the following structure:
response: {
Values: [
[ ]
]
}
where each item in Values is an array. In most cases, Values have a single item "" in the array (empty). I want to write a query that can filter all logs entry that have values different from an empty string (an array in fact).
Here's the query I tried to run:
severity="DEFAULT" AND
jsonPayload.response.Values != ''
This did not return any result. There are thousands of entries, most of which are empty. Can this be done? If so, what am I missing in this case?
Edit
I am checking to see if the first value inside Values is something other than an empty string. In entries I am looking for, the value of the first item will be an array.
Edit 2
Following the reference suggested, I tried looking for the opposite:
severity="DEFAULT" AND
jsonPayload.response.Values = ''
This shows me the all the results with empty Values array as expected. What's confusing me is why it's not working. The logs are generated by a cloud function that serves as a webhook for event processing. The jsonPayload represents the body of the request from the event source.
To filter non-empty strings in Google Cloud Logs Explorer as seen in the official documentation:
severity="DEFAULT" AND
jsonPayload.response.Values !~ ''
Another way would be:
severity="DEFAULT" AND
jsonPayload.response.Values:*
NOT jsonPayload.response.Values = ''
I have been struggling with this simple query in django. I have checked a lot over internet. I tried using similar syntax - yet no luck.
In my application on html page I need to display some specific record. And for that i want to use parameterized select statement. But the query is not working..
I have checked id2 is getting correct value from previous html page..
And I know I can use APIs but for my databases classes I need to use raw queries in my application.
Can someone please help me here...
def details(request, id2):
temp = 'test3'
data = Posts.objects.raw('''select * from posts_posts where posts_posts.id = %s ''', id2)
context ={
'post' : data,
If you run that code you will see an error, since that is not the correct format for a call to raw. If you can't see the error output anywhere, then you have yet another problem for another post.
The correct format for raw is: .raw(sql, iterable-values), like this:
posts = Posts.objects.raw("select * ..", [id2, ]) # or use (id2,) for a tuple
<RawQuerySet: select * from ... where id = 5>
To get the actual list, since that just gives you a Query, you need to evaluate it somehow:
posts_list = list(posts)
first_post = posts[0]
Be careful, if you don't evaluate the QuerySet then it can be re-run a second time. Please convert it to a list() before doing further operations on it.
Having WSO2AM 2.1.0-update12 (with mysql) with following use case
in the API Store we crete an application
assign code grant permission with a callback url
save
edit application
Updating the application keys will fail anyway (this stack is from using the default h2 database)
Caused by: org.h2.jdbc.JdbcSQLException: Syntax error in SQL statement " SELECT AKM.CONSUMER_KEY FROM AM_APPLICATION APP, AM_APPLICATION_KEY_MAPPING AKM, AM_SUBSCRIBER SUB WHERE SUB.SUBSCRIBER_ID=APP.SUBSCRIBER_ID AND APP.APPLICATION_ID = AKM.APPLICATION_ID AND APP.NAME = ? AND AKM.KEY_TYPE=? AND ( (APP.APPLICATION_ID IN (SELECT APPLICATION_ID FROM AM_APPLICATION_GROUP_MAPPING WHERE GROUP_ID IN ($PARAMS[*]) AND TENANT = ?)) OR LOWER(SUB.USER_ID) = ?) "; expected "[, ::, *, /, %, +, -, ||, ~, !~, NOT, LIKE, REGEXP, IS, IN, BETWEEN, AND, OR, ,, )"; SQL statement:
SELECT AKM.CONSUMER_KEY FROM AM_APPLICATION APP, AM_APPLICATION_KEY_MAPPING AKM, AM_SUBSCRIBER SUB WHERE SUB.SUBSCRIBER_ID=APP.SUBSCRIBER_ID AND APP.APPLICATION_ID = AKM.APPLICATION_ID AND APP.NAME = ? AND AKM.KEY_TYPE=? AND ( (APP.APPLICATION_ID IN (SELECT APPLICATION_ID FROM AM_APPLICATION_GROUP_MAPPING WHERE GROUP_ID IN ($params) AND TENANT = ?)) OR lower(SUB.USER_ID) = ?) [42001-175]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:332)
set a visibility group and save
now the application seems to be working, but the code grant is "unchecked" and it is not possible to set it again
org.wso2.carbon.apimgt.api.APIManagementException: Error when reading the application information from the persistence store.
at org.wso2.carbon.apimgt.impl.dao.ApiMgtDAO.handleException(ApiMgtDAO.java:6399)
at org.wso2.carbon.apimgt.impl.dao.ApiMgtDAO.getConsumerKeyForApplicationKeyType(ApiMgtDAO.java:4799)
at org.wso2.carbon.apimgt.impl.APIConsumerImpl.updateAuthClient(APIConsumerImpl.java:3313)
at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.updateAuthClient(UserAwareAPIConsumer.java:34)
...
Caused by: java.sql.SQLException: Parameter index out of range (9 > number of parameters, which is 4).
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
Is there any way to work around or compensate? (not to loose the code grant)
Note: we've created a few extensions (userstore manager, group extractor, AuthorizationHandler) and some are not compatible with newer versions
task = datastore.Entity(client.key('ModelDataTest', prod_id))
task.update({
'ProductId': '1234',
'ListOfRankedRelevantItems.ProductId': ['345', '456','567'],
'ListOfRankedRelevantItems.SimilarityScore': ['0.98', '0.89','0.77']
})
client.put(task)
Using the above code I'm creating an entity in GC-datastore
however I'm getting prod_id as blob like 'MTIzNDU=' instead of string '1234' and ProductId as ["MzIx","MzQ1","NDU2"] instead of ['345', '456','567'] and SimilarityScore as ["MC45OA==","MC44Nw==","MC43Nw=="] instead of ['0.98', '0.89','0.77'] any one having any idea why so and how to get the proper values.
The issue is that in python 2 string is represented as bytestream and you have to convert it to unicode to see the results properly in gcloud datastore.
Simply use unicode() method to solve this issue.