Maybe someone get this Dataflow BigQuery name value error? - google-cloud-platform

Creating new job from template in Dataflow and than it comes to writing BigQuery table dataset ID I get this error:
Error: value must be of the form ".+:.+..+"
Try to google it, but didn't find anything, maybe it some new rule, because before I create the same way, but never get this error.
Maybe someone know what does it mean?

As proposed by #kiranMathew in the above comment, you have to propose the table with following convention :
project:dataset.table
The error message indicates this and you can't add special characters like + .

Related

PowerBI subscription error : there is no data for the field at position x

We have run a PowerBI subscription to generate visualisations report in PDF format we have get many errors like this
There is no data for the field at position x
The problem is we searched many times about it we found that it may occurred due to missing data in dataset.
But we have about 30 datasets with a query to oracle database we cannot figure out which is the missing data and the log does not mention which report causes the error.
Is there a way to figure out which field is missing?
Or is there a way to enrich the reports error log to give us which report failed?
A sample of exact error is repeated with different positions :
processing!ReportServer_0-8!1e18!02/07/2022-09:56:36:: e
ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: , Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: There is no data for the field at position 29.;
Dears
I found a solution help me. I will share it.
The error is due to missing data, not missing values, which means the column name defined in the data set field has been changed in the database.
note:
When make the value equals null it will not give the same error; even if it is used in the report it will give a different error.
about how to detect ?
Simply install report builder in machine has connection to this database and open this report with report builder and make verify fields, it will give detailed error with the name of dataset fields not found so we tracked it in database we found it has been changed so fix it in either dataset or column name in database it fix the issue.
New challenge we are going to handle it either column name exist or changed to e, never get error and give empty report better as there is some database the report will connect may not have the same column names so it should give empty part of report instead of error.
thanks BR,

We can’t parse this SQL syntax shows up in AWS Quicksight after applying toString()

I have a calculated field which computes a total based on a particular type:
sumIf(amount, type = "sale")
Now I'm trying to convert the result to string and then concatenate some text to it, but doing toString(sumIf(amount, type = "sale")) gives the following message:
We can’t parse this SQL syntax. If you are using custom SQL, verify the syntax and try again. Otherwise, contact support.
Is there any way to make this work?
Did you try using the correct bracket type? i.e.
toString(sumIf({amount}, {type} = "sale"))
I tried an example like this and that worked fine, Quicksight can have issues when calling fields without the right brackets.

How to see 'full' SQL Error Messages in BigQuery?

I am writing a large MERGE statement in BigQuery.
When I attempt to run this query the validator gives me an error involving a lot of ...'s that hides the useful information as shown below:
Value has type ARRAY<STRUCT<eventName STRING, eventUUID STRING, eventDate DATE, ...>> which cannot be inserted into column Events, which has type ARRAY<STRUCT<eventName STRING, eventUUID STRING, eventDate DATE, ...>> at [535:1]
I am extremely confident these two array objects match exactly, however since I am struggling to get around this I would love to see the full error message.
Is there any way to see the full error?
I have looked into the Google Logging tool and cannot see any additional information.
I have also tried the following Cloud Shell command:
bq --format=prettyjson show -j [Job Id Goes Here]
Again, this seems to provide no additional information.
This approach feels pretty silly but it could be the last resort for really long nest type.
Use INFORMATION_SCHEMA.COLUMNS to get a full string of the target type, in your case, type of column Events.
Use CREATE TABLE <yourDataset>.<yourTempTable> AS SELECT ... to dump one row of the Value into a table. Use 1) again to see its full type string.

Why doesn't the parameter index work?

In the documentation, https://docs.spring.io/spring-data/neo4j/docs/current/reference/html/
it uses {0} to reference the parameter 'movieTitle'.
#Query("MATCH (movie:Movie {title={0}}) RETURN movie")
Movie getMovieFromTitle(String movieTitle);
However, in my own code, if I use "{title={0}", my IntelliJ always reports a syntax error. I can resolve the issue by changing it to
{title:{movieTitle}
Here I have to use the actual argument name and the colon plus {}.
Is there any trick for this? I don't think the documentation is wrong.
Question 2:
If I want the node label "Movie" to be a parameter, it also shows an error message:
#Query("MATCH (movie:{label} {title={0}}) RETURN movie")
Movie getMovieFromTitle(String movieTitle, String label);
I do not know what version of IntelliJ you are using but the first query is right. There is also a test case for this in the spring-data-neo4j project.
It is not possible to use the second query syntax because there is no support for this on the database level where the query gets executed. If it would be supported in SDN before making the call to the DB the query has to be parsed (and the pattern replaced) every time when the query get executed and SDN will loose the possibility to parse the query once and then just add the parameter values in subsequent calls. This will lower the performance of executing annotated query functions.

Amazon Athena: no viable alternative at input

While creating a table in Athena; it gives me following exception:
no viable alternative at input
hyphens are not allowed in table name.. ( though wizard allows it ) .. Just remove hyphen and it works like a charm
Unfortunately, at the moment the syntax validation error messages are not very descriptive in Athena, this error may mean "almost" any possible syntax errors on the create table statement.
Although this is annoying at the moment you will need to check if the syntax follows the Create table documentation
Some examples are:
Backticks not in place (as already pointed out)
Missing/extra commas (remember that the last column doesn't need the comma after column definition
Missing spaces
More ..
This error generally occurs when the syntax of DDL has some silly errors.There are several answers that explain different errors based on there state.The simple solution to this problem is to patiently look into DDL and verify following points line by line:-
Check for missing commas
Unbalanced `(backtick operator)
Incompatible datatype not supported by HIVE(HIVE DATA TYPES REFERENCE)
Unbalanced comma
Hypen in table name
In my case, it was because of a trailing comma after the last column in the table. For example:
CREATE EXTERNAL TABLE IF NOT EXISTS my_table (
one STRING,
two STRING,
) LOCATION 's3://my-bucket/some/path';
After I removed the comma at the end of two STRING, it worked fine.
My case: it was an external table and the location had a typo (hence didn't exist)
Couple of tips:
Click the "Format query" button so you can spot errors easily
Use the example at the bottom of the documentation - it works - and modify it with your parameters: https://docs.aws.amazon.com/athena/latest/ug/create-table.html
Slashes. Mine was slashes. I had the DDL from Athena, saved as a python string.
WITH SERDEPROPERTIES (
'escapeChar'='\\',
'quoteChar'='\"',
'separatorChar'=',')
was changed to
WITH SERDEPROPERTIES (
'escapeChar'='\',
'quoteChar'='"',
'separatorChar'=',')
And everything fell apart.
Had to make it:
WITH SERDEPROPERTIES (
'escapeChar'='\\\\',
'quoteChar'='\\\"',
'separatorChar'=',')
In my case, it was an extra comma in PARTITIONED BY section,
In my case, I was missing the singlequotes for the S3 URL
In my case, it was that one of the table column names was enclosed in single quotes, as per the AWS documentation :( ('bucket')
As other users have noted, the standard syntax validation error message that Athena provides is not particularly helpful. Thoroughly checking the required DDL syntax (see HIVE data types reference) that other users have mentioned can be pretty tedious since it is fairly extensive.
So, an additional troubleshooting trick is to let AWS's own data parsing engine (AWS Glue) give you a hint about where your DDL may be off. The idea here is to let AWS Glue parse the data using its own internal rules and then show you where you may have made your mistake.
Specifically, here are the steps that worked for me to troubleshoot my DDL statement, which was giving me lots of trouble:
create a data crawler in AWS Glue; AWS and lots of other places go through the very detailed steps this requires so I won't repeat it here
point the crawler to the same data that you wanted (but failed) to upload into Athena
set the crawler output to a table (in an Athena database you've already created)
run the crawler and wait for the table with populated data to be created
find the newly-created table in the Athena Query Editor tab, click on the three vertical dots (...), and select "Generate Create Table DLL":
this will make Athena create the DLL for this table that is guaranteed to be valid (since the table was already created using that DLL)
take a look at this DLL and see if/where/how it differs from the DLL that you originally wrote. Naturally, this automatically-generated DLL will not have the exact choices for the data types that you may find useful, but at least you will know that it is 100% valid
finally, update your DLL based on this new Glue/Athena-generated-DLL, adjusting the column/field names and data types for your particular use case
After searching and following all the good answers here.
My issue was that working in Node.js i needed to remove the optional
ESCAPED BY '\' used in the Row settings to get my query to work. Hope this helps others.
Something that wasn't obvious for me the first time I used the UI is that if you get an error in the create table 'wizard', you can then cancel and there should be the query used that failed written in a new query window, for you to edit and fix.
My database had a hypen, so I added backticks in the query and rerun it.
This happened to me due to having comments in the query.
I realized this was a possibility when I tried the "Format Query" button and it turned the entire thing into almost 1 line, mostly commented out. My guess is that the query parser runs this formatter before sending the query to Athena.
Removed the comments, ran the query, and an angel got its wings!