I wanted to know if tagging of tables is possible in aws glue in any way. I know that in documentation , it is not given that it is possible for tables. It also is given that the tagging of a database is also not possible but I was able to do so using tag_resource boto3 call as specifed in this link.
This is the error I am getting if I try same tagging a table.
botocore.errorfactory.InvalidInputException: An error occurred
(InvalidInputException) when calling the TagResource operation
Just wanted to double check to see if it is not possible or I am missing something here.
Related
I am trying to create a GCP Pub Sub BigQuery subscriptions using the console: https://cloud.google.com/pubsub/docs/bigquery
However, I get the following error message:
API returned error: ‘Request contains an invalid argument.’
Any help would be appreciated.
NOTE
When the big query table does not exist I get the following error:
2.Pub Sub schema was not deleted
Update:
Actually, despite the generic error message, turns out to be a straightforward issue.
To use the "write metadata" option the BigQuery table must previously have the following field structure:
subscription_name (string),
message_id (string)
publish_time (timestamp)
data (bytes, string or json)
attributes (string or json)
they are better described in the docs here
Once they are created, this option works fine.
I believe that problem with the "Use topic schema" is also related to the schema used in the topic, since the table must have already the same structure (but there is a need to check it in your configuration). If your topic follow an avro schema, this might help: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions
---------- previous answer
Not a definitive answer, but am having the same problem and figured out that it is somewhat related to the following options:
Use topic schema
Write metadata
Unchecking then make it work.
The same thing happens algo using terraform to try to set up the infrastructure.
I am still investigating if it is a bug or perhaps an error in my schema definition, but maybe it can be a starting point to you as well
A 404 error would indicate that either the BigQuery table does not exist or the Pub/Sub schema associated with the topic was deleted. For the former, ensure that the project, data set, and table names all match the names of the existing table to which you want to write data. For the latter, you could look at the topic details page and make sure that the schema name is not _deleted-schema_.
Beginning with a new stack I get the error message as in the title.
I am using SAM, and I am confused, why it wants to update the macro.
I thought, this macro is provided by aws and I wonder why it is requesting to modify it.
My template spins up a lambda, a database and a REST api, but does even try to touch existing macros.
My template did contain the TableName tag for a DynamoDb.
As I am aware, named tables cannot be updated, if resource replacement required. I was not trying to do updates on that resource though.
The table existed before I cloudformed that new stack though.
I am creating a JOB in AWS Glue but it shows below error in the last step.
{"service":"AWSGlue","statusCode":400,"errorCode":"InvalidInputException","requestId":"ad9ee511-adb8-11e9-9bbf-9d08424a9846","errorMessage":"No enum constant com.amazonaws.services.glue.FileFormat.UNKNOWN","type":"AwsServiceError"}
The column mapping is shown correctly as below screenshot.
what I don't understand is which input field has an invalid data? Glue doesn't give me more detailed information about this error. How can I debug this issue?
I faced the same issue, because accidentally I tried to write the data to my DynamoDB table (as my Target). When I changed to S3. It worked.
You can learn more about it on AWS Documentation
Here is the issue raised in amazon related to this.
I want to export error log , general log and slow-query log from RDS Mysql.
I am done with all the necessary settings on my DB Instance.
I have exported the general log and slow-query log to file. (log_output : FILE)
What is the best approach to do this.
I am thinking to use lambda for this. But I am not able to find a suitable way to trigger my lambda function, eg : When ever a new log file is created my lambda function must be triggered.
Is it possible to push this log files events to CloudWatch directly ?
I have gone through the documentation , but I am not able to find such mechanism.
How should I proceed ?
There is no Supported Event Source for RDS in Lambda. I would suggest using a table to store logs for each kind in the parameters. And call the rotation to remove the old data to save disk space.
When I try to download all log files from a RDS instance, in some cases, I found this error in my python output:
An error occurred (InvalidParameterValue) when calling the
DownloadDBLogFilePortion operation: This file contains binary data and
should be downloaded instead of viewed.
I manage correctly the pagination and the throttling (using The Marker parameter and the sleep function).
This is my calling:
log_page=request_paginated(rds,DBInstanceIdentifier=id_rds,LogFileName=log,NumberOfLines=1000)
rds-> boto3 resource
And this is the definition of my function:
def request_paginated(rds,**kwargs):
return rds.download_db_log_file_portion(**kwargs)
Like I said, most of time this function works but sometime it returns:
"An error occurred (InvalidParameterValue) when calling the
DownloadDBLogFilePortion operation: This file contains binary data and
should be downloaded instead of viewed"
Can you help me please? :)
UPDATE: the problem is a known issue with downloading log files that contain non printable sign. As soon as possible I will try the proposed solution provide by the aws support
LATEST UPDATE: This is an extract of my discussion with aws support team:
There is a known issue with non binary characters when using the boto based AWS cli, however this issue is not present when using the older Java based cli.
There is currently no way to fix the issue that you are experiencing while using the boto based AWS cli, the workaround is to make the API call from the Java based cli
the aws team are aware of this issue and are working on a way to resolve this, however the do not have an ETA for when this will be released.
So the solutions is: use the java API
Giuseppe
LATEST UPDATE: This is an extract of my discussion with aws support team:
There is a known issue with non binary characters when using the boto based AWS cli, however this issue is not present when using the older Java based cli.
There is currently no way to fix the issue that you are experiencing while using the boto based AWS cli, the workaround is to make the API call from the Java based cli
the aws team are aware of this issue and are working on a way to resolve this, however the do not have an ETA for when this will be released. So the solutions is: use the java API
Giuseppe
http://docs.aws.amazon.com/AmazonRDS/latest/APIReference/CommonErrors.html
InvalidParameterValue : An invalid or out-of-range value was supplied
for the input parameter.
Invalid parameter in boto means the data pass does not complied. Probably an invalid name that you specified, possible something wrong with your variable id_rds, or maybe your LogFileName, etc. You must complied with the function arguments requirement.
response = client.download_db_log_file_portion(
DBInstanceIdentifier='string',
LogFileName='string',
Marker='string',
NumberOfLines=123
)
(UPDATE)
For example, LogFileName must be te exact file name exist inside RDS instance.
For the logfile , please make sure the log file EXISTS inside the instance. Use this AWS CLI to get a quick check
aws rds describe-db-log-files --db-instance-identifier <my-rds-name>
Do check Marker (string) and NumberOfLines (Integer) as well. Mismatch type or out of range. Skip them since they are not required, then test it later.