What should be the proper format to provide Google storage image link while images.insert ?
Image file located at GS Bucket as *.tar.gz.
Creating new image with google client library in python with api images.insert(body=body,project=project)
my body config looks like:
body = { "name":"test", "sourceImage":"https://console.cloud.google.com/storage/browser/[BUCKET]/[IMAGEFILE]",}
Procedure fails with following error:
googleapiclient.errors.HttpError: <HttpError 400 when requesting returned "Invalid value for field 'resource.sourceImage': 'https://storag
e.cloud.google.com/[BUCKET]/[IMAGEFILE]'. The URL is malformed.">
In order to access an image from Google storage, please use “rawDisk” instead. Here is an example:
"name": "image-1",
"rawDisk": {
"source": "https://storage.googleapis.com/[bucket]/[imagefile]"
}
Where [bucket] is the name of your bucket and [imagefile] = *.tar.gz
Related
I've observed for failed multipart upload (like crash or stop in the middle), the partially uploaded object still exist in storage.
I want to configure lifecycle rules for these incomplete objects via either minio or S3 C++ SDk.
I want to configure something like
{
"ID": "bucket-lifecycle-incomplete-chunk-upload",
"Status": "Enabled",
"NoncurrentVersionExpiration": {
"NoncurrentDays": 1
},
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 1
}
},
My C++ code looks like the following:
Aws::S3::Model::AbortIncompleteMultipartUpload incomplete_upload_config;
incomplete_upload_config.SetDaysAfterInitiation(days);
Aws::S3::Model::NoncurrentVersionExpiration version_expire;
version_expire.SetNoncurrentDays(1);
auto status = Aws::S3::Model::ExpirationStatus::Enabled;
Aws::S3::Model::LifecycleRule rule;
rule.SetID("bucket-lifecycle-incomplete-chunk-upload");
rule.SetStatus(std::move(status));
rule.SetNoncurrentVersionExpiration(std::move(version_expire));
rule.SetAbortIncompleteMultipartUpload(std::move(incomplete_upload_config));
Aws::S3::Model::BucketLifecycleConfiguration bkt_config;
bkt_config.AddRules(std::move(rule));
Aws::S3::Model::PutBucketLifecycleConfigurationRequest config_req{};
config_req.SetBucket(bucket);
config_req.SetLifecycleConfiguration(std::move(bkt_config));
auto outcome = client->PutBucketLifecycleConfiguration(config_req);
And I get the following result:
Received HTTP return code: 400; Failed to update config for bucket <bucket-name> because MalformedXML: Unable to parse ExceptionName: MalformedXML Message:
The pain point for this error is: I cannot find which additional or missing fields lead to this error.
From documentation on https://developers.google.com/vault/guides/exports, I've been able to create, list, and retrieve exports, but I haven't found any way to download the exported data associated with a specific export. Is there any way to download the exported files via the API, or is this only available through the vault UI?
There is a cloudStorageSink key in the export metadata, but trying to use the values provided using the cloud storage API results in a generic permissions issue (403 Error).
There is error I got:
com.google.cloud.storage.StorageException: gcsadmin#gmail-acess-347301.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object.
71e6f8ba-dc92-494f-b584-1a9675074c0b/exportly-f1c1ecd8-254c-4078-b0c2-5228905720d3/My_first_mail_accounts_export-metadata.csv 4d17606f-0ebc-46e7-8369-d8e4ffd12b44
at com.google.cloud.storage.StorageException.translate(StorageException.java:118)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:287)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:512)
at com.google.cloud.storage.StorageImpl.lambda$get$6(StorageImpl.java:285)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.Retrying.run(Retrying.java:54)
at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1406)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:284)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:290)
at com.q1d.googlevaulttest.Quickstart.downloadObject(Quickstart.java:159)
at com.q1d.googlevaulttest.Quickstart.main(Quickstart.java:215)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://storage.googleapis.com/storage/v1/b/4d17606f-0ebc-46e7-8369-d8e4ffd12b44/o/71e6f8ba-dc92-494f-b584-1a9675074c0b%2Fexportly-f1c1ecd8-254c-4078-b0c2-5228905720d3%2FMy_first_mail_accounts_export-1.zip?projection=full
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "gcsadmin#gmail-acess-347301.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object.",
"reason" : "forbidden"
} ],
"message" : "gcsadmin#gmail-acess-347301.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:509)
... 10 more
As follow other user guide, I was trying to download the object not whole bucket, Here is the code I am using:
(gcs.json key file is the service account I created in Google cloud which have domain-wide delegation)
public static void downloadObject(String projectId, String bucketName, String objectName,
String destFilePath) throws IOException {
// The ID of your GCP project
// String projectId = "your-project-id";
// The ID of your GCS bucket
// String bucketName = "your-unique-bucket-name";
// The ID of your GCS object
// String objectName = "your-object-name";
// The path to which the file should be downloaded
// String destFilePath = "/local/path/to/file.txt";
// Load client secrets.
InputStream in = Quickstart.class.getResourceAsStream("/gcs.json");
if (in == null) {
throw new FileNotFoundException("Resource not found: " + "gcs.json");
}
Storage storage = StorageOptions.newBuilder().setCredentials(ServiceAccountCredentials.fromStream(in)).build()
.getService();
Blob blob = storage.get(BlobId.of(bucketName, objectName));
//Blob blob = storage.get(BlobId.fromGsUtilUri(objectURI));
blob.downloadTo(Paths.get(destFilePath));
System.out
.println("Downloaded object " + objectName + " from bucket name " + bucketName + " to " + destFilePath);
}
This part of the error indicates how to solve the problem:
gcsadmin#gmail-acess-347301.iam.gserviceaccount.com does not have
storage.objects.get access to the Google Cloud Storage object
The solution is to add a role that has the required permission. For example:
Storage Object Viewer (roles/storage.objectViewer)
The role must be added to the service account:
gcsadmin#gmail-acess-347301.iam.gserviceaccount.com
gcloud projects add-iam-policy-binding REPLACE_WITH_YOUR_PROJECT_ID \
--member='serviceAccount:gcsadmin#gmail-acess-347301.iam.gserviceaccount.com' \
--role='roles/storage.objectViewer'
Documentation for the command:
gcloud projects add-iam-policy-binding
IAM roles for Cloud Storage
Understanding roles
I am trying to upload a binary Image to S3 and update a DynamoDB table in the same AWS Lambda Function. The problem is, whenever I try to make an API call, I get the following error in postman:
{
"errorMessage": "'itemId'",
"errorType": "KeyError",
"requestId": "bccaead6-cb60-4a5e-9fc7-14ff25380451",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 14, in lambda_handler\n s3_upload = s3.put_object(Bucket=bucket, Key=event[\"itemId\"] + \".png\", Body=decode_content)\n"
]
}
My events section takes in 3 Strings and whenever I try and access those strings, I get this error. However, if I try and access them without trying to upload to an S3 Bucket, everything works fine. My Lambda Function looks like this:
import json
import boto3
import base64
dynamoclient = boto3.resource("dynamodb")
s3 = boto3.client("s3")
table = dynamoclient.Table("Items")
bucket = "images"
def lambda_handler(event, context):
get_file_content = event["content"]
decode_content = base64.b64decode(get_file_content)
s3_upload = s3.put_object(Bucket=bucket, Key=event["itemId"] + ".png", Body=decode_content)
table.put_item(
Item={
'itemID': event["itemId"],
'itemName': event['itemName'],
'itemDescription': event['itemDescription']
}
)
return {
"code":200,
"message": "Item was added successfully"
}
Again, if I remove everything about the S3 file upload, everything works fine and I am able to update the DynamoDB table successfully. As for the API Gateway side, I have added the image/png to the Binary Media Types section. Additionally, for the Mapping Templates section for AWS API Gateway, I have added the content type image/png. In the template for the content type, I have the following lines:
{
"content": "$input.body"
}
For my Postman POST request, in the headers section, I have put this:
Finally, for the body section, I have added the raw event data with this:
{
"itemId": "0fx170",
"itemName": "Mouse",
"itemDescription": "Smooth"
}
Lastly, for the binary section, I have uploaded my PNG file.
What could be going wrong?
I am following the steps in the SageMaker Monitoring Tutorial here:
https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_model_monitor/introduction/SageMaker-ModelMonitoring.html
And for the line:
bucket.Object(code_prefix + "/preprocessor.py").upload_file("preprocessor.py")
I get the error:
TypeError: expected string or bytes-like object
Which I dont understand, because the input to the upload_file() function is "preprocessor.py" which is a string.
To unblock you quickly, please take the sagemaker session route to upload the artifact to S3 bucket -
import sagemaker
from sagemaker import get_execution_role
# S3 bucket for saving code and model artifacts.
# Feel free to specify a different bucket and prefix
bucket = sagemaker.Session().default_bucket()
artifact_name = "preprocessor.py"
prefix = "code" // change as required
sample_url = sagemaker.Session().upload_data(
artifact_name, bucket=bucket, key_prefix=prefix"
)
I call google api when the return of "We can not access the URL currently." But the resources must exist and can be accessed.
https://vision.googleapis.com/v1/images:annotate
request content:
{
"requests": [
{
"image": {
"source": {
"imageUri": "http://yun.jybdfx.com/static/img/homebg.jpg"
}
},
"features": [
{
"type": "TEXT_DETECTION"
}
],
"imageContext": {
"languageHints": [
"zh"
]
}
}
]
}
response content:
{
"responses": [
{
"error": {
"code": 4,
"message": "We can not access the URL currently. Please download the content and pass it in."
}
}
]
}
As of August, 2017, this is a known issue with the Google Cloud Vision API (source). It appears to repro for some users but not deterministically, and I've run into it myself with many images.
Current workarounds include either uploading your content to Google Cloud Storage and passing its gs:// uri (note it does not have to be publicly readable on GCS) or downloading the image locally and passing it to the vision API in base64 format.
Here's an example in Node.js of the latter approach:
const request = require('request-promise-native').defaults({
encoding: 'base64'
})
const data = await request(image)
const response = await client.annotateImage({
image: {
content: data
},
features: [
{ type: vision.v1.types.Feature.Type.LABEL_DETECTION },
{ type: vision.v1.types.Feature.Type.CROP_HINTS }
]
})
I have faced the same issue when I was trying to call the api using the firebase storage download url (although it worked initially)
After looking around I found the below example in the api docs for NodeJs.
NodeJs example
// Imports the Google Cloud client libraries
const vision = require('#google-cloud/vision');
// Creates a client
const client = new vision.ImageAnnotatorClient();
/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// const bucketName = 'Bucket where the file resides, e.g. my-bucket';
// const fileName = 'Path to file within bucket, e.g. path/to/image.png';
// Performs text detection on the gcs file
const [result] = await client.textDetection(`gs://${bucketName}/${fileName}`);
const detections = result.textAnnotations;
console.log('Text:');
detections.forEach(text => console.log(text));
For me works only uploading image to google cloud platform and passing it to URI parameters.
In my case, I tried retrieving an image used by Cloudinary our main image hosting provider.
When I accessed the same image but hosted on our secondary Rackspace powered CDN, Google OCR was able to access the image.
Not sure why Cloudinary didn't work when I was able to access the image via my web browser, but just my little workaround situation.
I believe the error is caused by the Cloud Vision API refusing to download images on a domain whose robots.txt file blocks Googlebot or Googlebot-Image.
The workaround that others mentioned is in fact the proper solution: download the images yourself and either pass them in the image.content field or upload them to Google Cloud Storage and use the image.source.gcsImageUri field.
For me, I resolved this issue by requesting URI (e.g.: gs://bucketname/filename.jpg) instead of Public URL or Authenticated URL.
const vision = require('#google-cloud/vision');
function uploadToGoogleCloudlist (req, res, next) {
const originalfilename = req.file.originalname;
const bucketname = "yourbucketname";
const imageURI = "gs://"+bucketname+"/"+originalfilename;
const client = new vision.ImageAnnotatorClient(
{
projectId: 'yourprojectid',
keyFilename: './router/fb/yourprojectid-firebase.json'
}
);
var visionjson;
async function getimageannotation() {
const [result] = await client.imageProperties(imageURI);
visionjson = result;
console.log ("vision result: "+JSON.stringify(visionjson));
return visionjson;
}
getimageannotation().then( function (result){
var datatoup = {
url: imageURI || ' ',
filename: originalfilename || ' ',
available: true,
vision: result,
};
})
.catch(err => {
console.error('ERROR CODE:', err);
});
next();
}
I faced with the same issue several days ago.
In my case the problem happened due to using queues and call api requests in one time from the same ip. After changing the number of parallel processes from 8 to 1, the amount of such kind of errors was reduced from ~30% to less than 1%.
May be it will help somebody. I think there is some internal limits on google side for loading remote images (because as people reported, using google storage also solves the problem).
My hypothesis is that an overall (short) timeout exists on Google API side which limit the number of files that can actually be retrieved.
Sending 16 images for batch-labeling is possible but only 5 o 6 will labelled because the origin webserver hosting the images was unable to return all 16 files within <Google-Timeout> milliseconds.
In my case, the image uri that I was specifying in the request pointed at a large image ~ 4000px x 6000px. When I changed it to a smaller version of the image. The request succeeded
The very same request works for me. It is possible that the image host was temporarily down and/or had issues on their side. If you retry the request it will mostly work for you.