Getting a 403 while creating a user, followed every step right so far - google-admin-sdk

Looks like I have followed every step (given that the documentation is extremely lacking, it is sourced from multiple places). This is my code:
def create_user(cred_file_location, user_first_name, user_last_name, user_email):
cred_data = json.loads(open(cred_file_location).read())
access_email = cred_data['client_email']
private_key = cred_data['private_key']
# I have tried with the scope as a single string, and also
# as an array of a single string. Neither worked
credentials = SignedJwtAssertionCredentials(access_email, private_key, ["https://www.googleapis.com/auth/admin.directory.user"])
http = Http()
http = credentials.authorize(http)
service = build('admin', 'directory_v1', http=http)
users = service.users()
userinfo = {
'primaryEmail': user_email,
'name': {
'givenName': user_first_name,
'familyName': user_last_name
},
'password': ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(80))
}
users.insert(body=userinfo).execute()
I downloaded the JSON key right, and it is loading it correctly. This is my JSON key (I am redacting certain parts of identifying information, I have kept some of it there to show that I am loading the correct info):
{
"type": "service_account",
"private_key_id": "c6ae56a9cb267fe<<redacted>>",
"private_key": "<<redacted>>",
"client_email": "account-1#<<redacted>>.iam.gserviceaccount.com",
"client_id": "10931536<<redacted>>",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account-1%40<<redacted>>.iam.gserviceaccount.com"
}
This is how these credentials look in the developer console:
I have also enabled sitewide access for the service account:
I have no clue as to why I am still getting these 403s:
File "/usr/lib/python2.7/site-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?alt=json returned "Not Authorized to access this resource/api">
Any help is greatly appreciated.

Finally, on some random stackoverflow answer, I found the solutin. I have to sub as a user to execute any request. Esentially:
credentials = SignedJwtAssertionCredentials(
access_email,
private_key,
["https://www.googleapis.com/auth/admin.directory.user"])
changes to:
credentials = SignedJwtAssertionCredentials(
access_email,
private_key,
["https://www.googleapis.com/auth/admin.directory.user"],
sub="user#example.org")
Where all requests will now be made as if they were made on behalf of user#example.org.

Related

AWS SecretsManager works in Eclipse, can't connect to Service Endpoint in ColdFusion

I have the following class written in Java using Eclipse on my Amazon EC2 instance.
import java.nio.ByteBuffer;
import com.amazonaws.auth.*;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.secretsmanager.*;
import com.amazonaws.services.secretsmanager.model.*;
public class SMtest {
public static void main(String[] args) {
String x = getSecret();
System.out.println(x);
}
public SMtest()
{
}
public static String getSecret() {
String secretName = "mysecret";
String endpoint = "secretsmanager.us-east-1.amazonaws.com";
String region = "us-east-1";
String result = "";
//BasicAWSCredentials awsCreds = new BasicAWSCredentials("mypublickey", "mysecretkey");
AwsClientBuilder.EndpointConfiguration config = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AWSSecretsManagerClientBuilder clientBuilder = AWSSecretsManagerClientBuilder.standard()
.withCredentials(new InstanceProfileCredentialsProvider(false));
//.withCredentials(new AWSStaticCredentialsProvider(awsCreds));
clientBuilder.setEndpointConfiguration(config);
AWSSecretsManager client = clientBuilder.build();
String secret;
ByteBuffer binarySecretData;
GetSecretValueRequest getSecretValueRequest = new GetSecretValueRequest()
.withSecretId(secretName).withVersionStage("AWSCURRENT");
GetSecretValueResult getSecretValueResult = null;
try {
getSecretValueResult = client.getSecretValue(getSecretValueRequest);
} catch(ResourceNotFoundException e) {
System.out.println("The requested secret " + secretName + " was not found");
} catch (InvalidRequestException e) {
System.out.println("The request was invalid due to: " + e.getMessage());
} catch (InvalidParameterException e) {
System.out.println("The request had invalid params: " + e.getMessage());
}
if(getSecretValueResult == null) {
result = "";
}
// Depending on whether the secret was a string or binary, one of these fields will be populated
if(getSecretValueResult.getSecretString() != null) {
secret = getSecretValueResult.getSecretString();
result = secret;
//System.out.println(secret);
}
else {
binarySecretData = getSecretValueResult.getSecretBinary();
result = binarySecretData.toString();
}
return result;
}
}
When I execute it from within Eclipse, it works just fine. When I compile the class to use it in ColdFusion (2021) from the same EC2 with the following code:
<cfscript>
obj = CreateObject("java","SMtest");
obj.init();
result = obj.getSecret();
</cfscript>
<cfoutput>#result#</cfoutput>
I get a "Failed to connect to service endpoint" error. I believe I have all the IAM credentials set up properly since it is working in straight Java. However, when I change the credentials to use the Basic Credentials with my AWS Public and Secret Key (shown in comments in the code above), it works in both Java and ColdFusion.
I created an AWS Policy that manages the Secrets Manager permissions. I also added AmazonEC2FullAccess. I also tried to create a VPC Endpoint, but this had no effect.
Why would it be working in Java and not in ColdFusion (which is based on Java) ? What roles/policies would I have to add to get it to work in ColdFusion when it is already working in Java?
STACK TRACE ADDED:
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:165)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.doInvoke(AWSSecretsManagerClient.java:2454)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.invoke(AWSSecretsManagerClient.java:2421)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.invoke(AWSSecretsManagerClient.java:2410)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.executeGetSecretValue(AWSSecretsManagerClient.java:943)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.getSecretValue(AWSSecretsManagerClient.java:912)
at SMtest.getSecret(SMtest.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at coldfusion.runtime.java.JavaProxy.invoke(JavaProxy.java:106)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:4254)
at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:4217)
at cfsmtest2ecfm1508275519.runPage(D:\ColdFusion2021\cfusion\wwwroot\smtest.cfm:4)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:257)
at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:749)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:578)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:573)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
at coldfusion.filter.LicenseFilter.invoke(LicenseFilter.java:30)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:231)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at coldfusion.inspect.weinre.MobileDeviceDomInspectionFilter.doFilter(MobileDeviceDomInspectionFilter.java:57)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.MalformedURLException: Unknown protocol: http
at java.base/java.net.URL.<init>(URL.java:708)
at java.base/java.net.URL.fromURI(URL.java:748)
at java.base/java.net.URI.toURL(URI.java:1139)
at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:83)
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)
... 80 more
Caused by: java.lang.IllegalStateException: Unknown protocol: http
at org.apache.felix.framework.URLHandlersStreamHandlerProxy.parseURL(URLHandlersStreamHandlerProxy.java:373)
at java.base/java.net.URL.<init>(URL.java:703)
... 84 more
After noticing that the bottom end of the Stack Trace says "unknown protocol: http" and "Illegal State Exception: Unknown Protocol", I changed the endpoint in my class above to:
String endpoint = "https://secretsmanager.us-east-1.amazonaws.com";
and I am running the web-site via https now. It still works with no problem running it in Eclpse, and I am still getting the same error when using ColdFusion (i.e., when using a browser)
Since the basic credentials are working and since the error is not access denied, it suggests there is no issue with your IAM setup. Instead, likely your process failed to fetch the EC2 instance metadata. For example, timeout when calling the metadata endpoint, hence the Failed to connect to service endpoint error.
One way around this is to retrieve the accessKey and secretKey manually by calling the instance metadata API. For example, using cfhttp to populate variables.
Example curl command from docs: retrieve /api/token and use it to retrieve /meta-data/iam/security-credentials/<rolename>.
[ec2-user ~]$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
Output:
{
"Code" : "Success",
"LastUpdated" : "2012-04-26T16:39:16Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAIOSFODNN7EXAMPLE",
"SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"Token" : "token",
"Expiration" : "2017-05-17T15:09:54Z"
}
From here, if you still face any errors, at least it should be more explicit to you which step has failed and why.
Just a note, AWS recommends caching the credentials until near expiry instead of querying for every transaction to avoid throttling.

Uploading a zip from from Blazor WASM app to AWS-S3 - tutorials followed and no errors observed, but also no file in the bucket

I have a Blazor WASM app (i.e., client side) that users upload some data to. After some sanity and security checks on the data, I would like to upload these to a zip file in AWS S3.
Currently I have constructed the zip file as follows
using (var outStream = new MemoryStream())
{
using (var archive = new ZipArchive(outStream, ZipArchiveMode.Create, true))
{
foreach (var file in imagelist.Files)
{
json_name = file.Name + ".json";
var file_in_archive = archive.CreateEntry(file.Name, CompressionLevel.Optimal);
using (var entryStream = file_in_archive.Open())
{
file.Data.CopyToAsync(entryStream);
}
file_in_archive = archive.CreateEntry(json_name, CompressionLevel.Optimal);
using (var entryStream = file_in_archive.Open())
{
formMems[file_counter].CopyToAsync(entryStream);
}
file_counter = file_counter + 1;
}
}
Console.WriteLine("Finished zipping");
uploader.UploadZipFile(outStream, "test_title.zip");
}
And I am trying to upload the zip file via this method uploader.UploadZipFile - the code is given below
public class UploadZip
{
public BasicAWSCredentials credentials = new BasicAWSCredentials("access key", "secret key");
public void UploadZipFile(Stream zip_file, string file_name)
{
zip_file.Seek(0, SeekOrigin.Begin);
var config = new AmazonS3Config
{
RegionEndpoint = Amazon.RegionEndpoint.Region
};
using var client = new AmazonS3Client(credentials, config);
var uploadRequest = new TransferUtilityUploadRequest
{
InputStream = zip_file,
Key = file_name,
BucketName = "Bucket-Name",
CannedACL = S3CannedACL.Private
};
var fileTransferUtility = new TransferUtility(client);
fileTransferUtility.UploadAsync(uploadRequest);
Console.WriteLine("finshed uploading");
}
}
(uploader is an instance of UploadZip). I have removed the bucket name, region, access key and secret key for obvious reasons, but these are correct.
The Access Key and the Secret Key belong to a custom IAM user I made just for this purpose. The Policy summary is given below
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::Bucket-Name/*"
}
]
}
And the S3 bucket has Block *all* public access turned off, the CORS is given below
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]
The ACL grants the Bucket owner (your AWS account) List and Write for Objects, and Read and Write for Bucket ACL, with nothing else for the other 3 grantees. There is no bucket ploicy.
I simply don't understand why this isn't working. When I run the program I see no errors and the two Console.write() calls run fine.
edit
In response to advice from a comment, I tried running the code with Upload rather than UploadAsync - see below
public class UploadZip
{
public BasicAWSCredentials credentials = new BasicAWSCredentials("AKIAZWFJ7CFNER3WG5EP", "EE9eMD/vz8G3Ui3/MZdGDNBY9xv4+/3y0VQAhNzn");
public void UploadZipFile(Stream zip_file, string file_name)
{
zip_file.Seek(0, SeekOrigin.Begin);
var config = new AmazonS3Config
{
RegionEndpoint = Amazon.RegionEndpoint.EUWest2
};
using var client = new AmazonS3Client(credentials, config);
var uploadRequest = new TransferUtilityUploadRequest
{
InputStream = zip_file,
Key = file_name,
BucketName = "web-pupil-data-store",
CannedACL = S3CannedACL.Private
};
var fileTransferUtility = new TransferUtility(client);
fileTransferUtility.Upload(uploadRequest);
Console.WriteLine("finshed uploading");
}
}
This still completes the zipping of the files, but before printing finished uploading, it now throws an error with a large and quite inpenetrable (for me) stack trace, copied here
blazor.webassembly.js:1 crit: Microsoft.AspNetCore.Components.WebAssembly.Rendering.WebAssemblyRenderer[100]
Unhandled exception rendering component: Cannot start threads on this runtime.
System.NotSupportedException: Cannot start threads on this runtime.
at (wrapper managed-to-native) System.Threading.Thread.Thread_internal(System.Threading.Thread,System.MulticastDelegate)
at System.Threading.Thread.StartInternal (System.Object principal, System.Threading.StackCrawlMark& stackMark) <0x47bb958 + 0x00008> in <filename unknown>:0
at System.Threading.Thread.Start (System.Threading.StackCrawlMark& stackMark) <0x47bb818 + 0x0004e> in <filename unknown>:0
at System.Threading.Thread.Start () <0x47bb6d8 + 0x0000e> in <filename unknown>:0
at Amazon.Runtime.Internal.Util.BackgroundDispatcher`1[T]..ctor (System.Action`1[T] action) [0x0005c] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Util\Dispatcher.cs:55
at Amazon.Runtime.Internal.Util.BackgroundInvoker..ctor () [0x00000] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Util\Dispatcher.cs:180
at Amazon.Util.AWSSDKUtils.get_Dispatcher () [0x00007] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Util\AWSSDKUtils.cs:674
at Amazon.Util.AWSSDKUtils.InvokeInBackground[T] (System.EventHandler`1[TEventArgs] handler, T args, System.Object sender) [0x0004c] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Util\AWSSDKUtils.cs:661
at Amazon.Runtime.Internal.StreamReadTracker.ReadProgress (System.Int32 bytesRead) [0x00047] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\StreamReadTracker.cs:55
at Amazon.Runtime.Internal.Util.EventStream.ReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count, System.Threading.CancellationToken cancellationToken) [0x0009c] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Util\EventStream.cs:138
at Amazon.Runtime.Internal.Util.ChunkedUploadWrapperStream.FillInputBufferAsync (System.Threading.CancellationToken cancellationToken) [0x00090] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Util\ChunkedUploadWrapperStream.cs:189
at Amazon.Runtime.Internal.Util.ChunkedUploadWrapperStream.ReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count, System.Threading.CancellationToken cancellationToken) [0x00053] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Util\ChunkedUploadWrapperStream.cs:163
at System.Threading.Tasks.ValueTask`1[TResult].get_Result () <0x370ff80 + 0x00034> in <filename unknown>:0
at System.IO.Stream.CopyToAsyncInternal (System.IO.Stream destination, System.Int32 bufferSize, System.Threading.CancellationToken cancellationToken) <0x36ee098 + 0x00134> in <filename unknown>:0
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore (System.Threading.Tasks.Task serializeToStreamTask, System.IO.MemoryStream tempBuffer) <0x49ea648 + 0x00110> in <filename unknown>:0
at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult] (System.Threading.Tasks.Task waitTask, TState state, System.Func`2[T,TResult] returnFunc) <0x49eba60 + 0x000c2> in <filename unknown>:0
at System.Net.Http.WebAssemblyHttpHandler.doFetch (System.Threading.Tasks.TaskCompletionSource`1[TResult] tcs, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) <0x47cfe90 + 0x0038c> in <filename unknown>:0
at System.Net.Http.WebAssemblyHttpHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) <0x47af880 + 0x00174> in <filename unknown>:0
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered (System.Threading.Tasks.Task`1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) <0x49ef528 + 0x00134> in <filename unknown>:0
at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync (System.Threading.CancellationToken cancellationToken) [0x0003d] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_netstandard\HttpRequestMessageFactory.cs:520
at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) [0x00201] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:183
at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) [0x00036] in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:59
at Amazon.Runtime.Internal.U
f.printErr # blazor.webassembly.js:1
f.preRun.push.window.Blazor._internal.dotNetCriticalError # blazor.webassembly.js:1
_mono_wasm_invoke_js_unmarshalled # dotnet.3.2.0.js:1
do_icall # 00755c3a:0x10f924
do_icall_wrapper # 00755c3a:0x50b6a
interp_exec_method # 00755c3a:0x2588e
interp_runtime_invoke # 00755c3a:0xf7391
mono_jit_runtime_invoke # 00755c3a:0xddb3d
do_runtime_invoke # 00755c3a:0x3ba85
mono_runtime_try_invoke # 00755c3a:0xcfdb
try_invoke_perform_wait_callback # 00755c3a:0xb62e0
worker_callback # 00755c3a:0x9a7a3
fire_tp_callback # 00755c3a:0x105481
mono_background_exec # 00755c3a:0xeb5ee
Module._mono_background_exec # dotnet.3.2.0.js:1
pump_message # dotnet.3.2.0.js:1
setTimeout (async)
_schedule_background_exec # dotnet.3.2.0.js:1
mono_threads_schedule_background_job # 00755c3a:0x14dbd
ves_icall_System_Threading_ThreadPool_RequestWorkerThread # 00755c3a:0xa0025
ves_icall_System_Threading_ThreadPool_RequestWorkerThread_raw # 00755c3a:0x9fecf
do_icall # 00755c3a:0x10f790
do_icall_wrapper # 00755c3a:0x50b6a
interp_exec_method # 00755c3a:0x2588e
interp_runtime_invoke # 00755c3a:0xf7391
mono_jit_runtime_invoke # 00755c3a:0xddb3d
do_runtime_invoke # 00755c3a:0x3ba85
mono_runtime_try_invoke # 00755c3a:0xcfdb
mono_runtime_invoke # 00755c3a:0x44b39
mono_wasm_invoke_method # 00755c3a:0xca6a7
Module._mono_wasm_invoke_method # dotnet.3.2.0.js:1
call_method # dotnet.3.2.0.js:1
(anonymous) # dotnet.3.2.0.js:1
endInvokeJSFromDotNet # blazor.webassembly.js:1
(anonymous) # blazor.webassembly.js:1
Promise.then (async)
beginInvokeJSFromDotNet # blazor.webassembly.js:1
_mono_wasm_invoke_js_marshalled # dotnet.3.2.0.js:1
do_icall # 00755c3a:0x10f8b1
do_icall_wrapper # 00755c3a:0x50b6a
interp_exec_method # 00755c3a:0x2588e
interp_runtime_invoke # 00755c3a:0xf7391
mono_jit_runtime_invoke # 00755c3a:0xddb3d
do_runtime_invoke # 00755c3a:0x3ba85
mono_runtime_invoke_checked # 00755c3a:0x96a2
mono_runtime_try_invoke_array # 00755c3a:0x66c26
ves_icall_InternalInvoke # 00755c3a:0xb44ff
ves_icall_InternalInvoke_raw # 00755c3a:0xb3ffd
do_icall # 00755c3a:0x10f8b1
do_icall_wrapper # 00755c3a:0x50b6a
interp_exec_method # 00755c3a:0x2588e
interp_runtime_invoke # 00755c3a:0xf7391
mono_jit_runtime_invoke # 00755c3a:0xddb3d
do_runtime_invoke # 00755c3a:0x3ba85
mono_runtime_try_invoke # 00755c3a:0xcfdb
mono_runtime_invoke # 00755c3a:0x44b39
mono_wasm_invoke_method # 00755c3a:0xca6a7
Module._mono_wasm_invoke_method # dotnet.3.2.0.js:1
call_method # dotnet.3.2.0.js:1
(anonymous) # dotnet.3.2.0.js:1
beginInvokeDotNetFromJS # blazor.webassembly.js:1
s # blazor.webassembly.js:1
e.invokeMethodAsync # blazor.webassembly.js:1
(anonymous) # blazor.webassembly.js:1
t.dispatchEvent # blazor.webassembly.js:1
(anonymous) # blazor.webassembly.js:1
(anonymous) # blazor.webassembly.js:1
e.onGlobalEvent # blazor.webassembly.js:1
Please advise on what further tests I need to do to determine what the problem is, and how I can fix.
Edit and update
Following from one of the answers below, I switched to using a PutObjectRequest. The code is below. This now gives a new error, which I assume is to do with the permissions of the S3 bucket. Can anyone guide on what I need to change?
Code
public class UploadData
{
private readonly BasicAWSCredentials credentials = new BasicAWSCredentials("access key", "secret Key");
public async Task UploadZipFile(Stream zip_file, string file_name)
{
zip_file.Seek(0, SeekOrigin.Begin);
var config = new AmazonS3Config
{
RegionEndpoint = Amazon.RegionEndpoint.EUWest2
};
using var client = new AmazonS3Client(credentials, config);
var putRequest1 = new PutObjectRequest
{
BucketName = "Bucket_name",
Key = file_name,
InputStream = zip_file
};
PutObjectResponse response1 = await client.PutObjectAsync(putRequest1);
Console.WriteLine("finshed uploading");
}
}
And here is the error code
Unknown encountered on server. Message:'TypeError: Failed to execute 'append' on 'Headers': Invalid value' when putting an object
I do not know much about blazor and webassembly in particular, but I can give try and give you an advice what I would try:
It looks like the issue in the code above is a lack of multithreading support in webassembly, see also [1].
The TransferUtility class in the aws-sdk-net makes use of threads for multipart uploads, see [2]:
TransferUtility provides a simple API for uploading content to and downloading content from Amazon S3. It makes extensive use of Amazon S3 multipart uploads to achieve enhanced throughput, performance, and reliability.
When uploading large files by specifying file paths instead of a stream, TransferUtility uses multiple threads to upload multiple parts of a single upload at once. When dealing with large content sizes and high bandwidth, this can increase throughput significantly.
Thus, I would try to use some of the low-level constructs to accomplish a standard upload using PutObjectRequest, see: https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html
If the files to be uploaded are not too large, this should work without any issues because it is a single-threaded approach.
Another issue would be to enable multi-threading in Blazor WASM, but I am not sure if that is already an option, see [3].
[1] https://www.meziantou.net/don-t-freeze-ui-while-executing-cpu-intensive-work-in-blazor-webassembly.htm
[2] https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/TTransferUtility.html
[3] https://github.com/dotnet/aspnetcore/issues/17730

python, google cloud platform: unable to overwite a file from google bucket: CRC32 does not match

I am using python3 client to connect to google buckets and trying to the following
download 'my_rules_file.yaml'
modify the yaml file
overwrite the file
Here is the code that i used
from google.cloud import storage
import yaml
client = storage.Client()
bucket = client.get_bucket('bucket_name')
blob = bucket.blob('my_rules_file.yaml')
yaml_file = blob.download_as_string()
doc = yaml.load(yaml_file, Loader=yaml.FullLoader)
doc['email'].clear()
doc['email'].extend(["test#gmail.com"])
yaml_file = yaml.dump(doc)
blob.upload_from_string(yaml_file, content_type="application/octet-stream")
This is the error I get from the last line for upload
BadRequest: 400 POST https://storage.googleapis.com/upload/storage/v1/b/fc-sandbox-datastore/o?uploadType=multipart: {
"error": {
"code": 400,
"message": "Provided CRC32C \"YXQoSg==\" doesn't match calculated CRC32C \"EyDHsA==\".",
"errors": [
{
"message": "Provided CRC32C \"YXQoSg==\" doesn't match calculated CRC32C \"EyDHsA==\".",
"domain": "global",
"reason": "invalid"
},
{
"message": "Provided MD5 hash \"G/rQwQii9moEvc3ZDqW2qQ==\" doesn't match calculated MD5 hash \"GqyZzuvv6yE57q1bLg8HAg==\".",
"domain": "global",
"reason": "invalid"
}
]
}
}
: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>)
why is this happening. This seems to happen only for ".yaml files".
The reason for your error is because you are trying to use the same blob object for both downloading and uploading this will not work you need two separate instances... You can find some good examples here Python google.cloud.storage.Blob() Examples
You should use a seperate blob instance to handle the upload you are trying with only one...
.....
blob = bucket.blob('my_rules_file.yaml')
yaml_file = blob.download_as_string()
.....
the second instance is needed here
....
blob.upload_from_string(yaml_file, content_type="application/octet-stream")
...

When using lambda to generate elbv2 attributes (name specifically), receiving error from Lambda that name is longer than 32 characters

I am building a CloudFormation template that uses a Lambda function to generate the name of the load balancer built by the template.
When the function runs, it fails with the following error:
Failed to validate attributes of ELB arn:aws-us-gov:elasticloadbalancing:us-gov-west-1:273838691273:loadbalancer/app/dev-fu-WALB-18VHO2DJ4MHK/c69c48fd3464de01. An error occurred (ValidationError) when calling the DescribeLoadBalancers operation: The load balancer name 'arn:aws-us-gov:elasticloadbalancing:us-gov-west-1:273838691273:loadbalancer/app/dev-fu-WALB-18VHO2DJ4MHK/c69c48fd3464de01' cannot be longer than '32' characters.
It is obviously pulling the arn rather than the name of the elbv2.
I opened a ticket with AWS to no avail, and also with the company that wrote the script... same results.
I have attached the script and any help is greatly appreciated.
import cfn_resource
import boto3
import boto3.session
import logging
logger = logging.getLogger()
handler = cfn_resource.Resource()
# Retrieves DNSName and source security group name for the specified ELB
#handler.create
def get_elb_attribtes(event, context):
properties = event['ResourceProperties']
elb_name = properties['PORALBName']
elb_template = properties['PORALBTemplate']
elb_subnets = properties['PORALBSubnets']
try:
client = boto3.client('elbv2')
elb = client.describe_load_balancers(
Names=[
elb_name
]
)['LoadBalancers'][0]
for az in elb['AvailabilityZones']:
if not az['SubnetId'] in elb_subnets:
raise Exception("ELB does not include VPC subnet '" + az['SubnetId'] + "'.")
target_groups = client.describe_target_groups(
LoadBalancerArn=elb['LoadBalancerArn']
)['TargetGroups']
target_group_arns = []
for target_group in target_groups:
target_group_arns.append(target_group['TargetGroupArn'])
if elb_template == 'geoevent':
if elb['Type'] != 'network':
raise Exception("GeoEvent Server requires network ElasticLoadBalancer V2.")
response_data = {}
response_data['DNSName'] = elb['DNSName']
response_data['TargetGroupARNs'] = target_group_arns
msg = 'ELB {} found.'.format(elb_name)
logger.info(msg)
return {
'Status': 'SUCCESS',
'Reason': msg,
'PhysicalResourceId': context.log_stream_name,
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId'],
'Data': response_data
}
except Exception, e:
error_msg = 'Failed to validate attributes of ELB {}. {}'.format(elb_name, e)
logger.error(error_msg)
return {
'Status': 'FAILED',
'Reason': error_msg,
'PhysicalResourceId': context.log_stream_name,
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId']
}
The error says:
An error occurred (ValidationError) when calling the DescribeLoadBalancers operation
So, looking at where it calls DescribeLoadBalancers:
elb = client.describe_load_balancers(
Names=[
elb_name
]
)['LoadBalancers'][0]
The error also said:
The load balancer name ... cannot be longer than '32' characters.
The name comes from:
properties = event['ResourceProperties']
elb_name = properties['PORALBName']
So, the information is being passed into the Lambda function via event. This is coming from whatever is triggering the Lambda function. So, you'll need to find out what is triggering the function and discover what information it actually sending. Your problem is outside of the code listed.
Other options
In your code, you can send event to the debug logs (eg print (event)) and see whether they are passing the ELB name in a different field.
Alternatively, you could call describe_load_balancers without a Name filter to retrieve a list of all load balancers, then use the ARN (that you have) to find the load balancer of interest. Simply loop through all the results until you find the one that matches the ARN you have. Then, continue as normal.

"ValueError: Invalid control character" using Python OAuth client

I am trying to load data from a google spreadsheet into a postgres database.
The problem is when I am trying to authenticate my credentials, I get the following error:
File "/usr/local/lib/python2.7/dist-packages/oauth2client/_openssl_crypt.py", line 117, in from_string
pkey = crypto.load_privatekey(crypto.FILETYPE_PEM, parsed_pem_key)
OpenSSL.crypto.Error: [('PEM routines', 'PEM_read_bio', 'no start line')]
I have followed all steps in Using OAuth2 for Authorization, enabled the API and created a service account from which I got a .json file containing the necessary keys and authentication elements.
The way I am trying to authenticate is the following:
json_key = json.load(open('gdoc.json'),strict=False)
creds = SignedJwtAssertionCredentials(json_key['client_email'], json_key['private_key'], scope)
login = gspread.authorize(creds)
Perhaps the issue comes from the parameter in the json load strict=False, the problem is if I remove it, get the error:
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 5 column 38 (char 174)
I saw a couple other forums and they suggest using the strict=False parameter since there are non escaped \n inside the json key file.
This is a copy of the .json key file:
{
"type": "service_account",
"project_id": "geometric-shine-118101",
"private_key_id": "xxx",
"private_key": "-----BEGIN PRIVATE KEY-----\nxxx\n-----END PRIVATE KEY-----\n",
"client_email": "dataload#geometric-shine-118101.iam.gserviceaccount.com",
"client_id": "117076930343404252458",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/dataload%40geometric-shine-118101.iam.gserviceaccount.com"
}
"private_key": "-----BEGIN PRIVATE KEY-----\nxxx\n-----END PRIVATE KEY-----\n",
"client_email": "dataload#geometric-shine-118101.iam.gserviceaccount.com",
Copy and paste the actual values into single quotes (.e.g. ''). It worked for me.
'dataload#geometric-shine-118101.iam.gserviceaccount.com'
'-----BEGIN PRIVATE KEY-----\nxxx\n-----END PRIVATE KEY-----\n'