I am trying to retrieve images from my bucket to send to my mobile apps, I currently have the devices accessing AWS directly, however I am adding a layer of security and having my apps (IOS and Android) now make requests to my server which will then respond with DynamoDB and S3 data.
I am trying to follow the documentation and code samples provided by AWS for .Net and they worked seamlessly for DynamoDB, I am running into problems with S3.
S3 .NET Documentation
My problem is that if I provide no credentials, I get the error:
Failed to retrieve credentials from EC2 Instance Metadata Service
This is expected as I have IAM roles set up and only want my apps and this server (in the future, only this server) to have access to the buckets.
But when I provide the credentials, the same way I provided credentials for DynamoDB, my server waits forever and doesn't receive any responses from AWS.
Here is my C#:
<%# WebHandler Language="C#" Class="CheckaraRequestHandler" %>
using System;
using System.Web;
using System.Collections.Generic;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using System.IO;
using System.Threading.Tasks;
public class CheckaraRequestHandler : IHttpHandler
{
private const string bucketName = "MY_BUCKET_NAME";
private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USEast1;
public static IAmazonS3 client = new AmazonS3Client("MY_ACCESS_KEY", "MY_SECRET_KEY", RegionEndpoint.USEast1);
public void ProcessRequest(HttpContext context)
{
if (context.Request.HttpMethod.ToString() == "GET")
{
string userID = context.Request.QueryString["User"];
string Action = context.Request.QueryString["Action"];
if (userID == null)
{
context.Response.ContentType = "text/plain";
context.Response.Write("TRY AGAIN!");
return;
}
if (Action == "GetPhoto")
{
ReadObjectDataAsync(userID).Wait();
}
var client = new AmazonDynamoDBClient("MY_ACCESS_KEY", "MY_SECRET_KEY", RegionEndpoint.USEast1);
Console.WriteLine("Getting list of tables");
var table = Table.LoadTable(client, "TABLE_NAME");
var item = table.GetItem(userID);
if (item != null)
{
context.Response.ContentType = "application/json";
context.Response.Write(item.ToJson());
}
else
{
context.Response.ContentType = "text/plain";
context.Response.Write("0");
}
}
}
public bool IsReusable
{
get
{
return false;
}
}
static async Task ReadObjectDataAsync(string userID)
{
string responseBody = "";
try
{
string formattedKey = userID + "/" + userID + "_PROFILEPHOTO.jpeg";
//string formattedKey = userID + "_PROFILEPHOTO.jpeg";
//formattedKey = formattedKey.Replace(":", "%3A");
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucketName,
Key = formattedKey
};
using (GetObjectResponse response = await client.GetObjectAsync(request))
using (Stream responseStream = response.ResponseStream)
using (StreamReader reader = new StreamReader(responseStream))
{
string title = response.Metadata["x-amz-meta-title"]; // Assume you have "title" as medata added to the object.
string contentType = response.Headers["Content-Type"];
Console.WriteLine("Object metadata, Title: {0}", title);
Console.WriteLine("Content type: {0}", contentType);
responseBody = reader.ReadToEnd(); // Now you process the response body.
}
}
catch (AmazonS3Exception e)
{
Console.WriteLine("Error encountered ***. Message:'{0}' when writing an object", e.Message);
}
catch (Exception e)
{
Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message);
}
}
}
When I debug, this line waits forever:
using (GetObjectResponse response = await client.GetObjectAsync(request))
This is the same line that throws the credentials error when I don't provide them. Is there something that I am missing here?
Any help would be greatly appreciated.
I suspect that the AWS .NET SDK has some isses with it specifically with the async call to S3.
The async call to dynamoDB works perfect, but the S3 one hangs forever.
What fixed my problem was simply removing the async functionality (even tho in the AWS docs, the async call is supposed to be used)
Before:
using (GetObjectResponse response = await client.GetObjectAsync(request))
After:
using (GetObjectResponse response = myClient.GetObject(request))
Hopefully this helps anyone else encountering this issue.
Related
I am trying to update my appsync client to authenticate with IAM credentials. In case of API_KEY I set the API_KEY_HEADER like so: request.addHeader(API_KEY_HEADER, this.apiKey); Is there a similar way to authenticate in a Java client with IAM credentials? Is there a header I can pass in to pass in the secret and access keys like here: https://docs.amplify.aws/lib/graphqlapi/authz/q/platform/js#iam? Or should I just be using a cognito user pool as a way to authenticate the request?
According to AWS Documentation we need to use sign requests using the process documented here: https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html and steps listed here: https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html.
I also found an implementation here: https://medium.com/#tridibbolar/aws-lambda-as-an-appsync-client-fbb0c1ce927d. Using the code above:
private void signRequest(final Request<AmazonWebServiceRequest> request) {
final AWS4Signer signer = new AWS4Signer();
signer.setRegionName(this.region);
signer.setServiceName("appsync");
signer.sign(request, this.appsyncCredentials);
}
private Request<AmazonWebServiceRequest> getRequest(final String data) {
final Request<AmazonWebServiceRequest> request =
new DefaultRequest<AmazonWebServiceRequest>("appsync");
request.setHttpMethod(HttpMethodName.POST);
request.setEndpoint(URI.create(this.appSyncEndpoint));
final byte[] byteArray = data.getBytes(Charset.forName("UTF-8"));
request.setContent(new ByteArrayInputStream(byteArray));
request.addHeader(AUTH_TYPE_HEADER, AWS_IAM_AUTH_TYPE);
request.addHeader(HttpHeaders.CONTENT_TYPE, APPLICATION_GRAPHQL);
request.addHeader(HttpHeaders.CONTENT_LENGTH, String.valueOf(byteArray.length));
signRequest(request);
return request;
}
private HttpResponseHandler<String> getResponseHandler() {
final HttpResponseHandler<String> responseHandler = new HttpResponseHandler<String>() {
#Override
public String handle(com.amazonaws.http.HttpResponse httpResponse) throws Exception {
final String result = IOUtils.toString(httpResponse.getContent());
if(httpResponse.getStatusCode() != HttpStatus.SC_OK) {
final String errorText = String.format(
"Error posting request. Response status code was %s and text was %s. ",
httpResponse.getStatusCode(),
httpResponse.getStatusText());
throw new RuntimeException(errorText);
} else {
final ObjectMapper objectMapper = new ObjectMapper();
//custom class to parse appsync response.
final AppsyncResponse response = objectMapper.readValue(result, AppsyncResponse.class);
if(CollectionUtils.isNotEmpty(response.getErrors())){
final String errorMessages = response
.getErrors()
.stream()
.map(Error::getMessage)
.collect(Collectors.joining("\n"));
final String errorText = String.format(
"Error posting appsync request. Errors were %s. ",
errorMessages);
throw new RuntimeException(errorText);
}
}
return result;
}
#Override
public boolean needsConnectionLeftOpen() {
return false;
}
};
return responseHandler;
}
private Response<String> makeGraphQlRequest(final Request<AmazonWebServiceRequest> request) {
return this.httpClient.requestExecutionBuilder()
.executionContext(new ExecutionContext())
.request(request)
.execute(getResponseHandler());
}
I'm trying to upload a large video file (800mb) to my S3 bucket, but it appears to timeout. It works just fine for smaller files. My project is an ASP.Net Core 2.1 application.
This is the exception that is thrown:
An unhandled exception occurred while processing the request.
SocketException: The I/O operation has been aborted because of either a thread exit or an application request.
Unknown location
IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.
System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
TaskCanceledException: The operation was canceled.
GamerPilot.Video.AWSS3Helper.UploadFileAsync(Stream stream, string key, S3CannedACL acl, bool useReducedRedundancy, bool throwOnError, CancellationToken cancellationToken) in AWSS3Helper.cs, line 770
My source code looks like this:
public async Task<IVideo> AddVideoAsync(int instructorId, int lectureId, string videoName, string filePath, CancellationToken cancellationToken = default(CancellationToken))
{
if (string.IsNullOrEmpty(filePath)) { throw new ArgumentNullException("filePath", "Video filepath is missing"); }
if (!File.Exists(filePath)) { throw new ArgumentNullException("filePath", "Video filepath does not exists"); }
//Test video file upload and db row insertion
using (var stream = File.OpenRead(filePath))
{
return await AddVideoAsync(instructorId, lectureId, videoName, stream, cancellationToken);
}
}
public async Task<IVideo> AddVideoAsync(int instructorId, int lectureId, string videoName, Stream videoFile, CancellationToken cancellationToken = default(CancellationToken))
{
var video = (Video) await GamerPilot.Video.Helper.Create(_awsS3AccessKey, _awsS3SecretKey, _awsS3BucketName, _awsS3Region)
.AddVideoAsync(instructorId, lectureId, videoName, videoFile, cancellationToken);
using (var db = new DbContext(_connectionString))
{
db.Videos.Add(video);
var count = await db.SaveChangesAsync();
}
return video;
}`
public async Task<IVideo> AddVideoAsync(int instructorId, int lectureId, string videoName, Stream videoFile, CancellationToken cancellationToken = default(CancellationToken))
{
if (string.IsNullOrEmpty(videoName)) { throw new ArgumentNullException("videoName", "Video name cannot be empty or null"); }
if (videoFile == null) { throw new ArgumentNullException("video", "Video stream is missing"); }
var videoNameCleaned = videoName.Replace(" ", "-").ToLower().Replace(".mp4", "");
var videoKey = string.Join('/', "videos", instructorId, lectureId, videoNameCleaned + ".mp4");
using (var aws = new AWSS3Helper(_awsS3AccessKey, _awsS3SecretKey, _awsS3BucketName, _awsS3Region))
{
try
{
//THIS FAILS ------
await aws.UploadFileAsync(videoFile, videoKey, Amazon.S3.S3CannedACL.PublicRead, true, true, cancellationToken);
}
catch (Exception ex)
{
throw;
}
}
return new Video
{
InstructorId = instructorId,
LectureId = lectureId,
Name = videoName,
S3Key = videoKey,
S3Region = _awsS3Region.SystemName,
S3Bucket = _awsS3BucketName,
Created = DateTime.Now
};
}
How can I work around this?
There is no general constraint on S3 itself which would prevent you from uploading an 800MB file. However, there are requirements for the handling of retries and timeouts when working with AWS. It is not clear from your question whether or not you are using Amazon's SDK, (I can't find the origin of GamerPilot.Video.AWSS3Helper.UploadFileAsync). However, Amazon's SDK for .NET should handle this for you if you use it in accordance with the following:
Programming with the AWS SDK for .NET - Retries and Timeouts
Using the AWS SDK for .NET for Multipart Upload (High-Level API)
I followed Google's Quick-Start documentation for the Speech API to enable billing and API for an account. This account has authorized a service account to create Compute instances on its behalf. After creating an instance on the child account, hosting a binary to use the Speech API, I am unable to successfully use the example C# code provided by Google in the C# speech example:
try
{
var speech = SpeechClient.Create();
var response = speech.Recognize(new RecognitionConfig()
{
Encoding = RecognitionConfig.Types.AudioEncoding.Linear16,
LanguageCode = "en"
}, RecognitionAudio.FromFile(audioFiles[0]));
foreach (var result in response.Results)
{
foreach (var alternative in result.Alternatives)
{
Debug.WriteLine(alternative.Transcript);
}
}
} catch (Exception ex)
// ...
}
Requests fail on the SpeechClient.Create() line with the following error:
--------------------------- Grpc.Core.RpcException: Status(StatusCode=Unauthenticated, Detail="Exception occured in
metadata credentials plugin.")
at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at Grpc.Core.Internal.AsyncCall`2.UnaryCall(TRequest msg)
at
Grpc.Core.Calls.BlockingUnaryCall[TRequest,TResponse](CallInvocationDetails`2
call, TRequest req)
at
Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2
method, String host, CallOptions options, TRequest request)
at
Grpc.Core.Internal.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2
method, String host, CallOptions options, TRequest request)
at
Google.Cloud.Speech.V1.Speech.SpeechClient.Recognize(RecognizeRequest
request, CallOptions options)
at
Google.Api.Gax.Grpc.ApiCall.<>c__DisplayClass0_0`2.b__1(TRequest
req, CallSettings cs)
at
Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass1_0`2.b__0(TRequest
request, CallSettings callSettings)
at Google.Api.Gax.Grpc.ApiCall`2.Sync(TRequest request,
CallSettings perCallCallSettings)
at
Google.Cloud.Speech.V1.SpeechClientImpl.Recognize(RecognizeRequest
request, CallSettings callSettings)
at Google.Cloud.Speech.V1.SpeechClient.Recognize(RecognitionConfig
config, RecognitionAudio audio, CallSettings callSettings)
at Rc2Solver.frmMain.RecognizeWordsGoogleSpeechApi() in
C:\Users\jorda\Google
Drive\VSProjects\Rc2Solver\Rc2Solver\frmMain.cs:line 1770
--------------------------- OK
I have verified that the Speech API is activated. Here is the scope that the service account uses when creating the Compute instances:
credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(me)
{
Scopes = new[] { ComputeService.Scope.Compute, ComputeService.Scope.CloudPlatform }
}.FromPrivateKey(yk)
);
I have found no information or code online about specifically authorizing or authenticating the Speech API for service account actors. Any help is appreciated.
It turns out the issue was that the Cloud Compute instances needed to be created with a ServiceAccount parameter specified. Otherwise the Cloud instances were not part of a ServiceAccount default credential, which is referenced by the SpeechClient.Create() call. Here is the proper way to create an instance attached to a service account, and it will use the SA tied to the project ID:
service = new ComputeService(new BaseClientService.Initializer() {
HttpClientInitializer = credential,
ApplicationName = "YourAppName"
});
string MyProjectId = "example-project-27172";
var project = await service.Projects.Get(MyProjectId).ExecuteAsync();
ServiceAccount servAcct = new ServiceAccount() {
Email = project.DefaultServiceAccount,
Scopes = new [] {
"https://www.googleapis.com/auth/cloud-platform"
}
};
Instance instance = new Instance() {
MachineType = service.BaseUri + MyProjectId + "/zones/" + targetZone + "/machineTypes/" + "g1-small",
Name = name,
Description = name,
Disks = attachedDisks,
NetworkInterfaces = networkInterfaces,
ServiceAccounts = new [] {
servAcct
},
Metadata = md
};
batchRequest.Queue < Instance > (service.Instances.Insert(instance, MyProjectId, targetZone),
(content, error, i, message) => {
if (error != null) {
AddEventMsg("Error creating instance " + name + ": " + error.ToString());
} else {
AddEventMsg("Instance " + name + " created");
}
});
I have a website which exposes an ODatas $metadata but to do anything further requires the request to be authenticated (using a cookie).
I want to access this from a console app, not a browser.
I am using Microsofts Odata V4 client code generator.
1) Create a wrapper around the provided Container created by the OData client code generator.
2) Log in and get the cookie you need for authentication
3) Add a hook to the request builder, so you can apply cookies at request time. For my app, I specifically needed the cookie with the name .AspNet.ApplicationCookie
Here is a full working example. You can instantiate this container with the user and password needed as defined at the bottom. This MUST match whatever the controller at the Login API is expecting.
using Nito.AsyncEx;
using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
namespace MyAppOdataOdataService.Default
{
public class MyAppOdataContainer : Container
{
public Cookie[] _MyAppOdataAuthcookie;
public string cookieAuthName = ".AspNet.ApplicationCookie";
private string baseurl = "https://TheAppwWebsite.co.jp/";
public MyAppOdataContainer(MyAppOdataLoginInfo logininfo ) :
base(new Uri("https://TheAppwWebsite.co.jp/odata/"))
{
// init authorization
_MyAppOdataAuthcookie = AsyncContext.Run(() => AuthenticateUser(logininfo));
if (_MyAppOdataAuthcookie == null) throw new UnauthorizedAccessException();
this.BuildingRequest += AddCookie;
}
private void AddCookie(object sender, Microsoft.OData.Client.BuildingRequestEventArgs e)
{
e.Headers.Add("Cookie", cookieAuthName+"=" + _MyAppOdataAuthcookie.First(c=>c.Name == cookieAuthName).Value);
}
private async Task<Cookie[]> AuthenticateUser(MyAppOdataLoginInfo logininfo)
{
CookieContainer cookies = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = cookies;
HttpClient client = new HttpClient(handler);
client.BaseAddress = new Uri(baseurl);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
Uri uri = new Uri(baseurl + "/Login/Login");
HttpResponseMessage response = await client.PostAsJsonAsync(uri, logininfo);
response.EnsureSuccessStatusCode();
// Return the URI of the created resource.
return cookies.GetCookies(uri).Cast<Cookie>().ToArray();
}
}
public class MyAppOdataLoginInfo
{
public string Username { get; set; }
public string Password { get; set; }
}
}
Thanks:
How to apply the cookie:
Creating the client code:
CookieContainer explaination:
Post operation idea here - for authorizing - having to use PostAsJsonAsync
I'm using a simple example from Amazon documentation for ItemSearch and I get a strange error:
"The remote server returned an unexpected response: (400) Bad Request."
This is the code:
public static void Main()
{
//Remember to create an instance of the amazon service, including you Access ID.
AWSECommerceServicePortTypeClient service = new AWSECommerceServicePortTypeClient(new BasicHttpBinding(),
new EndpointAddress(
"http://webservices.amazon.com/onca/soap?Service=AWSECommerceService"));
AWSECommerceServicePortTypeClient client = new AWSECommerceServicePortTypeClient(
new BasicHttpBinding(),
new EndpointAddress("http://webservices.amazon.com/onca/soap?Service=AWSECommerceService"));
// prepare an ItemSearch request
ItemSearchRequest request = new ItemSearchRequest();
request.SearchIndex = "Books";
request.Title = "Harry+Potter";
request.ResponseGroup = new string[] { "Small" };
ItemSearch itemSearch = new ItemSearch();
itemSearch.Request = new ItemSearchRequest[] { request };
itemSearch.AWSAccessKeyId = accessKeyId;
// issue the ItemSearch request
try
{
ItemSearchResponse response = client.ItemSearch(itemSearch);
// write out the results
foreach (var item in response.Items[0].Item)
{
Console.WriteLine(item.ItemAttributes.Title);
}
}
catch(Exception e)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine(e.Message);
Console.ForegroundColor = ConsoleColor.White;
Console.WriteLine("Press any key to quit...");
Clipboard.SetText(e.Message);
}
Console.ReadKey();
What is wrong?
You receive this message because your request is not signed. Starting from August 2009 all requests must be signed.
Here you can see at the example about how to sign amazon requests:
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2480&categoryID=14