I want to create a youtube like Video streaming application but in a small scale. I am using Spring boot for backend rest endpoints and amazon S3 bucket for storing video files. I am able to upload and download video files to S3 bucket. But I am confused in streaming side. I want to show those video files in jsp page to play. I heard about Aws video on demand, aws kinesis, etc. Can someone suggest me or share some link which will be the best approach to follow for video streaming with spring boot. Or is there any other service apart from aws services which can be useful in this scenario. I am totally confused. Please help me out. Thank you.
I have created a sample project for streaming the AWS s3 resources using spring boot.
You can set a controller with mapping as required.
For this demo code the endpoint is http://localhost:port/bucket_name/object_key
#RestController("/")
public class ApiController {
#Value("${aws.region}")
private String awsRegion;
#GetMapping(value = "/**", produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE })
public ResponseEntity<StreamingResponseBody> getObject(HttpServletRequest request) {
try {
AmazonS3 s3client = AmazonS3ClientBuilder.standard().withRegion(awsRegion).build();
String uri = request.getRequestURI();
String uriParts[] = uri.split("/", 2)[1].split("/", 2);
String bucket = uriParts[0];
String key = uriParts[1];
System.out.println("Fetching " + uri);
S3Object object = s3client.getObject(bucket, key);
S3ObjectInputStream finalObject = object.getObjectContent();
final StreamingResponseBody body = outputStream -> {
int numberOfBytesToWrite = 0;
byte[] data = new byte[1024];
while ((numberOfBytesToWrite = finalObject.read(data, 0, data.length)) != -1) {
outputStream.write(data, 0, numberOfBytesToWrite);
}
finalObject.close();
};
return new ResponseEntity<StreamingResponseBody>(body, HttpStatus.OK);
} catch (Exception e) {
System.err.println("Error "+ e.getMessage());
return new ResponseEntity<StreamingResponseBody>(HttpStatus.BAD_REQUEST);
}
}
}
You need to use StreamingResponseBody in your ResponseEntity.
If you need a ready to use microservice feel free to explore the github project s3-streamer I wrote for very same purpose.
Related
Hello I am trying to download data from one of an Amazon S3 public bucket.
For example https://registry.opendata.aws/noaa-gfs-bdp-pds/
The bucket has web accessible folder and I want to download the files inside the bucket.
I know I can do this with AWS CLI tool
But I want to know if there anyway to do this with AWs SDK Api (s3 client) (c# visual studio)?
I think the issue is authentication when creating connection to s3 client it requires credentials like access key ,I don't have an AWS account,and the bucket I try to get to is public so
Does anyone know how to access to this public bucket without any credentials via API?
Thanks.
If you specify the AnonymousAWSCredentials as the credentials object, any requests that are made to S3 will be unsigned. After that, interacting with the bucket is done like any other call:
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using System;
namespace S3TestApp
{
class Program
{
static void Main(string[] args)
{
var unsigned = new AnonymousAWSCredentials();
var client = new AmazonS3Client(unsigned, Amazon.RegionEndpoint.USEast1);
var listRequest = new ListObjectsRequest
{
BucketName = "noaa-gfs-bdp-pds",
Delimiter = "/",
};
ListObjectsResponse listResponse;
do
{
listResponse = client.ListObjects(listRequest);
foreach (var obj in listResponse.CommonPrefixes)
{
Console.WriteLine("PRE {0}", obj);
}
foreach (var obj in listResponse.S3Objects)
{
Console.WriteLine("{0} {1}", obj.Size, obj.Key);
}
listRequest.Marker = listResponse.NextMarker;
} while (listResponse.IsTruncated);
}
}
}
when i make post request to upload image file to aws s3 bucket form my local dot net core aws lambda serverless application it works but form my deployed application the image still uploded to s3 bucket but the image broken (shows black empty image)
here is the code
[HttpPut("PostImageFile")]
public async Task FileImageAsync(string Id)
{
var s3Client = new AmazonS3Client("*******", "*******", Amazon.RegionEndpoint.USEast1);
try
{
var httpRequest = HttpContext.Request;
//posted file
var file = httpRequest.Form.Files[0];
byte[] fileBytes = new byte[file.Length];
file.OpenReadStream().Read(fileBytes, 0, Int32.Parse(file.Length.ToString()));
var fileName = Guid.NewGuid() + file.FileName;
PutObjectResponse response = null;
using (var stream = new MemoryStream())
{
file.CopyTo(stream);
var request = new PutObjectRequest
{
BucketName = "imageakal",
Key = fileName,
InputStream = stream,
ContentType = file.ContentType,
CannedACL = S3CannedACL.PublicReadWrite
};
response = await s3Client.PutObjectAsync(request);
};
}
catch (Exception ex)
{
Console.Write("Upload Failed: " + ex.Message);
}
}
Without many more details, I would guess that your AWS settings could have a list of permitted/denied domains. I would check that your AWS instance is configured to allow requests from your domain.
Just put "multipart/form-data" at 'Binary Media Type' section in Api Gateway setting tab, and deploy it(don't forget).
I am using aws unity sdk for my unity game to download the assets stored in amazon server.I used GetObjectAsync() for download.In the aws sdk for iOS,we get the progress value from NSUrlSessionDelegates.But I want to use aws mobile sdk for unity and get the progress value of download.How do I do this? Please help.
private void GetObject()
{
ResultText.text = string.Format("fetching {0} from bucket {1}",
SampleFileName, S3BucketName);
Client.GetObjectAsync(S3BucketName, SampleFileName, (responseObj) =>
{
string data = null;
var response = responseObj.Response;
if (response.ResponseStream != null)
{
using (StreamReader reader = new StreamReader(response.ResponseStream))
{
data = reader.ReadToEnd();
}
ResultText.text += "\n";
ResultText.text += data;
}
});
}
You can do that. Here is snippet from Amazon develper website.
public event EventHandler<WriteObjectProgressArgs> WriteObjectProgressEvent
TransferUtilityDownloadRequest request = new TransferUtilityDownloadRequest();
request.WriteObjectProgressEvent += displayProgress;
private void displayProgress(object sender, WriteObjectProgressArgs args)
{
Console.WriteLine(args);
}
I have the same problem and I tried to ask on AWS forum.
Unfortunately they doesn't support yet.
Here the link:
https://forums.aws.amazon.com/thread.jspa?threadID=248187&tstart=0
To access of forum is necessary to login.
I am trying to access my AWS DataPipelines using AWS Java SDK v1.7.5, but listPipelines is returning an empty list in the code below.
I have DataPipelines that are scheduled in the US East region, which I believe I should be able to list using the listPipelines method of the DataPipelineClient. I am already using the ProfilesConfigFile to authenticate and connect to S3, DynamoDB and Kinesis without a problem. I've granted the PowerUserAccess Access Policy to the IAM user specified in the config file. I've also tried applying the Administrator Access policy to the user, but it didn't change anything. Here's the code I'm using:
//Establish credentials for connecting to AWS.
File configFile = new File(System.getProperty("user.home"), ".aws/config");
ProfilesConfigFile profilesConfigFile = new ProfilesConfigFile(configFile);
AWSCredentialsProvider awsCredentialsProvider = new ProfileCredentialsProvider(profilesConfigFile, "default");
//Set up the AWS DataPipeline connection.
DataPipelineClient dataPipelineClient = new DataPipelineClient(awsCredentialsProvider);
Region usEast1 = Region.getRegion(Regions.US_EAST_1);
dataPipelineClient.setRegion(usEast1);
//List all pipelines we have access to.
ListPipelinesResult listPipelinesResult = dataPipelineClient.listPipelines(); //empty list returned here.
for (PipelineIdName p: listPipelinesResult.getPipelineIdList()) {
System.out.println(p.getId());
}
Make sure to check if there are more results - I've noticed sometimes the API returns only few pipelines (could even be empty), but has a flag for more results. You can retrieve them like this:
void listPipelines(DataPipelineClient dataPipelineClient, String marker) {
ListPipelinesRequest request = new ListPipelinesRequest();
if (marker != null) {
request.setMarker(marker);
}
ListPipelinesResult listPipelinesResult = client.listPipelines(request);
for (PipelineIdName p: listPipelinesResult.getPipelineIdList()) {
System.out.println(p.getId());
}
// Call recursively if there are more results:
if (pipelineList.getHasMoreResults()) {
listPipelines(dataPipelineClient, listPipelinesResult.getMarker());
}
}
Is it possible to request "Snapshot Logs" through AWS SDK somehow?
It's possible to do it through AWS console:
Cross posted to Amazon forum.
Requesting a log snapshot is a 3 step process. First you have to do an environment information request:
elasticBeanstalk.requestEnvironmentInfo(
new RequestEnvironmentInfoRequest()
.withEnvironmentName(envionmentName)
.withInfoType("tail"));
Then you have to retreive the environment information:
final List<EnvironmentInfoDescription> envInfos =
elasticBeanstalk.retrieveEnvironmentInfo(
new RetrieveEnvironmentInfoRequest()
.withEnvironmentName(environmentName)
.withInfoType("tail")).getEnvironmentInfo();
This returns a list of environment info descriptions, with the EC2 instance id and the URL to an S3 object that contains the log snapshot. You can then retreive the logs with:
DefaultHttpClient client = new DefaultHttpClient();
DefaultHttpRequestRetryHandler retryhandler =
new DefaultHttpRequestRetryHandler(3, true);
client.setHttpRequestRetryHandler(retryhandler);
for (EnvironmentInfoDescription environmentInfoDescription : envInfos) {
System.out.println(environmentInfoDescription.getEc2InstanceId());
HttpGet rq = new HttpGet(environmentInfoDescription.getMessage());
try {
HttpResponse response = client.execute(rq);
InputStream content = response.getEntity().getContent();
System.out.println(IOUtils.toString(content));
} catch ( Exception e ) {
System.out.println("Exception fetching " +
environmentInfoDescription.getMessage());
}
}
I hope this helps!