I am able to use the Athena API with startQueryExecution() to create a CSV file of the responses in S3. However, I would like to be able to return to my application a JSON response so I can further process the data. I am trying to return JSON results after I run startQueryExecution() via the API, how do I can get the results as a JSON response back?
I am using the AWS PHP SDK [https://aws.amazon.com/sdk-for-php/] , however this is relevant to any language since I can not find any answers to actually getting a response back, it just saves a CSV file to S3.
$athena = AWS::createClient('athena');
$queryx = 'SELECT * FROM elb_logs LIMIT 20';
$result = $athena->startQueryExecution([
'QueryExecutionContext' => [
'Database' => 'sampledb',
],
'QueryString' => 'SELECT request_ip FROM elb_logs LIMIT 20', // REQUIRED
'ResultConfiguration' => [ // REQUIRED
'EncryptionConfiguration' => [
'EncryptionOption' => 'SSE_S3' // REQUIRED
],
'OutputLocation' => 's3://xxxxxx/', // REQUIRED
],
]);
// check completion : getQueryExecution()
$exId = $result['QueryExecutionId'];
sleep(6);
$checkExecution = $athena->getQueryExecution([
'QueryExecutionId' => $exId, // REQUIRED
]);
if($checkExecution["QueryExecution"]["Status"]["State"] == 'SUCCEEDED')
{
$dataOutput = $athena->getQueryResults([
'QueryExecutionId' => $result['QueryExecutionId'], // REQUIRED
]);
while (($data = fgetcsv($dataOutput, 1000, ",")) !== FALSE) {
$num = count($data);
echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
echo $data[$c] . "<br />\n";
}
}
}
The Amazon Athena SDK will return the results of a query and then you can write (send) this as JSON. The SDK will not do this for you itself.
The API startQueryExecution() retuns QueryExecutionId. Use this to call getQueryExecution() to determine if the query is complete. Once the query completes call getQueryResults().
You can then process each row in the result set.
Related
I'm trying to get past a road block situation related to S3.
Background: From my mobile app users can pick upto 10 images and post it to server. This is received and moved to concerned folders in AWS instance - Let's call it instance "A" for the purpose of our discussion. As and when the file is moved to instance "A" we are calling "moveFileToS3" . The code looks something like this.
foreach($_FILES as $file)
{
$file_info = $file['name'];
$extension = pathinfo($file['name'],PATHINFO_EXTENSION);
$destination_file_name = $imagesMoved . '-'. $sku . '.'. $extension;
$file_path = $images_directory . $destination_file_name;
if (move_uploaded_file($file['tmp_name'], $file_path))
{
$imagesMoved++;
//Move the file to S3
moveFileToS3($destination_file_name, $images_directory, $parent_folder . '/' . $images_folder . '/');
}
if (intval($imagesMoved) == intval($totalFileCount))
{
$image_moved = true;
//Begin to update the DB
}
}
When this call (moveFileToS3) is NOT made.. all the files selected by the user make it to instance "A".
But when this call(moveFileToS3) is made... not all the files selected by the user make it to instance "A" and only a few files from instance "A" get moved to S3 location. Neither set of instructions after $image_moved = true get executed.
Any assistance to get past this situation would be very much appreciated. I have attached the file that has the method "moveFileToS3" for your quick reference.
<?php
require '../api/vendor/autoload.php';
use Aws\Common\Exception\MultipartUploadException;
use Aws\S3\MultipartUploader;
use Aws\S3\S3Client;
function moveFileToS3($fileName, $fileLocation, $targetLocation)
{
date_default_timezone_set('Asia/Kolkata');
$region = 'xxxxx';
$version = 'xxxxx';
$bucket = 'xx-xxxxxx-xxxxx';
$endpoint = 'xxxxxxxx.s3-accelerate.amazonaws.com';
$key = 'xxxxxx';
$secret = 'xxxxxxxxx';
$fileFullPathLocal = $fileLocation.$fileName;
$s3 = new S3Client([
'version' => $version,
'region' => $region,
'debug' => true,
'credentials' => [
'key' => $key,
'secret' => $secret,
]
]);
// Prepare the upload parameters.
$uploader = new MultipartUploader($s3, $fileFullPathLocal, [
'bucket' => $bucket,
'key' => $targetLocation.$fileName
]);
// Perform the upload.
try
{
$responseLogFile = fopen("../log/S3UploadLog_".date("Y-m-d").".log", "a+");
fwrite($responseLogFile, '['.date("Y-m-d H:i:s").']: Upload Started : '.$fileName. PHP_EOL. PHP_EOL);
fclose($responseLogFile);
$result = $uploader->upload();
$responseLogFile = fopen("../log/S3UploadLog_".date("Y-m-d").".log", "a+");
fwrite($responseLogFile, '['.date("Y-m-d H:i:s").']: Upload Finished : '.$fileName. PHP_EOL. PHP_EOL);
// fwrite($responseLogFile, '['.date("Y-m-d H:i:s").']: Upload Result : '.$result. PHP_EOL. PHP_EOL);
fwrite($responseLogFile, '['.date("Y-m-d H:i:s").']: Object Url : '.$result['ObjectURL']. PHP_EOL. PHP_EOL);
fclose($responseLogFile);
// echo "Upload complete: {$result['ObjectURL']}" . PHP_EOL;
unlink($fileFullPathLocal);
}
catch (MultipartUploadException $e)
{
$responseLogFile = fopen("../log/S3UploadLog_".date("Y-m-d").".log", "a+");
fwrite($responseLogFile, '['.date("Y-m-d H:i:s").']: Upload Failed : '.$fileName. PHP_EOL. PHP_EOL);
fclose($responseLogFile);
echo $e->getMessage() . PHP_EOL;
}
}
?>
Seems to issue when you move file from your local dev sever to a bucket.. But when moved to AWS environment.. works without any issues
I am trying to create an archive and retrieval system in php. When the user click on archive button particular files will move to glacier storage from standard storage and when click on restore button file in the glacier storage will retrieved to standard storage.
Using AWS php SDK 3.0 api I have successfully move files to glacier vault and for retrieval an archive-retrieval job is initiated and I got the job Id after 3-5 hours and using that Job id after 5 hours I tried getJobOutput function. And I am getting the response same as mentioned in the api documentation but I am not getting the restored file in my s3 bucket.
Here is my code to upload to glacier and restore from glacier
public function archiveAndRestore() {
$this->s3Client = new S3Client(Configure::read('AWScredentials'));
$this->glacier = GlacierClient::factory(Configure::read('AWScredentials'));
// Upload to glacier
$this->s3Client->registerStreamWrapper();
$context = stream_context_create([
's3' => ['seekable' => true]
]);
$result = $this->glacier->uploadArchive(array(
'vaultName' => 'archiveTest',
'archiveDescription' => 'File Name is archiveTest.txt ',
'body' => fopen('s3://storage-bucket/Videos/archiveTest.txt', 'r', false, $context),
));
$archiveid = $result->get('archiveId');
$jobId = $this->glacier->initiateJob([
'accountId' => '-',
'vaultName' => 'archiveTest',
'jobParameters' => [
'Type' => 'archive-retrieval',
'ArchiveId' => 'ORgyyyqsKwoopp110EvFoyqj3G-csmOKLyy3IJnWF9Dpd8BJfwerEhg241nxHf6y6kNUUyhUHOaY4y8QvWBGESmAopa80f6GZ9C05tyyKANhY-qfBUB6YkfTABg',
],
]);
$this->s3Client->registerStreamWrapper();
$context = stream_context_create([
's3' => ['seekable' => true]
]);
$stream = fopen('s3://storage-bucket/RetrivedFiles/test1.txt', 'w');
$result = $this->glacier->getJobOutput([
'accountId' => '-',
'jobId' => '2dddfffffff9SwZIOPWxcB7TLm_3apNx--2rIiD7SgjOJjjkrerrcN1YCtivh_zsmpLyczY4br-bhyyX0Ev5B7e6-D1',
'vaultName' => 'archiveTest',
'saveAs' => $stream,
]);
fclose($stream);
}
According to the documentation (aws GetJobOutput operation documentation) saveAs attribute of getJobOutput function is to Specify where the contents of the operation should be downloaded. Can be the path to a file, a resource returned by fopen, or a Guzzle\Http\EntityBodyInterface object. As I am giving a path to the file in s3 also. What will be the issue. Any help is really appreciated. Thanks in advance.
This is the result contained in the response $result which is exactly same as mentioned in documentation
Aws\Result Object ( [data:Aws\Result:private] => Array ( [body] => GuzzleHttp\Psr7\Stream Object ( [stream:GuzzleHttp\Psr7\Stream:private] => Resource id #25 [size:GuzzleHttp\Psr7\Stream:private] => [seekable:GuzzleHttp\Psr7\Stream:private] => 1 [readable:GuzzleHttp\Psr7\Stream:private] => 1 [writable:GuzzleHttp\Psr7\Stream:private] => 1 [uri:GuzzleHttp\Psr7\Stream:private] => php://temp [customMetadata:GuzzleHttp\Psr7\Stream:private] => Array ( ) ) [checksum] => c176c1843fd0c0fc662lh9bb8de916540e6f9dpk9b22020bbb8388jk6f81d1c2 [status] => 200 [contentRange] => [acceptRanges] => bytes [contentType] => application/octet-stream [archiveDescription] => File Name is children-wide.jpg [#metadata] => Array ( [statusCode] => 200 [effectiveUri] => https://glacier.region-name.amazonaws.com/-/vaults/vaultname/jobs/gFdjAl4xhTAVEnmffgfg-Ao3-xmmjghfmqkCLOR1m34gHLQpMd0a3WKCiRRrItv2bklawwZnq9KeIch3LKs8suZoJwk2_/output [headers] => Array ( [x-amzn-requestid] => NzAiVAfrMQbpSjj-2228iiKWK_VteDwNyFTUR7Kyu0duno [x-amz-sha256-tree-hash] => c176c1843khfullc662f09bb8de916540e6f9dcc9b22020bbb8388de6f81d1c2 [accept-ranges] => bytes [x-amz-archive-description] => File Name is children-wide.jpg [content-type] => application/octet-stream [content-length] => 1452770 [date] => Tue, 31 Jan 2017 03:34:26 GMT [connection] => close ) [transferStats] => Array ( [http] => Array ( [0] => Array ( ) ) ) ) ) )
When you are restoring files from Glacier it will not get Standard as storage class anymore. it will still show Glacier. to determine either files came down from Glacier or not,
Use GetObject instead, and look at the Restore value of the result. And set the Range to "bytes=0-0" to skip retrieving the content of the file itself. And be sure to trap for exceptions
if the object is in Glacier and not restored, AWS will throw an InvalidObjectStateError and the script will die if the error is not caught.
This is what you will see it the item resored.
["Restore"] => string(68)"ongoing-request="false ", expiry-date=" Thu,
12 Oct 2017 00: 00: 00 GMT ""
And this is what you will get if Item is still in Glacier
Fatal error: Uncaught exception 'Aws\S3\Exception\S3Exception' with
message 'Error executing "GetObject" on
"OBJ PATH";
AWS HTTP error: Client error: GET
OBJ PATH
resulted in a 403 Forbidden response: InvalidObjectStateThe
operation is not valid for the (truncated...) InvalidObjectState
(client): The operation is not valid for the object's storage class -
InvalidObjectStateThe operation is not valid for the object 's storage
class879A42BDC3939282VjgBNmLxhqesAaOnnUKkIahdr9OlUnTPASmjh8zZNVzLeYEDz+QooqoFjyaeoyXGeAa/IPxTBrA='
GuzzleHttp\ Exception\ ClientException: Client error: `GET
OBJ PATH
in
C:\inetpub\wwwroot\cruisecheap.com\php_includes\SDKs\AWS\vendor\aws\aws-sdk-php\src\WrappedHttpHandler.php
on line 192
I hope this can help you and other people that having the same problem.
I've created an Amazon S3 bucket and I've uploaded the files/images from mobile phone app. I've to show the posts with a lot of images and the images are automatically bind for image URLs. But I don't know how to get the URL because images should not be public to show directly. How can I show them in my app?
$cmd = $client->getCommand('GetObject',[
'Bucket' => 'myinstaclassbucket',
'Key' => 'e12e682c-936d-4a97-a049-6f104dd7c904.jpg',
]);
$request = $client->createPresignedRequest($cmd,$timetoexpire);
$presignedurl = (string) $request->getUri();
echo $presignedurl;
First of all you need to use AWS PHP SDK. Also make sure you have valid Access key and Secret key.
Than everything is straight forward.
$bucket = 'some-bucket';
$key = 'mainFolder/subFolder/file.xx';
// Init client
$client = new S3Client([
'key' => '*YOUR ACCESS KEY*',
'secret' => '*YOUR SECRET KEY*',
]);
if ($client->doesObjectExists($bucket, $key)) {
// If passing `expire` time you will get signed URL
$url = $client->getObjectUrl($bucket, $key, time() + (60 * 60 * 2));
} else {
$url = null;
}
Is there any better solution implement to get aws cloudtrail logs to kibana, here I am using ElasticSearch Service from AWS
Heres the logstash input that I use with 1.4.2. It works well, though I suspect it is noisy (it requires a lot of S3 GET/HEAD/LIST requests).
input {
s3 {
bucket => "bucketname"
delete => false
interval => 60 # seconds
prefix => "cloudtrail/"
type => "cloudtrail"
codec => "cloudtrail"
credentials => "/etc/logstash/s3_credentials.ini"
sincedb_path => "/opt/logstash_cloudtrail/sincedb"
}
}
filter {
if [type] == "cloudtrail" {
mutate {
gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
add_field => {
"document_id" => "%{eventID}"
}
}
if ! [ingest_time] {
ruby {
code => "event['ingest_time'] = Time.now.utc.strftime '%FT%TZ'"
}
}
ruby {
code => "event.cancel if (Time.now.to_f - event['#timestamp'].to_f) > (60 * 60 * 24 * 1)"
}
ruby {
code => "event['ingest_delay_hours'] = (Time.now.to_f - event['#timestamp'].to_f) / 3600"
}
# drop events more than a day old, we're probably catching up very poorly
if [ingest_delay_hours] > 24 {
drop {}
}
# example of an event that is noisy and I don't care about
if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "deploy-s3" {
drop {}
}
}
}
The credentials.ini format is explained on the s3 input page; it's just this:
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
I also have a search that sends results to our #chatops but I'm not posting that here.
If you haven't tried it already, you can use cloudtrail and cloudwatch logs together. Then use cloudwatch logs to create a subscription to send the cloudtrail data to elasticsearch.
Once that is done you should be able to define a kibana index that starts with cwl* that is time based.
Cheers-
I am completely new to Perl and I need to create a tool that will consume a SOAP Webservice and I need to save the XML provided by this WS in an output file. At this point I can consume the webservice and save it as hash data, but I need it to be in XML format.
My code is pretty simple and goes like this:
#!/usr/bin/perl -w
use SOAP::Lite ( +trace => "all", maptype => {} );
use IO::File;
use Data::Dump "pp";
sub SOAP::Transport::HTTP::Client::get_basic_credentials {
return 'username' => 'password';
}
my $soap = SOAP::Lite
-> proxy('https://.../WebService.do?SOAP', ssl_opts => [ SSL_verify_mode => 0 ] );
my $method = SOAP::Data->name('execute') -> attr({xmlns => 'http://.../'});
my $output = IO::File->new(">output.xml");
my %keyHash = %{ $soap->call($method)->body};
print $output pp({%keyHash});
$output->close();
As I have the trace full on, I can see the XML that the webservice provide in the console while my program is executed, but, when it gets printed in the output file, I see the hash as defined in Perl, with the pairs of key => values, organized just like if it was a json:
{
Docs => {
AssetDefinition => "AccountNumber",
BatchId => 1,
Doc => [
{
AssetDefinitionId => "CNTR0016716",
DateForRetention => "",
FileName => "",
FilePath => "",
SequenceNumber => "",
},
],
},
}
The data is completely correct, but I needed it saved in the file as XML and at this point I think I am going in the wrong direction.
Any help will be fairly appreciated.
Thanks and regards,
Felipe
You are on the right track. The soap call just returns a perl data structure, a hash of hashes. You need an additional step to convert it to XML.
I would recommedn this module http://search.cpan.org/~grantm/XML-Simple-2.20/lib/XML/Simple.pm
use XML::Simple qw(:strict);
my $xml = XMLout(\%keyHash);
You can supply options to give more control over the XML formatting