I want to use AWS S3 to store image files for my website. I create a bucket name images.mydomain.com which was referred by dns cname images.mydomain.com from AWS Route 53.
I want to check whether a folder or file exists; if not I will create one.
The following php codes work fine for regular bucket name using stream wrapper but fails for this type of bucket name such as xxxx.mydomain.com. This kind of bucket name fails in doesObjectExist() method too.
// $new_dir = "s3://aaaa/akak3/kk1/yy3/ww4" ; // this line works !
$new_dir = "s3://images.mydomain.com/us000000/10000" ; // this line fails !
if( !file_exists( $new_dir) ){
if( !mkdir( $new_dir , 0777 , true ) ) {
echo "create new dir $new_dir failed ! <br>" ;
} else {
echo "SUCCEED in creating new dir $new_dir <br" ;
}
} else {
echo "dir $new_dir already exists. Skip creating dir ! <br>" ;
}
I got the following message
Warning: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint: "images.mydomain.com.s3.amazonaws.com". in C:\AppServ\www\ecity\vendor\aws\aws-sdk-php\src\Aws\S3\StreamWrapper.php on line 737
What is the problem here?
Any advise on what to do for this case?
Thanks!
Related
When i try to execute below code, i am getting this error. Am i missing anything in the json format ?
```
Error parsing parameter '--change-batch': Invalid JSON: Expecting value: line 7 column 14 (char 160)
JSON received: {
"Comment":"To update private dns name recordset",
"Changes":[
{
"Action":"UPSERT",
"ResourceRecordSet":{
"Name":devdnstest.test.com.,
"Type":,
"TTL":300,
"ResourceRecords":[
{
"Value":"10.0.0.2"
}
]
}
}
]
}
#!/bin/bash
#Variable Declaration
HOSTED_ZONE_ID="Z2UERB3PG6W9Y"
NAME="devdnstest.test.com."
TYPE="CNAME"
TTL=300
#get current recordset IP from Route 53
aws route53 list-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID | \
jq -r '.ResourceRecordSets[] | select (.Name == "'"$NAME"'") | select (.Type == "'"$TYPE"'") | .ResourceRecords[0].Value' > /tmp/current_route53_value
cat /tmp/current_route53_value
#prepare route 53 file
file=/home/route53_changes.json
cat << EOF > $file
{
"Comment":"To update private dns name recordset",
"Changes":[
{
"Action":"UPSERT",
"ResourceRecordSet":{
"Name":$NAME,
"Type":$CNAME,
"TTL":$TTL,
"ResourceRecords":[
{
"Value":"10.0.0.2"
}
]
}
}
]
}
EOF
#update records
aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch file://$file
In this code i am trying to update a private hosted zone record set from 10.0.0.1 to 10.0.0.2.
It's complaining about the "Type":$CNAME. But not sure what else to keep in there.
Appreciate your inputs.
Thanks,
The data you're sending is not JSON - in particular,
"Name":devdnstest.test.com.,
"Type":,
There may be other tiny errors in there, but this will get you started.
In your original shell script, you'll need to add quotes and fix $NAME to $TYPE, something along the lines of
"Name":"$NAME",
"Type":"$TYPE",
For other people coming across this issue, you'll get a (slightly different) "Invalid JSON" message for syntactically correct JSON that doesn't conform to what AWS Route53 is expecting also.
In AWS SSM, I use RunRemoteScript document to run a PowerShell script to install some software on SSM managed instances. The script is hosted in a public accessible S3 bucket.
The RunCommand works fine with the script not taking any parameters. Software was successfully deployed to managed instances. But my script has a unique CID embedded in the code. For security reasons, I need to take it out and set it as a parameter for the PS script. Ever since then, the RunCommand just keeps failing.
My script looks like below (with parameter CID):
param (
[Parameter(Position = 0, Mandatory = 1)]
[string]$CID
)
Start-Transcript -Path "$([System.Environment]::GetEnvironmentVariable('TEMP','Machine'))\app_install.log" -Append
function Install-App {
<#
Installs App
#>
[CmdletBinding()]
[OutputType([PSCustomObject])]
param (
[Parameter(Position = 0, Mandatory = 1)]
[string]$msiURL,
[Parameter(Position = 2, Mandatory = 1)]
[string]$InstallCheck,
[Parameter(Position = 3, Mandatory = 1)]
[string]$CustomerID
)
if ( -not(Test-Path $installCheck)) {
# Do stuff
...
}
else {
Write-Host ("$installCheck - Already Installed")
Return "Already Installed, Skipped $(($msiURL -split '([^\\/]+$)')[1])"
}
}
Install-App -msiURL "https://s3.amazonaws.com/app.foo.com/Windows/app.exe" -InstallCheck "C:\Program Files\App\app.exe" -CustomerID $CID
Stop-Transcript
By following AWS SSM documentation below, I run the command below to kick off the RunCommand.
https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-remote-scripts.html
aws ssm send-command --document-name "AWS-RunRemoteScript" --targets "Key=instanceids,Values=mi-abc12345"
--parameters '{"sourceType":["S3"],"sourceInfo":["{\"path\": "https://s3.amazonaws.com/app.foo.com/Windows/app_install.ps1\"}"],"commandLine":["app_install.ps1 abcd123456"]}'
The RunCommand keeps failing with error below:
----------ERROR-------
app_install.ps1 : The term 'app_install.ps1' is not recognized
as the name of a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path is
correct and try again.
At C:\ProgramData\Amazon\SSM\InstanceData\mi-abcd1234\document\orchest
ration\a6811111d-c411-411-a222-bad123456\runPowerShellScript\_script.ps1:4
char:2
+ app_install.ps1 abcd123456
+ ~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (app_install.ps1:String)
[], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
failed to run commands: exit status 255
I suspect this is to do with the way how RunCommand handles the argument for the PowerShell script. But I cannnot find any examples other than the official document, which I followed. Anyone can point out what the issue is here?
BTW, I already tried putting the ps1 after ".\" without luck.
I found out the cause of the issue. The IAM role attached to the instance did not have sufficient rights to access the S3 bucket holds the script. As a result SSM wasn't able to download the script to the instance, hence the error "...ps1 is not recognized".
So it's not related to the code actually.
I am using 'terraform apply' in a shell script to create multiple EC2 instances. I need to output the list of generated IPs to a script variable & use the list in another sub-script. I have defined output variables for the ips in a terraform config file - 'instance_ips'
output "instance_ips" {
value = [
"${aws_instance.gocd_master.private_ip}",
"${aws_instance.gocd_agent.*.private_ip}"
]
}
However, the terraform apply command is printing entire EC2 generation output apart from the output variables.
terraform init \
-backend-config="region=$AWS_DEFAULT_REGION" \
-backend-config="bucket=$TERRAFORM_STATE_BUCKET_NAME" \
-backend-config="role_arn=$PROVISIONING_ROLE" \
-reconfigure \
"$TERRAFORM_DIR"
OUTPUT = $( terraform apply <input variables e.g -
var="aws_region=$AWS_DEFAULT_REGION">
-auto-approve \
-input=false \
"$TERRAFORM_DIR"
)
terraform output instance_ips
So the 'OUTPUT' script variable content is
Terraform command: apply Initialising the backend... Successfully
configured the backend "s3"! Terraform will automatically use this
backend unless the backend configuration changes. Initialising provider
plugins... Terraform has been successfully initialised!
.
.
.
aws_route53_record.gocd_agent_dns_entry[2]: Creation complete after 52s
(ID:<zone ............................)
aws_route53_record.gocd_master_dns_entry: Creation complete after 52s
(ID:<zone ............................)
aws_route53_record.gocd_agent_dns_entry[1]: Creation complete after 53s
(ID:<zone ............................)
Apply complete! Resources: 9 added, 0 changed, 0 destroyed. Outputs:
instance_ips = [ 10.39.209.155, 10.39.208.44, 10.39.208.251,
10.39.209.227 ]
instead of just the EC2 ips.
Firing the 'terraform output instance_ips' is throwing a 'Initialisation Required' error which I understand means 'terraform init' is required.
Is there any way to suppress ec2 generation & just print output variables. if not, how to retrieve the IPs using 'terraform output' command w/o needing to do a terraform init ?
If I understood the context correctly, you can actually create a file in that directory & that file can be used by your sub-shell script. You can do it by using a null_resource OR "local_file".
Here is how we can use it in a modularized structure -
Using null_resource -
resource "null_resource" "instance_ips" {
triggers {
ip_file = "${sha1(file("${path.module}/instance_ips.txt"))}"
}
provisioner "local-exec" {
command = "echo ${module.ec2.instance_ips} >> instance_ips.txt"
}
}
Using local_file -
resource "local_file" "instance_ips" {
content = "${module.ec2.instance_ips}"
filename = "${path.module}/instance_ips.txt"
}
Has anyone had any success migrating files from the Parse S3 Bucket to an S3 Bucket of their own? I have an app that contains many files (images) and I have them serving from both my own S3 Bucket and from the Parse Bucket using the S3 File Adapter but would like to migrate the physical files to my own Bucket on AWS where the app will now be hosted.
Thanks in advance!
If you've configured your new Parse instance to host files with the S3 file adapter, you could write a PHP script that downloads files from Parse S3 Bucket and upload it to your own. In my example (using Parse-PHP-SDK):
I do a loop through every entry.
I download the binary of that file (hosted in Parse)
I upload it as a new ParseFile (if your server is configured for S3, it will be uploaded to S3 bucket of your own).
Apply that new ParseFile to your entry.
Voilà
<?php
require 'vendor/autoload.php';
use Parse\ParseObject;
use Parse\ParseQuery;
use Parse\ParseACL;
use Parse\ParsePush;
use Parse\ParseUser;
use Parse\ParseInstallation;
use Parse\ParseException;
use Parse\ParseAnalytics;
use Parse\ParseFile;
use Parse\ParseCloud;
use Parse\ParseClient;
$app_id = "AAA";
$rest_key = "BBB";
$master_key = "CCC";
ParseClient::initialize( $app_id, $rest_key, $master_key );
ParseClient::setServerURL('http://localhost:1338/','parse');
$query = new ParseQuery("YourClass");
$query->descending("createdAt"); // just because of my preference
$count = $query->count();
for ($i = 0; $i < $count; $i++) {
try {
$query->skip($i);
// get Entry
$entryWithFile = $query->first();
// get file
$parseFile = $entryWithFile->get("file");
// filename
$fileName = $parseFile->getName();
echo "\nFilename #".$i.": ". $fileName;
echo "\nObjectId: ".$entryWithFile->getObjectId();
// if the file is hosted in Parse, do the job, otherwise continue with the next one
if (strpos($fileName, "tfss-") === false) {
echo "\nThis is already an internal file, skipping...";
continue;
}
$newFileName = str_replace("tfss-", "", $fileName);
$binaryFile = file_get_contents($parseFile->getURL());
// null by default, you don't need to specify if you don't want to.
$fileType = "binary/octet-stream";
$newFile = ParseFile::createFromData($binaryFile, $newFileName, $fileType);
$entryWithFile->set("file", $newFile);
$entryWithFile->save(true);
echo "\nFile saved\n";
} catch (Exception $e) {
// The conection with mongo or the server could be off for some second, let's retry it ;)
$i = $i - 1;
sleep(10);
continue;
}
}
echo "\n";
echo "¡FIN!";
?>
I am trying to add multiple DNS records using this script add_multipleDNSrecord.sh and i am getting this error
A client error (InvalidChangeBatch) occurred when calling the ChangeResourceRecordSets operation: FATAL problem: UnsupportedCharacter (Value contains unsupported characters) encountered with ' '
But i am able to add single record without any issue from aws cli. can anyone please tell me what went wrong in this script?
#!/bin/bash
# declare STRING variable
STRING="Hello World"
#print variable on a screen
echo $STRING
# Hosted Zone ID
ZONEID="Z24*************"
#Comment
COMMENT="Add new entry to the zone"
# The Time-To-Live of this recordset
TTL=300
# Type
TYPE="A"
# Input File Name
FILENAME=/home/ec2-user/awscli/route53/scripts/test.json
cat >> $FILENAME << EOF
{
"Comment":"$COMMENT",
"Changes":[
{
"Action":"CREATE",
"ResourceRecordSet":{
"ResourceRecords":[
{
"Value":"$IP"
}
],
"Name":"$RECORDSET",
"Type":"$TYPE",
"TTL":$TTL
}
}
]
}
EOF
echo $FILENAME
After Replacing the space and using dot instead of space solves the problem.
Now,The script works fine and its able to add multiple records to the hosted zone.