I want to upload all the files/folder inside a directory to a S3 bucket. I want to upload including the all the files in all the sub directories. I thought of using TransferUtility to do this. Though the link here says that 'By default, Amazon S3 only uploads the files at the root of the specified directory. You can, however, specify to recursively upload files in all the subdirectories.' but I couldnt find a way to do this. I coundnt find any property where I can mention to include all the sub directories. I tried using SearchOption = System.IO.SearchOption.AllDirectories and SearchPattern = "*" to achieve this, but still it uploaded only the files in the top most directory. Please help me in this. Thanks.
I'm using the below code,
TransferUtility directoryTransferUtility = new TransferUtility(s3Client);
TransferUtilityUploadDirectoryRequest uRequest = new TransferUtilityUploadDirectoryRequest()
{
Directory = dirPath,
BucketName = bucketName,
SearchOption = System.IO.SearchOption.AllDirectories,
SearchPattern = "*"
};
directoryTransferUtility.UploadDirectory(dirPath, bucketName);
This is what worked for me: I set the options on the UploadDirectory method and used "*.*" as the search pattern.
directoryTransferUtility.UploadDirectory(dirPath,
bucketName,
"*.*",
SearchOption.AllDirectories);
Related
I have a django app that allows files to be uploaded to an S3 bucket using django-storages. The file is for a part that needs to be approved. Once it is approved, I'd like to move the file to a different S3 bucket.
class DesignData(models.Model):
file = models.FileField(storage=PublicMediaStorage())
...
class PublicMediaStorage(S3Boto3Storage):
location = "media"
default_acl = "public-read"
file_overwrite = False
After approval, I copy the file over to the new bucket using:
client.copy_object(
Bucket=settings.AWS_APPROVED_STORAGE_BUCKET_NAME,
CopySource=copy_source,
Key=design_data["s3key"],
)
The file gets moved correctly however I need to update my object. How can I update the object? Trying something like myObject.file = "newbucket/myfile.txt" won't work as it is expecting an actual file. I've read I should be able to update the url with myObject.file.url = "newbucketaddress/myfile.txt" but I get an error AttributeError: can't set attribute.
Is there a way in django-storages with s3 to update an existing file s3 bucket?
You may have to manually build the path to the new bucket. Boto3 docs will guide you.
Get the bucket location, name and object key. You can build the path to the object
I ended up using somewhat of a workaround to resolve my issue. I ended up changing my model to include another file which would then store the new s3 bucket location.
class DesignData(models.Model):
file = models.FileField(storage=PublicMediaStorage())
approved_file = models.FileField(storage=ApprovedPublicMediaStorage())
I added ApprovedPublicMediaStorage():
class PublicMediaStorage(S3Boto3Storage):
location = "media"
default_acl = "public-read"
file_overwrite = False
custom_domain = "newbucketlocation0.s3...."
I did learn the hard way so make sure to include the custom_domain otherwise it will use the default_storage in the settings if it has been assigned. After I copy the file over to the new storage, I delete the old file and change file = null so only approved_file contains an s3 object.
I am using AWS S3 JS SDK. I have folders within folders in my S3 bucket and I would like to list only folders at a certain level.
This is the structure:
bucket/folder1/folder2/folder3a/file1
bucket/folder1/folder2/folder3a/file2
bucket/folder1/folder2/folder3a/file3
bucket/folder1/folder2/folder3a/...
bucket/folder1/folder2/folder3b/file1
bucket/folder1/folder2/folder3b/file2
bucket/folder1/folder2/folder3b/file3
bucket/folder1/folder2/folder3b/...
bucket/folder1/folder2/folder3c/file1
bucket/folder1/folder2/folder3c/file2
bucket/folder1/folder2/folder3c/file3
bucket/folder1/folder2/folder3c/...
As you can see, at the level of folder 3, I have multiple folders and each of those folders contain multiple items. I don't care about the items. I just want to list the folder names at level 3. Is there a good way to do this?
The only way I found is to use ListObjectsV2. But this gives me also the files which inflates the results set and I would need to do a manual filtering afterwards. Is there a way to get just the folder names at the API level?
This article answers all my questions. https://realguess.net/2014/05/24/amazon-s3-delimiter-and-prefix/
The solution can be done using the combination of prefix and delimiter. In my examples the parameters should contain the following:
const params = {
Bucket: 'bucket',
Prefix: 'folder1/folder2/',
Delimiter: '/',
};
Be sure to not forget the slash at the end of the Prefix parameter.
The list of folders will be in the CommonPrefixes attribute of the response object.
To give you a real life example:
...
const params = {
Bucket: bucketName,
Prefix: prefix + '/',
MaxKeys: 25,
Delimiter: '/',
};
const command = new ListObjectsV2Command(params);
const results = await s3client.send(command);
const foldersList = results.CommonPrefixes;
...
I set the projectId, the bucket name and content type of a file, but I need te upload it to a particular folder into the destination bucket. How do I set the complete url?
This is my preliminary code:
I tried to add a directory inside bucketname or inside filename, but it doesn't work. It seems to be a parameter, but I don't know where do I have to set it.
var newObject = new Google.Apis.Storage.v1.Data.Object{
Bucket = "bucketName",
Name = "filename",
ContentType = "fileContentType"};
var credential = Google.Apis.Auth.OAuth2.GoogleCredential.FromJson(System.IO.File.ReadAllText("credentials.json"));
using (var storageClient = Google.Cloud.Storage.V1.StorageClient.Create(credential)){
using (var fileStream = new System.IO.FileStream(filePath, System.IO.FileMode.Open)){
var uploadObjectOptions = new Google.Cloud.Storage.V1.UploadObjectOptions();
await storageClient.UploadObjectAsync(newObject,fileStream,uploadObjectOptions,progress: null).ConfigureAwait(false);
}
}
return Ok();
You're adding the "folder" in the wrong place. Note that Google Cloud Storage doesn't have real folders or directories, instead it uses simulated directories which is really just an object with a prefix in its name.
bucket = bucket
object = folderName/objectname
Hey guys I was trying to delete a folder from s3 with stuff in it but deleteObjects wasn't working so I found this script online and it works great my question is why does it work? Why do you have to listObjects when deleting a folder on s3 why cant I just pass it the objects name? Why doesn't It error when I attempt to delete the folder without listing the objects first.
first attempt (doesnt work)
var filePath2 = "templates/" + key + "/test/";
var toPush = { Key: filePath2 };
deleteParams.Delete.Objects.push(toPush);
console.log("deleteParams", deleteParams);
console.log("deleteParams.Delete", deleteParams.Delete);
const deleteResult = await s3.deleteObjects(deleteParams).promise();
console.log("deleteResult", deleteResult);
keep in mind folderPath2 is a folder that has other stuff in it I get no error but yet the catch isn't triggered and it says deleted and than the folder name.
second attempt (works)
async function deleteFromS3(bucket, path) {
const listParams = {
Bucket: bucket,
Prefix: path
};
const listedObjects = await s3.listObjectsV2(listParams).promise();
console.log("listedObjects", listedObjects);
if (listedObjects.Contents.length === 0) return;
const deleteParams = {
Bucket: bucket,
Delete: { Objects: [] }
};
listedObjects.Contents.forEach(({ Key }) => {
deleteParams.Delete.Objects.push({ Key });
});
console.log("deleteParams", deleteParams);
const deleteResult = await s3.deleteObjects(deleteParams).promise();
console.log("deleteResult", deleteResult);
if (listedObjects.IsTruncated && deleteResult)
await deleteFromS3(bucket, path);
}
than I call the function like so
const result = await deleteFromS3(myBucketName, folderPath);
Folders do not exist in Amazon S3. It is a flat object storage system, where the filename (Key) for each object contains the full path.
While Amazon S3 does support the concept of a Common Prefix, which can make things appear as though they are in folders/directories, folders do not actually exist.
For example, you could run a command like this:
aws s3 cp foo.txt s3://my-bucket/folder1/folder2/foo.txt
This would work even if the folders do not exist! It is merely storing an object with a Key of folder1/folder2/foo.txt.
If you were then to delete that object, the 'folder' would disappear because no object has it as a path. That is because the folder never actually existed.
Sometimes people want an empty folder to appear, so they create a zero-length object with the same name as the folder, eg folder1/folder2/.
So, your first program did not work because it deleted the 'folder', which has nothing to do with deleting the content of the folder (since there is no concept of 'content' of a folder).
I was able to create a bucket in an amazon S3 using this link.
I used the following code to create a bucket :
resource "aws_s3_bucket" "b" {
bucket = "my_tf_test_bucket"
acl = "private"
}
Now I wanted to create folders inside the bucket, say Folder1.
I found the link for creating an S3 object. But this has a mandatory parameter source. I am not sure what this value have to , since my intent is to create a folder inside the S3 bucket.
For running terraform on Mac or Linux, the following will do what you want
resource "aws_s3_bucket_object" "folder1" {
bucket = "${aws_s3_bucket.b.id}"
acl = "private"
key = "Folder1/"
source = "/dev/null"
}
If you're on windows you can use an empty file.
While folks will be pedantic about s3 not having folders, there are a number of operations where having an object placeholder for a key prefix (otherwise called a folder) make life easier. Like s3 sync for example.
Actually, there is a canonical way to create it, without being OS dependent, by inspecting the Network on a UI put you see the content headers, as stated by : https://stackoverflow.com/users/1554386/alastair-mccormack ,
And S3 does support folders these days as visible from the UI.
So this is how you can achieve it:
resource "aws_s3_bucket_object" "base_folder" {
bucket = "${aws_s3_bucket.default.id}"
acl = "private"
key = "${var.named_folder}/"
content_type = "application/x-directory"
kms_key_id = "key_arn_if_used"
}
Please notice the trailing slash otherwise it creates an empty file
Above has been used with a Windows OS to successfully create a folder using terraform s3_bucket_object.
The answers here are outdated, it's now definitely possible to create an empty folder in S3 via Terraform. Using the aws_s3_object resource, as follows:
resource "aws_s3_bucket" "this_bucket" {
bucket = "demo_bucket"
}
resource "aws_s3_object" "object" {
bucket = aws_s3_bucket.this_bucket.id
key = "demo/directory/"
}
If you don't supply a source for the object then terraform will create an empty directory.
IMPORTANT - Note the trailing slash this will ensure you get a directory and not an empty file
S3 doesn't support folders. Objects can have prefix names with slashes that look like folders, but that's just part of the object name. So there's no way to create a folder in terraform or anything else, because there's no such thing as a folder in S3.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
http://docs.aws.amazon.com/AWSImportExport/latest/DG/ManipulatingS3KeyNames.html
If you want to pretend, you could create a zero-byte object in the bucket named "Folder1/" but that's not required. You can just create objects with key names like "Folder1/File1" and it will work.
old answer but if you specify the key with the folder (that doesn't exist yet) terraform will create the folder automatically for you
terraform {
backend "s3" {
bucket = "mysql-staging"
key = "rds-mysql-state/terraform.tfstate"
region = "us-west-2"
encrypt = true
}
}
I would like to add to this discussion that you can create a set of empty folders by providing the resource a set of strings:
resource "aws_s3_object" "default_s3_content" {
for_each = var.default_s3_content
bucket = aws_s3_bucket.bucket.id
key = "${each.value}/"
}
where var.default_s3_content is a set of strings:
variable "default_s3_content" {
description = "The default content of the s3 bucket upon creation of the bucket"
type = set(string)
default = ["folder1", "folder2", "folder3", "folder4", "folder5"]
}
v0.12.8 introduces a new fileset() function which can be used in combination with for_each to support this natively :
NEW FEATURES:
lang/funcs: New fileset function, for finding static local files that
match a glob pattern. (#22523)
A sample usage of this function is as follows (from here):
# Given the file structure from the initial issue:
# my-dir
# |- file_1
# |- dir_a
# | |- file_a_1
# | |- file_a_2
# |- dir_b
# | |- file_b_1
# |- dir_c
# And given the expected behavior of the base_s3_key prefix in the initial issue
resource "aws_s3_bucket_object" "example" {
for_each = fileset(path.module, "my-dir/**/file_*")
bucket = aws_s3_bucket.example.id
key = replace(each.value, "my-dir", "base_s3_key")
source = each.value
}
At the time of this writing, v0.12.8 is a day old (Released on 2019-09-04) so the documentation on https://www.terraform.io/docs/providers/aws/r/s3_bucket_object.html does not yet reference it. I am not certain if that's intentional.
As an aside, if you use the above, remember to update/create version.tf in your project like so:
terraform {
required_version = ">= 0.12.8"
}