I have created SMB file_share to share files to s3 bucket by using below terraform code.
resource "aws_storagegateway_smb_file_share" "smb-file-shares" {
file_share_name = "file-share"
authentication = "ActiveDirectory"
gateway_arn = "arn:aws:storagegateway:us-east-1:123456789:gateway/gw-5GHFtg"
location_arn = "arn:aws:s3:::${module.s3-bucket.name}"
role_arn = module.gateway_role.role_arn
smb_acl_enabled = true
oplocks_enabled = false
object_acl = "bucket-owner-full-control"
valid_user_list = ["#Group1"]
}
But i am unable to copy or download files from s3 bucket by using SMB file_share mapped drive in my local. Is there any way to copy or download files from s3 bucket by using SMB file_share ?
If i tried to copy file through SMB file_share (mapped drive) i am getting below error.
Related
I'm using Databricks - python runtime 8.2.
There are two S3 buckets:
Remote S3 - belongs to someone else, I have access ONLY to bucketname/my_prefix only
Local S3 - belongs to me, I have full access to the bucket policy (can mount it to dbfs for example, copy files from dbfs to local s3.
My mission is to copy LOTs of files from remote S3 which I can't mount to local S3.
From Databricks Mount S3 using AWS access keys:
Important
When you mount an S3 bucket using keys, all users have read and write access to all the objects in the S3 bucket.
I have tried to use boto3 library but I cant seem to get much luck.
Essentially, is there a way to copy files from remote S3 to dbfs?
Here is one version of the code I am using - I have tried SO many other ways.
# test if list of days to copy is not empty; loop through each
day and copy the s3 folder contents
if len(days_to_copy) != 0:
for day in days_to_copy:
#convert the items in the list from date to string and format yyyy/mm/dd
day_to_copy = day.strftime('%Y/%m/%d')
print(day_to_copy)
source_bucket = "Remote-S3"
source_key = "myclient/2021/10/06/part-000d339cb-
c000.snappy.parquet"
target_bucket = "Local-s3"
target_key = "{}/".format(day_to_copy)
copy_source = {
'Bucket': source_bucket,
'Key': source_key
}
copy_target = {
'Bucket': target_bucket,
'Key': target_key
}
print('copy_source',copy_source)
print('copy_target',copy_target)
s3.meta.client.copy(copy_source, 'Local-s3','2021/10/06/part-000d339cb-c000.snappy.parquet' )
This code gives this error:
ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
Thanks.
I am fairly new at terraform and wanted to know if there is a way for terraform to access a directory stored in local computer (for example in the Downloads folder) to be used to upload to s3?
If not what would be the best way to go about it. I would appreciate any help on this.
Thank you
Terraform provides path.root which will get the root path to the script. You can combine your path to the root path & upload the file.
resource "aws_s3_bucket_object" "object" {
bucket = aws_s3_bucket.b1.id # bucket
key = "profile"
acl = "private" # or can be "public-read"
source = "${abspath(path.root)/../Downloads}" # ubuntu
etag = filemd5("myfiles/yourfile.txt")
}
I have access only to the specific folder in S3 bucket.
For S3 client builder I was using the following code for uploading to S3 bucket to the specified folder.
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(region)
.build();
(I am running the code from server which has access to S3 so credentials not required,I was able to view the buckets and folders from cli)
putObjectRequest = new PutObjectRequest(awsBucketName, fileName, multipartfile.getInputStream(), null)
I even tried giving bucketname along with prefix because I have access only for the specific folder.I was getting access denied status 403.
So for S3 client builder, I am trying to use endpoint configuration rather than just specifying the region. I got the following error.
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withEndpointConfiguration(endpointConfiguration)
.withClientConfiguration(new ClientConfiguration().withProtocol(Protocol.HTTP))
.build();
com.amazonaws.SdkClientException: Unable to execute HTTP request: null
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064)
What should I do or how should I verify if it correctly maps to the bucket and folder.
I got it when I used Presigned URL(refer aws presigned url documentation which has the example code as well for java) to upload to a folder only for which you have access (that is you dont have access for the bucket)
I need to enable the bucket replication for my s3 bucket.
I referred below 2 links for this and written the code (see below). But replication not enabled. But weirdly, a file created in the source bucket.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_bucket_replication
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTreplication.html
Could anyone help me with this?
source bucket name = tst;
Destination bucket name = rst
import boto3
s3 = boto3.client('s3',endpoint_url=S3_URL,aws_access_key_id=ACCESS_ID,aws_secret_access_key=SECRET_KEY,region_name=REGION)
s3.put_bucket_replication(Bucket='tst',ReplicationConfiguration={'Role': 'arn:aws:iam::10000003:root','Rules': [{'Status': 'Enabled','Destination': {'Bucket': 'arn:aws:s3:::rst'}}]})
I try to transfer a directory and its sub-directories to an s3 bucket by code.
In my case a sub-directory is a partition in the bucket.
As in the picture:
Are you trying to upload your directory structure to s3?
Then you can try this with the CLI:
aws s3 sync . s3://mybucket
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
You could also achieve this programatically:
TransferManager tm = new TransferManager(new ProfileCredentialsProvider());
MultipleFileUpload upload = tm.uploadDirectory(bucket, folder, new File(filePath), true);
try
{
// Or you can block and wait for the upload to finish
upload.waitForCompletion();
LOGGER.debug("Upload complete.", bucket, folder);
}
catch (AmazonClientException amazonClientException)
{
LOGGER.error("Unable to upload file, upload was aborted.", amazonClientException);
throw amazonClientException;
}
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-transfermanager.html