I am trying to add external storage to my Nextcloud to use. That would be an AWS S3 bucket. However, this is not possible because I get the following error message:
Exception: Creation of bucket \"nextcloud-modul346\" failed. Error executing \"CreateBucket\" on \"http:\/\/nextcloud-modul346.s3.eu-west-1.amazonaws.com\/\"; AWS HTTP error: Client error: `PUT http:\/\/nextcloud-modul346.s3.eu-west-1.amazonaws.com\/` resulted in a `403 Forbidden` response:\n\u003C?xml version=\"1.0\" encoding=\"UTF-8\"?\u003E\n\u003CError\u003E\u003CCode\u003EInvalidAccessKeyId\u003C\/Code\u003E\u003CMessage\u003EThe AWS Access Key Id you provided (truncated...)\n InvalidAccessKeyId (client): The AWS Access Key Id you provided does not exist in our records. - \u003C?xml version=\"1.0\" encoding=\"UTF-8\"?\u003E\n\u003CError\u003E\u003CCode\u003EInvalidAccessKeyId\u003C\/Code\u003E\u003CMessage\u003EThe AWS Access Key Id you provided does not exist in our records.\u003C\/Message\u003E\u003CAWSAccessKeyId\u003EASIARERFVIEWRBG5WD63\u003C\/AWSAccessKeyId\u003E\u003CRequestId\u003EM6BN3MC6F0214DQM\u003C\/RequestId\u003E\u003CHostId\u003EgVf0nUVJXQDL2VV50pP0qSzbTi+N+8OMbgvj4nUMv10pg\/T5VVccb4IstfopzzhuxuUCtY+1E58=\u003C\/HostId\u003E\u003C\/Error\u003E
However, I cannot use IAM users or groups as this is blocked by my organization. Also, I work with the AWS Learner Lab and I have to use S3.
As credentials I have specified in Nextcloud the aws_access_key_id and aws_secret_access_key from Learnerlab. However, I cannot connect to it. This Post havn't helped either.
Does anyone know a solution to this problem which does not involve IAM?
Thanks for any help!
Related
I have moved my application from a Xampp server to a Centos7 server. I have an s3 bucket on AWS which contains the code for the front end of my application and to deploy my code to that bucket I have to run the following command in the terminal:
aws s3 sync build/ s3://application_name --profile=profile_name -- acl public-read -- exclude 'node_modules/*|.env
The error I got was that the profile was not recognised so I then the following command:
aws configure –profile=”profile_name”
Then I provided the following details which I will not give all the answers too:
AWS_ACCESS_KEY_ID =
AWS_SECRET_ACCESS_KEY =
AWS_REGION = eu-west-1
Output = json
I then got the following error
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
I am now unsure what to do. Can someone help? Thank you.
As per the error, it seems that you don't have s3:ListBucket permission associated with the access credentials you are using.
You also need to check the bucket policy to see if there is an explicit deny present over there for other users and if you are falling into that category. Generally, to secure the bucket security team allow only a few users to access the bucket.
Background
I am attempting to upload a file to an AWS S3 bucket in Jenkins. I am using the steps/closures provided by the AWS Steps plugin. I am using an Access Key ID and an Access Key Secret and storing it as a username and password, respectively, in Credential Manager.
Code
Below is the code I am using in a declarative pipeline script
sh('echo "test" > someFile')
withAWS(credentials:'AwsS3', region:'us-east-1') {
s3Upload(file:'someFile', bucket:'ec-sis-integration-test', acl:'BucketOwnerFullControl')
}
sh('rm -f someFile')
Here is a screenshot of the credentials as they are stored globally in Credential Manager.
Issue
Whenever I run the pipeline I get the following error
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5N9VEJBY5MDZ2K0W; S3 Extended Request ID: AJmuP635cME8m035nA6rQVltCCJqHDPXsjVk+sLziTyuAiSN23Q1j5RtoQwfHCDXAOexPVVecA4=; Proxy: null), S3 Extended Request ID: AJmuP635cME8m035nA6rQVltCCJqHDPXsjVk+sLziTyuAiSN23Q1j5RtoQwfHCDXAOexPVVecA4=
Does anyone know why this isn't working?
Trouble Shooting
I have verified the Access Key ID and Access Key Secret combination works by testing it out through a small Java application I wrote. Additionally I set the id/secret via Java system properties ( through the script console ), but still get the same error.
System.setProperty("aws.accessKeyId", "<KEY_ID>")
System.setProperty("aws.secretKey", "<KEY_SECRET>")
I also tried to change the credential manager type from username/password to aws credentials as seen below. It made no difference
it might be a bucket and object ownership issue. check if the credentials you use allow you to upload to the bucket ec-sis-integration-test.
I'm trying to get at the Common Crawl news S3 bucket, but I keep getting a "fatal error: Unable to locate credentials" message. Any suggestions for how to get around this? As far as I was aware Common Crawl doesn't even require credentials?
From News Dataset Available – Common Crawl:
You can access the data even without a AWS account by adding the command-line option --no-sign-request.
I tested this by launching a new Amazon EC2 instance (without an IAM role) and issuing the command:
aws s3 ls s3://commoncrawl/crawl-data/CC-NEWS/
It gave me the error: Unable to locate credentials
I then ran it with the additional parameter:
aws s3 ls s3://commoncrawl/crawl-data/CC-NEWS/ --no-sign-request
It successfully listed the directories.
I am using Boto 3 to get s3 bucket access control information with real only permission on aws account, but unable to get it and getting an error
response2 = client.get_public_access_block(Bucket='BUCKETNAME')
print(response2['PublicAccessBlockConfiguration']['BlockPublicAcls'])
botocore.exceptions.ClientError: An error occurred (NoSuchPublicAccessBlockConfiguration) when calling the GetPublicAccessBlock operation: The public access block configuration was not found
Please suggest what all permissions as AWS Account user I need to get access information of s3 buckets
I had a same problem with boto3. But i resolved the problem, changing region.
so the region us-east-1 doesn't work for it.
Thanks
I'm running Spark 2.4 on an EC2 instance. I am assuming an IAM role and setting the key/secret key/token in the sparkSession.sparkContext.hadoopConfiguration, along with the credentials provider as "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider".
When I try to read a dataset from s3 (using s3a, which is also set in the hadoop config), I get an error that says
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 7376FE009AD36330, AWS Error Code: null, AWS Error Message: Forbidden
read command:
val myData = sparkSession.read.parquet("s3a://myBucket/myKey")
I've repeatedly checked the S3 path and it's correct. My assumed IAM role has the right privileges on the S3 bucket. The only thing I can figure at this point is that spark has some sort of hidden credential chain ordering and even though I have set the credentials in the hadoop config, it is still grabbing credentials from somewhere else (my instance profile???). But I have no way to diagnose that.
Any help is appreciated. Happy to provide any more details.
spark-submit will pick up your env vars and set them as the fs.s3a access +secret + session key, overwriting any you've already set.
If you only want to use the IAM credentials, just set fs.s3a.aws.credentials.provider to com.amazonaws.auth.InstanceProfileCredentialsProvider; it'll be the only one used
Further Reading: Troubleshooting S3A