Rails Sitemap Generator Uploading to S3 - ruby-on-rails-4

Trying to generate a sitemap and uploading it to my current existing bucket in Amazon's S3, however, I'm getting
Excon::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)
This is my sitemap.rb file
SitemapGenerator::Sitemap.default_host = "http://www.example.com"
SitemapGenerator::Sitemap.public_path = 'tmp/sitemaps/'
SitemapGenerator::Sitemap.sitemaps_host = "http://s3.amazonaws.com/#{ENV['S3_BUCKET_NAME']}/"
SitemapGenerator::Sitemap.create do
add about_path
add landing_index_path
add new_user_session_path, priority: 0.0
Trip.find_each do |trip|
add trip_path(trip.slug), lastmod: trip.updated_at
end
end
I have this in my s3.rb file
CarrierWave.configure do |config|
config.storage = :fog
config.fog_credentials = {
:provider => 'AWS',
:aws_access_key_id => Rails::AWS.config['access_key_id'],
:aws_secret_access_key => Rails::AWS.config['secret_access_key'],
:region => 'us-east-1'
}
config.fog_directory = Rails::AWS.config['bucket_name']
end
Would someone be able to know what the issue is with this?

My working config (which I use in heroku) is a little different than yours, here is what I have:
SitemapGenerator::Sitemap.default_host = 'http://example.com'
SitemapGenerator::Sitemap.public_path = 'tmp/'
SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new(fog_provider: 'AWS', fog_directory: 'sitemap-bucket')
SitemapGenerator::Sitemap.sitemaps_host = "http://#{ENV['FOG_DIRECTORY']}.s3.amazonaws.com/"
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'
I don't use a S3.rb, instead, I set the following environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
FOG_DIRECTORY
FOG_REGION
I used the tutorial in here: https://github.com/kjvarga/sitemap_generator/wiki/Generate-Sitemaps-on-read-only-filesystems-like-Heroku
I hope it helps!

I was experiencing a similar error:
In '/app/tmp/':
rake aborted!
ArgumentError: is not a recognized provider
Going off the help of renatolond's answer above, this is the configuration that worked for me. The key is to make sure that all of your variables, such as "fog_region:" actually match up to valid values. Do not blindly copy + paste configuration credentials.
SitemapGenerator::Sitemap.default_host = "https://yourwebsitename.com"
SitemapGenerator::Sitemap.public_path = 'tmp/'
SitemapGenerator::Sitemap.adapter = SitemapGenerator::S3Adapter.new(
fog_provider: 'AWS',
aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
fog_directory: ENV['S3_BUCKET'],
fog_region: ENV['AWS_REGION'])
SitemapGenerator::Sitemap.sitemaps_host = "http://{ENV['S3_BUCKET']}.s3.amazonaws.com/"
SitemapGenerator::Sitemap.sitemaps_path = 'sitemaps/'

Related

AWS S3 presign url, check file exist

I use AWS php SDK.
How can I check if file exist using presign request commands?
Currently I use "GetObject" command but I do not need it download file. I only need check if file exist.
$cmd = $s3->getCommand('GetObject', [
'Bucket' => 's3.test.bucket',
'Key' => $fileKey
]);
$request = $s3->createPresignedRequest($cmd, '+60 minutes')->withMethod('GET');
return (string)$request->getUri();
Is there any command to achieve it?
Thank you.
I found solution. The proper command is HeadObject and method is HEAD.
Return 200 or 404.

AWS - Centos7 - /home/.aws/credentials not working

I have a Centos7 VPS with AWS CLI installed on the /home directory. I've added my credentials into aws configure and it's generated the following files:
/home/.aws/credentials
/home/.aws/config
If I run the following code, it fails:
$client = new Aws\Lightsail\LightsailClient([
'region' => 'eu-west-2',
'version' => '2016-11-28'
]);
The error message is:
AccessDeniedException (client): User: arn:aws:sts::523423432423:assumed-role/AmazonLightsailInstanceRole/i-0eb5b2155b08e5185 is not authorized to perform
However if I add my credentials like so it works:
$credentials = new Aws\Credentials\Credentials('key', 'secret');
$client = new Aws\Lightsail\LightsailClient([
'region' => 'eu-west-2',
'version' => '2016-11-28',
'credentials' => $credentials
]);
Do I need to do something extra in order to get my script to read the /home/.aws/credentials file?
Do I need to do something extra in order to get my script to read the /home/.aws/credentials file?
Yes, you need to put the .aws/credentials directory in the home directory of the user running the command. This will be something like /home/username instead meaning that the full path to the credentials will be /home/username/.aws/credentials. It does not matter where you installed the aws command to.

s3cmd obfuscate file names (change to random value) on Amazon S3 side (local original file name)

my .s3cfg with GPG encryption passphrase and other security settings. Would you recommend other security hardening?
[default]
access_key = $USERNAME
access_token =
add_encoding_exts =
add_headers =
bucket_location = eu-central-1
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/local/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = $PASSPHRASE
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
secret_key = $PASSWORD
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class =
urlencoding_mode = normal
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html
I use this command to upload/sync my local folder to Amazon S3.
s3cmd -e -v put --recursive --dry-run /Users/$USERNAME/Downloads/ s3://dgtrtrtgth777
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 15957 files, this may take some time...
INFO: [1000/15957]
INFO: [2000/15957]
INFO: [3000/15957]
INFO: [4000/15957]
INFO: [5000/15957]
INFO: [6000/15957]
INFO: [7000/15957]
INFO: [8000/15957]
INFO: [9000/15957]
INFO: [10000/15957]
INFO: [11000/15957]
INFO: [12000/15957]
INFO: [13000/15957]
INFO: [14000/15957]
INFO: [15000/15957]
I tested the encryption with Transmit GUI S3 Client and didn't get plain text files.
But I see the original filename. I wish to change the filename to a random value, but have local the original filename (mapping?). How can I do this?
What are downsides doing so if I need to restore the files? I use Amazon S3 only as a backup, in addition to my TimeMachine backup.
If you use "random" names, then it isn't sync.
If your only record on the filenames/mapping is local, it will be impossible to restore your backup in case of a local failure.
If you don't need all versions of your files I'd suggest putting everything in a (possibly encrypted) compressed tarball before uploading it.
Otherwise, you will have to write a small script that lists all files and individually does an s3cmd put specifying a random destination, where the mapping is appended to a log file, which should be the first thing you s3cmd put to your server. I don't recommend this for something as crucial as storing your backups.
A skeleton showing how this could work:
# Save all files in backupX.sh where X is the version number
find /Users/$USERNAME/Downloads/ | awk '{print "s3cmd -e -v put "$0" s3://dgtrshitcrapola/"rand()*1000000}' > backupX.sh
# Upload the mapping file
s3cmd -e -v put backupX.sh s3://dgtrshitcrapola/
# Upload the actual files
sh backupX.sh
# Add cleanup code here
However, you will need to handle filename collisions, failed uploads, versioning clashes, ... why not use an existing tool that backs up to S3?

Rails 4 precompiled assets upload to Amazon s3 and cloudfront

I have used asset-sync and fog gem for assets upload to aws s3.I can uploaded precompiled assets files to aws-s3 while deploying but it taking lot of time.The deploy is not complete then it was stopped on bundle exec rake assets:precompile command.
Gem file:
gem "fog", "~>1.20", require: "fog/aws/storage"
gem 'asset_sync'
Asset sync configuration file:
AssetSync.configure do |config|
config.fog_provider = 'AWS'
config.aws_access_key_id = "Your aws access key"
config.aws_secret_access_key = "Your aws secret access key"
config.fog_directory = ENV['FOGDIRECTORY']
config.fog_region = 'us-west-2'
end
Staging.rb file:
config.assets.enabled = true
config.assets.digest = true
config.action_controller.asset_host = "http://djrcjofcge7nb.cloudfront.net"
config.action_mailer.asset_host = "http://djrcjofcge7nb.cloudfront.net"
config.assets.initialize_on_precompile = true
Add the following configurations in your staging.rb file
config.assets.compile = true
config.eager_load = true

Asset_Sync not pushing to S3

I am building a rails app on heroku, and want to deploy the js, css, and image files to a bucket on Amazon. I haven't found many resources for this, but I am using this (2012) tutorial for guidance; https://firmhouse.com/blog/complete-guide-to-serving-your-rails-assets-over-s3-with-asset_sync
The site is mainly css and js at the moment. Here is my code so far;
production.rb
Rails.application.configure do
config.action_controller.asset_host = "http://localize.s3.amazonaws.com"
config.cache_classes = true
config.consider_all_requests_local = false
config.action_controller.perform_caching = true
config.serve_static_assets = false
config.assets.compress = true
config.assets.compile = false
config.assets.digest = true
end
initializers/asset_sync.rb
if defined?(AssetSync)
AssetSync.configure do |config|
config.fog_provider = ENV['FOG_PROVIDER']
config.aws_access_key_id = ENV['AWS_ACCESS_KEY_ID']
config.aws_secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
config.fog_directory = ENV['FOG_DIRECTORY']
config.fog_region = ENV['FOG_REGION']
# Don't delete files from the store
config.existing_remote_files = "delete"
# Automatically replace files with their equivalent gzip compressed version
config.gzip_compression = true
# Use the Rails generated 'manifest.yml' file to produce the list of files to
# upload instead of searching the assets directory.
config.manifest = true
config.custom_headers = { '.*' => { cache_control: 'max-age=31536000', expires: 1.year.from_now.httpdate } }
end
end
Heroku Vars
AWS_ACCESS_KEY_ID: *****************
AWS_SECRET_ACCESS_KEY: *****************************
FOG_DIRECTORY: localize
FOG_PROVIDER: AWS
FOG_REGION: us-west-2
gemfile
gem 'rails', '4.1.1'
gem 'uglifier', '>= 1.3.0'
gem 'jquery-rails'
gem 'sdoc', '~> 0.4.0', group: :doc
#aws
gem "fog", "~>1.20"
gem 'asset_sync'
group :development do
gem 'thin'
end
group :production do
gem 'newrelic_rpm'
gem 'rails_12factor'
gem 'pg'
end
I also ran:
heroku config:add FOG_PROVIDER=AWS AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy
Along with
heroku config:add FOG_DIRECTORY=localize
And then when I run
bundle exec rake assets:precompile
Or
RAILS_ENV=production bundle exec rake assets:precompile
I get this output;
rake aborted!
AssetSync::Config::Invalid: Fog directory can't be blank, Aws access key can't be blank, Aws secret access key can't be blank
Anyone who has experience with rails, heroku, and S3 who could guide me in the right direction would be much appreciated. Thanks in advance.
Ok looking at your settings there seems to be a few things wrong, Ill add what i normally use and hopefully it will help you
Production.rb
ExampleApp::Application.configure do
config.action_controller.asset_host = "http://exampleapp.s3.amazonaws.com"
config.cache_classes = true
config.consider_all_requests_local = false
config.action_controller.perform_caching = true
config.serve_static_assets = true
config.assets.compress = true
config.assets.compile = true
config.assets.digest = true
end
asset_sync.rb
if defined?(AssetSync)
AssetSync.configure do |config|
config.fog_provider = ENV['FOG_PROVIDER']
config.aws_access_key_id = ENV['AWS_ACCESS_KEY_ID']
config.aws_secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
config.fog_directory = ENV['FOG_DIRECTORY']
config.fog_region = ENV['FOG_REGION']
# Don't delete files from the store
config.existing_remote_files = "delete"
# Automatically replace files with their equivalent gzip compressed version
config.gzip_compression = true
# Use the Rails generated 'manifest.yml' file to produce the list of files to
# upload instead of searching the assets directory.
config.manifest = true
config.custom_headers = { '.*' => { cache_control: 'max-age=31536000', expires: 1.year.from_now.httpdate } }
end
end