I have a PyCharm project that is stored on my local disk. The project scripts download a file folder that is quite large. I have run out of space on the hard disk and am looking for ways to migrate the data files or download them to an aws s3 bucket I have created.
my current directory:
/Users/$User/PycharmProjects/Project
s3 path:
s3://project-bucket/
I have installed Big Data Tools and connected my aws account using the AWS plugin. Im wondering if its possible to move the entire project directory or point the part of my project directory with the large files to the bucket for later processing. I have seen tools such as Boto3 in other posts but am wondering if there is an alternative method or easier way to do this.
Related
I am trying to upload a folder on SageMaker. However, I cannot select any folders, I need to go through the files and upload them one by one. Is there any way to upload the whole directory from local computer to SageMaker?
zip the folder in local, upload it as a zip file and unzip it in Sagemaker terminal
I want to create a MERN stack application from where I can auth to google drive, upload files and download zip folders. But I want to upload files in unzip format and when I want to download folder from google drive I want similar behaviors as google drive does i.e: It first zip the folder and the download it.
I checked it's documentation various npm modules but I am not able found and figure out how to do it.
Any one can help me in this
I want to integrate AWS S3 in my django project and i have successfully integrated that but the problem is many of my client's media files are already on the pythonanywhere server and now i want to change it to AWS S3 so the upcoming media files should be stored in S3. But now what happens is that the new files are storing correctly and i can open them but the previous files, when i open them because of the media files url i suppose django looks it in AWS S3 but they are not there. So i need django to understand to look for the media files where they are stored like for the previous files i want django to look in the server directory and for new files in S3 instance.
Hope that makes sense.
This is my AWS S3 configurations file for my django project
This is my django settings file, mainly focuses on media files
So if you want to move your static files that were uploaded before you configured AWS S3 with django you can run this command in your command line: python manage.py collectstatic.
I have a simple Django app which has been pushed to AWS Lambda using Zappa.
This process has worked properly, with one exception : cannot load library 'pango-1.0': pango-1.0: cannot open shared object file: No such file or directory. Additionally, ctypes.util.find_library() did not manage to locate a library called 'pango-1.0'
I'm using Weasyprint to generate PDF files. Weasyprint needs Cairo and Pango.
I don't know how to get Pango to work on my AWS Lambda install.
What should I do to make it work ?
So, after asking around in multiple locations, I found out that I needed the static versions of all the libraries required by Weasyprint, and that I needed to push these in my zappa package.
Luckily, a github user has uploaded a working repo of the static requirements : https://github.com/Prasengupta/weasyprint_for_awslambda
So all I had to do was download it and extract all the folders at the root of my django app (the folders must be at the same level as the zappa_settings.json file).
I then just had to do a zappa update command to upload all these files to my AWS Lambda install, and it worked!
My Django app is now full of weird directories, but at least the whole thing works.
I want to make a Django project compatible with AWS Beanstalk.
I dont want this to be like in AWS tutorial, since they use git and need to setup the whole project as they tell.
I just want to know if there is a way of converting an already created Python-Django project to be AWS Beanstalk compatible. I mean, isn't there a standard project layout to download or a plugin or command-line tool that creates the .ebsettings folder for me? I want to convert my project and upload it throw the AWS web gui, dont need all the git stuff.
You can do this without using git route. You just need to zip your source bundle and upload to the Beanstalk Web Console. The code structure can be kept the way you want.
Key configurations are:
1. WSGIPath : This should point to the .py file which you need to start the app (WSGI app)
2. static: This should point to the path containing the static files
You can add the configurations in the .ebextensions folder, which should at the root of your app zip. You can read more details here: Customizing and Configuring a Python Container - AWS Elastic Beanstalk