Is it possible to send files from a mobile application to ES2 that has a python script file that processes the file and the final product will be save into S3?
Deploy a simple webapp in EC2 to receive the data from your mobile app, run the python script you mentioned with the data, use the S3 API and save the data there. As for how you're going to deploy that webapp, there are tons of ways/languages/technologies, fit for another question.
Related
I was hoping I could get some direction about creating a website using AWS that will run a python script. I created an EC2 Instance running Ubuntu and made it talk with a relational database made with the same account.
In a nutshell, the site I am creating is a YouTube Library of captions. The user will input a title and AWS will retrieve links to XML documents that contains the captions to the related videos from YouTube. I would like to know where and how to run a Python script to scrape the text from these XML documents every time a user makes a request.
My research has taken me in multiple directions, but I am not sure what is best for my purpose. For example, I am trying to run a remote script from GitHub, but don't know if there's a better way to store the script?
It's my first time working with AWS so please keep explanations simple. Thanks!
I am trying to create DF v2 custom activity. I have console application, but i am not sure where to put it.
Should i only build it and upload to blob storage or is there some special publishment?
Thx
As stated here https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-dotnet-custom-activity you need to upload your code to a Compute Environment which will be an Azure Batch pool.
The custom activity runs your customized code logic on an Azure Batch
pool of virtual machines.
Then just configure the linked service and the activity as shown in the official documentation.
Hope this helped!
I have following process that is run nightly:
Grab XML from an FTP server
Transform the XML with a number of XSLT's to XML formatted for MySql
Load the new XML using the "LOAD XML" mysql command
I've been reading about AWS Data Pipelines and instead of me having this process run on an Ec2 instance it sounds like aws pipelines may be suited for this but I have a couple questions:
With step 2, the xslt's have some custom functions that are run. Currently the transforms are done with a .NET console app but I could convert this to Node if there is a way to do this in a cloud lambda
Can the pipeline run a LOAD XML command on a database? I assume I'd have to out the xml to an s3 bucket?
Is AWS pipelines a good idea for this task or and I heading in the wrong direction?
This is very much possible with AWS Data Pipeline. See the following examples among many others in the github repository https://github.com/awslabs/data-pipeline-samples
ShellCommandWithFTP
RedshiftToRDS
You can transform your xml to CSV and use CopyActivity
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-copyactivity.html
I need to upload a file to AWS Simple Storage Service from a PHP script. The script gets called from an external program and for some unknown reason the script bombs out as soon as I load the AWS PHP SDK. I've tried everything to get it to work without any success. I'm therefore thinking of rather using the AWS S3 REST API to upload the file.
My question is, what is the major drawback of using the REST API compared to the PHP SDK? I know it will be a bit harder to use the REST APIs, but if I only need to upload files to S3, would it take significantly more time? Or would it be worth spending another half a day (hopefully) trying to get the script to run while using the SDK?
On my in-house jenkins server I use the S3 plugin to upload warfiles to a s3 bucket. How can I do this wtih cloudbees ? Any plan to integrate this plugin ?
This plugin is available on the Pro and Enterprise tiers of DEV#cloud, just not the Free and Base tiers.
If the base account level is perfect for you other than S3 file transfer and you are not yet ready for a significant price increase just to get S3 file copy, you may consider this workaround to get files to your EC2:
It is possible to use scripting to get stuff out of your build environment:
http://wiki.cloudbees.com/bin/view/DEV/Accessing+Jenkins+using+HTTP+Basic+Authentication
For example:
curl -u EMAIL:JENKINS_API_KEY https://your_account_name.ci.cloudbees.com/job/job_name/ws/your_file_name > your_file_name
You can do a fingerprint post build process on your file and then check that fingerprint page for md5 hash.