My requirement is to FTP files that are arriving from client-server to our landing zone. Need a python script to automate the FTP whenever the file arrives in the client-server to the landing zone. I am new to python and looking for options on the net to get it somehow. Would be very helpful if you could throw some light on it
Related
As a part of my project, I am trying to integrate ESP32 with some of the AWS services.
One of the task demands to store certain codes(stored as '.ino' files) of ESP32(in Arduino IDE) in Amazon S3 buckets(cloud storage). Whenever the user(For example: a mobile app) or ESP32 itself sends requests to AWS S3 server to retrieve any .ino file, that particular .ino file should be received and executed instantaneously by ESP32.
While trying to figure out the solution for the above, I am stuck thinking about how exactly will ESP32 execute the code retrieved from cloud while maintaining a web-socket connection with AWS Servers.
One of the possible solutions which I thought was using Dual Core Processing in ESP32 but I am not exactly sure how and where this would be used in my Arduino IDE code
It would be great if someone could throw some light on the above query!
I'm using Django+celery for my 1st ever web development project, and rabbitmq as the broker. My celery workers are running on a different system from the web server and are executing long-running tasks. During the task execution, the task output will be dumped to local log files on the workers. I'd like to display these task log files through the web server so the user can know in real-time where the execution is, but I've no idea how I should transfer these log files between the workers and the system where the web server is. Any suggestion is appreciated.
Do not move logs, just log to the same place. It can be really any database (relational or non-relational) accessible from the web server and Celery workers. You can even create (or look for) appropriate python logging handler, saving log records to the centralized storage.
Maybe the solution isn't to move the logs, but to aggregate them. Take a look at some logging tools like splunk, loggly or logscape.
I'm looking for some direction on an AWS architictural decision. My goal is to allow users to ftp a file to an EC2 instance and then run some analysis on the file. My focus is to build this in as much a service-oriented way as possible.. and in the future scale it out for multiple clients where each would have there own ftp server and processing queue with no co-mingling of data.
Currently I have a dev EC2 instance with vsftpd installed and a node.js process running Chokidar that is continuously watching for new files to be dropped. When that file drops I'd like for another server or group of servers to be notified to get the file and process it.
Should the ftp server move the file to S3 and then use SQS to let the pool of processing servers know that it's ready for processing? Should I use SQS and then have the pool of servers ssh into the ftp instance (or other approach) to get the file rather than use S3 as a intermediary? Are there better approaches?
Any guidance is very much appreciated. Feel free to school me on any alternate ideas that might save money at high file volume.
I'd segregate it right down into small components.
Load balancer
FTP Servers in scaling group
Daemon on FTP Servers to move to S3 and then queue a job
Processing servers in scaling group
This way you can scale the ftp servers if necessary, or scale the processing servers (on SQS queue length or processor utilisation). You may end up with one ftp server and 5 processing servers, or vice versa - but at least this way you only scale at the bottleneck.
The other thing you may want to look at is DataPipeline - which (whilst not knowing the details of your job) sounds like it's tailor made for your use case.
S3 and queues are cheap, and it gives you more granular control around the different components to scale as appropriate. There are potentially some smarts around wildcard policies and IAM you could use to tighten the data segregation too.
Ideally I would try to process the file on the server that it is currently placed.
This will save a lot of network traffic and CPU load.
However if you want one of the servers to be like reverse proxy and to load balance between farm of servers, then I will notify the server with http call that the file has arrived. I would made the file available via ftp since you already has working vsftp that will not be a problem and will include the file ftp url in the http call, so the server that will do the processing can get the file and start working on it immediately.
This way you will save money by not using any extra S3 or SQS or any other additional services.
If the farm of servers are made of equal type of servers, then the algorithm for distributing the load should be RoundRobin if the servers are with different capacity then the load distribution should be made according to the server performance.
For example if server ONE has 3 times more performnce then server THREE and server TWO has 2 times better performance than server THREE, then you can do:
1: Server ONE - forward 3 request
2: Server TWO - forward 2 request
3: Server THREE - forward 1 request
4: GOTO 1
Ideally there should be feedback from the serves that report the current load so the load-balancer knows who is the best candidate for the next request instead of using hard-coded algorithms, since probably the requests do not need exactly equal amount of resources to be processed, but this start looking like Map-Reduce paradigm and is out of the scope ... at least for the begining. :)
If you want to stick with S3, you could use RioFS to mount S3 bucket as a local filesystem on your FTP and processing servers. Then you could do the usual file operations (e.g.: get the notification when a file was created / modified).
As well as RioFS s3fs-fuse utilizes FUSE to provide a filesystem that is (virtual locally) mountable; s3fs-fuse is currently well-maintained.
In contrast the Filesystem Abstraction for S3, HDFS and normal filesystem
swineherd-fs allows to have a different (locally virtual) approach:
All filesystem abstractions implement the following core methods, based on standard UNIX functions, and Ruby's File class [...].
As the 'local abstraction layer' has been Only thoroughly tested on Ubuntu Linux i'd personally go for a more mainstream/solid/less experimental stack, i.e.:
a (sandboxed) vsftpd for
FTP transfers
(optionally) listen for filesystem changes and finally
trigger
middleman-s3_sync to do the cloud lift (or synchronize all by itself).
Alternatively, and more experimental, there are some github projects that might fit it:
s3-ftp: FTP server front-end
that forwards all uploads to an S3 bucket (Clojure)
ftp-to-s3: An FTP server that
uploads every file it receives to S3 (Python)
ftp-s3: FTP frontend to S3 in Python.
Last but not least i do recommend using donationware Cyberduck if on OSX - a comfortable (and very FTP-like) client interfacing S3 directly. For Windows there is a (PRO optional) freeware named S3 Browser.
What I'm trying to design is a system where-by a user can upload a compiled .hex file to a web site, and then that web server sends it on to another server which has a microprocessor connected to it via USB. The web server would then trigger a script to run on that sever which would load the .hex file to the micro.
So my question is: Is it possible for a web server to trigger a shell script or C/Java program on another (trusted) machine?
Yes it's possible. Webservers often support invocation of scripts, usually with the CGI interface. This scripts can perform the required interaction with the other server. How this has to be implemented depends on the webserver and its configuration.
If there is a microcontroller is involved and how this is connected to the second server is not related to this question. Therefore I suggest to remove the microcontroller tag. It's also doubful if it's somehow related to shell at all.
We are trying to use monit to monitor services on our Ubuntu machine. I have successfully setup a host url check to make sure that coldfusion can render web pages and it there is an error to restart coldfusion.
I was wondering if there is a way to get more stats into monit by monitoring the coldfusion process. I have been unable to find out if coldfusion creates a pid file.
Does Coldfusion 9 or Jrun create a pid file for monit to use? Is there another way to monitor coldfusion with monit?
ColdFusion can output real-time performance metrics such as:
Page hits per second
Database accesses per second
Number of queued requests
Number of running requests
Number of timed out requests
Average queue time
Average request time
Average database transaction time
Bytes incoming per second
Bytes outgoing per second
You can learn more about the output of this logging here: http://help.adobe.com/en_US/ColdFusion/9.0/Admin/WSc3ff6d0ea77859461172e0811cbf3638e6-7fe0.html#WS9F365555-357A-4a15-AC72-449EF611E342
I would be interested to learn how you set this up once complete. I'll have the same task in a few weeks.
Thanks!
You will need to create the PID file with a wrapper script around your Java application. I'm doing the same thing myself these days. To the best of my understanding monit has to have the PID file to check the life of your service.