I am building a lab utility to deploy my teams development environments (testing / stress etc).
At present, the pipeline is as follows:
Trigger pipeline via HTTP request, args contain the distribution, web server and web server version using ARGs that are passed too multi stage dockerfiles.
Dockerx builds the container (if it doesn't exist in ECR)
Pipeline job pushes that container to ECR (if it doesn't already exist).
Terraform deploys the container using Fargate, sets up VPCs and a ALB to handle ingress externally.
FQDN / TLS is then provisioned on ...com
Previously when I've made tools like this that create environments, environments were managed and deleted solely at project level, given each environment had it's own project, given this is best practice for isolation and billing tracking purposes, however given the organisation security constraints of my company, I am limited to only 1 project wherein I can create all the resources.
This means I have to find a way to manage/deploy 30 (the max) environments in one project without it being a bit of a clustered duck.
More or less, I am looking for a way that allows me to keep track, and tear down environments (autonomously) and their associated resources relevant to a certain identifier, most likely these environments can be separated by resource tags/groups.
It appears the CDKTF/Pulumi looks like a neat way of achieving some form of "high-level" structure, but I am struggling to find ways to use them to do what I want. If anyone can recommend an approach, it'd be appreciated.
I have not tried anything yet, mainly because this is something that requires planning before I start work on it (don't like reaching deadends ha).
Ok, so I would like to build my application in a way that allows for each organization to get their own instance.
My way of thinking here, is that I could do something with AWS or digital ocean or whatever to deploy my java (dropwizard) application every time a new client registers their company with us.
This would be virtualized, I would be hoping, so I would have those instances running on various virtual servers.
Basically, when a company registers... I would like to spin up an instance of the core API, and an instance of the DB server (or the two could be one instance here, I guess)
Is this a thing? I would google it, but I am not fully sure what to be looking for!
I know this is not a dropwizard question - but I tagged it this way because it is a dropwizard application I am building - and I figure people in that community may have had similar concerns! Please feel free to edit!
You would need to automate the process of spinning up an environment using something like CloudFormation, Ansible, Terraform, Chef, Puppet, etc. There are a lot of tools in this space. These tools are called Infrastructure as Code (IaC). Once you have it automated, setting up a new environment for a new customer would be a simple task of kicking off the appropriate script.
I'm developing an tool using Django for internal use at my organization. It's used to search and tag documents (using Haystack and Solr), and will be employed on different projects. My team currently has a working prototype and we want to deploy it 'in the wild.'
Our security environment is strict. Project documents are located on subfolders on a network drive, and access to these folders is restricted based on users' Windows credentials (we also have an MS SQL server that uses the same credentials). A user can only access the projects they are involved in. Since we're an exclusively Microsoft shop, if we want to deploy our app on the company intranet, we'll need to use an IIS server to deal with these permissions. No one on the team has the requisite knowledge to work with IIS, Active Directory, and our IT department is already over-extended. In short, we're not web developers and we don't have immediate access to anybody experienced.
My hacky solution is to forgo IIS entirely and have each end user run a lightweight server locally (namely, CherryPy) while each retaining access to a common project-specific database (e.g. a SQLite DB living on the network drive or a DB on the MS SQL server). In order to use the tool, they would just launch an all-in-one batch script and point their browser to 127.0.0.1:8000. I recognize how ugly this is, but I feel like it leverages the security measures already in place (note that never expect more than 10 simultaneous users on a given project). Is this a terrible idea, and if so, what's a better solution?
I've dealt with a similar situation (primary development was geared toward a normal deployment situation, but some users have a requirement to use the application on a standalone workstation). Rather than deploy web and db servers on a standalone workstation, I just run the app with the Django internal development server and a SQLite DB. I didn't use CherryPy, but hopefully this is somewhat useful to you.
My current solution makes a nice executable for users not familiar with the command line (who also have trouble remembering the URL to put in their browser) but is also relatively easy development:
Use PyInstaller to package up the Django app into single executable. Once you figure this out, don't continue to do it by hand, add it to your continuous integration system (or at least write a script).
Modify the manage.py to:
Detect if the app is frozen by PyInstaller and there are no arguments (i.e.: user executed it by double clicking it) and if so, then run execute_from_command_line(..) with arguments to start the Django development server.
Right before running the execute_from_command_line(..), pop off a thread that does a time.sleep(2) (to let the development server come up fully) and then webbrowser.open_new("http://127.0.0.1:8000").
Modify the app's settings.py to detect if frozen and change things around such as the path to the DB server, enabling the development server, etc.
A couple additional notes.
If you go with SQLite, Windows file locking on network shares may not be adequate if you have concurrent writing to the DB; concurrent readers should be fine. Additionally, since you'll have different DB files for different projects you'll have to figure out a way for the user to indicate which file to use. Maybe prompt in app, or build the same app multiple times with different settings.py files. Variety of a ways to hit this nail...
If you go with MSSQL (or any client/server DB), the app will have to know the DB credentials (which means they could be extracted by a knowledgable user). This presents a security risk that may not be acceptable. Basically, don't try to have the only layer of security within the app that the user is executing. The DB credentials used by the app that a user is executing should only have the access that the user is allowed.
I'm trying to add a web setup project for my web services project to make a reasonably complex installation easier for the end user. But the defaults are getting in my way and I'm not sure how to adjust them.
My requirements are:
Create a new website & app pool
When creating the new app pool, needs to be set to .Net 4, classic mode with user selectable user account to run the app pool
Install the web service to a custom path under C:\Program Files\<comapany name>\Services folder to meet company standards
Create a SQL database adding a login & db_datawriter roles to the account selected for the app pool
Run SQL statement to build initial database
Creating a SQL database and Logins (4 & 5) seems possible through Custom Actions as I'll get to run my code at install time. But the web setup project by default only allows users to select from existing websites and app pools (1 & 2) and doesn't seem to allow me to customise this process except to add an image, this also means that the physical path for the selected website is used.
An option would be to create a suitable website & app pool prior to installation and remove it if it goes unused, then the user would simply need to select the website/app-pool I've created for them (not ideal but..) but there isn't a pre-installation step for me to attach such custom code to and I'm not sure how to interface with IIS reliably & programmatically.
Can anyone suggest the best way for me to approach creating the installer outlined above as I don't think web setup projects are fit for purpose at this stage, but would prefer not to have to waste a day learning WIX or NSIS.
1 & 3) Creating a new app pool isn't possible as there is no way to run custom code pre-installation in web setup projects, in my case it turned out to be acceptable for a user to create the website/app pool in IIS before installation. You can modify the Virtual Directory of the existing site through Microsoft.Web.Administration.dll (got the idea from here) if desired, by passing [TARGETSITE] to CustomActionData so you can find the Application object needed to modify the Virtual Directory.
2) This is achieved by referencing the Microsoft.Web.Administration.dll (got the idea from here) and using it to change application pool properties, you will need to pass the appPool name to the Custom action by adding /AppPool="[TARGETAPPPOOL]" to the CustomActionData for the CustomAction.
FYI - The available properties you may pass to CustomActionData are documented here but are incomplete (excluding [TARGETAPPPOOL] and [TARGETSITE] for example)
I used this code to modify the app pool:
Microsoft.Web.Administration.ServerManager serverManager = new ServerManager();
var appPoolName = base.Context.Parameters["AppPool"].ToString();
var appPool = serverManager.ApplicationPools.SingleOrDefault( pool => pool.Name == appPoolName );
appPool.ManagedRuntimeVersion = "v4.0";
appPool.ManagedPipelineMode = ManagedPipelineMode.Classic;
serverManager.CommitChanges();
4 - 5) Simply open a SQL Connection from a custom action and run the relevant SQL Scripts, you won't be able to add the true accounts ie: NT AUTHORITY\NETWORK SERVICE or NT AUTHORITY\ANONYMOUS, but may use the virtual user accounts IIS creates IIS APPPOOL\<app pool name>.
If you need to develop a sophisticated installer but don't want to spend even a day learning installer technologies then your best bet is probably to hire a consultant to do it for you.
Suppose I create a site in Wordpress, which is running on Elastic Beanstalk. Now, on the running app I will create posts /pages, upload images, etc. That is, some data, videos, files and records in a database will be added to the running application.
3 questions:
If WordPress is running on Elastic Beanstalk with multiple Amazon EC2 instances actually running my WordPress install, then will those files propagate automatically to all running instances? And will this also happen, if a new EC2 instance is fired up - for example, to handle increased load?
From what I see in AWS console, I can deploy different versions of an app-- but as per scenario above, if I deploy a new version, wont I lose all the files uploaded directly into running app (i.e. files and database records)? How do I keep those and at the same time deploy a new version of my app?
The WordPress team keeps issuing upgrades. Can I directly upgrade my running WordPress install, through the web interface? Or do I have to first upgrade my local version of WordPress, and then upload the new version of the app to Beanstalk? If I have to upgrade my local version and then upload, then again I am back to point 1, i.e. changes made by users directly to the older version of running app. How do I preserve those changes?
I've been working on this as well, and have learned a couple of things that are relevant here -- your question about uploads in particular has been on my mind:
(1) The best way to handle uploads, it seems to me, is to either go the NFS/NAS route like you suggest, but one better than that is to use an Amazon S3 plugin for WordPress, so that any uploads automatically copy up to S3 and the URLs in your WordPress media library reflect the FQDN of your bucket and not your specific site. That way you could have one or ten WP nodes in your Beanstalk and media/images are independent of any one of those servers.
(2) You should absolutely be using RDS here. Few things are easier to work with and as stress-free as a Multi-AZ, reserved MySQL RDS instance. Either that or your own EC2 running MySQL that is independent of the Beanstalk, but why run that when RDS is so much easier?
(3) Yes you definitely have to commit changes to your Git repository or local file first (new plugins, changes to themes, WP upgrades) and then upload/install as a revision to the Beanstalk code. Otherwise, all the changes you make via the web interface to one node will never be in the new load for a new node -- in fact you'll have an upgraded database but an older set of code in the Beanstalk application, so it's likely to create errors of some kind or another.
I took an AWS architecture course, and their advice for EC2 and the Beanstalk is to start to think about server instances as very disposable -- so you should try to think about easy ways for your boxes to provision themselves in the bootstrapping process and to take over work for one another without any precious resources on just one box. So losing an instance should never be a big deal. (This is definitely not how we thought in the world of physical servers, where we got everything tweaked 'just so'.)
Good luck!
Well, I'm no expert, but since no one has answered, I'll give it my best shot.
You are absolutely right--kind of. While each EC2 instance does have some local storage, it is destroyed and reset with each new instance. Because of this, Amazon has things like Elastic Block Storage and S3 for persistent files. I don't know how one would configure WP to utilize this, but that will likely be the solution.
I think this problem is solved by my answer to #1. As for the database, all of your EC2 instances should be pulling from the same RDS location. Again, while you could have MySQL running on each EC2 instance, in the interest of persistence, having a separate database makes more sense.
You, again, have most everything right. Local development should always precede live deployment. Upgrading locally then pushing to the live servers will make sure all of your instances remain identical.
Truth-be-told, I am still in the process of learning all of this too, as I said, I'm not an expert. Hopefully someone else will come along and give a more informed answer. However, the key conceptual hurdle here is the idea of elastic scalability--and the salient point of this idea is the separation of elements between what is elastic/scalable/disposable and what is persistent.
Hopefully that helps.
I have deployed a small Wordpress site on EB, S3 and RDS. S3 holds all static data, such as media uploads. This works through a plugin. RDS holds the database. EB holds the latest deployed application. The application is deployed from my dev environment, with a build script. This way, I just have to press one button and I redeploy.
I wrote an article about it here: http://www.cortexcode.com/wordpress-to-aws-code-example/
While it was at first annoying to work with, the speed of AWS is nice and now it's easier than ever. It used to be that I had to upload a bunch of files over FTP, this is way more efficient. :-)
As an addition to all the great answers already:
1) I can highly recommend EFS but also S3 for media files, so they are served from high availability regions in combination with cloudfront. For Wordpress there is one plugin that really speeds up this ( not affiliated to them, just really like the plugin ). There is also an assets plugin, if you'd like to serve JS, CSS files from S3. For the EFS solution, take a look at the AWSlabs docs on git, and specifically this file on how they mount the uploads file.
In general, EBS is really great for Wordpress, but you'll need to think in a different mindset as compared to other hosting solutions ( shared hosting, managed hosting ).
OK I researched a lot on this particular issue, and this is what I learned--
(1) If a wordpress user uploads some files, then his files will be uploaded only to the virtual machine that is actually serving his request at that time. Eg if currently the wordpress site is cloud-deployed and is using 5 virtual machines, now when user makes request he is directed to one virtual machine-- the one with the lowest load at that point... His uploads are stored only on that server. Current Platform-as-a-service solutions (like Amazon Elastic Beanstalk and App Fog) do not have the ability to propagate the changes to all the running instances. Either that (propagate changes to all servers) or use a common storage by all running instances-- these are the only 2 solutions to this problem. (Eg of common storage would be all 5 running virtual machines using Network-Attached-Storage (NAS)... )
(2) With ref to platforms available currently like Amazon Elastic Beanstalk and App Fog, for example, even if user made changes directly to running app- these platforms rely on the local version of code (which the admin deployed initially to cloud)- and there is no way to update the local version of code (on admin's PC) with the changes made by a user to running app-- hence these changes viz, files are lost-- Similarly, changes in database by user to running app are also lost-- unless the admin is using exactly the same database for his local app (that he deployed to cloud)
(3) Any changes to running apps first have to be made to the local app on admin's PC and then pushed to cloud.
I am working on a Cloud PaaS that addresses all these concerns-- viz updates can be made to running apps, code changes made to running app are also updated in code repository accessible by user...The Proof of concept is ready, hopefully it will be as good as I hope it should be :) -- currently the only thing that is actually there is the website (anyacloudpanel.com) -- design work is going on :)
If there is some rule that I should not mention my website( Anya Cloud Panel) -- then I am sorry -- pls feel free to edit and remove my website URL from my answer :)
Thanks,
Arvind.
Deploying WordPress to AWS Elastic Beanstalk does require some change to the normal WordPress deployment as mentioned here a few times. To answer your questions, here is a great tutorial explaining stateless applications and how to deploy to Elastic Beanstalk:
Deploying WordPress to Amazon Web Services AWS EC2 and RDS via ElasticBeanstalk
Be careful if you use a theme from themeforest for example. Some of them are incompatible with wordpress S3 plugin. Then you're screwed, you can not deploy your wordpress on the cloud.