I have a working setup of puppet open-source 3.8 with a puppet master and several nodes.
I would like to install puppetDB and a dashboard so I can get a good overview about my nodes.
To not destroy anything from the current setup, I wish to install puppetDB on a separate server. ( Everything is running on AWS EC2 instances.)
I was trying to use the following descriptions and let puppet install puppetDB:
PuppetDB 2.3.8-1.el6 - I believe this is the latest version compatible with puppet 3.8
I've managed to install puppetDB on the DB node, however I can't connect my puppet master to the node.
Based on this documentation:
Connecting Puppet Masters to PuppetDB
I need to install puppetdb-terminus. However I'm using Puppet Open-source, so "sudo puppet resource package puppetdb-terminus ensure=latest" is failing on puppet agent as a dependency.
It's fine, but I have also issues with the alternative solution:
Download the PuppetDB source code, unzip it and navigate into the resulting directory in your terminal.
Run sudo cp -R puppet/lib/puppet/ /usr/lib/ruby/site_ruby/1.8/puppet.
There is no puppet/lib/puppet in neither puppetdb-2.3.8.tar.gz nor puppetdb-3.2.4.tar.gz archives.
As my last hope, I went to github to grab the source:
https://github.com/puppetlabs/puppetlabs-puppetdb/tree/master/lib/puppet
I've copied these files to /usr/lib/ruby/site_ruby/1.8/puppet.
/var/log/messages:Mar 18 13:08:03 ip-10-84-4-172 puppet-master[25616]: Could not configure routes from /etc/puppet/routes.yaml: Could not find terminus puppetdb for indirection facts
At this point I'm completely stuck. How can I verify my puppet-terminus installation? If this way is not good, how can I install it on my puppet master?
(I'm using RHEL6, Puppet Open-source 3.8, I've did all the other changes on puppet master based on the documentation.)
Just for the sake of completeness, here is my puppetDB puppet manifest:
class { 'puppetdb::globals':
version => '2.3.8-1.el6',
}
class { 'puppetdb::database::postgresql':
listen_addresses => $postgres_host,
}
class { 'puppetdb::server':
database_host => $puppetdb_host,
}
I've just migrate my PuppetDB from puppetmaster server to a standalone one. To handle the installation of PuppetDB I've used this module from Puppet Labs.
Its was straightforward. The db migration was done with puppetdb --export from master server and puppetdb --import in the new server. The last thing was change the address for PuppetDB within puppet master config.
[]'s
The puppet repo pages is tricky, there is a separate repo for the Pre4.0 Open Source binaries:
https://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#pre-40-open-source-repositories
After using this repo, there was no more issue during installation.
Related
I'm using AWS Lambda, which involves creating an archive of my node.js script, including the node_modules folder and uploading that to their infrastructure to run.
This works fine, except when it comes to node modules with native bindings (using node-gyp). Because the binding was complied and project archived on my local computer (OS X), it is not compatible with AWS's (Amazon Linux) servers.
How can I cross-compile/install a node module (specifically, node-sqlite3) so when I upload it to another server arch it runs?
While not really a solution to your problem, a very easy workaround could be to simply compile the native addons on a Linux machine.
For your particular situation, I would use Vagrant. Vagrant can create virtual machines and configure them within seconds.
Find an OS image that resembles Amazon's Linux distro (Fedora, CentOS, others that use yum as package manager - see Wiki)
Use a simple configuration script that, when run by Vagrant on machine startup, will run npm install (optionally it might also remove the node_modules folder before to ensure a clean installation)
For extra comfort, the script can also create the zip file for deployment
Once the installation finishes, the script will shutdown the VM to avoid unnecessary consumption of system resources
Deploy!
It might require some tuning if the linked libraries are not at the same place on the target machine but generally this seems to me like the best and quickest solution.
While installing the app using Vagrant might be sufficient in some cases, I have found it necessary to build the app on Linux which is as close to Lambda's Amazon Linux AMI as possible.
You can read the original answer here: https://stackoverflow.com/a/34019739/303184
Steps to make it work:
Spawn new EC2 instance. Make sure it is based on exactly the same image as your AWS Lambda runtime. You can review Lambda env details here: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. In our case, it was Amazon Linux AMI called amzn-ami-hvm-2015.03.0.x86_64-gp2.
Install nvm and use it to install the same version of Node.js as on the AWS Lambda. At the time of writing this, it was v0.10.36. You can refer to http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html again to find out.
You will probably need to install git & g++ compiler on the EC2. You can do this running
sudo yum install git gcc-c++
Finally, clone your app to your new EC2 and install your app's dependecies:
nvm use 0.10.36
npm install --production
You can then easily download the node_modules using scp or such.
Same lines as Robert's answer, when I had to work on my MAC in a different OS I use vm ware like Oracle's free virtualizer VirtualBox to get a linux on my mac, no cost to me. Or sign up for a new AWS account, you get a micro for a year free. Use that to get your linux box, do whatever you need there.
AWS has a page describing how to deal with native NPM modules: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/
Just FYI ... context here is AWS Elastic Beanstalk. I'm trying to the install audiowaveform program on 64bit Amazon Linux 2015.03 v1.4.3 (the customer AMI ID is ami-6b50291c). Running this ... 👇
$ sudo yum install git cmake libmad-devel libsndfile-devel gd-devel boost-devel
... successfully installs all packages except libmad-devel and libsndfile-devel. Below is the relevant output ...
Failed to set locale, defaulting to C
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/2015.03 | 2.1 kB 00:00
amzn-updates/2015.03 | 2.3 kB 00:00
Package git-2.1.0-1.38.amzn1.x86_64 already installed and latest version
Package cmake-2.8.12-2.20.amzn1.x86_64 already installed and latest version
No package libmad-devel available.
No package libsndfile-devel available.
Package gd-devel-2.0.35-11.10.amzn1.x86_64 already installed and latest version
Package boost-devel-1.53.0-14.21.amzn1.x86_64 already installed and latest version
Nothing to do
That said, this is not a problem with audiowaveform ... all this means is that the repositories enabled for Amazon Linux AMIs do not have libmad-devel and libsndfile-devel by default. I probably have to simply add my own sources I guess.
Also to note is that no yum packages exist for audio waveform so I have to build this manually.
Obtain the source ... 👇
$ git clone https://github.com/bbcrd/audiowaveform.git
$ cd audio waveform
Then build and install ... 👇
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install
Question 1
On AWS EB ... the EC2 instances are configured to use Amazon sources which don't have the above packages i.e. libmad-devel and libsndfile-devel. What would be the recommended approach to adding these packages so that they are available to yum?
I stress recommended because I feel that changing the sources from Amazon's could not be the best approach. Nor is adding another source that could conflict with Amazon's packages ... etc etc etc ...
Question 2
Assuming I'm able to install libmad-devel and libsndfile-devel. I still have to build this manually since there are no packages of audiowaveform. On AWS EB I could write a script to do this as each instance is being instantiated ... but I feel this isn't ideal, slow and kinda error-prone. Anyone have advice on how I can do this better?
Probably prepare an AMI with this already built that's based off ami-6b50291c. Thoughts?
Core Objective
I don't have to use audiowaveform ... my objective really is to extract the peak points of some audio (MP3). I will set this up as a separate question.
Amazon Elastic Beanstalk tends to be very restricted in terms of what software you can install on it. I solved it by dockerizing my application environment. This is possible now even on Elastic Beanstalk.
Learn more about Elastic Beanstalk's support for Docker ...
AWS Elastic Beanstalk makes it easy for you to deploy and manage
applications in the AWS cloud. After you upload your application,
Elastic Beanstalk will provision, monitor, and scale capacity (Amazon
EC2 instances), while also load balancing incoming requests across all
of the healthy instances.
Docker automates the deployment of applications in the form of
lightweight, portable, self-sufficient containers that can run in a
variety of environments. Containers can be populated from pre-built
Docker images or from a simple recipe known as a Dockerfile.
Docker’s container-based model is very flexible. You can, for example,
build and test a container locally and then upload it to the AWS Cloud
for deployment and scalability. Docker’s automated deployment model
ensures that the runtime environment for your application is always
properly installed and configured, regardless of where you decide to
host the application.
This way ... you can do whatever you want in the container and that container will run on the kernel provided by the Amazon Linux AMI instance (obviously completely isolated).
I'm also somehow having hard time getting yum to find libsndfile on Amazon Linux AMI (RedHat 7.4). Repositories I've added to yum never seem to contain it. (How to add new repos is described here )
Finally I just downloaded and installed the rpms directly:
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/x86_64/RPMS.classic//libsndfile-1.0.28-alt1.x86_64.rpm
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/x86_64/RPMS.classic//libsndfile-devel-1.0.28-alt1.x86_64.rpm
sudo yum localinstall libsndfile-devel-1.0.28-alt1.x86_64.rpm
This way I got PySoundfile working finally.
I'm using Vagrant to deploy chef scripts to an AWS server (and it mostly works awesome). I have set up a local rsync in my Vagrantfile to mirror my dev directory onto the server.
config.vm.synced_folder "../geoevents", "/vagrant/geoevents-repo"
And this syncs find on 'vagrant provision'. I'm wondering if there is an easier way that I can have vagrant only trigger that rsync, or to control how often rsync occurs?
Or, should I not be using rsync, but instead mount a shared file system?
Vagrant CLI now has two new commands, vagrant rsync and vagrant rsync-auto which can do the job.
Command: vagrant rsync
This command forces a re-sync of any rsync synced folders.
Note that if you change any settings within the rsync synced folders such as exclude paths, you will need to vagrant reload before this command will pick up those changes.
https://www.vagrantup.com/docs/cli/rsync.html
https://www.vagrantup.com/docs/cli/rsync-auto.html
https://www.vagrantup.com/docs/cli/non-primary.html
Currently, you can fit your needs with the following plugin:
https://github.com/cromulus/vagrant-rsync
By the way, most of the plugin features will be available in Vagrant 1.5 (currently in development).
The vagrant-rsync is deprecated as of Vagrant 1.5. One solution out there is vagrant-unison. You may also check out this discussion. What should also work is a vagrant reload.
I have a provisioning setup with vagrant and puppet that works well locally and I'm now tryign to move it to AWS using vagrant-aws.
As I understand it I can make use the AWS user-data field in vagrant as specified to run commands on the first boot of a new vm like so:
aws.user_data = File.read("user_data.txt")
Where user_data.txt contains:
#!/bin/bash
sudo apt-get install -y puppet-common
Then my existing puppet provisioning scripts should be able to run. However this errors out on the vagrant up command with:
[aws] Running provisioner: puppet...
The `puppet` binary appears to not be in the PATH of the guest. This
could be because the PATH is not properly setup or perhaps Puppet is not
installed on this guest. Puppet provisioning can not continue without
Puppet properly installed.
But when I ssh into the machine I see that the user-data did get parsed and puppet is installed successfully. Is the puppet provisioner running before the user-data install puppet maybe? Or is there some better way to install puppet on a vm before trying to provision?
It is broken, but there's a workaround if you're using Ubuntu which is far simpler than building your own AMI.
Add the following line to your config:
aws.user_data = "#cloud-config\nbootcmd:\n - echo 'manual' > /etc/init/ssh.override\npackages:\n - puppet\nruncmd:\n - [ 'rm', '/etc/init/ssh.override' ]\n - [ 'service', 'ssh', 'start' ]\n"
This tells Cloudinit to disable SSH startup early in the boot process and re-enable it once your packages are installed. Now Vagrant can only SSH in to run puppet once the packages are fully installed.
This will probably work for other distros that use Cloudinit aside from Ubuntu, altho it is Upstart specific so the commands may need tweaking.
Well I worked around this by building my own AMI with puppet and other things I need installed, still seems like vagrant-aws is broken or I'm misunderstanding something else here.
Is there a pre-existing, best practices project skeleton for Chef + Django web applications on Linux (Ubuntu preferably)?
For production Django systems our preferred setup is Supervisor, Nginx, Ubuntu and Uwsgi. Additionally we use Chef to do configuration management and Vagrant + Chef to do development environment management.
While this system is great once they're all up and running they can be very time consuming to setup properly.
My ideal solution would be pre-made Chef Github repository which was a skeleton for a best-practices Django deployment. (It would come with a chef-solo.rb ready to be used to deploy to some cloud ubuntu instance and a Vagrantfile ready to be used to create Vagrant dev machine.) Basically all you would have to do is add a Chef cookbook to deploy your application code and tweak a few settings.
Does anything like that ideal solution exist?
Here's typical chef based configuration solution:
one git repo saves chef-repo. you can use knife solo init <repo-name> to create it. Or just clone that from git repo of opscode.com
one git repo per cookbook. you can use berkshelf cookbook <your-cookbook-name> to create a full set of cookbook content including cookbook itself, test-kitchen, vagrant and berks. Please install berkshelf first via gem install berkshelf
For any other cookbooks that from cookbook community or git repo, you can use berkshelf to download them and managed as local cookbooks.