How can I monitor VM services in GCP? [closed] - google-cloud-platform

Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last month.
Improve this question
I have a VM instance that contains SQL service, I want to monitor this SQL service how can I do that?

Google provides a Cloud Monitoring agent that can be used in Compute Engine to collect more detailed metrics from the Compute Engine instance itself and third-party applications it might contain, displaying this information at the Cloud Monitoring page. Here's a list of the applications it can gather the metrics from. Here's a guide on how to install the agent.
Assuming you're using an Ubuntu-based distro and the SQL service is MySQL perhaps this guide may be more relevant. In essence what you'd have to do is:
Add the Monitoring agent's package repository:
curl -sSO https://dl.google.com/cloudagents/add-monitoring-agent-repo.sh
sudo bash add-monitoring-agent-repo.sh
sudo apt-get update
Install the Monitoring agent:
sudo apt-get install stackdriver-agent
Start it:
sudo service stackdriver-agent start
Install MySQL monitoring:
sudo apt-get install libmysqlclient20
Download mysql.conf and place it in the directory /opt/stackdriver/collectd/etc/collectd.d/ which is where the Monitoring agent operates on:
(cd /opt/stackdriver/collectd/etc/collectd.d/ && sudo curl -O https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/mysql.conf)
Edit the configuration file and replace DATABASE_NAME with the
name of the database you want to monitor, and any other variables
that might apply to your database.
Restart the Monitoring agent:
sudo service stackdriver-agent restart
Once that's done, go to the Cloud Monitoring page, select the VM instance as Resource Type: and scroll down until you see Agent Metrics to view the metrics collected by the Monitoring agent.
Optionally you can also install a Cloud Logging agent which will ingest logs from the application. You'd install it with:
curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
sudo bash add-logging-agent-repo.sh
sudo apt-get update
sudo apt-get install -y google-fluentd-catch-all-config-structured
sudo service google-fluentd start
Keep in mind a certain pricing is applied to Cloud Logging once you pass a specific threshold of data storage so that's something to keep in mind if you don't want to accidentally incurr costs. Lastly and again, these steps are applicable if you're indeed using an Ubuntu distro and MySQL, if not, check the links and select the options that meet your configuration accordingly.

Related

Clone a google cloud VM

I have google cloud VM with Ubuntu installed along with various services and libraries. I need to make a similar bootable VM with the same OS and all the data, libraries etc in the already configured VM. How do I clone the VM with these requirements?
I tried to create an image from the already existing VM and could not SSH into it.
So I retraced my installations step by step trying to figure out which step is breaking the image.
I created an Ubuntu(18.04) VM and used that to create an image. The instance I created using the image did allow me to SSH into.
Next installed Ubuntu desktop and xorg server and created an image after that. Using that image, I created a new VM and tried to SSH into it.
But unfortunately, the SSH connection could not be established. So I think it is these installations that are causing the error if it is not some sort of system error.
Below are the exact commands I ran to install these after creating an Ubuntu(18.04) VM:
sudo passwd username
sudo su -
passwd
apt update && apt upgrade -y
adduser username root
adduser username admin
adduser username sudo
apt-get install ubuntu-desktop -y
apt-get install xserver-xorg-video-dummy
nano /etc/X11/xorg.conf
and pasted the following into the .conf file
Section "Device"
Identifier "Configured Video Device"
Driver "dummy"
EndSection
Section "Monitor"
Identifier "Configured Monitor"
HorizSync 31.5-48.5
VertRefresh 50-70
EndSection
Section "Screen"
Identifier "Default Screen"
Monitor "Configured Monitor"
Device "Configured Video Device"
DefaultDepth 24
SubSection "Display"
Depth 24
Modes "1600x900"
EndSubSection
EndSection
After this state, I created the image using which I could not instantiate a VM that I could SSH into.
Since you have your VM ready and running; backup your image as per this GCP document. Follow the guidelines before you begin the process which were mentioned in the document like updating Google cloud CLI setting default region and zone and for general image guidelines.
Few networking features may require guest operating system mode. You can also check how to export a custom image to cloud storage.
You can also consider the Snapshot Approach.
Follow this process in order to create the image exactly as the one you have already set up and you know is working correctly. As you may already know, this is a custom image so they are available only to your Cloud project. You can create a custom image from boot disks and other images if you would like also. Then, use the custom image to create an instance.
I will also suggest you to give a look at this document which would give you a deeper knowledge on the task.
Regards,
Just spin up a new container from a disk snapshot, if you need an exact copy. And if you cannot SSH, you may either not have a SSH public key provisioned, no external IP assigned, or :22 closed.
gcloud ssh always works. One can as well provision project-wide SSH keys, which all VM in the project will inherit then. The documentation below: About VM metadata explains this all in detail.
My personal favorite are rather startup scripts, which describe the configuration, instead of copying it.
And it's not so difficult to get started with these: cat ~/.bash_history > rocky8_startup.sh. In a software-defined data-center, it might make sense to use software-defined configurations (one simply cannot alternate the installation per VM instance, when starting with a disk snapshot).
xserver-xorg-video-dummy is questionable, because one can enable display device -but unless recording the screen, this driver might still suffice; eg. for VNC sessions.

Google cloud compute engine - disable automatic updates (centos)

I wonder if there is a way to disable automatic updates of our Linux machines on Google Cloud (yum update)
As far as I know during maintenance window our servers get new packages of software installed. (I checked yum.log). Since our installed software must be specific version (not latest) we don't want Google to run updates for us because it usually breaks all kind of dependencies...
I have searched on Google but didn't find any info about that.
Thanks.
The centOS 7 image used in Compute Engine includes the yum-cron installed and enabled by default. You can verify it by either using one of the following commands:
sudo yum list installed yum-cron
sudo systemctl status yum-cron.service
The yum-cron will periodically check for updates and apply them if there are updates available.
Solution
If you have yum-cron running on your instance, you can disable auto-updates by accessing the configuration file /etc/yum/yum-cron.conf. Then change the following variables to ‘no’:
update_messages = no
download_updates = no
apply_updates = no
This will prevent the system from updating automatically.
As an alternative, you can opt for uninstalling the package on your system using the following command.
sudo yum remove yum-cron
This part is missing in the official documentation so It will be added soon.

how do I install python library on AWS EMR notebook?

I want to install additional libraries on AWS notebook (connected to EMR cluster), however I do not see any option to connect from Notebook to internet. If I do "pip install ", it always come back saying that network is not reachable. I am not sure which network need to be changed for network connection and library installation.
I did login to Jupyter terminal, and ping to google.com, which just timed out. I do not see any network / security group etc... configuration under Notebook section for making any relevant changes.
May be I need to take some additional steps?
If you use PySpark kernel then you can install libraries using
sc.install_pypi_package("celery")
Or by running
sudo pip-3.6 install boto3
The following document has more details
If you use python 3 kernel, then only the packages are installed and there is no direct way to install extra libraries except uploading the python package to the notebook then using jupyterlab terminal to run
pip install package.tar.gz

Difference between Hyperledger setups

I have a question about the difference between setting up Hyperledger v1 using getting start method :
http://hyperledger-fabric.readthedocs.io/en/latest/getting_started.html
and the other methods like using vagrant and make peer,
and which method is good and fits for both configuring roles and privacy also for configuring chaincodes.
Thank you,
The getting started has it all contained within. You can bring up an entire Hyperledger Fabric V1.0 network on your local box (or in a hosted cloud environment if you'd like). You can also see how to break it down, to simply bring up a peer, or ordering service, etc, if you don't want the entire fabric.
Vagrant is good for development when you don't want to alter anything on your local box, easy to tear down, and start from scratch. The getting started has it all contained within. You can bring up an entire Hyperledger Fabric V1.0 network on your local box (or in a hosted cloud environment if you'd like). You can also see how to break it down, to simply bring up a peer, or ordering service, etc, if you don't want the entire fabric.
Vagrant is good for development when you don't want to alter anything on your local box, easy to tear down, and start from scratch.
Most would suggest to just use the getting started straight away which uses Docker containers. The only exception might be is if you are running on windows, in which case you might want to use vagrant.
There are 5 independent Hyperledger blockchain technologies. The setup for each one is different.
Hyperledger Fabric--explained above
Hyperledger Sawtooth--explained below
Hyperledger Indy (Identity Managment focus)
Hyperledger Iroha (Mobile app focus)
Hyperledger Burrow (Ethereum EVM implementation)
The last three I will let others explain how to install and setup (since I do not know).
For Hyperledger Sawtooth, here are the instructions for the 1.x release of Sawtooth: https://sawtooth.hyperledger.org/docs/core/releases/latest/app_developers_guide/installing_sawtooth.html
Here's a brief summary of the package installation steps and initial setup:
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8AA7AF1F1091A5FD
$ sudo add-apt-repository 'deb http://repo.sawtooth.me/ubuntu/1.0/stable xenial universe'
$ sudo apt-get update
$ sudo apt-get install -y sawtooth
$ sawtooth keygen
$ sawset genesis
$ sudo -u sawtooth sawadm genesis config-genesis.batch
$ sudo sawadm keygen

How to install audiowaveform program on AWS Elastic Beanstalk

Just FYI ... context here is AWS Elastic Beanstalk. I'm trying to the install audiowaveform program on 64bit Amazon Linux 2015.03 v1.4.3 (the customer AMI ID is ami-6b50291c). Running this ... 👇
$ sudo yum install git cmake libmad-devel libsndfile-devel gd-devel boost-devel
... successfully installs all packages except libmad-devel and libsndfile-devel. Below is the relevant output ...
Failed to set locale, defaulting to C
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/2015.03 | 2.1 kB 00:00
amzn-updates/2015.03 | 2.3 kB 00:00
Package git-2.1.0-1.38.amzn1.x86_64 already installed and latest version
Package cmake-2.8.12-2.20.amzn1.x86_64 already installed and latest version
No package libmad-devel available.
No package libsndfile-devel available.
Package gd-devel-2.0.35-11.10.amzn1.x86_64 already installed and latest version
Package boost-devel-1.53.0-14.21.amzn1.x86_64 already installed and latest version
Nothing to do
That said, this is not a problem with audiowaveform ... all this means is that the repositories enabled for Amazon Linux AMIs do not have libmad-devel and libsndfile-devel by default. I probably have to simply add my own sources I guess.
Also to note is that no yum packages exist for audio waveform so I have to build this manually.
Obtain the source ... 👇
$ git clone https://github.com/bbcrd/audiowaveform.git
$ cd audio waveform
Then build and install ... 👇
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install
Question 1
On AWS EB ... the EC2 instances are configured to use Amazon sources which don't have the above packages i.e. libmad-devel and libsndfile-devel. What would be the recommended approach to adding these packages so that they are available to yum?
I stress recommended because I feel that changing the sources from Amazon's could not be the best approach. Nor is adding another source that could conflict with Amazon's packages ... etc etc etc ...
Question 2
Assuming I'm able to install libmad-devel and libsndfile-devel. I still have to build this manually since there are no packages of audiowaveform. On AWS EB I could write a script to do this as each instance is being instantiated ... but I feel this isn't ideal, slow and kinda error-prone. Anyone have advice on how I can do this better?
Probably prepare an AMI with this already built that's based off ami-6b50291c. Thoughts?
Core Objective
I don't have to use audiowaveform ... my objective really is to extract the peak points of some audio (MP3). I will set this up as a separate question.
Amazon Elastic Beanstalk tends to be very restricted in terms of what software you can install on it. I solved it by dockerizing my application environment. This is possible now even on Elastic Beanstalk.
Learn more about Elastic Beanstalk's support for Docker ...
AWS Elastic Beanstalk makes it easy for you to deploy and manage
applications in the AWS cloud. After you upload your application,
Elastic Beanstalk will provision, monitor, and scale capacity (Amazon
EC2 instances), while also load balancing incoming requests across all
of the healthy instances.
Docker automates the deployment of applications in the form of
lightweight, portable, self-sufficient containers that can run in a
variety of environments. Containers can be populated from pre-built
Docker images or from a simple recipe known as a Dockerfile.
Docker’s container-based model is very flexible. You can, for example,
build and test a container locally and then upload it to the AWS Cloud
for deployment and scalability. Docker’s automated deployment model
ensures that the runtime environment for your application is always
properly installed and configured, regardless of where you decide to
host the application.
This way ... you can do whatever you want in the container and that container will run on the kernel provided by the Amazon Linux AMI instance (obviously completely isolated).
I'm also somehow having hard time getting yum to find libsndfile on Amazon Linux AMI (RedHat 7.4). Repositories I've added to yum never seem to contain it. (How to add new repos is described here )
Finally I just downloaded and installed the rpms directly:
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/x86_64/RPMS.classic//libsndfile-1.0.28-alt1.x86_64.rpm
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/x86_64/RPMS.classic//libsndfile-devel-1.0.28-alt1.x86_64.rpm
sudo yum localinstall libsndfile-devel-1.0.28-alt1.x86_64.rpm
This way I got PySoundfile working finally.