I am evaluating stackdriver from GCP for logging across multiple micro services.
Some of these services are deployed on premise and some of them are on AWS/GCP.
Our services are either .NET or nodejs based apps and we are invested in winston for nodejs and nlog in .net.
I was looking # integrating our on-premise nodejs application with stackdriver logging. Looking # https://cloud.google.com/logging/docs/setup/nodejs the documentation it seems that there we need to install the agent for any machine other than the google compute instances. Is this correct?
if we need to install the agent then is there any way where I can test the logging during my development? The development environment is either a windows 10/mac.
There's a new option for ingesting logs (and metrics) with Stackdriver as most of the non-google environment agents look like they are being deprecated. https://cloud.google.com/stackdriver/docs/deprecations/third-party-apps
A Google post on logging on-prem resources with stackdriver and Blue Medora
https://cloud.google.com/solutions/logging-on-premises-resources-with-stackdriver-and-blue-medora
for logs you still need to install an agent on each box to collect the logs, it's a BindPlane agent not a Google agent.
For node.js, you can use the #google-cloud/logging-winston and #google-cloud/logging-bunyan modules from anywhere (on-prem, AWS, GCP, etc.). You will need to provide projectId and auth credentials manually if not running on GCP. Instructions on how to set these up is available in the linked pages.
When running on GCP we figure out the exact environment (App Engine, Compute Engine, etc.) automatically and the logs should up under those resources in the Logging UI. If you are going to use the modules from your development machines, we will report the logs against the 'global' resource by default. You can customize this by passing a specific resource descriptor yourself.
Let us know if you run into any trouble.
I tried setting this up on my local k8s cluster. By following this: https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/
But i couldnt get it to work, the fluentd-gcp-v2.0-qhqzt keeps crashing.
Also, the page mentions that there are multiple issues with stackdriver logging if you DONT use it on google GKE. See the screenshot.
I think google is trying to lock you in into GKE.
Related
I am working on creating monitoring based on SLO. So far I have been using Google Cloud Monitoring solutions like Dashboards, Alerting and Uptime Checks.
I have noticed GCP has now a Managed Service for Prometheus.
My question is what would be the advantage of using Prometheus(not only Google managed one)for monitoring. Is there anything that could be achieved with Prometheus that I could not achive with Google Cloud Monitoring?
Managed service for prometheus is a managed and automatically scalable prometheus endpoint. You can request the metrics with PromQL language instead of MQL (Monitoring Query Language).
What's the advantage? If you deploy an application instrumented with Open Telemetry (for example), you don't have to change anything. On Kubernetes (GKE), the managed collector do the job for you. Else you have to configure the collector to use Managed Service for Prometheus.
If you build an app from scratch, and you want it portable, Open Telemetry and Prometheus are standard tools to instrument your app.
If not, use Cloud Monitoring!
Important note
That feature is very new and, for now, only the metrics sinks with Managed Service for Prometheus can be query with PromQL. The other metrics must be requested by MQL. It could change in the future.
So, for now, if you can use built in Cloud Monitoring metrics, it's a better solution.
We have setup a fluentd agent on a GCP VM to push logs from syslog server (the VM) to GCP's Google Cloud Logging. The current setup is working fine and is pushing more than 300k log entries to Stackdriver (Google Cloud Logging) per hour.
Due to increased traffic, we are planning to increase the number of VMs employed behind the load balancer. However, the new VM with fluentd agent is not being able to push logs to Stackdriver. After the first time activation of VM, it does send a few entries to Stackdriver and after that, it does not work.
I tried below options to setup the fluentd agent and to resolve the issue:
Create a new VM from scratch and install fluentd logging agent using this Google Cloud documentation.
Duplicate the already working VM (with logging agent) by creating Images
Restart the VM
Reinstall the logging agent
Debugging I did:
All the configurations for google fluentd agent. Everything is correct and is also exactly similar to the currently working VM instance.
I checked the "/var/log/google-fluentd/google-fluentd.log" for any logging errors. But there are none.
Checked if the logging API is enabled. As there are already a few million logs per day, I assume we are fine on that front.
Checked the CPU and memory consumption. It is close to 0.
All the solutions I could find on Google (there are not many)
It would be great if someone can help me identify where exactly I am going wrong. I have checked configurations/setup files multiple times and they look fine.
Troubleshooting steps to resolve the issue:
Check whether you are using the latest version of the fluentd agent or not. If not, try upgrading the fluentd agent. Refer to upgrade the agent for information.
If you are running very old Compute Engine instances or Compute Engine instances created without the default credentials you must complete the Authorizing the agent procedures.
Another point to focus is, how you are Configuring an HTTP Proxy. If you are using an HTTP proxy for proxying requests to the Logging and Monitoring APIs, check whether the metadata server is reachable. The metadata server has to be reachable (and do it directly; no proxy) when Configuring an HTTP Proxy.
Check if you have any log exclusions configured which is preventing the logs from arriving. Refer Exclusion filters for information.
Try uninstalling the Fluentd agent and try to use Ops agent instead (note that syslog logs are collected by it with no setup) and check whether you were able to see the logs. Combining logging and metrics into a single agent, the Ops Agent uses Fluent Bit for logs, which supports high-throughput logging, and the OpenTelemetry Collector for metrics. Refer Ops agent for more information.
I am running cloud foundry on a Kubernetes cluster on the Digital Ocean platform. I am able to deploy apps successfully via cf push APP_NAME without a database. Now I would like to run my Django app with a PostgreSQL database. When I run from terminal cf marketplace it does now show me the list of offerings/services available in the marketplace.
cf marketplace
Output
Getting services from marketplace in org abc-cforg / space abc-cfspace as admin...
OK
No service offerings found
Output from cf version
cf version 6.53.0+8e2b70a4a.2020-10-01
I have tried with cf version 7 as well but no luck.
I am quoting from this doc -
No problem. The Cloud Foundry marketplace is a collection of services that can be
provisioned on demand. Your marketplace may differ depending on the Cloud Foundry
distribution you are using.
What should I be doing now to get the list of service offerings in the marketplace? I googled quite some time but could not find a fix.
I have an account in pivotal as well but this is deprecated already as per this link.
By default, there will not be any services in the marketplace. As a platform operator, you'll need to add the services that you want to expose to your CloudFoundry users.
If you look at a public CloudFoundry offering, you can see that this is done for you, and when you run cf m you'll get the list of services that the public provider and their operations team set up for you.
When you run your own CF, that's on you to set up.
There are a couple of things you can do:
The easy option is to use user-provided services. These are not set up through the marketplace, so you simply ignore that command altogether.
You would instead go procure your service from somewhere else. You mentioned using Digital Ocean, so you could procure one of their managed databases. Once you have your database credentials, you would run cf cups -p username,password,host my-service (these are free-form fields names, enter whatever makes sense for your service) and, when prompted, enter the info. This creates a user-provided service, which can be bound to your apps and works just like a service you'd acquire through the marketplace.
The more involved option requires deploying more infrastructure to run a service broker. The service broker talks to Cloud Controller and provides a catalog of available services. Those services are what Cloud Controller displays when you run cf m.
There are some community-provided brokers and commercial ones as well. I think a lot of these brokers also assume a Bosh deployment and not Kubernetes, so be careful to read the instructions and see if that's a requirement.
A quick scan through and here are a few that seem like they should work:
https://github.com/cloudfoundry-community/cf-containers-broker
https://github.com/cloudfoundry-community/s3-broker
https://github.com/cloudfoundry-community/rds-broker
Does Metric Registrar works in Cloud Foundry without Pivotal?
I have open source Cloud Foundry and I need to get custom metrics from app. I installed Metric Registrar community plugin for CF, I registered my application with endpoint, I also defined log format. Unfortunately I see no traffic on registered endpoint.
If open source Cloud Foundry do not support Metric Registrar, is there any other way to get support for custom app metrics?
Does Metric Registrar works in Cloud Foundry without Pivotal?
The Metric Registrar is part of the VMware Tanzu Application Service product, it's not part of the Open Source Cloud Foundry project. It's a value-add feature for those using the paid product.
If open source Cloud Foundry do not support Metric Registrar, is there any other way to get support for custom app metrics?
You don't strictly need the Metric Registrar to do this. The Metric Registrar's main purpose is to take metrics from your apps and inject them into the Loggregator log/metric stream. This is convenient if you have other software that is already consuming log & metric streams from Loggregator.
You don't have to do that though, as there are other ways to export metrics from your app.
If you want them to go through Loggregator, you could export structured log messages (perhaps JSON?) via STDOUT that contains your metrics. Those will, like your other log messages, go out through Loggregator. You would then just need to have something ingesting your logs, identifying the structured messages, and parsing out your metrics. This is similar to what Metric Registrar does, you're just parsing out the structured log entries after they leave the platform.
If you have an ELK stack or similar running, you can probably make this solution work easily enough. ELK can ingest your logs & structured log metrics, then you can search/filter through the metrics and create dashboards.
Another option you could do is to run Prometheus/Grafana. You then just need to make sure your app has a Prometheus Exposition metrics endpoint (this is super easy with Java/Spring Boot & Spring Boot Actuator, but can be done in any language). Point Prometheus at your app and it will then be able to scrape metrics from your apps & you can use Grafana to view them. None of this goes through Loggregator.
If you're looking for a solution that's more automatic, you could run an APM agent (NewRelic, DataDog, AppDynamics, Dynatrace, etc..) with your apps. These will capture metrics directly from the process and export them to a SaaS platform where you can monitor/review them.
There are probably other options as well. This is just what comes to mind as I write this up.
Google Cloud Platform announced "Stackdriver Kubernetes Monitoring" at Kubecon 2018. It looks awesome.
I am an AWS user running a few Kubernetes clusters and immediately had envy, until I saw that it also supported AWS and "on prem".
Stackdriver Kubernetes Engine Monitoring
This is where I am getting a bit lost.
I cannot find any documentation for helping me deploy the agents onto my Kubernetes clusters. The closest example I could find was here: Manual installation of Stackdriver support, but the agents are polling for "internal" GCP metadata services.
E0512 05:14:12 7f47b6ff5700 environment.cc:100 Exception: Host not found (authoritative): 'http://metadata.google.internal./computeMetadata/v1/instance/attributes/cluster-name'
I'm not sure the Stackdriver dashboard has "Stackdriver Kubernetes Monitoring" turned on. I don't seem to have the same interface as the demo on YouTube here
I'm not sure if this is something which will get turned on when I configure the agents correctly, or something I'm missing.
I think I might be missing some "getting started" documentation which takes me through the setup.
You can use a Stackdriver partner service, Blue Medora BindPlane, to monitor AWS Kubernetes or almost anything else in AWS for that matter or on-premise. Here's an article from Google Docs about the partnership: About Blue Medora; you can signup for BindPlane through the Google Cloud Platform Marketplace.
It looks like BindPlane is handling deprecated Stackdriver monitoring agents. Google Cloud: Transition guide for deprecated third-party integrations
As per this article, currently Stackdriver Kubernetes Monitoring beta release only supports Kubernetes version v1.10.2 clusters running on Google Cloud Platform's Kubernetes Engine. To track when this feature will be available in AWS, I suggest creating a feature request in Public Issue Tracker.
Stackdriver monitoring of Amazon EKS, Azure AKS, and general purpose Kubernetes running on non--GCP hosted VMs is available if you enable the BindPlane option for Stackdriver.
https://cloud.google.com/stackdriver/blue-medora