VM Manager - OS Policy Assignment for a Windows VM in GCP - google-cloud-platform

I am trying to create a couple of os policy assignments to configure - run some scripts with PowerShell - and install some security agents on a Windows VM (Windows Server 2022), by using the VM Manager. I am following the official Google documentation to setup the os policies. The VM Manager is already enabled, nevertheless I have difficulties creating the appropriate .yaml file which is required for the policy assignment since I haven't found any detailed examples.
Related topics I have found:
Google documentation offers a very simple example of installing an .msi file - Example OS policies.
An example of a fixed policy assignment in Terraform registry - google_os_config_os_policy_assignment, from where I managed to better comprehend the required structure for the .yaml file even though it is in a .json format.
Few examples provided at GCP GitHub repository (OSPolicyAssignments).
OS Policy resources in JSON representation - REST Resource, from where you can navigate to sample cases based on the selected resource.
But, it is still not very clear how to create the desired .yaml file. (ie. Copy some files, run a PowerShell script to perform an installation or an authentication). According to the Google documentation pkg, repository, exec, and file are the supported resource types.
Are there any more detailed examples I could use to understand what is needed? Have you already tried something similar?
Update: Adding an additional source.

You need to follow these steps:
Ensure that the OS Config agent is installed in your VM by running the below command in PowerShell:
PowerShell Get-Service google_osconfig_agent
you should see an output like this:
Status Name DisplayName
------ ---- -----------
Running google_osconfig... Google OSConfig Agent
if the agent is not installed, refer to this tutorial.
Set the metadata values to enable OSConfig agent with Cloud Shell command:
gcloud compute instances add-metadata $YOUR_VM_NAME \
--metadata=enable-osconfig=TRUE
Generate an OS policy and OS policy assignment yaml file. As an example, I am generating an OS policy that installs a msi file retrieved from a GCS bucket, and an OS policy assignment to run it in all Windows VMs:
# An OS policy assignment to install a Windows MSI downloaded from a Google Cloud Storage bucket
# on all VMs running Windows Server OS.
osPolicies:
- id: install-msi-policy
mode: ENFORCEMENT
resourceGroups:
- resources:
- id: install-msi
pkg:
desiredState: INSTALLED
msi:
source:
gcs:
bucket: <your_bucket_name>
object: chrome.msi
generation: 1656698823636455
instanceFilter:
inventories:
- osShortName: windows
rollout:
disruptionBudget:
fixed: 10
minWaitDuration: 300s
Note: Every file has its own generation number, you can get it with the command gsutil stat gs://<your_bucket_name>/<your_file_name>.
Apply the policies created in the previous step using Cloud Shell command:
gcloud compute os-config os-policy-assignments create $POLICY_NAME --location=$YOUR_ZONE --file=/<your-file-path>/<your_file_name.yaml> --async
Refer to the Examples of OS policy assignments for more scenarios, and check out this example of a PowerShell script.

Down below you can find the the .yaml file that worked, in my case. It copies a file, and executes a PowerShell command, so as to configure and deploy a sample agent (TrendMicro) - again this is specifically for a Windows VM.
.yaml file:
id: trendmicro-windows-policy
mode: ENFORCEMENT
resourceGroups:
- resources:
- id: copy-exe-file
file:
path: C:/Program Files/TrendMicro_Windows.ps1
state: CONTENTS_MATCH
permissions: '755'
file:
gcs:
bucket: [your_bucket_name]
generation: [your_generation_number]
object: Windows/TrendMicro/TrendMicro_Windows.ps1
- id: validate-running
exec:
validate:
interpreter: POWERSHELL
script: |
$service = Get-Service -Name 'ds_agent'
if ($service.Status -eq 'Running') {exit 100} else {exit 101}
enforce:
interpreter: POWERSHELL
script: |
Start-Process PowerShell -ArgumentList '-ExecutionPolicy Unrestricted','-File "C:\Program Files\TrendMicro_Windows.ps1"' -Verb RunAs
To elaborate a bit more, this .yaml file:
copy-exe-file: It copies the necessary installation script from GCS to a specified location on the VM. Generation number can be easily found on "VERSION HISTORY" when you select the object on GCS.
validate-running: This stage contains two different steps. On the validate it checks if the specific agent is up and running on the VM. If not, then it proceeds with the enforce step, where it executes the "TrendMicro_Windows.ps1" file with PowerShell. This .ps1 file downloads, configures and installs the agent. Note 1: This command is executed as Administrator and the full path of the file is specified. Note 2: Instead of Start-Process PowerShell a Start-Process pwsh can also be utilized. It was vital for one of my cases.
Essentially, a PowerShell command can be directly run at the enforce
step, nonetheless, I found it much easier to pass it first to a .ps1
file, and then just run this file. There are some restriction with the
.yaml file anywise.
PS: Passing osconfig-log-level - debug as a key-value pair as Metadata - directly to a VM or applied to all of them (Compute Engine > Setting - Metadata > EDIT > ADD ITEM) - provide some additional information and may help you on dealing with errors.

Related

Migrate Secrets from SecretManager in GCP

Hi I have my secrets in Secretmanager in one project and want to know how to copy them or migrate them to other project.
Is there a mechanism to do it smoothly.
As of today there is no way to have GCP move the Secret between projects for you.
It's a good feature request that you can file here: https://b.corp.google.com/issues/new?component=784854&pli=1&template=1380926
edited according to John Hanley's comment
I just had to deal with something similar myself, and came up with a simple bash script that does what I need. I run Linux.
there are some prerequisites:
download the gcloud cli for your OS.
get the list of secrets you want to migrate (you can do it by setting up the gcloud with the source project gcloud config set project [SOURCE_PROJECT], and then running gcloud secrets list)
then once you have the list, convert it textually to a list in
format "secret_a" "secret_b" ...
the last version of each secret is taken, so it must not be in a "disabled" state, or it won't be able to move it.
then you can run:
$(gcloud config set project [SOURCE_PROJECT])
declare -a secret_array=("secret_a" "secret_b" ...)
for i in "${secret_array[#]}"
do
SECRET_NAME="${i}_env_file"
SECRET_VALUE=$(gcloud secrets versions access "latest" --secret=${SECRET_NAME})
echo $SECRET_VALUE > secret_migrate
$(gcloud secrets create ${SECRET_NAME} --project [TARGET_PROJECT] --data-file=secret_migrate)
done
rm secret_migrate
what this script does, is set the project to the source one, then get the secrets, and one by one save it to file, and upload it to the target project.
the file is rewritten for each secret and deleted at the end.
you need to replace the secrets array (secret_array), and the project names ([SOURCE_PROJECT], [TARGET_PROJECT]) with your own data.
I used this version below, which also sets a different name, and labels according to the secret name:
$(gcloud config set project [SOURCE_PROJECT])
declare -a secret_array=("secret_a" "secret_b" ...)
for i in "${secret_array[#]}"
do
SECRET_NAME="${i}"
SECRET_VALUE=$(gcloud secrets versions access "latest" --secret=${SECRET_NAME})
echo $SECRET_VALUE > secret_migrate
$(gcloud secrets create ${SECRET_NAME} --project [TARGET_PROJECT] --data-file=secret_migrate --labels=environment=test,service="${i}")
done
rm secret_migrate
All "secrets" MUST be decrypted and compiled in order to be processed by a CPU as hardware decryption isn't practical for commercial use. Because of this getting your passwords/configuration (in PLAIN TEXT) is as simple as logging into one of your deployments that has the so called "secrets" (plain text secrets...) and typing 'env' a command used to list all environment variables on most Linux systems.
If your secret is a text file just use the program 'cat' to read the file. I haven't found a way to read these tools from GCP directly because "security" is paramount.
GCP has methods of exec'ing into a running container but you could also look into kubectl commands for this too. I believe the "PLAIN TEXT" secrets are encrypted on googles servers then decrypted when they're put into your cluser/pod.

How to clone an AWS EB environment across platform branches

Background
Our AWS Elastic Beanstalk environment, running the latest version of the pre-configured "Python 3.7 on 64-bit Amazon Linux 2" platform branch, has a lot of custom configuration and environment properties.
Now we would like to switch this environment to the "Python 3.8 on 64-bit Amazon Linux 2" platform branch.
Basically, the goal is to clone the environment, keeping the current configuration (other than platform branch and version) and environment properties.
Problem
Unfortunately, when cloning, it is not possible to switch between different platform branches (we can only switch between platform versions within the same platform branch).
The documentation suggests that a blue/green deployment is required here. However, a blue/green deployment involves creating a new environment from scratch, so we would still need some other way to copy our configuration settings and environment properties.
Question
What would be the recommended way to copy the configuration settings and/or environment properties from the original environment into a newly created environment?
I suppose we could use eb config to download the original configuration, modify the environment name, platform branch and version, and so on, and then use eb config --update on the new environment. However, that feels like a hack.
Summary
save current config: eb config save <env name>
use a text editor to modify the platform branch in the saved config file
create new environment based on modified config file: eb create --cfg <config name> (add --sample to use the sample application)
if necessary, delete local config files
if necessary, use eb printenv and eb setenv to copy environment properties
EDIT: For some reason the saved config does not include all security group settings, so it may be necessary to check those manually, using the EB console (configuration->instances).
Background
AWS support have confirmed that using eb config is the way to go, and they referred to the online documentation for details.
Unfortunately, the documentation for the eb cli does not provide all the answers.
The following is based on my own adventures using the latest version of the eb cli (3.20.2) with botocore 1.21.50, and documentation at the time of writing (Sep 30, 2021). Note there's a documentation repo on github but it was last updated six months ago and does not match the latest online docs...
eb config
Here's a screenshot from the eb config docs:
Indeed, if you call eb config my-env or eb config my-env --display, environment properties are not shown.
However, this does not hold for eb config save: YAML files created using eb config save actually do include environment properties*.
*Beware, if your environment properties include secrets (e.g. passwords), these also end up in your saved configs, so make sure you don't commit those to version control.
Moreover, it is currently also possible to set environment properties using eb config --update.
This implies we should be able to "copy" both configuration settings and environment properties in one go.
EDIT: After some testing it turns out eb config save does not always get the complete set of environment properties: some properties may be skipped. Not yet sure why... Step 5 below might help in those cases.
Walk-through
Not sure if this is the best way to do it, but here's what seems to work for me:
Suppose we have an existing EB environment called py37-env with lots of custom configuration and properties, running the Python 3.7 platform branch.
The simplest way to "clone" this would be as follows:
Step 1: download the existing configuration
Download the configuration for the existing environment:
eb config save py37-env
By default, the config file will end up in our project directory as .elasticbeanstalk/saved_configs/py37-env-sc.cfg.yml.
The saved config file could look like this (just an example, also see environment manifest):
EnvironmentConfigurationMetadata:
Description: Configuration created from the EB CLI using "eb config save".
DateCreated: '1632989892000'
DateModified: '1632989892000'
Platform:
PlatformArn: arn:aws:elasticbeanstalk:eu-west-1::platform/Python 3.7 running on 64bit Amazon Linux 2/3.3.5
OptionSettings:
aws:elasticbeanstalk:application:environment:
MY_ENVIRONMENT_PROPERTY: myvalue
aws:elasticbeanstalk:command:
BatchSize: '30'
BatchSizeType: Percentage
aws:elb:policies:
ConnectionDrainingEnabled: true
aws:elb:loadbalancer:
CrossZone: true
aws:elasticbeanstalk:environment:
ServiceRole: aws-elasticbeanstalk-service-role
aws:elasticbeanstalk:healthreporting:system:
SystemType: enhanced
aws:autoscaling:launchconfiguration:
IamInstanceProfile: aws-elasticbeanstalk-ec2-role
EC2KeyName: my-key
aws:autoscaling:updatepolicy:rollingupdate:
RollingUpdateType: Health
RollingUpdateEnabled: true
EnvironmentTier:
Type: Standard
Name: WebServer
AWSConfigurationTemplateVersion: 1.1.0.0
Also see the list of available configuration options in the documentation.
Step 2: modify the saved configuration
We are only interested in the Platform, so it is sufficient here to replace 3.7 by 3.8 in the PlatformArn value.
If necessary, you can use e.g. eb platform list to get an overview of valid platform names.
Step 3: create a new environment based on the modified config file
eb create --cfg py37-env-sc
This will deploy the most recent application version. Use --version <my version> to deploy a specific version, or use --sample to deploy the sample application, as described in the docs.
This will automatically look for files in the default saved config folder, .elasticbeanstalk/saved_configs/.
If you get a ServiceError or InvalidParameterValueError at this point, make sure only to pass in the name of the file, i.e. without the file extension .cfg.yml and without the folders.
Step 4: clean up local saved configuration file
Just in case you have any secrets stored in the environment properties.
Step 5: alternative method for copying environment properties
If environment properties are not included in the saved config files, or if some of them are missing, here's an alternative way to copy them (using bash).
This might not be the most efficient implementation, but I think it serves to illustrate the approach. Error handling was omitted, for clarity.
source_env="py37-env" # or "$1"
target_env="py38-env" # or "$2"
# get the properties from the source environment
source_env_properties="$(eb printenv "$source_env")"
# format the output so it can be used with `eb setenv`
mapfile -t arg_array < <(echo "$source_env_properties" | grep "=" | sed -e 's/ =/=/g' -e 's/= /=/g' -e 's/^ *//g')
# copy the properties to the target environment
eb setenv -e "$target_env" "${arg_array[#]}"
This has the advantage that it does not store any secrets in local files.

How to specify the root folder to deploy an app using the Cloud SDK?

I'm using "Google App Engine" from GCP to host a static website. I already created the website files (HTML, JS) and yaml using Visual Studio Code. I have the folder with those files stored locally in my local computer.
I downloaded the Cloud SDK Shell for Windows. I logged in to my account, and selected the project. According to videos and tutorials, I need to deploy the app using "gcloud app deploy".
However I got an error saying that an "app.yaml" file is required to deploy this directory...
I'm trying to follow this tutorial:
https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website#before_you_begin
I'm also trying to follow the steps contained in this video:
https://www.youtube.com/watch?v=mlcO7nfQzSg
How do I specify the root folder where I have my "app.yaml" file?
Thanks in advance!!
I already tried with many commands and unfortunately none of them have worked
The particular case in which gcloud app deploy works without additional arguments is for single-service applications only and only if the command is executed in the directory in which the service's app.yaml configuration file exists (and has that exact name, can't use a different name).
For other cases deployables can/must be specified. From gcloud app deploy:
SYNOPSIS
gcloud app deploy [DEPLOYABLES …] [--bucket=BUCKET] [--image-url=IMAGE_URL] [--no-promote] [--no-stop-previous-version]
[--version=VERSION, -v VERSION] [GCLOUD_WIDE_FLAG …]
DESCRIPTION
This command is used to deploy both code and configuration to the App
Engine server. As an input it takes one or more DEPLOYABLES that
should be uploaded. A DEPLOYABLE can be a service's .yaml file or a
configuration's .yaml file (for more information about configuration
files specific to your App Engine environment, refer to
https://cloud.google.com/appengine/docs/standard/python/configuration-files
or
https://cloud.google.com/appengine/docs/flexible/python/configuration-files).
Note, for Java Standard apps, you must add the path to the
appengine-web.xml file inside the WEB-INF directory. gcloud app
deploy skips files specified in the .gcloudignore file (see gcloud
topic gcloudignore for more information).
So apart from running the command with no arguments in the directory in which your app.yaml exists is to specify the app.yaml (with a full or relative path if needed) as a deployable:
gcloud app deploy path/to/your/app.yaml
IMHO doing this is a good habit - specifying deployables is more reliable and is the only way to deploy apps with multiple services or using routing via a dispatch.yaml file.
gcloud app deploy will look at the current directory first for app.yaml. Generally you will change to the directory with app.yaml and your other files before deploying

AWS Elastic Beanstalk - .ebextensions

My app currently uses a folder called "Documents" that is located in the root of the app. This is where it stores supporting docs, temporary files, uploaded files etc. I'm trying to move my app from Azure to Beanstalk and I don't know how to give permissions to this folder and sub-folders. I think it's supposed to be done using .ebextensions but I don't know how to format the config file. Can someone suggest how this config file should look? This is an ASP.NET app running on Windows/IIS.
Unfortunately, you cannot use .ebextensions to set permissions to files/folders within your deployment directory.
If you look at the event hooks for an elastic beanstalk deployment:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-windows-ec2.html#windows-container-commands
You'll find that commands run before the ec2 app and web server are set up, and
container_commands run after the ec2 app and web server are setup, but before your application version is deployed.
The solution is to use a wpp.targets file to set the necessary ACLs.
The following SO post is most useful
Can Web Deploy's setAcl provider be used on a sub-directory?
Given below is the sample .ebextensions config file to create a directory/file and modify the permissions and add some content to the file
====== .ebextensions/custom_directory.config ======
commands:
create_directory:
command: mkdir C:\inetpub\AspNetCoreWebApps\backgroundtasks\mydirectory
command: cacls C:\inetpub\AspNetCoreWebApps\backgroundtasks\mydirectory /t /e /g username:W
files:
"C:/inetpub/AspNetCoreWebApps/backgroundtasks/mydirectory/mytestfile.txt":
content: |
This is my Sample file created from ebextensions
ebextensions go into the root of the application source code through a directory called .ebextensions. For more information on how to use ebextensions, please go through the documentation here
Place a file 01_fix_permissions.config inside .ebextensions folder.
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/49_change_permissions.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo chown -R ec2-user:ec2-user tmp/
Following that you can set your folder permissions as you want.
See this answer on Serverfault.
There are platform hooks that you can use to run scripts at various points during deployment that can get you around the shortcomings of the .ebextension Commands and Platform Commands that Napoli describes.
There seems to be some debate on whether or not this setup is officially supported, but judging by comments made on the AWS github, it seems to be not explicitly prohibited.
I can see where Napoli's answer could be the more standard MS way of doing things, but wpp.targets looks like hot trash IMO.
The general scheme of that answer is to use Commands/Platform commands to copy a script file into the appropriate platform hook directory (/opt/elasticbeanstalk/hooks or C:\Program Files\Amazon\ElasticBeanstalk\hooks\ ) to run at your desired stage of deployment.
I think its worth noting that differences exist between platforms and versions such as Amazon Linux 1 and Linux 2.
I hope this helps someone. It took me a day to gather that info and what's on this page and pick what I liked best.
Edit 11/4 - I would like to note that I saw some inconsistencies with the File .ebextension directive when trying to place scripts drirectly into the platform hook dir's during repeated deployments. Specifically the File directive failed to correctly move the backup copies named .bak/.bak1/etc. I would suggest using a Container Command to copy with overwriting from another directory into the desired hook directory to overcome this issue.

A sane way to set up CloudWatch logs (awslogs-agent)

tl;dr The configuration of cloudwatch agent is #$%^. Any straightforward way?
I wanted one place to store the logs, so I used Amazon CloudWatch Logs Agent. At first it seemed like I'd just add a Resource saying something like "create a log group, then a log stream and send this file, thank you" - all declarative and neat, but...
According to this doc I had to setup JSON configuration that created a BASH script that downloaded a Python script that set up the service that used a generated config in yet-another-language somewhere else.
I'd think logging is something frequently used, so there must be a declarative configuration way, not this 4-language crazy combo. Am I missing something, or is ops world so painful?
Thanks for ideas!
"Agent" is just an aws-cli plugin and a bunch of scripts. You can install the plugin with pip install awscli-cwlogs on most systems (assuming you already installed awscli itself). NOTE: I think Amazon Linux is not "most systems" and might require a different approach.
Then you'll need two configs: awscli config with the following content (also add credentials if needed and replace us-east-1 with your region):
[plugins]
cwlogs = cwlogs
[default]
region = us-east-1
and logging config with something like this (adjust to your needs according to the docs):
[general]
state_file = push-state
[logstream-cfn-init.log]
datetime_format = %Y-%m-%d %H:%M:%S,%f
file = /var/log/cfn-init.log
file_fingerprint_lines = 1-3
multi_line_start_pattern = {datetime_format}
log_group_name = ec2-logs
log_stream_name = {hostname}-{instance_id}/cfn-init.log
initial_position = start_of_file
encoding = utf_8
buffer_duration = 5000
after that, to start the daemon automatically you can create a systemd unit like this (change config paths to where you actually put them):
[Unit]
Description=CloudWatch logging daemon
[Service]
ExecStart=/usr/local/bin/aws logs push --config-file /etc/aws/cwlogs
Environment=AWS_CONFIG_FILE=/etc/aws/config
Restart=always
Type=simple
[Install]
WantedBy=multi-user.target
after that you can systemctl enable and systemctl start as usual. That's assuming your instance running a distribution that uses systemd (which is most of them nowadays but if not you should consult documentation to your distribution to learn how to run daemons).
Official setup script also adds a config for logrotate, I skipped that part because it wasn't required in my case but if your logs are rotated you might want to do something with it. Consult the setup script and logrotate documentation for details (essentially you just need to restart the daemon whenever files are rotated).
You've linked doco particular to CloudFormation so a bunch of the complexity is probably associated with that context.
Here's the stand-alone documentation for the Cloudwatch Logs Agent:
Quick Start
Agent Reference
If you're on Amazon Linux, you can install the 'awslogs' system package via yum. Once that's done, you can enable the logs plugin for the AWS CLI by making sure you have the following section in the CLI's config file:
[plugins]
cwlogs = cwlogs
E.g., the system package should create a file under /etc/awslogs/awscli.conf . You can use that file by setting the...
AWS_CONFIG_FILE=/etc/awslogs/awscli.conf
...environment variable.
Once that's all done, you can:
$ aws logs push help
and
$ cat /path/to/some/file | aws logs push [options]
The agent also comes with helpers to keep various log files in sync.