I would like to include --optional-components ANACONDA,JUPYTER in my CICD deployment using Deployment Manager
I have tried to place it in python template configuration under metadata secion as well as directly in properties, I checked schemas, existing templates, however I cant find proper spot for it nor anything related in documentation(maybe I missed something, but I only reached gcloud CLI part not DM equvalent)
My expected result would have --optional-components ANACONDA,JUPYTER inside :
`
resources = []
# cluster X
resources.append({
'name': 'X',
'type': 'dataproc.py',
'subnetwork': 'default',
'properties': {
'zone': ZONE,
'region': REGION,
'serviceAccountEmail': 'X',
'softwareConfig': {
'imageVersion': 'X',
'properties': {
'dataproc:dataproc.conscrypt.provider.enable' : 'False'
}
},
'master': {
'numInstances': 1,
'machineType': 'n1-standard-1',
'diskSizeGb': 50,
'diskType': 'pd-standard',
'numLocalSsds': 0
},
'worker': {
'numInstances': 2,
'machineType': 'n1-standard-1',
'diskType': 'pd-standard',
'diskSizeGb': 50,
'numLocalSsds': 0
},
'initializationActions':[{
'executableFile': 'X'
}],
'metadata': {
'PIP_PACKAGES':'requests_toolbelt==0.9.1 google-auth==1.6.31'
},
'labels': {
'environment': 'dev',
'data_type': 'X'
}
}
})`
Deployment manager only uses the APIs and only supports certain APIs. For your case to write --optional-components ANACONDA,JUPYTER , please follow this link as you almost close.To have details about the cloud dataproc components refer to the dataproc API reference.
Related
I'm trying to use Boto3 to get the number of vulnerabilities from my images in my repositories. I have a list of repository names and image IDs that are getting passed into this function. Based off their documentation
I'm expecting a response like this when I filter for ['imageScanFindings']
'imageScanFindings': {
'imageScanCompletedAt': datetime(2015, 1, 1),
'vulnerabilitySourceUpdatedAt': datetime(2015, 1, 1),
'findingSeverityCounts': {
'string': 123
},
'findings': [
{
'name': 'string',
'description': 'string',
'uri': 'string',
'severity': 'INFORMATIONAL'|'LOW'|'MEDIUM'|'HIGH'|'CRITICAL'|'UNDEFINED',
'attributes': [
{
'key': 'string',
'value': 'string'
},
]
},
],
What I really need is the
'findingSeverityCounts' number, however, it's not showing up in my response. Here's my code and the response I get:
main.py
repo_names = ['cftest/repo1', 'your-repo-name', 'cftest/repo2']
image_ids = ['1.1.1', 'latest', '2.2.2']
def get_vuln_count(repo_names, image_ids):
container_inventory = []
client = boto3.client('ecr')
for n, i in zip(repo_names, image_ids):
response = client.describe_image_scan_findings(
repositoryName=n,
imageId={'imageTag': i}
)
findings = response['imageScanFindings']
print(findings)
Output
{'findings': []}
The only thing that shows up is findings and I was expecting findingSeverityCounts in the response along with the others, but nothing else is showing up.
THEORY
I have 3 repositories and an image in each repository that I uploaded. One of my theories is that I'm not getting the other responses, such as findingSeverityCounts because my images don't have vulnerabilities? I have inspector set-up to scan on push, but they don't have vulnerabilities so nothing shows up in the inspector dashboard. Could that be causing the issue? If so, how would I be able to generate a vulnerability in one of my images to test this out?
My theory was correct and when there are no vulnerabilities, the response completely omits certain values, including the 'findingSeverityCounts' value that I needed.
I created a docker image using python 2.7 to generate vulnerabilities in my scan to test out my script properly. My work around was to implement this if statement- if there's vulnerabilities it will return them, if there aren't any vulnerabilities, that means 'findingSeverityCounts' is omitted from the response, so I'll have it return 0 instead of giving me a key error.
Example Solution:
response = client.describe_image_scan_findings(
repositoryName=n,
imageId={'imageTag': i}
)
if 'findingSeverityCounts' in response['imageScanFindings']:
print(response['imageScanFindings']['findingSeverityCounts'])
else:
print(0)
import json
import boto3,datetime
def lambda_handler(event, context):
cloudwatch = boto3.client('cloudwatch',region_name=AWS_REGION)
response = cloudwatch.get_metric_data(
MetricDataQueries=[
{
'Id': 'memory',
'MetricStat': {
'Metric': {
'Namespace': 'AWS/RDS',
'MetricName': 'TotalMemory',
'Dimensions': [
{
"Name": "DBInstanceIdentifier",
"Value": "mydb"
}]
},
'Period': 30,
'Stat': 'Average',
}
}
],
StartTime=(datetime.datetime.now() - datetime.timedelta(seconds=300)).timestamp(),
EndTime=datetime.datetime.now().timestamp()
)
print(response)
The result is like below:
{'MetricDataResults': [{'Id': 'memory', 'Label': 'TotalMemory', 'Timestamps': [], 'Values': [], 'StatusCode': 'Complete'}]
If you are looking to get the configured vCPU/Memory then it seems like we need to call DescribeDBInstances API to get DBInstanceClass, which contains the hardware information from here
You would need to use one of the CloudWatch metric names from https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MonitoringOverview.html#rds-metrics and it seems like we can retrieve the currently available memory metric using the FreeableMemory. I was able to get data (in bytes) as seen from the RDS' Monitoring console while using this metric name from your sample code.
You can check the total amount of memory and other useful information associated with the RDS in the CloudWatch console.
Step1: Go to the CloudWatch console. Navigate to Log groups.
Step2: Search for RDSOSMetrics in the search bar.
Step3: Click on the log stream. You will be able to find all the details in the JSON. Your total memory would be present in the field titled memory.total. Sample result would be like this
{
"engine": "MYSQL",
"instanceID": "dbName",
"uptime": "283 days, 21:08:36",
"memory": {
"writeback": 0,
"free": 171696,
"hugePagesTotal": 0,
"inactive": 1652000,
"pageTables": 19716,
"dirty": 324,
"active": 5850016,
"total": 7877180,
"buffers": 244312
}
}
I have intentionally reduced the message in the JSON because of the size, but there will be many other useful fields that you can find here.
You can use custom jq command-line utility to extract the field that you want from these log groups.
You can read more about this here cloudwatch enhanced monitoring.
I am about at my wits end with this one. I am currently working on setting up an auto deployment for a aws application. The application is running inside of ECS without a load balancer.
The idea here is that when a new docker image get pushed to ECR, cloudTrail picks up the log, cloudWatch triggers and alarm, said alarm goes to SNS, and SNS triggers a lambda to deploy. So far I have everything working except the lambda.
The code I am using is in python and is as follows:
import boto3
import pprint
import os
region = "us-east-2"
client = boto3.client('ecs', region_name=region)
response = client.list_task_definitions(familyPrefix= 'stage-dcs-task-000', status='ACTIVE')
def lambda_handler(event, context):
response = client.register_task_definition(
family='stage-dcs-task-000',
networkMode='bridge',
containerDefinitions=[
{
'name': 'dcs-stage-container-000',
'image': 'ecrUrlHere:latest',
'cpu': 10,
'memory': 500,
'portMappings': [
{
'containerPort': 8080,
'hostPort': 80,
'protocol': 'tcp'
},
],
'essential': True,
},
],
)
taskDefinitionRev = response['taskDefinition']['family'] + ':' + str(response['taskDefinition']['revision'])
response = client.update_service(
cluster='stage-secretName-conversation-service',
service='stage-dcs-service-000',
desiredCount=1,
taskDefinition=taskDefinitionRev,
deploymentConfiguration={
'maximumPercent': 200,
'minimumHealthyPercent': 100
},
forceNewDeployment=True
)
Now, this lambda fails with the error
"errorMessage": "An error occurred (InvalidParameterException) when calling the UpdateService operation: The container stage-dcs-container-000is not valid. The value for 'containerName' must already be specified in the task definition. Registry: ARN-LINK-HERE"
Looking at what gets created in AWS, the task definition was created correctly, and the container was named and configured correctly. Based on what I understand from the docs, my call to update-service is correct, I just have no idea why the container name would be invalid.
Something interesting to note, is I though the container names might have to be unique. So I started add the task revision number to the end of the container name in place of the 000, however, I always get back stage-dcs-container-000 back as the invalid name and have no idea where it might be coming from. Any ideas here?
After doing some more debugging I have figured out that it is the container name within the current task definition that is throwing the error. Based on container name rules, I cannot have dashes, so I changed them to underscores, but the same issues still persists.
resources: this is the tutorial I have been following https://medium.com/#YadavPrakshi/automate-zero-downtime-deployment-with-amazon-ecs-and-lambda-c4e49953273d
It took a little while for me to figure this one out, but I eventually got it solved. Truth be told I am not 100% sure what the fix was, but I know there were at least two errors.
The first was the fact that I had somehow managed to fall into a state where I had illegal characters in the name of different parts of the stack. To give an explicit example, I have dashes (-) in the name of the containers, which is an illegal character. To solve this is just ended up recreating the whole tech stack using camel casing.
The second was I had some error in the lambda function, specifically in the creation of the task revision. What I ended up noticing what that when I created things via the cli I somehow missed the docker requirement. How I got around this was I created the task revision in the AWS consul and then just copied the json into to register_task_definition method inside the lambda. This is what I started with that DID NOT WORK.
response = client.register_task_definition(
family='stage-dcs-task-000',
networkMode='bridge',
containerDefinitions=[
{
'name': 'dcs-stage-container-000',
'image': 'ecrUrlHere:latest',
'cpu': 10,
'memory': 500,
'portMappings': [
{
'containerPort': 8080,
'hostPort': 80,
'protocol': 'tcp'
},
],
'essential': True,
},
],
)
this is what I finished with that DOES WORK.
response = client.register_task_definition(
containerDefinitions=[
{
"portMappings": [
{
"hostPort": 80,
"protocol": "tcp",
"containerPort": 8080
}
],
"cpu": 10,
"environment": [],
"mountPoints": [],
"memoryReservation": 500,
"volumesFrom": [],
"image": "image uri here",
"essential": True,
"name": "productionDCSContainer"
}
],
family="productionDCSTask",
requiresCompatibilities=[
"EC2"
],
networkMode="bridge"
)
It should be noted that it is easy to fall into a state where you dead lock on the revision update, that is to say you have something like minPercentage = 100, maxPercentage = 100 and desired = 1. In this case, AWS will never kill the running instance to start the update. A way around this is have minInstance = 0, maxInstance = 200% and have forceNewDeployment = True.
I am trying to define a create cluster function to create a cluster on Cloud Dataproc. While going through the reference material I came across an idle delete parameter (idleDeleteTtl) which would auto-delete the cluster if not in use for the amount of time defined. When I try to include it in cluster config it gives me a ValueError: Protocol message ClusterConfig has no "lifecycleConfig" field.
The create cluster function for reference:
def create_cluster(dataproc, project, zone, region, cluster_name, pip_packages):
"""Create the cluster."""
print('Creating cluster...')
zone_uri = \
'https://www.googleapis.com/compute/v1/projects/{}/zones/{}'.format(
project, zone)
cluster_data = {
'project_id': project,
'cluster_name': cluster_name,
'config': {
'initialization_actions': [{
'executable_file': 'gs://<some_path>/python/pip-install.sh'
}],
'gce_cluster_config': {
'zone_uri': zone_uri,
'metadata': {
'PIP_PACKAGES': pip_packages
}
},
'master_config': {
'num_instances': 1,
'machine_type_uri': 'n1-standard-1'
},
'worker_config': {
'num_instances': 2,
'machine_type_uri': 'n1-standard-1'
},
'lifecycleConfig': { #### PROBLEM AREA ####
'idleDeleteTtl': '30m'
}
}
}
cluster = dataproc.create_cluster(project, region, cluster_data)
cluster.add_done_callback(callback)
global waiting_callback
waiting_callback = True
I want similar functionality if not in the same function itself. I already have a manual delete function defined but I want to add the functionality to auto delete clusters when not in use.
You are calling the v1 API passing a parameter that is part of the v1beta2 API.
Change your endpoint from:
https://www.googleapis.com/compute/v1/projects/{}/zones/{}
To this:
https://www.googleapis.com/compute/v1beta2/projects/{}/zones/{}
Im looking at this example: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/examples/v2/cloud_functions
which uses this template. I added a print statement to it, but how do I see this output?
import base64
import hashlib
from StringIO import StringIO
import zipfile
def GenerateConfig(ctx):
"""Generate YAML resource configuration."""
in_memory_output_file = StringIO()
function_name = ctx.env['deployment'] + 'cf'
zip_file = zipfile.ZipFile(
in_memory_output_file, mode='w', compression=zipfile.ZIP_DEFLATED)
####################################################
############ HOW DO I SEE THIS????? ################
print('heelo wworrld')
####################################################
####################################################
for imp in ctx.imports:
if imp.startswith(ctx.properties['codeLocation']):
zip_file.writestr(imp[len(ctx.properties['codeLocation']):],
ctx.imports[imp])
zip_file.close()
content = base64.b64encode(in_memory_output_file.getvalue())
m = hashlib.md5()
m.update(content)
source_archive_url = 'gs://%s/%s' % (ctx.properties['codeBucket'],
m.hexdigest() + '.zip')
cmd = "echo '%s' | base64 -d > /function/function.zip;" % (content)
volumes = [{'name': 'function-code', 'path': '/function'}]
build_step = {
'name': 'upload-function-code',
'action': 'gcp-types/cloudbuild-v1:cloudbuild.projects.builds.create',
'metadata': {
'runtimePolicy': ['UPDATE_ON_CHANGE']
},
'properties': {
'steps': [{
'name': 'ubuntu',
'args': ['bash', '-c', cmd],
'volumes': volumes,
}, {
'name': 'gcr.io/cloud-builders/gsutil',
'args': ['cp', '/function/function.zip', source_archive_url],
'volumes': volumes
}],
'timeout':
'120s'
}
}
cloud_function = {
'type': 'gcp-types/cloudfunctions-v1:projects.locations.functions',
'name': function_name,
'properties': {
'parent':
'/'.join([
'projects', ctx.env['project'], 'locations',
ctx.properties['location']
]),
'function':
function_name,
'labels': {
# Add the hash of the contents to trigger an update if the bucket
# object changes
'content-md5': m.hexdigest()
},
'sourceArchiveUrl':
source_archive_url,
'environmentVariables': {
'codeHash': m.hexdigest()
},
'entryPoint':
ctx.properties['entryPoint'],
'httpsTrigger': {},
'timeout':
ctx.properties['timeout'],
'availableMemoryMb':
ctx.properties['availableMemoryMb'],
'runtime':
ctx.properties['runtime']
},
'metadata': {
'dependsOn': ['upload-function-code']
}
}
resources = [build_step, cloud_function]
return {
'resources':
resources,
'outputs': [{
'name': 'sourceArchiveUrl',
'value': source_archive_url
}, {
'name': 'name',
'value': '$(ref.' + function_name + '.name)'
}]
}
EDIT: this is in no way a solution to this problem but I found that if I set a bunch of outputs for info im interested in seeing it kind of helps. So I guess you could roll your own sort of log-ish thing by collecting info/output into a list or something in your python template and then passing all that back as an output- not great but its better than nothing
Deployment Manager is an infrastructure deployment service that automates the creation and management of Google Cloud Platform (GCP) resources. What you are trying to do on deployment manager is not possible due to its managed environment.
As of now, the only way to troubleshoot is to rely on the expanded template from the Deployment Manager Dashboard. There is already a feature request in order to address your use case here. I advise you to star the feature request in order to get updates via email and to place a comment in order to show the interest of the community. All the official communication regarding that feature will be posted there.