I am producing from VMware datastores collection a list of volumes and their associated tags
I formated them in a JSON type output to be able to feed another system later. The output is working but for the tags section I would like to keep only the name and category_name not the other properties.
This is my playbook :
---
- name: Fetching Datastores
hosts: localhost
tasks:
- name: Gather info from datacenter about specific datastore
community.vmware.vmware_datastore_info:
hostname: 'myesxiHost'
username: 'administrator'
password: 'Password'
validate_certs: False
show_tag: true
delegate_to: localhost
register: ds_info
- set_fact:
ds_info_eph: "{{ ds_info_eph|default([]) + [ {
'name': item.name,
'type': item.type,
'capacity': item.capacity,
'tag': item.tags
} ] }}"
loop: "{{ ds_info.datastores }}"
- name: Output info
debug:
msg: "{{ ds_info_eph }}"
This playbook give me the following output :
ok: [localhost] => {
"msg": [
{
"capacity": 499826819072,
"name": "ESX01_SSD4",
"tag": [
{
"category_id": "urn:vmomi:InventoryServiceCategory:017ad796-b931-4f5b-bd4a-8fee7adaf124:GLOBAL",
"category_name": "Tier",
"description": "",
"id": "urn:vmomi:InventoryServiceTag:6112eeac-ed81-48d4-b7d9-81084b8a415a:GLOBAL",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 499826819072,
"name": "ESX01_SSD3",
"tag": [
{
"category_id": "urn:vmomi:InventoryServiceCategory:017ad796-b931-4f5b-bd4a-8fee7adaf124:GLOBAL",
"category_name": "Tier",
"description": "",
"id": "urn:vmomi:InventoryServiceTag:6112eeac-ed81-48d4-b7d9-81084b8a415a:GLOBAL",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 499826819072,
"name": "ESX01_SSD5",
"tag": [
{
"category_id": "urn:vmomi:InventoryServiceCategory:017ad796-b931-4f5b-bd4a-8fee7adaf124:GLOBAL",
"category_name": "Tier",
"description": "",
"id": "urn:vmomi:InventoryServiceTag:6112eeac-ed81-48d4-b7d9-81084b8a415a:GLOBAL",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 239981297664,
"name": "ESX01_SSD2",
"tag": [
{
"category_id": "urn:vmomi:InventoryServiceCategory:017ad796-b931-4f5b-bd4a-8fee7adaf124:GLOBAL",
"category_name": "Tier",
"description": "",
"id": "urn:vmomi:InventoryServiceTag:6112eeac-ed81-48d4-b7d9-81084b8a415a:GLOBAL",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 248034361344,
"name": "ESX01_SSD1",
"tag": [
{
"category_id": "urn:vmomi:InventoryServiceCategory:017ad796-b931-4f5b-bd4a-8fee7adaf124:GLOBAL",
"category_name": "Tier",
"description": "",
"id": "urn:vmomi:InventoryServiceTag:6112eeac-ed81-48d4-b7d9-81084b8a415a:GLOBAL",
"name": "Gold"
}
],
"type": "VMFS"
}
]
}
Instead I would like to have this output :
ok: [localhost] => {
"msg": [
{
"capacity": 499826819072,
"name": "ESX01_SSD4",
"tag": [
{
"category_name": "Tier",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 499826819072,
"name": "ESX01_SSD3",
"tag": [
{
"category_name": "Tier",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 499826819072,
"name": "ESX01_SSD5",
"tag": [
{
"category_name": "Tier",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 239981297664,
"name": "ESX01_SSD2",
"tag": [
{
"category_name": "Tier",
"name": "Gold"
}
],
"type": "VMFS"
},
{
"capacity": 248034361344,
"name": "ESX01_SSD1",
"tag": [
{
"category_name": "Tier",
"name": "Gold"
}
],
"type": "VMFS"
}
]
}
But on the tag section I tried to do in the playbook the following to get only the name and not all the tag item list : 'tag': item.tags.name but I'v got the following error fatal: [localhost]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ ds_info_eph|default([]) + [ { 'name': item.name, 'type': item.type, 'capacity': item.capacity, 'tag': item.tags.name } ] }}): can only concatenate str (not \"list\") to str"}
My understanding is somehow in that set_fac I have to loop again to get only the tag name and tag category_name but I didn't figure out how to.
Thank you for your help !
Select the attributes from the tag lists, e.g.
- set_fact:
_tags: "{{ ds_info_eph|
map(attribute='tag')|
map('items2dict', 'category_name', 'name')}}"
gives
_tags:
- Tier: Gold
- Tier: Gold
- Tier: Gold
- Tier: Gold
- Tier: Gold
Combine the dictionaries
- set_fact:
result: "{{ result|d([]) + [item.0|combine({'tag': item.1})] }}"
with_together:
- "{{ ds_info_eph }}"
- "{{ _tags|map('dict2items', 'category_name', 'name') }}"
gives
result:
- capacity: 499826819072
name: ESX01_SSD4
tag:
category_name: Tier
name: Gold
type: VMFS
- capacity: 499826819072
name: ESX01_SSD3
tag:
category_name: Tier
name: Gold
type: VMFS
- capacity: 499826819072
name: ESX01_SSD5
tag:
category_name: Tier
name: Gold
type: VMFS
- capacity: 239981297664
name: ESX01_SSD2
tag:
category_name: Tier
name: Gold
type: VMFS
- capacity: 248034361344
name: ESX01_SSD1
tag:
category_name: Tier
name: Gold
type: VMFS
Related
Is there any way to overwrite the default operationid,name,tag,type,summary, description of generated drf api schema?
When use drf-yasg or drf-spectacular,
is there any way to overwrite the default description shown in the sample picture
or does drf-yasg or drf-spectacular provide other language support for the default description?
I tried i18n support of drf-spectacular but I think it's not for this problem.
And to change the language_code settings of django is not work.
I appriciate your help very much
generated schema is :
Schema
GET /schema
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/vnd.oai.openapi
Vary: Accept
openapi: 3.0.2
info:
title: Persons
version: 1.0.0
paths:
/persons:
get:
operationId: listPersonApis
description: Get list of persons
parameters: []
responses:
'200':
content:
application/json:
schema:
type: array
items: {}
description: ''
tags:
- persons
post:
operationId: createPersonApi
description: ":Process: Post a Person \n:summary: \"Create New User\"\n:method:\
\ `POST`\n:HTTPRequest:\n\n** Context **\n\n:firstName: `string`\n\
:lastName: `string`\n:email: `string`\n:Response: `Posted\
\ Data in Json`"
parameters: []
requestBody:
content:
application/json:
schema: {}
application/x-www-form-urlencoded:
schema: {}
multipart/form-data:
schema: {}
responses:
'201':
content:
application/json:
schema: {}
description: ''
tags:
- persons
/persons/{id}/:
get:
operationId: retrievePersonDetail
description: ":Process: Person details\n:method: `GET`\n:HTTPRequest:\n \
\ \n\n** Context **\n:param: `id`\n\n:Response: `Retrieve List of Json Array\
\ Data by ID, And Status Code According method`"
parameters:
- name: id
in: path
required: true
description: ''
schema:
type: string
responses:
'200':
content:
application/json:
schema: {}
description: ''
tags:
- persons
put:
operationId: updatePersonDetail
description: ':Process: Person details
:method: `PUT`
:HTTPRequest:
** Context **
:param: `id`
:Response: `Retrieve List of Json Array Data by ID, And Status Code According
method`'
parameters:
- name: id
in: path
required: true
description: ''
schema:
type: string
requestBody:
content:
application/json:
schema: {}
application/x-www-form-urlencoded:
schema: {}
multipart/form-data:
schema: {}
responses:
'200':
content:
application/json:
schema: {}
description: ''
tags:
- persons
delete:
operationId: destroyPersonDetail
description: ':Process: Person details
:method: `DELETE`
:HTTPRequest:
** Context **
:param: `id`
:"description": "Remove person specified by id"'
parameters:
- name: id
in: path
required: true
description: ''
schema:
type: string
responses:
'204':
description: ''
tags:
- persons
but this schema should be in this format:final schema**
{
"openapi": "3.0.3",
"info": {
"title": "Persons",
"version": "1.0"
},
"servers": [
{
"url": "http://localhost:3000"
}
],
"paths": {
"/persons/{id}": {
"parameters": [
{
"schema": {
"type": "string"
},
"name": "id",
"in": "path",
"required": true
}
],
"get": {
"summary": "Get User Info by User ID",
"tags": [],
"responses": {
"200": {
"description": "User Found",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Result"
},
"examples": {}
}
}
},
"404": {
"description": "User Not Found"
}
},
"operationId": "get-persons-id",
"description": "Retrieve entry for person with the specified id (still delivered as a single entry inside an array)"
},
"delete": {
"summary": "",
"operationId": "delete-persons-id",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Result"
}
}
}
}
},
"description": "Remove person specified by id"
},
"put": {
"summary": "",
"operationId": "put-persons-id",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Result"
}
}
}
}
},
"description": "Update person specified by id",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Person"
}
}
}
}
}
},
"/persons": {
"post": {
"summary": "Create New User",
"operationId": "post-persons",
"responses": {
"200": {
"description": "User Created",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Result"
},
"examples": {}
}
}
},
"400": {
"description": "Missing Required Information"
},
"409": {
"description": "Email Already Taken"
}
},
"description": "Create a new person.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Person"
}
}
}
}
},
"parameters": [],
"get": {
"summary": "",
"operationId": "get-persons",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Result"
}
}
}
}
},
"description": "Get list of persons"
}
}
},
"components": {
"schemas": {
"Person": {
"title": "Person",
"type": "object",
"properties": {
"id": {
"type": "integer",
"readOnly": true
},
"first_name": {
"type": "string"
},
"last_name": {
"type": "string"
},
"email": {
"type": "string",
"format": "email"
}
},
"x-examples": {
"example-1": {
"id": 0,
"first_name": "John",
"last_name": "Smith",
"email": "user#example.com"
}
}
},
"Result": {
"title": "Result",
"type": "object",
"properties": {
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Person"
}
},
"error": {
"type": "object",
"description": "object literal holding error messages keyed to either a field name or \"non_field_errors\" if not applicable\n\n```json\n{\n \"first_name\": [\"required field\"],\n \"non_field_errors\": [\"something more general went wrong\"]\n}\n```"
}
},
"required": [
"data"
],
"x-examples": {
"Example": {
"data": [
{
"id": 0,
"first_name": "John",
"last_name": "Smith",
"email": "user#example.com"
}
],
"error": {}
},
"Error-example": {
"data": [],
"error": {
"first_name": [
"This field is required"
]
}
}
}
}
}
}
}
I have created api using class based api view.
how to override schema this in django rest api like as below json format???
I've written some CDK code to programmatically create a data pipeline that backs up a DynamoDB table into an S3 bucket on a daily basis.
But it keeps running into this error:
amazonaws.datapipeline.taskrunner.TaskExecutionException: Failed to complete EMR transform. at amazonaws.datapipeline.activity.EmrActivity.runActivity(EmrActivity.java:67) at amazonaws.datapipeline.objects.AbstractActivity.run(AbstractActivity.java:16) at amazonaws.datapipeline.taskrunner.TaskPoller.executeRemoteRunner(TaskPoller.java:136) at amazonaws.datapipeline.taskrunner.TaskPoller.executeTask(TaskPoller.java:105) at amazonaws.datapipeline.taskrunner.TaskPoller$1.run(TaskPoller.java:81) at private.com.amazonaws.services.datapipeline.poller.PollWorker.executeWork(PollWorker.java:76) at private.com.amazonaws.services.datapipeline.poller.PollWorker.run(PollWorker.java:53) at java.lang.Thread.run(Thread.java:750) Caused by:
....
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied at amazonaws.datapipeline.activity.mapreduce.HadoopJobRunner.executeCommand(HadoopJobRunner.java:175) at amazonaws.datapipeline.activity.mapreduce.HadoopJobRunner.ex
I'm using the DataPipelineDefaultResourceRole and DataPipelineDefaultRole for this data pipeline which has S3:* permission, so I'm puzzled why this is happening.
On top of that, I'm not sure why logging is not enabled on my EMR cluster which is spun off by this data pipeline although I've specified the logLocation parameter: myLogUri
Any pointers please?
import { CfnPipeline } from "monocdk/aws-datapipeline";
private createDataPipeline(props: InfrastructureStackProps) {
const dataPipelineName = "a-nice-datapipeline8.23";
const pipeline = new CfnPipeline(this, dataPipelineName, {
name: dataPipelineName,
parameterObjects: [
{
id: "myDDBTableName",
attributes: [
{
key: "Description",
stringValue: "Source table"
},
{
key: "Type",
stringValue: "String"
},
{
key: "Default",
stringValue: "Attributes"
}
]
},
{
id: "myOutputS3Location",
attributes: [
{
key: "Description",
stringValue: "Output S3 Location"
},
{
key: "Type",
stringValue: "String"
},
{
key: "Default",
stringValue: "s3://ddb-table-backup/"
}
]
},
{
id: "myDdbReadThroughputRatio",
attributes: [
{
key: "Description",
stringValue: "DynamoDB Read Throughput Ratio"
},
{
key: "Type",
stringValue: "Double"
},
{
key: "Default",
stringValue: "0.15"
}
]
},
{
id: 'myLogUri',
attributes: [
{
key: 'type',
stringValue: 'AWS::S3::ObjectKey',
},
{
key: 'description',
stringValue: 'DataPipeline Log Uri',
},
],
},
{
id: "myDDBRegion",
attributes: [
{
key: "Description",
stringValue: "Region of the DynamoDB Table"
},
{
key: "Type",
stringValue: "String"
},
{
key: "Default",
stringValue: props.region
}
]
}
],
parameterValues: [
{
id: "myDDBTableName",
stringValue: "Attributes"
},
{
id: "myOutputS3Location",
stringValue: "s3://ddb-table-backup/"
},
{
id: "myDdbReadThroughputRatio",
stringValue: "0.15"
},
{
id: 'myLogUri',
stringValue: `s3://data_pipeline_log/`,
},
{
id: "myDDBRegion",
stringValue: props.region
}
],
pipelineObjects: [
{
"id": "EmrClusterForBackup",
"name": "EmrClusterForBackup",
"fields": [
{
"key": "resourceRole",
"stringValue": "DataPipelineDefaultResourceRole"
},
{
"key": "role",
"stringValue": "DataPipelineDefaultRole"
},
{
"key": "coreInstanceCount",
"stringValue": "1"
},
{
"key": "coreInstanceType",
"stringValue": "m4.xlarge"
},
{
"key": "releaseLabel",
"stringValue": "emr-5.29.0"
},
{
"key": "masterInstanceType",
"stringValue": "m4.xlarge"
},
{
"key": "region",
"stringValue": props.region
},
{
"key": "type",
"stringValue": "EmrCluster"
},
{
"key": "terminateAfter",
"stringValue": "2 Hours"
}
]
},
{
"id": "S3BackupLocation",
"name": "S3BackupLocation",
"fields": [
{
"key": "directoryPath",
"stringValue": "s3://ddb-table-backup/"
},
{
"key": "type",
"stringValue": "S3DataNode"
}
]
},
{
"id": "DDBSourceTable",
"name": "DDBSourceTable",
"fields": [
{
"key": "readThroughputPercent",
"stringValue": "0.15"
},
{
"key": "type",
"stringValue": "DynamoDBDataNode"
},
{
"key": "tableName",
"stringValue": "Attributes"
}
]
},
{
"id": "Default",
"name": "Default",
"fields": [
{
"key": "failureAndRerunMode",
"stringValue": "CASCADE"
},
{
"key": "resourceRole",
"stringValue": "DataPipelineDefaultResourceRole"
},
{
"key": "role",
"stringValue": "DataPipelineDefaultRole"
},
{
"key": "scheduleType",
"stringValue": "cron"
},
{
key: 'schedule',
refValue: 'DailySchedule'
},
{
key: 'pipelineLogUri',
stringValue: 's3://data_pipeline_log/',
},
{
"key": "type",
"stringValue": "Default"
}
]
},
{
"name": "Every 1 day",
"id": "DailySchedule",
"fields": [
{
"key": 'type',
"stringValue": 'Schedule'
},
{
"key": 'period',
"stringValue": '1 Day'
},
{
"key": 'startDateTime',
"stringValue": "2021-12-20T00:00:00"
}
]
},
{
"id": "TableBackupActivity",
"name": "TableBackupActivity",
"fields": [
{
"key": "type",
"stringValue": "EmrActivity"
},
{
"key": "output",
"refValue": "S3BackupLocation"
},
{
"key": "input",
"refValue": "DDBSourceTable"
},
{
"key": "maximumRetries",
"stringValue": "2"
},
{
"key": "preStepCommand",
"stringValue": "(sudo yum -y update aws-cli) && (aws s3 rm #{output.directoryPath} --recursive)"
},
{
"key": "step",
"stringValue": "s3://dynamodb-dpl-#{myDDBRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar,org.apache.hadoop.dynamodb.tools.DynamoDBExport,#{output.directoryPath},#{input.tableName},#{input.readThroughputPercent}"
},
{
"key": "runsOn",
"refValue": "EmrClusterForBackup"
},
{
"key": "resizeClusterBeforeRunning",
"stringValue": "false"
}
]
}
],
activate: true
});
return pipeline;
}
I may be avoiding the direct issue at hand here, but I'm curious to know why you are using DataPipeline for this? You would probably be better served using AWS Backup which will allow you to take periodic backups in a manged fashion as well as other features such as expiring backups or sending to cold storage.
On the particular issue at hand, please check that your S3 bucket does not have a resource based policy blocking you: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html
Also check the EC2 role used for your DP, commonly called AmazonEC2RoleforDataPipelineRole. More info on IAM roles for DP here.
I am trying to create Amazon Connect Contact flows using the cloudformation template and Amazon Contact flow language which is new for me. But I am
facing this issue and the stack fails to create. I am not sure If I am having syntax errors in my yml file.
Complete error message
Resource handler returned message: "Invalid request (Service: Connect, Status Code: 400, Request ID: aa1e1049-47bb-43ea-8065-0f8b1cf503d2)" (RequestToken: ff62940b-0e60-2308-d33d-917eeb399402, HandlerErrorCode: InvalidRequest)
Here is the cloud formation template
---
AWSTemplateFormatVersion: "2010-09-09"
Description: "Amazon Connect - Contact Flow"
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "General Information"
Parameters:
- SystemName
- EnvironmentName
- Label:
default: "Amazon Connect Information"
Parameters:
- InstanceID
- ContactFlowName
Parameters:
SystemName:
Type: String
EnvironmentName:
Type: String
AllowedValues: [prd, stg, dev]
InstanceID:
Type: String
Description: "Enter the Amazon Connect Instance ID (e.g. 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx')"
ContactFlowName:
Type: String
Description: "Enter the Name of the Contact flow"
Resources:
ContactFlow:
Type: 'AWS::Connect::ContactFlow'
Properties:
Name: !Sub "${ContactFlowName}"
Description: Example contact flow
InstanceArn: !Sub "arn:aws:connect:${AWS::Region}:${AWS::AccountId}:instance/${InstanceID}"
Type: CONTACT_FLOW
Content: |
{
"Version": "2019-10-30",
"StartAction": "c8275c19-08aa-4ee5-9b42-bcc47d32e914",
"Metadata": {
"entryPointPosition": {
"x": 20,
"y": 20
},
"snapToGrid": false,
"ActionMetadata": {
"5c47b68e-d790-45eb-a0bd-654027af4868": {
"position": {
"x": 463,
"y": 491
},
"useDynamic": false
},
"c8275c19-08aa-4ee5-9b42-bcc47d32e914": {
"position": {
"x": 180,
"y": 20
},
"conditionMetadata": [
{
"id": "6a7b09ef-d441-4183-8ef3-2f0f360ba747",
"value": "1"
},
{
"id": "f3516737-c758-4d6f-8d31-ea43d1e27990",
"value": "2"
},
{
"id": "21ad3977-2559-4449-ba54-b84d4a0521b6",
"value": "3"
}
],
"useDynamic": false,
"useLexBotDropdown": true,
"useDynamicLexBotArn": false
},
"c36d9f86-780d-4c44-8c79-80c8353a699a": {
"position": {
"x": 1068,
"y": 317
}
},
"14837518-cc08-4d54-ac93-ea7e063481a8": {
"position": {
"x": 492,
"y": 340
},
"overrideConsoleVoice": false,
"defaultVoice": "Standard"
},
"1e435e5f-7b62-4938-8c9f-fa05106c72b3": {
"position": {
"x": 501,
"y": 174
},
"overrideConsoleVoice": false,
"defaultVoice": "Standard"
},
"98768cdb-f5e6-4642-90e3-6e06fd6813e8": {
"position": {
"x": 478,
"y": 21
},
"overrideConsoleVoice": false,
"defaultVoice": "Conversational",
"fragments": {
"SetContactData": "bc981918-e0b3-4840-afa5-12b6d7078248"
}
},
"bc981918-e0b3-4840-afa5-12b6d7078248": {
"position": {
"x": 0,
"y": 0
}
},
"3a521089-edfe-4821-8156-19014b7b8926": {
"position": {
"x": 716,
"y": 45
},
"useDynamic": false,
"ContactFlow": {
"id": "arn:aws:connect:eu-central-1:132387760551:instance/a65f970f-0f59-444e-b421-5ec6ec74c116/contact-flow/d727e34f-6707-437f-8013-20e4f2a867d6",
"text": "Transaction Banking Authentication"
}
}
}
},
"Actions": [
{
"Identifier": "5c47b68e-d790-45eb-a0bd-654027af4868",
"Parameters": {
"Text": "An Error Occured!"
},
"Transitions": {
"NextAction": "c36d9f86-780d-4c44-8c79-80c8353a699a",
"Errors": [
{
"NextAction": "c36d9f86-780d-4c44-8c79-80c8353a699a",
"ErrorType": "NoMatchingError"
}
],
"Conditions": []
},
"Type": "MessageParticipant"
},
{
"Identifier": "c8275c19-08aa-4ee5-9b42-bcc47d32e914",
"Parameters": {
"Text": "To continue in English press 1, Ukuqhubeka ngesiZulu cindezela u- 2, Vir Afrikaans Druk 3.",
"StoreInput": "False",
"InputTimeLimitSeconds": "5"
},
"Transitions": {
"NextAction": "5c47b68e-d790-45eb-a0bd-654027af4868",
"Errors": [
{
"NextAction": "5c47b68e-d790-45eb-a0bd-654027af4868",
"ErrorType": "NoMatchingError"
},
{
"NextAction": "5c47b68e-d790-45eb-a0bd-654027af4868",
"ErrorType": "NoMatchingCondition"
},
{
"NextAction": "5c47b68e-d790-45eb-a0bd-654027af4868",
"ErrorType": "InputTimeLimitExceeded"
}
],
"Conditions": [
{
"NextAction": "98768cdb-f5e6-4642-90e3-6e06fd6813e8",
"Condition": {
"Operator": "Equals",
"Operands": [
"1"
]
}
},
{
"NextAction": "1e435e5f-7b62-4938-8c9f-fa05106c72b3",
"Condition": {
"Operator": "Equals",
"Operands": [
"2"
]
}
},
{
"NextAction": "14837518-cc08-4d54-ac93-ea7e063481a8",
"Condition": {
"Operator": "Equals",
"Operands": [
"3"
]
}
}
]
},
"Type": "GetParticipantInput"
},
{
"Identifier": "c36d9f86-780d-4c44-8c79-80c8353a699a",
"Type": "DisconnectParticipant",
"Parameters": {},
"Transitions": {}
},
{
"Identifier": "14837518-cc08-4d54-ac93-ea7e063481a8",
"Parameters": {
"TextToSpeechVoice": "Lotte"
},
"Transitions": {
"NextAction": "3a521089-edfe-4821-8156-19014b7b8926",
"Errors": [],
"Conditions": []
},
"Type": "UpdateContactTextToSpeechVoice"
},
{
"Identifier": "1e435e5f-7b62-4938-8c9f-fa05106c72b3",
"Parameters": {
"TextToSpeechVoice": "Marlene"
},
"Transitions": {
"NextAction": "3a521089-edfe-4821-8156-19014b7b8926",
"Errors": [],
"Conditions": []
},
"Type": "UpdateContactTextToSpeechVoice"
},
{
"Identifier": "98768cdb-f5e6-4642-90e3-6e06fd6813e8",
"Parameters": {
"TextToSpeechVoice": "Joanna"
},
"Transitions": {
"NextAction": "bc981918-e0b3-4840-afa5-12b6d7078248",
"Errors": [],
"Conditions": []
},
"Type": "UpdateContactTextToSpeechVoice"
},
{
"Identifier": "bc981918-e0b3-4840-afa5-12b6d7078248",
"Parameters": {
"LanguageCode": "en-US"
},
"Transitions": {
"NextAction": "3a521089-edfe-4821-8156-19014b7b8926",
"Errors": [
{
"NextAction": "3a521089-edfe-4821-8156-19014b7b8926",
"ErrorType": "NoMatchingError"
}
],
"Conditions": []
},
"Type": "UpdateContactData"
},
{
"Identifier": "3a521089-edfe-4821-8156-19014b7b8926",
"Parameters": {
"ContactFlowId": "arn:aws:connect:eu-central-1:132387760551:instance/a65f970f-0f59-444e-b421-5ec6ec74c116/contact-flow/d727e34f-6707-437f-8013-20e4f2a867d6"
},
"Transitions": {
"NextAction": "c36d9f86-780d-4c44-8c79-80c8353a699a",
"Errors": [
{
"NextAction": "c36d9f86-780d-4c44-8c79-80c8353a699a",
"ErrorType": "NoMatchingError"
}
],
"Conditions": []
},
"Type": "TransferToFlow"
}
]
Tags:
- Key: System
Value: !Ref SystemName
- Key: Environment
Value: !Ref EnvironmentName
Any kinda help will Appreciated
Okay, looks like the Content object is not valid. For starters it ends with a ] character instead of a } But even fixing that it has issues.
I'd suggest creating a flow in the gui and export it, then use the json from that as a template.
Also, check the Flow language documentation to get the syntax and structure correct.
I am using google cloud-build as my CI system and Spinnaker as my CD.
I have configured cloud-build to copy my manifest and include it as an artifact.
that's my cloudbuild.yaml:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '--tag=gcr.io/$PROJECT_ID/gordion-backend:$COMMIT_SHA', '.']
- name: 'gcr.io/cloud-builders/gsutil'
args: ['cp', 'manifests/manifest.yaml', 'gs://$PROJECT_ID-kubernetes-manifests/gordion-backend/']
images: ['gcr.io/$PROJECT_ID/gordion-backend']
artifacts:
objects:
location: 'gs://${_BUCKET_NAME}/$COMMIT_SHA/'
paths: ['manifests/manifest.yaml']
When the build is over, that's the message being broadcast by pub/sub:
{
"id": "ba084f1c-494a-42c0-8499-1b84fbb30551",
"projectId": "gordion-2",
"status": "SUCCESS",
"source": {
"repoSource": {
"projectId": "gordion-2",
"repoName": "github_gordion-beam_gordion-backend",
"branchName": "feature/build-yaml"
}
},
"steps": [
{
"name": "gcr.io/cloud-builders/docker",
"args": [
"build",
"--tag=gcr.io/gordion-2/gordion-backend:936543554ad5fab4017319005413dd38beefda04",
"."
],
"timing": {
"startTime": "2019-10-04T12:03:46.996926010Z",
"endTime": "2019-10-04T12:03:57.357589900Z"
},
"pullTiming": {
"startTime": "2019-10-04T12:03:46.996926010Z",
"endTime": "2019-10-04T12:03:47.056257759Z"
},
"status": "SUCCESS"
},
{
"name": "gcr.io/cloud-builders/gsutil",
"args": [
"cp",
"manifests/manifest.yaml",
"gs://gordion-2-kubernetes-manifests/gordion-backend/"
],
"timing": {
"startTime": "2019-10-04T12:03:57.357662249Z",
"endTime": "2019-10-04T12:04:00.089381056Z"
},
"pullTiming": {
"startTime": "2019-10-04T12:03:57.357662249Z",
"endTime": "2019-10-04T12:03:57.436904050Z"
},
"status": "SUCCESS"
}
],
"results": {
"images": [
{
"name": "gcr.io/gordion-2/gordion-backend:936543554ad5fab4017319005413dd38beefda04",
"digest": "sha256:538771fe1be2abd54f6d092519c131019d64bd1001ba38138c565faa9fe343b6",
"pushTiming": {
"startTime": "2019-10-04T12:04:00.404451498Z",
"endTime": "2019-10-04T12:04:06.089323750Z"
}
}
],
"buildStepImages": [
"sha256:c0525aac022b1a92e97f9c6cf4ede4dd82979a9f014f05b2ec8843012a03aa60",
"sha256:fbd483fa382118462a136f916b17d3197325881b6d966ba82ee3a54f4b550e76"
],
"artifactManifest": "gs://gordion-manifests/gordion-2/936543554ad5fab4017319005413dd38beefda04/artifacts-ba084f1c-494a-42c0-8499-1b84fbb30551.json",
"numArtifacts": "1"
},
"createTime": "2019-10-04T12:03:32.668464866Z",
"startTime": "2019-10-04T12:03:38.343591796Z",
"finishTime": "2019-10-04T12:04:15.243708Z",
"timeout": "600s",
"images": [
"gcr.io/gordion-2/gordion-backend"
],
"artifacts": {
"images": [
"gcr.io/gordion-2/gordion-backend"
],
"objects": {
"location": "gs://gordion-manifests/gordion-2/936543554ad5fab4017319005413dd38beefda04/",
"paths": [
"manifests/manifest.yaml"
],
"timing": {
"startTime": "2019-10-04T12:04:08.060372060Z",
"endTime": "2019-10-04T12:04:12.248879364Z"
}
}
},
"logsBucket": "gs://1039931537996.cloudbuild-logs.googleusercontent.com",
"sourceProvenance": {
"resolvedRepoSource": {
"projectId": "gordion-2",
"repoName": "github_gordion-beam_gordion-backend",
"commitSha": "936543554ad5fab4017319005413dd38beefda04"
}
},
"buildTriggerId": "6b7405e7-c2db-4412-9f1a-186c7f7d5975",
"options": {
"substitutionOption": "ALLOW_LOOSE",
"logging": "LEGACY"
},
"logUrl": "https://console.cloud.google.com/gcr/builds/ba084f1c-494a-42c0-8499-1b84fbb30551?project=1039931537996",
"substitutions": {
"_BUCKET_NAME": "gordion-manifests"
},
"tags": [
"trigger-6b7405e7-c2db-4412-9f1a-186c7f7d5975"
],
"timing": {
"BUILD": {
"startTime": "2019-10-04T12:03:46.996877211Z",
"endTime": "2019-10-04T12:04:00.404416077Z"
},
"FETCHSOURCE": {
"startTime": "2019-10-04T12:03:39.677294695Z",
"endTime": "2019-10-04T12:03:46.930866092Z"
},
"PUSH": {
"startTime": "2019-10-04T12:04:00.404450167Z",
"endTime": "2019-10-04T12:04:14.358324463Z"
}
}
}
I can also see in cloud-build history that it produced 2 artifacts:
docker image
Kubernetes manifest
the manifest is located in this location:
gs://gordion-manifests/a91cb009fa184713b86eb8b532a75dc088a25713/manifest.yaml gs://"gordion-manifests
As the location of the artifact is dynamic (due to folders named after SHA), How do I consume this artifact in expected artifacts in spinnaker?
You can use wildcards in artifact's matcher.
Please refer to documentation - https://www.spinnaker.io/reference/artifacts/in-pipelines/#expected-artifacts for overall description and
https://www.spinnaker.io/reference/artifacts-with-artifactsrewrite/types/gcs-object/#in-a-trigger for GCS artifact details.
For example following matcher should handle your case (I don't have instance to test it properly):
{
"type": "gcs/object",
"name": "gs://gordion-manifests/.*/manifest.yaml"
}
When I deploy the below AWS CloudFormation script, I am getting the following error: "Encountered unsupported property InstanceGroups"
I have used InstanceGroups in the past without any issues. Here is an example of how others using it: https://noise.getoto.net/tag/amazon-emr/
I am using EMR 5.17.0, which I have used to set up before.
{
"Description": "Spark ETL EMR CloudFormation",
"Resources": {
"EMRCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Applications": [
{
"Name": "Hadoop"
},
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"AutoScalingRole": "EMR_AutoScaling_DefaultRole",
"BootstrapActions": [
{
"Path": "s3://somepath/scripts/install_pip36_dependencies.sh",
"Args": [
"relay==0.0.1"
],
"Name": "install_pip36_dependencies"
}
],
"Configurations": [
{
"Classification": "yarn-site",
"Properties": {
"yarn.scheduler.fair.preemption": "False",
"yarn.resourcemanager.am.max-attempts": "1"
},
"Configurations": []
},
{
"Classification": "core-site",
"Properties": {
"fs.s3.canned.acl": "BucketOwnerFullControl"
},
"Configurations": []
}
],
"EbsRootVolumeSize": 10,
"InstanceGroups": [
{
"Name": "Master",
"Market": "ON_DEMAND",
"InstanceRole": "MASTER",
"InstanceType": "m5.2xlarge",
"InstanceCount": 1,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "64"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Core",
"Market": "ON_DEMAND",
"InstanceGroupType": "CORE",
"InstanceType": "m5.2xlarge",
"InstanceCount": 5,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Task - 3",
"Market": "ON_DEMAND",
"InstanceGroupType": "TASK",
"InstanceType": "m5.2xlarge",
"InstanceCount": 2,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 32,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
}
],
"LogUri": "s3://somepath/emr-logs/",
"Name": "EMR CF",
"ReleaseLabel": "emr-5.17.0",
"ServiceRole": "EMR_DefaultRole",
"VisibleToAllUsers": "True"
}
}
}
}
When the CF script is loaded, it should create an AWS EMR cluster
Aws recommends that you set MasterInstanceGroup and CoreInstanceGroup under Instances
I give you an example of the Instances property of an EMR Cluster with Hadoop, Hbase, Spark, Ganglia and Zookeeper:
Instances:
Ec2KeyName: !Ref KeyName
Ec2SubnetId: !ImportValue MySubnetPrivateA
EmrManagedMasterSecurityGroup: !ImportValue EmrMasterSgId
AdditionalMasterSecurityGroups:
- !ImportValue EmrMasterAdditionalSgId
EmrManagedSlaveSecurityGroup: !ImportValue EmrSlaveSgId
AdditionalSlaveSecurityGroups:
- !ImportValue EmrSlaveAdditionalSgId
ServiceAccessSecurityGroup: !ImportValue EmrServiceSgId
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref MasterInstanceType
Market: ON_DEMAND
Name: Master
CoreInstanceGroup:
InstanceCount: !Ref NumberOfCoreInstances
InstanceType: !Ref CoreInstanceType
Market: ON_DEMAND
Name: Core
TerminationProtected: false
VisibleToAllUsers: true
JobFlowRole: !Ref EMRClusterinstanceProfile
ReleaseLabel: !Ref ReleaseLabel
LogUri: !Ref LogUri
Name: !Ref EMRClusterName
AutoScalingRole: EMR_AutoScaling_DefaultRole
ServiceRole: !Ref EMRClusterServiceRole
Tags:
-
Key: "cluster_name"
Value: "master.emr.my.com"
You can see the complete AWS template here.