Why is AWS CloudFormation throwing "Encountered unsupported property InstanceGroups"? - amazon-web-services

Question

When I deploy the below AWS CloudFormation script, I am getting the following error: "Encountered unsupported property InstanceGroups"
I have used InstanceGroups in the past without any issues. Here is an example of how others using it: https://noise.getoto.net/tag/amazon-emr/
I am using EMR 5.17.0, which I have used to set up before.
{
"Description": "Spark ETL EMR CloudFormation",
"Resources": {
"EMRCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Applications": [
{
"Name": "Hadoop"
},
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"AutoScalingRole": "EMR_AutoScaling_DefaultRole",
"BootstrapActions": [
{
"Path": "s3://somepath/scripts/install_pip36_dependencies.sh",
"Args": [
"relay==0.0.1"
],
"Name": "install_pip36_dependencies"
}
],
"Configurations": [
{
"Classification": "yarn-site",
"Properties": {
"yarn.scheduler.fair.preemption": "False",
"yarn.resourcemanager.am.max-attempts": "1"
},
"Configurations": []
},
{
"Classification": "core-site",
"Properties": {
"fs.s3.canned.acl": "BucketOwnerFullControl"
},
"Configurations": []
}
],
"EbsRootVolumeSize": 10,
"InstanceGroups": [
{
"Name": "Master",
"Market": "ON_DEMAND",
"InstanceRole": "MASTER",
"InstanceType": "m5.2xlarge",
"InstanceCount": 1,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "64"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Core",
"Market": "ON_DEMAND",
"InstanceGroupType": "CORE",
"InstanceType": "m5.2xlarge",
"InstanceCount": 5,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Task - 3",
"Market": "ON_DEMAND",
"InstanceGroupType": "TASK",
"InstanceType": "m5.2xlarge",
"InstanceCount": 2,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 32,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
}
],
"LogUri": "s3://somepath/emr-logs/",
"Name": "EMR CF",
"ReleaseLabel": "emr-5.17.0",
"ServiceRole": "EMR_DefaultRole",
"VisibleToAllUsers": "True"
}
}
}
}
When the CF script is loaded, it should create an AWS EMR cluster

Answer 1

Aws recommends that you set MasterInstanceGroup and CoreInstanceGroup under Instances
I give you an example of the Instances property of an EMR Cluster with Hadoop, Hbase, Spark, Ganglia and Zookeeper:
Instances:
Ec2KeyName: !Ref KeyName
Ec2SubnetId: !ImportValue MySubnetPrivateA
EmrManagedMasterSecurityGroup: !ImportValue EmrMasterSgId
AdditionalMasterSecurityGroups:
- !ImportValue EmrMasterAdditionalSgId
EmrManagedSlaveSecurityGroup: !ImportValue EmrSlaveSgId
AdditionalSlaveSecurityGroups:
- !ImportValue EmrSlaveAdditionalSgId
ServiceAccessSecurityGroup: !ImportValue EmrServiceSgId
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref MasterInstanceType
Market: ON_DEMAND
Name: Master
CoreInstanceGroup:
InstanceCount: !Ref NumberOfCoreInstances
InstanceType: !Ref CoreInstanceType
Market: ON_DEMAND
Name: Core
TerminationProtected: false
VisibleToAllUsers: true
JobFlowRole: !Ref EMRClusterinstanceProfile
ReleaseLabel: !Ref ReleaseLabel
LogUri: !Ref LogUri
Name: !Ref EMRClusterName
AutoScalingRole: EMR_AutoScaling_DefaultRole
ServiceRole: !Ref EMRClusterServiceRole
Tags:
-
Key: "cluster_name"
Value: "master.emr.my.com"
You can see the complete AWS template here.