add-instance-fleet¶

Description¶

Adds an instance fleet to a running cluster.

Note

The instance fleet configuration is available only in Amazon EMR versions 4.8.0 and later, excluding 5.0.x.

Synopsis¶

  add-instance-fleet
--cluster-id <value>
--instance-fleet <value>
[--cli-input-json | --cli-input-yaml]
[--generate-cli-skeleton <value>]

Options¶

--cluster-id (string)

The unique identifier of the cluster.

--instance-fleet (structure)

Specifies the configuration of the instance fleet.

Name -> (string)

The friendly name of the instance fleet.

InstanceFleetType -> (string)

The node type that the instance fleet hosts. Valid values are MASTER, CORE, and TASK.

TargetOnDemandCapacity -> (integer)

The target capacity of On-Demand units for the instance fleet, which determines how many On-Demand Instances to provision. When the instance fleet launches, Amazon EMR tries to provision On-Demand Instances as specified by InstanceTypeConfig . Each instance configuration has a specified WeightedCapacity . When an On-Demand Instance is provisioned, the WeightedCapacity units count toward the target capacity. Amazon EMR provisions instances until the target capacity is totally fulfilled, even if this results in an overage. For example, if there are 2 units remaining to fulfill capacity, and Amazon EMR can only provision an instance with a WeightedCapacity of 5 units, the instance is provisioned, and the target capacity is exceeded by 3 units.

Note

If not specified or set to 0, only Spot Instances are provisioned for the instance fleet using TargetSpotCapacity . At least one of TargetSpotCapacity and TargetOnDemandCapacity should be greater than 0. For a master instance fleet, only one of TargetSpotCapacity and TargetOnDemandCapacity can be specified, and its value must be 1.

TargetSpotCapacity -> (integer)

The target capacity of Spot units for the instance fleet, which determines how many Spot Instances to provision. When the instance fleet launches, Amazon EMR tries to provision Spot Instances as specified by InstanceTypeConfig . Each instance configuration has a specified WeightedCapacity . When a Spot Instance is provisioned, the WeightedCapacity units count toward the target capacity. Amazon EMR provisions instances until the target capacity is totally fulfilled, even if this results in an overage. For example, if there are 2 units remaining to fulfill capacity, and Amazon EMR can only provision an instance with a WeightedCapacity of 5 units, the instance is provisioned, and the target capacity is exceeded by 3 units.

Note

If not specified or set to 0, only On-Demand Instances are provisioned for the instance fleet. At least one of TargetSpotCapacity and TargetOnDemandCapacity should be greater than 0. For a master instance fleet, only one of TargetSpotCapacity and TargetOnDemandCapacity can be specified, and its value must be 1.

InstanceTypeConfigs -> (list)

The instance type configurations that define the EC2 instances in the instance fleet.

(structure)

An instance type configuration for each instance type in an instance fleet, which determines the EC2 instances Amazon EMR attempts to provision to fulfill On-Demand and Spot target capacities. There can be a maximum of five instance type configurations in a fleet.

Note

The instance fleet configuration is available only in Amazon EMR versions 4.8.0 and later, excluding 5.0.x versions.

InstanceType -> (string)

An EC2 instance type, such as m3.xlarge .

WeightedCapacity -> (integer)

The number of units that a provisioned instance of this type provides toward fulfilling the target capacities defined in InstanceFleetConfig . This value is 1 for a master instance fleet, and must be 1 or greater for core and task instance fleets. Defaults to 1 if not specified.

BidPrice -> (string)

The bid price for each EC2 Spot Instance type as defined by InstanceType . Expressed in USD. If neither BidPrice nor BidPriceAsPercentageOfOnDemandPrice is provided, BidPriceAsPercentageOfOnDemandPrice defaults to 100%.

BidPriceAsPercentageOfOnDemandPrice -> (double)

The bid price, as a percentage of On-Demand price, for each EC2 Spot Instance as defined by InstanceType . Expressed as a number (for example, 20 specifies 20%). If neither BidPrice nor BidPriceAsPercentageOfOnDemandPrice is provided, BidPriceAsPercentageOfOnDemandPrice defaults to 100%.

EbsConfiguration -> (structure)

The configuration of Amazon Elastic Block Storage (Amazon EBS) attached to each instance as defined by InstanceType .

EbsBlockDeviceConfigs -> (list)

An array of Amazon EBS volume specifications attached to a cluster instance.

(structure)

Configuration of requested EBS block device associated with the instance group with count of volumes that will be associated to every instance.

VolumeSpecification -> (structure)

EBS volume specifications such as volume type, IOPS, and size (GiB) that will be requested for the EBS volume attached to an EC2 instance in the cluster.

VolumeType -> (string)

The volume type. Volume types supported are gp2, io1, standard.

Iops -> (integer)

The number of I/O operations per second (IOPS) that the volume supports.

SizeInGB -> (integer)

The volume size, in gibibytes (GiB). This can be a number from 1 - 1024. If the volume type is EBS-optimized, the minimum value is 10.

VolumesPerInstance -> (integer)

Number of EBS volumes with a specific volume configuration that will be associated with every instance in the instance group

EbsOptimized -> (boolean)

Indicates whether an Amazon EBS volume is EBS-optimized.

Configurations -> (list)

A configuration classification that applies when provisioning cluster instances, which can include configurations for applications and software that run on the cluster.

(structure)

Note

Amazon EMR releases 4.x or later.

An optional configuration specification to be used when provisioning cluster instances, which can include configurations for applications and software bundled with Amazon EMR. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file. For more information, see Configuring Applications .

Classification -> (string)

The classification within a configuration.

Configurations -> (list)

A list of additional configurations to apply within a configuration object.

(structure)

Note

Amazon EMR releases 4.x or later.

An optional configuration specification to be used when provisioning cluster instances, which can include configurations for applications and software bundled with Amazon EMR. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file. For more information, see Configuring Applications .

Classification -> (string)

The classification within a configuration.

Properties -> (map)

A set of properties specified within a configuration classification.

key -> (string)

value -> (string)

Properties -> (map)

A set of properties specified within a configuration classification.

key -> (string)

value -> (string)

LaunchSpecifications -> (structure)

The launch specification for the instance fleet.

SpotSpecification -> (structure)

The launch specification for Spot Instances in the fleet, which determines the defined duration, provisioning timeout behavior, and allocation strategy.

TimeoutDurationMinutes -> (integer)

The spot provisioning timeout period in minutes. If Spot Instances are not provisioned within this time period, the TimeOutAction is taken. Minimum value is 5 and maximum value is 1440. The timeout applies only during initial provisioning, when the cluster is first created.

TimeoutAction -> (string)

The action to take when TargetSpotCapacity has not been fulfilled when the TimeoutDurationMinutes has expired; that is, when all Spot Instances could not be provisioned within the Spot provisioning timeout. Valid values are TERMINATE_CLUSTER and SWITCH_TO_ON_DEMAND . SWITCH_TO_ON_DEMAND specifies that if no Spot Instances are available, On-Demand Instances should be provisioned to fulfill any remaining Spot capacity.

BlockDurationMinutes -> (integer)

The defined duration for Spot Instances (also known as Spot blocks) in minutes. When specified, the Spot Instance does not terminate before the defined duration expires, and defined duration pricing for Spot Instances applies. Valid values are 60, 120, 180, 240, 300, or 360. The duration period starts as soon as a Spot Instance receives its instance ID. At the end of the duration, Amazon EC2 marks the Spot Instance for termination and provides a Spot Instance termination notice, which gives the instance a two-minute warning before it terminates.

AllocationStrategy -> (string)

Specifies the strategy to use in launching Spot Instance fleets. Currently, the only option is capacity-optimized (the default), which launches instances from Spot Instance pools with optimal capacity for the number of instances that are launching.

OnDemandSpecification -> (structure)

The launch specification for On-Demand Instances in the instance fleet, which determines the allocation strategy.

Note

The instance fleet configuration is available only in Amazon EMR versions 4.8.0 and later, excluding 5.0.x versions. On-Demand Instances allocation strategy is available in Amazon EMR version 5.12.1 and later.

AllocationStrategy -> (string)

Specifies the strategy to use in launching On-Demand instance fleets. Currently, the only option is lowest-price (the default), which launches the lowest price first.

CapacityReservationOptions -> (structure)

The launch specification for On-Demand instances in the instance fleet, which determines the allocation strategy.

UsageStrategy -> (string)

Indicates whether to use unused Capacity Reservations for fulfilling On-Demand capacity.

If you specify use-capacity-reservations-first , the fleet uses unused Capacity Reservations to fulfill On-Demand capacity up to the target On-Demand capacity. If multiple instance pools have unused Capacity Reservations, the On-Demand allocation strategy (lowest-price ) is applied. If the number of unused Capacity Reservations is less than the On-Demand target capacity, the remaining On-Demand target capacity is launched according to the On-Demand allocation strategy (lowest-price ).

If you do not specify a value, the fleet fulfils the On-Demand capacity according to the chosen On-Demand allocation strategy.

CapacityReservationPreference -> (string)

Indicates the instance’s Capacity Reservation preferences. Possible preferences include:

open - The instance can run in any open Capacity Reservation that has matching attributes (instance type, platform, Availability Zone).

none - The instance avoids running in a Capacity Reservation even if one is available. The instance runs as an On-Demand Instance.

JSON Syntax:

{
  "Name": "string",
  "InstanceFleetType": "MASTER"|"CORE"|"TASK",
  "TargetOnDemandCapacity": integer,
  "TargetSpotCapacity": integer,
  "InstanceTypeConfigs": [
    {
      "InstanceType": "string",
      "WeightedCapacity": integer,
      "BidPrice": "string",
      "BidPriceAsPercentageOfOnDemandPrice": double,
      "EbsConfiguration": {
        "EbsBlockDeviceConfigs": [
          {
            "VolumeSpecification": {
              "VolumeType": "string",
              "Iops": integer,
              "SizeInGB": integer
            },
            "VolumesPerInstance": integer
          }
          ...
        ],
        "EbsOptimized": true|false
      },
      "Configurations": [
        {
          "Classification": "string",
          "Configurations": [
            {
              "Classification": "string",
              "Configurations": ,
              "Properties": {"string": "string"
                ...}
            }
            ...
          ],
          "Properties": {"string": "string"
            ...}
        }
        ...
      ]
    }
    ...
  ],
  "LaunchSpecifications": {
    "SpotSpecification": {
      "TimeoutDurationMinutes": integer,
      "TimeoutAction": "SWITCH_TO_ON_DEMAND"|"TERMINATE_CLUSTER",
      "BlockDurationMinutes": integer,
      "AllocationStrategy": "capacity-optimized"
    },
    "OnDemandSpecification": {
      "AllocationStrategy": "lowest-price",
      "CapacityReservationOptions": {
        "UsageStrategy": "use-capacity-reservations-first",
        "CapacityReservationPreference": "open"|"none"
      }
    }
  }
}

--cli-input-json | --cli-input-yaml (string) Reads arguments from the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton. If other arguments are provided on the command line, those values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally. This may not be specified along with --cli-input-yaml.

--generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. Similarly, if provided yaml-input it will print a sample input YAML that can be used with --cli-input-yaml. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command.

See ‘aws help’ for descriptions of global parameters.

Examples¶

To add a task instance fleet to a cluster

This example adds a new task instance fleet to the cluster specified.

Command:

aws emr add-instance-fleet --cluster-id 'j-12ABCDEFGHI34JK' --instance-fleet  InstanceFleetType=TASK,TargetSpotCapacity=1,LaunchSpecifications={SpotSpecification='{TimeoutDurationMinutes=20,TimeoutAction=TERMINATE_CLUSTER}'},InstanceTypeConfigs=['{InstanceType=m3.xlarge,BidPrice=0.5}']

Output:

{
   "ClusterId": "j-12ABCDEFGHI34JK",
   "InstanceFleetId": "if-23ABCDEFGHI45JJ"
}

Output¶

ClusterId -> (string)

The unique identifier of the cluster.

InstanceFleetId -> (string)

The unique identifier of the instance fleet.

ClusterArn -> (string)

The Amazon Resource Name of the cluster.

Table of Contents

Feedback

User Guide