Discovery

The discovery resource dynamically creates resources based on a discovery workflow, such as cloud deployments or trouble tickets. This allows Maeztro models to provide real-time observability and management including elements which are not explicitly described in IaC or known beforehand.

Sensors, effectors, and policies can be applied to these resources just as with explicitly represented resources. They can also be partitioned like group resources.

As an example which lists EC2 instances in an autoscaling group, consider Terraform code which defines a resource "aws_autoscaling_group" "my_asg"; Maeztro code might then:

# TODO indicative pseudocode -- needs fleshing out, and ideally an accompanying cookbook

resource discovery "ec2s_in_asg" {
  parent = aws_autoscaling_group.my_asg
  steps [
    "shell aws autoscaling describe-auto-scaling-instances --query 'AutoScalingInstances[?AutoScalingGroupName!=`${parent.name}`]'"
    step "foreach asg_instance in ${jsondecode(stdout).AutoScalingInstances}" [
      "discovery upsert ${asg_instance} as item"
    ]
  ]

  resource "${item.InstanceId}" {
    config {
      instance_id = item.InstanceId
      az = item.AvailabilityZone
    }
    effector "describe-instance" {
      "shell aws describe-instances --instance-id ${self.instance_id}"
      "set-config instance_data = ${jsondecode(stdout).Reservations[0].Instances[0]}"
      "set-sensor launch_time = ${self.instance_data.LaunchTime}"
      "set-sensor public_ip = ${try(self.instance_data.PublicIpAddress,"")}"
      "set-sensor state = ${self.instance_data.State.Name}"
    }
  }
}

Continuing in this way, Maeztro could ensure that monitoring and alerting is automatically set up on all ASG instances.

Syntax

The essential syntax is as follows:

resource discovery "<DISCOVERY_ID>" {
  discovery {
    steps = [
      ...         # required: workflow steps to run to discover external items 
    ]
    
    resource "<ITEM_ID_${item.<field>}>" {
                  # required: the resource block including the unique identifier in terms of the "item"
      ...         # optional: any blocks or attributes applicable to a resource, 
                  # applied to the Maeztro resource corresponding to the discovered item
    }
    
    period = "<DURATION>"        # optional: run periodically
    triggers = [ "<SENSOR>" ]    # optional: run on triggers
    
    partition ... {}             # optional: organize discovered resources into partitions
  }
  
  ...             # optional: anything applicable to a resource, applied to the discovery resource
}

Workflow steps should call the following two steps, as shown in the examples:

• discovery upsert ${item} as <var_name> on each discovered item; typically item is used unless in a nested discovery template (see below)
• discovery complete when all items have been seen on a run, to trigger removal of items no longer present.

The resource block is required and should refer to the variable item specified in the upsert instruction in order to set a unique identifier.

Notes:

• Any of the resource-definition blocks and attributes described here can be used inside the resource blocks
• Discovery will run on full apply/reapply cycle, and according to any period or triggers set, and whenever the runDiscoverySteps effector is invoked
• Partition blocks should include an identifier in terms of ${item} or on variables, as described for groups; they can include apply blocks to apply to discovered resources placed into the partiton
• The variable discovery can be used to reference the discovery resource, and partition can refer to the immeidate containing partition (or the root discovery resource if no partition present)
• The variable item can be referenced in the discovery resource template, but to avoid recording sometimes quite large item records, only static references to the item are permitted; to reference a dynamic subfield, store the parent explicitly as config, and then reference that config
• The identifier of the discovery resource template must be unique inside the corresponding partition, or if partitions are not used, they must be unique across the discovery element; these resources are given an address of the form discovery.<DISCOVERY_ID>["<ITEM_ID>"], or discovery.<DISCOVERY_ID>["<PARTITION_ID>"]["<ITEM_ID>"] if partitions are used

Advanced: Nested Discovery

Maeztro supports discovery creating additional discovery resources: for example the autoscaling groups themselves could be discovered, instantiated as Maeztro discovery resources, and then each of these acts like the previous example to discover the VMs within the ASG each respresents.

The things to note in this case are that the resource should include a type discovery before the identifier, and it should then declare a discovery block as usual, except in the steps block the discovery upsert ${item} as <var_name> must to specify a different var_name. The reason for this is that item will be expanded at resource creation time everywhere it is used as a variable within the outer discovery -> resource block. Often subitem is used, as below:

resource discovery "nested" {
  discovery {
    steps [
      "let list<map> items = [ { id: '001', name: 'One' }, { id: '002', name: 'Two' }, { id: '004', name: 'Four' } ]"
      "foreach item in ${items} do discovery upsert ${item} as item"
      "discovery complete"
    ]

    resource discovery "d-${item.id}" {
      config {
        name = item.name
      }
      discovery {
        steps [
          "let list<map> items = [ { id: '001', name: 'One' }, { id: '002', name: 'Two' }, { id: '004', name: 'Four' } ]"
          "foreach subitem in ${items} do discovery upsert ${subitem} as subitem"
          "discovery complete"
        ]
        resource "sub-${subitem.id}" {
        }
      }
    }
  }
}