Recently, Everyday Hero has been doing a heap of work around the automation and provisioning of resources with AWS cloud services. This entails working frequently with the AWS API. A useful filter we have been using in Ansible is taking output from a shell action and turning it into something we can consume via variables. An example playbook is below: ``` - shell: | export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY lib/ec2.py --list --refresh-cache register: output - set_fact: ec2_output: "{{ output.stdout|from_json }}" - shell: "rm addresses.txt" - shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt" when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}" with_dict: ec2_output._meta.hostvars - shell: "cat addresses.txt" register: output - add_host: hostname: "{{ output.stdout_lines.0 }}" groupname: "deis_node" - debug: var={{ groups['deis_node'].0 }} ``` This example uses an Ansible plugin script ec2.py to retrieve a list of instances running in an AWS account. It then filters out the ones that we want via the `ec2_tag_Name` key and adds the private IP address to a file. After running the following shell action, `output` holds a heap of information about what the commands did and the output from stdout and stderr (if there was any.) ``` - shell: | export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY lib/ec2.py --list --refresh-cache register: output ``` The Ansible output for the registered variable output looks something like this: ``` TASK: [debug var=output] ****************************************************** ok: [localhost] => { "output": { "changed": true, "cmd": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache", "delta": "0:00:05.268682", "end": "2014-10-05 11:43:05.235594", "invocation": { "module_args": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache", "module_name": "shell" }, "rc": 0, "start": "2013-10-05 11:42:59.966912", "stderr": "", "stdout": "", "stdout_lines": [ "{", " \"_meta\": {", " \"hostvars\": {", " \"xx.xxx.xxx.xxx\": {", " \"ec2__in_monitoring_element\": false, ", " \"ec2_ami_launch_index\": \"0\", ", " \"ec2_architecture\": \"x86_64\", ", " \"ec2_client_token\": \"xxxxxxxx-xxxx-xxxx-xxxx-c55977fc0029_us-east-1a_1\", ", " \"ec2_dns_name\": \"ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com\", ", " \"ec2_ebs_optimized\": false, ", " \"ec2_eventsSet\": \"\", ", " \"ec2_group_name\": \"\", ", " \"ec2_hypervisor\": \"xen\", ", " \"ec2_id\": \"i-xxxxxx\", ", " \"ec2_image_id\": \"ami-xxxxxx\", ", " \"ec2_instance_profile\": \"\", ", " \"ec2_instance_type\": \"m1.large\", ", " \"ec2_ip_address\": \"xx.xxx.xxx.xxx\", ", " \"ec2_item\": \"\", ", " \"ec2_kernel\": \"aki-xxxxxxxx\", ", " \"ec2_key_name\": \"ambari\", ", " \"ec2_launch_time\": \"2013-10-01T04:53:00.000Z\", ", " \"ec2_monitored\": true, ", " \"ec2_monitoring\": \"\", ", " \"ec2_monitoring_state\": \"enabled\", ", " \"ec2_persistent\": false, ", " \"ec2_placement\": \"us-east-1a\", ", " \"ec2_platform\": \"\", ", " \"ec2_previous_state\": \"\", ", " \"ec2_previous_state_code\": 0, ", " \"ec2_private_dns_name\": \"ip-xx-xx-x-xxx.ec2.internal\", ", " \"ec2_private_ip_address\": \"xx.xx.x.xxx\", ", " \"ec2_public_dns_name\": \"ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com\", ", " \"ec2_ramdisk\": \"\", ", " \"ec2_reason\": \"\", ", " \"ec2_region\": \"us-east-1\", ", " \"ec2_requester_id\": \"\", ", " \"ec2_root_device_name\": \"/dev/sda1\", ", " \"ec2_root_device_type\": \"ebs\", ", " \"ec2_security_group_ids\": \"sg-xxxxxxxx\", ", " \"ec2_security_group_names\": \"ambari\", ", " \"ec2_sourceDestCheck\": \"true\", ", " \"ec2_spot_instance_request_id\": \"\", ", " \"ec2_state\": \"running\", ", " \"ec2_state_code\": 16, ", " \"ec2_state_reason\": \"\", ", " \"ec2_subnet_id\": \"subnet-xxxxxxxx\", ", " \"ec2_tag_Name\": \"hdpmaster1\", ", " \"ec2_tag_aws_autoscaling_groupName\": \"ambari-LargeClusterGroup-148W3OVR9LSSE\", ", " \"ec2_tag_aws_cloudformation_logical-id\": \"LargeClusterGroup\", ", " \"ec2_tag_aws_cloudformation_stack-id\": \"arn:aws:cloudformation:us-east-1:167609138788:stack/ambari/xxxxxxxx-xxxx-xxxx-xxxx-50fa526be49c\", ", " \"ec2_tag_aws_cloudformation_stack-name\": \"ambari\", ", " \"ec2_tag_long_hostname\": \"hdpmaster1.domain.com\", ", " \"ec2_virtualization_type\": \"paravirtual\", ", " \"ec2_vpc_id\": \"vpc-xxxxxxxx\"", " }, ", "}" ] } } ``` As you can see stdout and stdout_lines aren't in a great format to do much with, here enters from_json: ``` - set_fact: ec2_output: "{{ output.stdout|from_json }}" ``` This then allows us to do the following and `ec2_output` is now in a standard JSON format that Ansible can then use to iterate over like so: ``` - shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt" when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}" with_dict: ec2_output._meta.hostvars ``` This Ansible task creates a file that contains all of the private IP addresses of ec2 instances that match the `ec2_tag_Name` criteria. We could then take the first of these IP addresses and use it later in our playbook: ``` - shell: "cat addresses.txt" register: output - debug: var={{ output.stdout_lines.0 }} ``` This becomes a very powerful pattern when you mix in RESTful APIs, calls to them and processing output from them.