Skip to content

Instantly share code, notes, and snippets.

@mkoertgen
Last active June 3, 2021 16:40
Show Gist options
  • Select an option

  • Save mkoertgen/b39bc813f9c4a27056f9e5742d4ea1ab to your computer and use it in GitHub Desktop.

Select an option

Save mkoertgen/b39bc813f9c4a27056f9e5742d4ea1ab to your computer and use it in GitHub Desktop.

Revisions

  1. mkoertgen revised this gist Jun 3, 2021. 4 changed files with 30 additions and 94 deletions.
    27 changes: 3 additions & 24 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -35,31 +35,10 @@ GET _alias/.kibana_*
    Say you have a terraform module for provisioning Index Patterns, Dashboards & Visualizations for an Open Distro Cluster
    [elastic.tf](./elastic.tf)
    ```terraform
    # file: elastic.tf
    # Create a tenant (similar to a Kibana space), cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/opendistro_kibana_tenant.md
    # Create a tenant (similar to a Kibana space)
    resource "elasticsearch_opendistro_kibana_tenant" "monitoring" {
    tenant_name = "monitoring"
    description = "Monitoring Tenant"
    }
    # Provision Kibana Objects, cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/kibana_object.md
    # NOTE: The current version of the provider can only provision a single object per resource, cf.:
    # - (https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/es/resource_elasticsearch_kibana_object.go#L196)
    resource "elasticsearch_kibana_object" "kibana" {
    for_each = fileset("${path.module}/templates/kibana", "*.json")
    body = templatefile("${path.module}/templates/kibana/${each.key}", local.template_context)
    # index = ".kibana" # default
    }
    ```
    After successfully applying it the resources will be only visible in the global tenant and not in the private tenants `admin` or `monitoring`.
    Until this is possible you may use [kibana_alias.tf](./kibana_alias.tf) and [kibana_alias.py](./kibana_alias.py) as a workaround.
    Until this is possible you may use [kibana_alias.py](./kibana_alias.py) as a workaround.
    See example usage in [elastic.tf](./elastic.tf)
    ## References
    15 changes: 7 additions & 8 deletions elastic.tf
    Original file line number Diff line number Diff line change
    @@ -5,18 +5,17 @@ resource "elasticsearch_opendistro_kibana_tenant" "monitoring" {
    description = "Monitoring Tenant"
    }

    locals {
    tenant = elasticsearch_opendistro_kibana_tenant.monitoring.tenant_name
    kibana_alias = data.external.kibana_alias.result.alias
    # empty for now. Nothing to interpolate
    template_context = {
    }
    # Workaround for looking up the kibana index-alias for an Open Distro Tenant, cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140
    # https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source
    data "external" "kibana_alias" {
    program = ["python", "${path.module}/kibana_alias.py", elasticsearch_opendistro_kibana_tenant.monitoring.tenant_name ]
    }

    # Provision Kibana Objects, cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/kibana_object.md
    resource "elasticsearch_kibana_object" "kibana" {
    for_each = fileset("${path.module}/templates/kibana", "*.json")
    body = templatefile("${path.module}/templates/kibana/${each.key}", local.template_context)
    index = local.kibana_alias
    body = templatefile("${path.module}/templates/kibana/${each.key}", {})
    index = data.external.kibana_alias.result.alias
    }
    55 changes: 20 additions & 35 deletions kibana_alias.py
    Original file line number Diff line number Diff line change
    @@ -1,41 +1,26 @@
    #!/usr/bin/env python3
    # coding: utf-8
    #!/bin/env python
    # -*- coding: utf-8 -*-

    import sys
    import json
    import requests
    # - https://raw.githubusercontent.com/jcantrill/cluster-logging-tools/master/scripts/kibana-index-name
    '''
    OD toUserIndexName()
    originalKibanaIndex+"_"+tenant.hashCode()+"_"+tenant.toLowerCase().replaceAll("[^a-z0-9]+",EMPTY_STRING);
    '''

    import urllib3
    import sys,re,json

    """Gets the Open Distro Kibana Alias for the specified tenant
    username = sys.argv[1]

    This is a helper script to work around current limitations of the terraform elasticsearch provider related to Open Distro tenants, cf.:
    def java_string_hashcode(s):
    """https://stackoverflow.com/questions/22845913/function-to-replicate-the-output-of-java-lang-string-hashcode-in-python-and-no"""
    """Mimic Java's hashCode in python 2"""
    h = 0
    for c in list(s.encode('utf8')):
    v = ord(c) if isinstance(c, str) else c
    h = int((((31 * h + v) ^ 0x80000000) & 0xFFFFFFFF) - 0x80000000)
    return h

    - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/169
    - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140
    """
    hashCode = java_string_hashcode(username)
    alias = ".kibana_" + str(hashCode) + "_" + re.sub("[^a-z0-9]","",username.lower())

    # Disable warnings about unsecure SSL polluting stdout
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

    def fetch():
    input_json = sys.stdin.read() or '{}'
    try:
    input_dict = json.loads(input_json)
    url = input_dict.get('url', 'https://localhost:9200')
    user = input_dict.get('user', 'admin')
    pwd = input_dict.get('pwd', 'admin')
    tenant = input_dict.get('tenant', 'monitoring')

    response = requests.get(f'{url}/_cat/aliases/.kibana_*_{tenant}?format=json', auth=requests.auth.HTTPBasicAuth(username=user, password=pwd), verify=False)
    output_json = response.json()

    alias = output_json[0].get("alias") # Take first matching alias (should be unique)
    output = json.dumps({'alias':alias})

    sys.stdout.write(output)
    except ValueError as e:
    sys.exit(e)

    if __name__ == "__main__":
    fetch()
    print(json.dumps({"alias": alias}))
    27 changes: 0 additions & 27 deletions kibana_alias.tf
    Original file line number Diff line number Diff line change
    @@ -1,27 +0,0 @@
    #-----------
    # Workaround for looking up the kibana index-alias for an Open Distro Tenant, cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140
    # $ curl -k -u "admin:admin" https://localhost:9200/_cat/aliases/.kibana_*_monitoring?format=json | jq .[0].alias
    variable "elasticsearch_url" {
    type = string
    }

    variable "elasticsearch_username" {
    type = string
    }

    variable "elasticsearch_password" {
    type = string
    sensitive = true
    }

    # https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source
    data "external" "kibana_alias" {
    program = ["python", "${path.module}/kibana_alias.py"]
    query = {
    url = var.elasticsearch_url
    user = var.elasticsearch_username
    pwd = var.elasticsearch_password
    tenant = local.tenant
    }
    }
  2. mkoertgen revised this gist Jun 3, 2021. 4 changed files with 123 additions and 4 deletions.
    37 changes: 33 additions & 4 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -1,10 +1,9 @@
    # Lookup Kibana Alias for Open Distro Tenant

    The "Open Distro"-fork of Elasticsearch uses [Tenants](https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/multi-tenancy/) instead of [Kibana Spaces](https://www.elastic.co/guide/en/kibana/master/xpack-spaces.html)
    The "Open Distro"-fork of Elasticsearch uses [Tenants](https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/multi-tenancy/) instead of [Kibana Spaces](https://www.elastic.co/guide/en/kibana/master/xpack-spaces.html)
    where each tenant corresponds to a separate Kibana Index / Alias.

    When provisioning Kibana Objects on an "Open Distro"-cluster using the [Phil Bakers Terraform Elasticsearch community provider](https://registry.terraform.io/providers/phillbaker/elasticsearch/latest)
    it will by default create resources in `.kibana`-index which corresponds to the global tenant.
    When provisioning Kibana Objects on an "Open Distro"-cluster using the [Phil Bakers Terraform Elasticsearch community provider](https://registry.terraform.io/providers/phillbaker/elasticsearch/latest) it will by default create resources in `.kibana`-index which corresponds to the global tenant.

    For example after spinning up a single node cluster the different Kibana indexes may look like

    @@ -32,7 +31,37 @@ GET _alias/.kibana_*
    }
    ```
    ## Usage Example
    Say you have a terraform module for provisioning Index Patterns, Dashboards & Visualizations for an Open Distro Cluster
    [elastic.tf](./elastic.tf)
    ```terraform
    # file: elastic.tf
    # Create a tenant (similar to a Kibana space), cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/opendistro_kibana_tenant.md
    # Create a tenant (similar to a Kibana space)
    resource "elasticsearch_opendistro_kibana_tenant" "monitoring" {
    tenant_name = "monitoring"
    description = "Monitoring Tenant"
    }
    # Provision Kibana Objects, cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/kibana_object.md
    # NOTE: The current version of the provider can only provision a single object per resource, cf.:
    # - (https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/es/resource_elasticsearch_kibana_object.go#L196)
    resource "elasticsearch_kibana_object" "kibana" {
    for_each = fileset("${path.module}/templates/kibana", "*.json")
    body = templatefile("${path.module}/templates/kibana/${each.key}", local.template_context)
    # index = ".kibana" # default
    }
    ```
    After successfully applying it the resources will be only visible in the global tenant and not in the private tenants `admin` or `monitoring`.
    Until this is possible you may use [kibana_alias.tf](./kibana_alias.tf) and [kibana_alias.py](./kibana_alias.py) as a workaround.
    ## References
    - GitHub Issue: [Failed to create index patterns via elasticsearch_kibana_object resource](https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140)
    - Terraform: [External Data Source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source)
    - Terraform: [External Data Source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source)
    22 changes: 22 additions & 0 deletions elastic.tf
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,22 @@
    # Create a tenant (similar to a Kibana space), cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/opendistro_kibana_tenant.md
    resource "elasticsearch_opendistro_kibana_tenant" "monitoring" {
    tenant_name = "monitoring"
    description = "Monitoring Tenant"
    }

    locals {
    tenant = elasticsearch_opendistro_kibana_tenant.monitoring.tenant_name
    kibana_alias = data.external.kibana_alias.result.alias
    # empty for now. Nothing to interpolate
    template_context = {
    }
    }

    # Provision Kibana Objects, cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/kibana_object.md
    resource "elasticsearch_kibana_object" "kibana" {
    for_each = fileset("${path.module}/templates/kibana", "*.json")
    body = templatefile("${path.module}/templates/kibana/${each.key}", local.template_context)
    index = local.kibana_alias
    }
    41 changes: 41 additions & 0 deletions kibana_alias.py
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,41 @@
    #!/usr/bin/env python3
    # coding: utf-8

    import sys
    import json
    import requests

    import urllib3

    """Gets the Open Distro Kibana Alias for the specified tenant
    This is a helper script to work around current limitations of the terraform elasticsearch provider related to Open Distro tenants, cf.:
    - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/169
    - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140
    """

    # Disable warnings about unsecure SSL polluting stdout
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

    def fetch():
    input_json = sys.stdin.read() or '{}'
    try:
    input_dict = json.loads(input_json)
    url = input_dict.get('url', 'https://localhost:9200')
    user = input_dict.get('user', 'admin')
    pwd = input_dict.get('pwd', 'admin')
    tenant = input_dict.get('tenant', 'monitoring')

    response = requests.get(f'{url}/_cat/aliases/.kibana_*_{tenant}?format=json', auth=requests.auth.HTTPBasicAuth(username=user, password=pwd), verify=False)
    output_json = response.json()

    alias = output_json[0].get("alias") # Take first matching alias (should be unique)
    output = json.dumps({'alias':alias})

    sys.stdout.write(output)
    except ValueError as e:
    sys.exit(e)

    if __name__ == "__main__":
    fetch()
    27 changes: 27 additions & 0 deletions kibana_alias.tf
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,27 @@
    #-----------
    # Workaround for looking up the kibana index-alias for an Open Distro Tenant, cf.:
    # - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140
    # $ curl -k -u "admin:admin" https://localhost:9200/_cat/aliases/.kibana_*_monitoring?format=json | jq .[0].alias
    variable "elasticsearch_url" {
    type = string
    }

    variable "elasticsearch_username" {
    type = string
    }

    variable "elasticsearch_password" {
    type = string
    sensitive = true
    }

    # https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source
    data "external" "kibana_alias" {
    program = ["python", "${path.module}/kibana_alias.py"]
    query = {
    url = var.elasticsearch_url
    user = var.elasticsearch_username
    pwd = var.elasticsearch_password
    tenant = local.tenant
    }
    }
  3. mkoertgen created this gist Jun 3, 2021.
    38 changes: 38 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,38 @@
    # Lookup Kibana Alias for Open Distro Tenant

    The "Open Distro"-fork of Elasticsearch uses [Tenants](https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/multi-tenancy/) instead of [Kibana Spaces](https://www.elastic.co/guide/en/kibana/master/xpack-spaces.html)
    where each tenant corresponds to a separate Kibana Index / Alias.

    When provisioning Kibana Objects on an "Open Distro"-cluster using the [Phil Bakers Terraform Elasticsearch community provider](https://registry.terraform.io/providers/phillbaker/elasticsearch/latest)
    it will by default create resources in `.kibana`-index which corresponds to the global tenant.

    For example after spinning up a single node cluster the different Kibana indexes may look like

    ```shell
    GET _alias/.kibana_*
    {
    // The private admin tenant
    ".kibana_92668751_admin_1" : {
    "aliases" : {
    ".kibana_92668751_admin" : { }
    }
    },
    // The global tenant'
    ".kibana_1" : {
    "aliases" : {
    ".kibana" : { }
    }
    }
    // Some custom private tenant 'monitoring'
    ".kibana_1852089416_monitoring_1" : {
    "aliases" : {
    ".kibana_1852089416_monitoring" : { }
    }
    }
    }
    ```
    ## References
    - GitHub Issue: [Failed to create index patterns via elasticsearch_kibana_object resource](https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140)
    - Terraform: [External Data Source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source)