Last active
June 3, 2021 16:40
-
-
Save mkoertgen/b39bc813f9c4a27056f9e5742d4ea1ab to your computer and use it in GitHub Desktop.
Revisions
-
mkoertgen revised this gist
Jun 3, 2021 . 4 changed files with 30 additions and 94 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -35,31 +35,10 @@ GET _alias/.kibana_* Say you have a terraform module for provisioning Index Patterns, Dashboards & Visualizations for an Open Distro Cluster After successfully applying it the resources will be only visible in the global tenant and not in the private tenants `admin` or `monitoring`. Until this is possible you may use [kibana_alias.py](./kibana_alias.py) as a workaround. See example usage in [elastic.tf](./elastic.tf) ## References This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -5,18 +5,17 @@ resource "elasticsearch_opendistro_kibana_tenant" "monitoring" { description = "Monitoring Tenant" } # Workaround for looking up the kibana index-alias for an Open Distro Tenant, cf.: # - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140 # https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source data "external" "kibana_alias" { program = ["python", "${path.module}/kibana_alias.py", elasticsearch_opendistro_kibana_tenant.monitoring.tenant_name ] } # Provision Kibana Objects, cf.: # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/kibana_object.md resource "elasticsearch_kibana_object" "kibana" { for_each = fileset("${path.module}/templates/kibana", "*.json") body = templatefile("${path.module}/templates/kibana/${each.key}", {}) index = data.external.kibana_alias.result.alias } This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,41 +1,26 @@ #!/bin/env python # -*- coding: utf-8 -*- # - https://raw.githubusercontent.com/jcantrill/cluster-logging-tools/master/scripts/kibana-index-name ''' OD toUserIndexName() originalKibanaIndex+"_"+tenant.hashCode()+"_"+tenant.toLowerCase().replaceAll("[^a-z0-9]+",EMPTY_STRING); ''' import sys,re,json username = sys.argv[1] def java_string_hashcode(s): """https://stackoverflow.com/questions/22845913/function-to-replicate-the-output-of-java-lang-string-hashcode-in-python-and-no""" """Mimic Java's hashCode in python 2""" h = 0 for c in list(s.encode('utf8')): v = ord(c) if isinstance(c, str) else c h = int((((31 * h + v) ^ 0x80000000) & 0xFFFFFFFF) - 0x80000000) return h hashCode = java_string_hashcode(username) alias = ".kibana_" + str(hashCode) + "_" + re.sub("[^a-z0-9]","",username.lower()) print(json.dumps({"alias": alias})) This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,27 +0,0 @@ -
mkoertgen revised this gist
Jun 3, 2021 . 4 changed files with 123 additions and 4 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,10 +1,9 @@ # Lookup Kibana Alias for Open Distro Tenant The "Open Distro"-fork of Elasticsearch uses [Tenants](https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/multi-tenancy/) instead of [Kibana Spaces](https://www.elastic.co/guide/en/kibana/master/xpack-spaces.html) where each tenant corresponds to a separate Kibana Index / Alias. When provisioning Kibana Objects on an "Open Distro"-cluster using the [Phil Bakers Terraform Elasticsearch community provider](https://registry.terraform.io/providers/phillbaker/elasticsearch/latest) it will by default create resources in `.kibana`-index which corresponds to the global tenant. For example after spinning up a single node cluster the different Kibana indexes may look like @@ -32,7 +31,37 @@ GET _alias/.kibana_* } ``` ## Usage Example Say you have a terraform module for provisioning Index Patterns, Dashboards & Visualizations for an Open Distro Cluster [elastic.tf](./elastic.tf) ```terraform # file: elastic.tf # Create a tenant (similar to a Kibana space), cf.: # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/opendistro_kibana_tenant.md # Create a tenant (similar to a Kibana space) resource "elasticsearch_opendistro_kibana_tenant" "monitoring" { tenant_name = "monitoring" description = "Monitoring Tenant" } # Provision Kibana Objects, cf.: # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/kibana_object.md # NOTE: The current version of the provider can only provision a single object per resource, cf.: # - (https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/es/resource_elasticsearch_kibana_object.go#L196) resource "elasticsearch_kibana_object" "kibana" { for_each = fileset("${path.module}/templates/kibana", "*.json") body = templatefile("${path.module}/templates/kibana/${each.key}", local.template_context) # index = ".kibana" # default } ``` After successfully applying it the resources will be only visible in the global tenant and not in the private tenants `admin` or `monitoring`. Until this is possible you may use [kibana_alias.tf](./kibana_alias.tf) and [kibana_alias.py](./kibana_alias.py) as a workaround. ## References - GitHub Issue: [Failed to create index patterns via elasticsearch_kibana_object resource](https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140) - Terraform: [External Data Source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source) This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,22 @@ # Create a tenant (similar to a Kibana space), cf.: # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/opendistro_kibana_tenant.md resource "elasticsearch_opendistro_kibana_tenant" "monitoring" { tenant_name = "monitoring" description = "Monitoring Tenant" } locals { tenant = elasticsearch_opendistro_kibana_tenant.monitoring.tenant_name kibana_alias = data.external.kibana_alias.result.alias # empty for now. Nothing to interpolate template_context = { } } # Provision Kibana Objects, cf.: # - https://github.com/phillbaker/terraform-provider-elasticsearch/blob/master/docs/resources/kibana_object.md resource "elasticsearch_kibana_object" "kibana" { for_each = fileset("${path.module}/templates/kibana", "*.json") body = templatefile("${path.module}/templates/kibana/${each.key}", local.template_context) index = local.kibana_alias } This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,41 @@ #!/usr/bin/env python3 # coding: utf-8 import sys import json import requests import urllib3 """Gets the Open Distro Kibana Alias for the specified tenant This is a helper script to work around current limitations of the terraform elasticsearch provider related to Open Distro tenants, cf.: - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/169 - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140 """ # Disable warnings about unsecure SSL polluting stdout urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) def fetch(): input_json = sys.stdin.read() or '{}' try: input_dict = json.loads(input_json) url = input_dict.get('url', 'https://localhost:9200') user = input_dict.get('user', 'admin') pwd = input_dict.get('pwd', 'admin') tenant = input_dict.get('tenant', 'monitoring') response = requests.get(f'{url}/_cat/aliases/.kibana_*_{tenant}?format=json', auth=requests.auth.HTTPBasicAuth(username=user, password=pwd), verify=False) output_json = response.json() alias = output_json[0].get("alias") # Take first matching alias (should be unique) output = json.dumps({'alias':alias}) sys.stdout.write(output) except ValueError as e: sys.exit(e) if __name__ == "__main__": fetch() This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,27 @@ #----------- # Workaround for looking up the kibana index-alias for an Open Distro Tenant, cf.: # - https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140 # $ curl -k -u "admin:admin" https://localhost:9200/_cat/aliases/.kibana_*_monitoring?format=json | jq .[0].alias variable "elasticsearch_url" { type = string } variable "elasticsearch_username" { type = string } variable "elasticsearch_password" { type = string sensitive = true } # https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source data "external" "kibana_alias" { program = ["python", "${path.module}/kibana_alias.py"] query = { url = var.elasticsearch_url user = var.elasticsearch_username pwd = var.elasticsearch_password tenant = local.tenant } } -
mkoertgen created this gist
Jun 3, 2021 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,38 @@ # Lookup Kibana Alias for Open Distro Tenant The "Open Distro"-fork of Elasticsearch uses [Tenants](https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/multi-tenancy/) instead of [Kibana Spaces](https://www.elastic.co/guide/en/kibana/master/xpack-spaces.html) where each tenant corresponds to a separate Kibana Index / Alias. When provisioning Kibana Objects on an "Open Distro"-cluster using the [Phil Bakers Terraform Elasticsearch community provider](https://registry.terraform.io/providers/phillbaker/elasticsearch/latest) it will by default create resources in `.kibana`-index which corresponds to the global tenant. For example after spinning up a single node cluster the different Kibana indexes may look like ```shell GET _alias/.kibana_* { // The private admin tenant ".kibana_92668751_admin_1" : { "aliases" : { ".kibana_92668751_admin" : { } } }, // The global tenant' ".kibana_1" : { "aliases" : { ".kibana" : { } } } // Some custom private tenant 'monitoring' ".kibana_1852089416_monitoring_1" : { "aliases" : { ".kibana_1852089416_monitoring" : { } } } } ``` ## References - GitHub Issue: [Failed to create index patterns via elasticsearch_kibana_object resource](https://github.com/phillbaker/terraform-provider-elasticsearch/issues/140) - Terraform: [External Data Source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source)