Skip to content

Instantly share code, notes, and snippets.

@nerdalert

nerdalert/e2e-kuadrant-metrics.md

Last active August 20, 2025 07:03
Show Gist options
  • Save nerdalert/0df4874cdc74c8f676686ce77f352f7b to your computer and use it in GitHub Desktop.
Save nerdalert/0df4874cdc74c8f676686ce77f352f7b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
e2e-kuadrant-metrics.md

Instructions for enabling and disabling metrics in a running Kuadrant OLM deployment

Load the patched Kuadrant operator:

kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/image", "value": "ghcr.io/nerdalert/kuadrant-operator:metrics-test"}]'
    
kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager

# Verify the operator image
kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}'

Load the patched wasm-shim image

kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
     -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/1/value", "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"}]'

kubectl -n llm rollout restart deployment/inference-gateway-istio

# Verify the wasm image
kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager \
     -o jsonpath='{.spec.template.spec.containers[0].env[1]}' | jq

Enable Metrics

  1. Update CSV to set KUADRANT_METRICS_ENABLED=true

Add the env:

kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' -p='[
    {
      "op": "add",
      "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/-",
      "value": {
        "name": "KUADRANT_METRICS_ENABLED",
        "value": "true"
      }
    }
  ]'

Or update the env if it already exists:

   kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
     -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "true"}]'

  1. Restart operator

    kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager

  2. Verify environment variable

    kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq

  3. Restart gateway and verify metrics initialization

    kubectl -n llm rollout restart deployment/inference-gateway-istio
kubectl -n llm delete pod -l app=inference-gateway-istio -n llm

  4. Test metrics collection Make request:

    curl -H 'Authorization: APIKEY <INSERT_KEY>' \
     -H 'Content-Type: application/json' \
     -d '{"model": "simulator-model", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10}' \
     http://<ENDPOINT_ADDRESS>/v1/chat/completions

  5. Check token metrics

    kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token

Disable Metrics

Set the ENV in the running operator:

Disable Metrics via OLM

  1. Update CSV to set KUADRANT_METRICS_ENABLED=false

    kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
  -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "false"}]'

  2. Restart operator

    kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager

  3. Verify environment variable

    kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq

  4. Restart gateway and verify no metrics initialization (kick the pod as needed)

    kubectl -n llm rollout restart deployment/inference-gateway-istio
kubectl -n llm delete pod -l app=inference-gateway-istio -n llm

  5. Test metrics collection Make request:

    curl -H 'Authorization: APIKEY <INSERT_KEY>' \
     -H 'Content-Type: application/json' \
     -d '{"model": "simulator-model", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10}' \
     http://simulator-llm.apps.summit-gpu.octo-emerging.redhataicoe.com/v1/chat/completions

  6. Make sure there are no metrics

    kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token

Raw Output

Enable metrics output

🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "true"}]'
clusterserviceversion.operators.coreos.com/kuadrant-operator.v0.0.0 patched
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
deployment.apps/kuadrant-operator-controller-manager restarted
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env[3]}' | jq
{
  "name": "KUADRANT_METRICS_ENABLED",
  "value": "true"
}
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$   kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
[
  {
    "name": "WITH_EXTENSIONS",
    "value": "true"
  },
  {
    "name": "RELATED_IMAGE_WASMSHIM",
    "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
  },
  {
    "name": "OPERATOR_NAMESPACE",
    "valueFrom": {
      "fieldRef": {
        "fieldPath": "metadata.namespace"
      }
    }
  },
  {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "true"
  }
]
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager \
     -o jsonpath='{.spec.template.spec.containers[0].env[1]}' | jq
{
  "name": "RELATED_IMAGE_WASMSHIM",
  "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
}
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$   kubectl -n llm rollout restart deployment/inference-gateway-istio
  kubectl -n llm delete pod -l app=inference-gateway-istio -n llm
deployment.apps/inference-gateway-istio restarted
No resources found
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl get pods     # --all-namespaces
NAME                                            READY   STATUS      RESTARTS      AGE
inference-gateway-istio-97846c4c9-djhvh         0/1     Completed   0             22m
inference-gateway-istio-cd9967644-h55px         1/1     Running     0             10s
qwen3-0-6b-instruct-predictor-5b96f458d-d5hm4   1/1     Running     0             9d
vllm-simulator-predictor-6d748c55b5-dtc66       1/1     Running     5 (22h ago)   9d

# GENERATED REQUESTS HERE

🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
# TYPE token_usage_total counter
token_usage_total{} 1140
# TYPE token_usage_with_user_and_group__user___freeuser1___group___free___namespace__llm counter
token_usage_with_user_and_group__user___freeuser1___group___free___namespace__llm{} 750
# TYPE token_usage_with_user_and_group__user___premiumuser1___group___premium___namespace__llm counter
token_usage_with_user_and_group__user___premiumuser1___group___premium___namespace__llm{} 390

Disable metrics output

$ kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "false"}]'
clusterserviceversion.operators.coreos.com/kuadrant-operator.v0.0.0 patched
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
deployment.apps/kuadrant-operator-controller-manager restarted
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
[
  {
    "name": "WITH_EXTENSIONS",
    "value": "true"
  },
  {
    "name": "RELATED_IMAGE_WASMSHIM",
    "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
  },
  {
    "name": "OPERATOR_NAMESPACE",
    "valueFrom": {
      "fieldRef": {
        "fieldPath": "metadata.namespace"
      }
    }
  },
  {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "false"
  }
]

🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm rollout restart deployment/inference-gateway-istio
deployment.apps/inference-gateway-istio restarted

# GENERATED REQUESTS HERE

🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token

# NO METRICS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment