Skip to content

Instantly share code, notes, and snippets.

@nerdalert

nerdalert/e2e-kuadrant-metrics.md

Last active August 20, 2025 07:03
Show Gist options
  • Save nerdalert/0df4874cdc74c8f676686ce77f352f7b to your computer and use it in GitHub Desktop.
Save nerdalert/0df4874cdc74c8f676686ce77f352f7b to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. nerdalert revised this gist Aug 20, 2025. 1 changed file with 44 additions and 1 deletion.
    45 changes: 44 additions & 1 deletion e2e-kuadrant-metrics.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,9 +4,52 @@
    - **[Kuadrant/kuadrant-operator patch (PR not opened yet)](https://github.com/Kuadrant/kuadrant-operator/compare/main...nerdalert:kuadrant-operator:metrics-enable-env?expand=1)**
    - **[Kuadrant/wasm-shim patch](https://github.com/Kuadrant/wasm-shim/pull/208)**

    ### Load the patched Kuadrant operator:

    ```bash
    kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/image", "value": "ghcr.io/nerdalert/kuadrant-operator:metrics-test"}]'

    kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager

    # Verify the operator image
    kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}'
    ```

    ### Load the patched wasm-shim image

    ```
    kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/1/value", "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"}]'
    kubectl -n llm rollout restart deployment/inference-gateway-istio
    # Verify the wasm image
    kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager \
    -o jsonpath='{.spec.template.spec.containers[0].env[1]}' | jq
    ```

    ### Enable Metrics

    1. Update CSV to set `KUADRANT_METRICS_ENABLED=true`

    Add the env:

    ```bash
    kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' -p='[
    {
    "op": "add",
    "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/-",
    "value": {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "true"
    }
    }
    ]'
    ```

    Or update the env if it already exists:

    ```bash
    kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "true"}]'
    @@ -38,7 +81,7 @@
    curl -H 'Authorization: APIKEY <INSERT_KEY>' \
    -H 'Content-Type: application/json' \
    -d '{"model": "simulator-model", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10}' \
    http://simulator-llm.apps.summit-gpu.octo-emerging.redhataicoe.com/v1/chat/completions
    http://<ENDPOINT_ADDRESS>/v1/chat/completions
    ```

    6. Check token metrics
  2. nerdalert revised this gist Aug 19, 2025. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion e2e-kuadrant-metrics.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,8 @@

    #### Instructions for enabling and disabling metrics in a running Kuadrant OLM deployment ########
    ### Instructions for enabling and disabling metrics in a running Kuadrant OLM deployment

    - **[Kuadrant/kuadrant-operator patch (PR not opened yet)](https://github.com/Kuadrant/kuadrant-operator/compare/main...nerdalert:kuadrant-operator:metrics-enable-env?expand=1)**
    - **[Kuadrant/wasm-shim patch](https://github.com/Kuadrant/wasm-shim/pull/208)**

    ### Enable Metrics

  3. nerdalert created this gist Aug 19, 2025.
    203 changes: 203 additions & 0 deletions e2e-kuadrant-metrics.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,203 @@

    #### Instructions for enabling and disabling metrics in a running Kuadrant OLM deployment ########

    ### Enable Metrics

    1. Update CSV to set `KUADRANT_METRICS_ENABLED=true`
    ```bash
    kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "true"}]'
    ```

    2. Restart operator

    ```bash
    kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
    ```

    3. Verify environment variable

    ```bash
    kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
    ```

    4. Restart gateway and verify metrics initialization

    ```bash
    kubectl -n llm rollout restart deployment/inference-gateway-istio
    kubectl -n llm delete pod -l app=inference-gateway-istio -n llm
    ```

    5. Test metrics collection
    Make request:

    ```bash
    curl -H 'Authorization: APIKEY <INSERT_KEY>' \
    -H 'Content-Type: application/json' \
    -d '{"model": "simulator-model", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10}' \
    http://simulator-llm.apps.summit-gpu.octo-emerging.redhataicoe.com/v1/chat/completions
    ```

    6. Check token metrics

    ```bash
    kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
    ```

    ---

    ### Disable Metrics

    Set the ENV in the running operator:

    **Disable Metrics via OLM**

    1. Update CSV to set `KUADRANT_METRICS_ENABLED=false`

    ```bash
    kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "false"}]'
    ```

    2. Restart operator

    ```bash
    kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
    ```

    3. Verify environment variable

    ```bash
    kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
    ```

    4. Restart gateway and verify no metrics initialization (kick the pod as needed)

    ```bash
    kubectl -n llm rollout restart deployment/inference-gateway-istio
    kubectl -n llm delete pod -l app=inference-gateway-istio -n llm
    ```

    5. Test metrics collection
    Make request:

    ```bash
    curl -H 'Authorization: APIKEY <INSERT_KEY>' \
    -H 'Content-Type: application/json' \
    -d '{"model": "simulator-model", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10}' \
    http://simulator-llm.apps.summit-gpu.octo-emerging.redhataicoe.com/v1/chat/completions
    ```

    6. Make sure there are no metrics

    ```bash
    kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
    ```

    ## Raw Output

    ### Enable metrics output

    ```
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "true"}]'
    clusterserviceversion.operators.coreos.com/kuadrant-operator.v0.0.0 patched
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
    deployment.apps/kuadrant-operator-controller-manager restarted
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env[3]}' | jq
    {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "true"
    }
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
    [
    {
    "name": "WITH_EXTENSIONS",
    "value": "true"
    },
    {
    "name": "RELATED_IMAGE_WASMSHIM",
    "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
    },
    {
    "name": "OPERATOR_NAMESPACE",
    "valueFrom": {
    "fieldRef": {
    "fieldPath": "metadata.namespace"
    }
    }
    },
    {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "true"
    }
    ]
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager \
    -o jsonpath='{.spec.template.spec.containers[0].env[1]}' | jq
    {
    "name": "RELATED_IMAGE_WASMSHIM",
    "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
    }
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm rollout restart deployment/inference-gateway-istio
    kubectl -n llm delete pod -l app=inference-gateway-istio -n llm
    deployment.apps/inference-gateway-istio restarted
    No resources found
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl get pods # --all-namespaces
    NAME READY STATUS RESTARTS AGE
    inference-gateway-istio-97846c4c9-djhvh 0/1 Completed 0 22m
    inference-gateway-istio-cd9967644-h55px 1/1 Running 0 10s
    qwen3-0-6b-instruct-predictor-5b96f458d-d5hm4 1/1 Running 0 9d
    vllm-simulator-predictor-6d748c55b5-dtc66 1/1 Running 5 (22h ago) 9d
    # GENERATED REQUESTS HERE
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
    # TYPE token_usage_total counter
    token_usage_total{} 1140
    # TYPE token_usage_with_user_and_group__user___freeuser1___group___free___namespace__llm counter
    token_usage_with_user_and_group__user___freeuser1___group___free___namespace__llm{} 750
    # TYPE token_usage_with_user_and_group__user___premiumuser1___group___premium___namespace__llm counter
    token_usage_with_user_and_group__user___premiumuser1___group___premium___namespace__llm{} 390
    ```

    ### Disable metrics output

    ```
    $ kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "false"}]'
    clusterserviceversion.operators.coreos.com/kuadrant-operator.v0.0.0 patched
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
    deployment.apps/kuadrant-operator-controller-manager restarted
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
    [
    {
    "name": "WITH_EXTENSIONS",
    "value": "true"
    },
    {
    "name": "RELATED_IMAGE_WASMSHIM",
    "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
    },
    {
    "name": "OPERATOR_NAMESPACE",
    "valueFrom": {
    "fieldRef": {
    "fieldPath": "metadata.namespace"
    }
    }
    },
    {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "false"
    }
    ]
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm rollout restart deployment/inference-gateway-istio
    deployment.apps/inference-gateway-istio restarted
    # GENERATED REQUESTS HERE
    🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
    # NO METRICS
    ```