### Instructions for enabling and disabling metrics in a running Kuadrant OLM deployment

- **[Kuadrant/kuadrant-operator patch (PR not opened yet)](https://github.com/Kuadrant/kuadrant-operator/compare/main...nerdalert:kuadrant-operator:metrics-enable-env?expand=1)**
- **[Kuadrant/wasm-shim patch](https://github.com/Kuadrant/wasm-shim/pull/208)**

### Load the patched Kuadrant operator:

```bash
kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/image", "value": "ghcr.io/nerdalert/kuadrant-operator:metrics-test"}]'
    
kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager

# Verify the operator image
kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}'
```

### Load the patched wasm-shim image 

```
kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
     -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/1/value", "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"}]'

kubectl -n llm rollout restart deployment/inference-gateway-istio

# Verify the wasm image
kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager \
     -o jsonpath='{.spec.template.spec.containers[0].env[1]}' | jq
```

### Enable Metrics

1. Update CSV to set `KUADRANT_METRICS_ENABLED=true`

Add the env:

```bash
kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' -p='[
    {
      "op": "add",
      "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/-",
      "value": {
        "name": "KUADRANT_METRICS_ENABLED",
        "value": "true"
      }
    }
  ]'
```

Or update the env if it already exists:

```bash
   kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
     -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "true"}]'
```

2. Restart operator

   ```bash
   kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
   ```

3. Verify environment variable

   ```bash
   kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
   ```

4. Restart gateway and verify metrics initialization

   ```bash
   kubectl -n llm rollout restart deployment/inference-gateway-istio
   kubectl -n llm delete pod -l app=inference-gateway-istio -n llm
   ```

5. Test metrics collection
   Make request:

   ```bash
   curl -H 'Authorization: APIKEY <INSERT_KEY>' \
        -H 'Content-Type: application/json' \
        -d '{"model": "simulator-model", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10}' \
        http://<ENDPOINT_ADDRESS>/v1/chat/completions
   ```

6. Check token metrics

   ```bash
   kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
   ```

---

### Disable Metrics

Set the ENV in the running operator:

**Disable Metrics via OLM**

1. Update CSV to set `KUADRANT_METRICS_ENABLED=false`

   ```bash
   kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
     -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "false"}]'
   ```

2. Restart operator

   ```bash
   kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
   ```

3. Verify environment variable

   ```bash
   kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
   ```

4. Restart gateway and verify no metrics initialization (kick the pod as needed)

   ```bash
   kubectl -n llm rollout restart deployment/inference-gateway-istio
   kubectl -n llm delete pod -l app=inference-gateway-istio -n llm
   ```

5. Test metrics collection
   Make request:

   ```bash
   curl -H 'Authorization: APIKEY <INSERT_KEY>' \
        -H 'Content-Type: application/json' \
        -d '{"model": "simulator-model", "messages": [{"role": "user", "content": "test"}], "max_tokens": 10}' \
        http://simulator-llm.apps.summit-gpu.octo-emerging.redhataicoe.com/v1/chat/completions
   ```

6. Make sure there are no metrics

   ```bash
   kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
   ```

## Raw Output

### Enable metrics output

```
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "true"}]'
clusterserviceversion.operators.coreos.com/kuadrant-operator.v0.0.0 patched
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
deployment.apps/kuadrant-operator-controller-manager restarted
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env[3]}' | jq
{
  "name": "KUADRANT_METRICS_ENABLED",
  "value": "true"
}
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$   kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
[
  {
    "name": "WITH_EXTENSIONS",
    "value": "true"
  },
  {
    "name": "RELATED_IMAGE_WASMSHIM",
    "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
  },
  {
    "name": "OPERATOR_NAMESPACE",
    "valueFrom": {
      "fieldRef": {
        "fieldPath": "metadata.namespace"
      }
    }
  },
  {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "true"
  }
]
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get deployment kuadrant-operator-controller-manager \
     -o jsonpath='{.spec.template.spec.containers[0].env[1]}' | jq
{
  "name": "RELATED_IMAGE_WASMSHIM",
  "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
}
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$   kubectl -n llm rollout restart deployment/inference-gateway-istio
  kubectl -n llm delete pod -l app=inference-gateway-istio -n llm
deployment.apps/inference-gateway-istio restarted
No resources found
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl get pods     # --all-namespaces
NAME                                            READY   STATUS      RESTARTS      AGE
inference-gateway-istio-97846c4c9-djhvh         0/1     Completed   0             22m
inference-gateway-istio-cd9967644-h55px         1/1     Running     0             10s
qwen3-0-6b-instruct-predictor-5b96f458d-d5hm4   1/1     Running     0             9d
vllm-simulator-predictor-6d748c55b5-dtc66       1/1     Running     5 (22h ago)   9d

# GENERATED REQUESTS HERE

🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token
# TYPE token_usage_total counter
token_usage_total{} 1140
# TYPE token_usage_with_user_and_group__user___freeuser1___group___free___namespace__llm counter
token_usage_with_user_and_group__user___freeuser1___group___free___namespace__llm{} 750
# TYPE token_usage_with_user_and_group__user___premiumuser1___group___premium___namespace__llm counter
token_usage_with_user_and_group__user___premiumuser1___group___premium___namespace__llm{} 390
```

### Disable metrics output

```
$ kubectl -n kuadrant-system patch csv kuadrant-operator.v0.0.0 --type='json' \
    -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/3/value", "value": "false"}]'
clusterserviceversion.operators.coreos.com/kuadrant-operator.v0.0.0 patched
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system rollout restart deployment/kuadrant-operator-controller-manager
deployment.apps/kuadrant-operator-controller-manager restarted
🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n kuadrant-system get csv kuadrant-operator.v0.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' | jq
[
  {
    "name": "WITH_EXTENSIONS",
    "value": "true"
  },
  {
    "name": "RELATED_IMAGE_WASMSHIM",
    "value": "oci://ghcr.io/nerdalert/wasm-shim:metrics"
  },
  {
    "name": "OPERATOR_NAMESPACE",
    "valueFrom": {
      "fieldRef": {
        "fieldPath": "metadata.namespace"
      }
    }
  },
  {
    "name": "KUADRANT_METRICS_ENABLED",
    "value": "false"
  }
]

🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm rollout restart deployment/inference-gateway-istio
deployment.apps/inference-gateway-istio restarted

# GENERATED REQUESTS HERE

🐱 ip-172-31-45-224:~/wasm-pr-v2/kuadrant-operator$ kubectl -n llm exec deployment/inference-gateway-istio -c istio-proxy -- curl -s http://127.0.0.1:15090/stats/prometheus | grep token

# NO METRICS
```