Skip to content

Instantly share code, notes, and snippets.

@justinmoon

justinmoon/WHY_OPENCODE_WORKS_MAPLE_DOESNT.md

Last active October 27, 2025 15:04
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save justinmoon/062f73444cb13ee3a7ea9b981ffb456a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save justinmoon/062f73444cb13ee3a7ea9b981ffb456a to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. justinmoon revised this gist Oct 27, 2025. 1 changed file with 69 additions and 0 deletions.
    69 changes: 69 additions & 0 deletions maple-bug-summary.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,69 @@
    # Maple AI Tool Calling Bug Report

    ## Issue
    Maple AI models don't use OpenAI-style tool calling format. All tested models return empty responses or text instead of making tool calls.

    ## Reproduction
    See `maple-tool-calling-bug-repro.ts` for a minimal, self-contained reproduction.

    **Test model:** `gpt-oss-120b` (chosen because it DOES support tool calling on Cerebras)

    **Expected behavior:**
    ```json
    {
    "role": "assistant",
    "tool_calls": [{
    "type": "function",
    "function": {
    "name": "write",
    "arguments": "{\"path\":\"/tmp/maple-test.txt\",\"content\":\"Hello World from Maple!\"}"
    }
    }]
    }
    ```

    **Actual behavior:**
    - Empty response (0 text chunks)
    - `hasText = true` but `textContent.length = 0`
    - No tool calls generated

    ## Models Tested (All Fail)
    1. ❌ `mistral-small-3-1-24b` - Refuses tools ("I don't have the capability...")
    2. ❌ `qwen2-5-72b` - Outputs tool syntax as text
    3. ❌ `llama-3.3-70b` - Outputs tool syntax as text
    4. ❌ `deepseek-r1-0528` - Uses custom format with special tokens
    5. ❌ `gpt-oss-120b` - Returns empty response (this reproduction)
    6. ❌ `deepseek-v31-terminus` - 400 Bad Request
    7. ❌ `leon-se/gemma-3-27b-it-fp8-dynamic` - 500 Internal Server Error

    ## Why This Matters
    The **same base models work fine** through other providers:

    | Model | Cerebras API | Maple API |
    |-------|--------------|-----------|
    | gpt-oss-120b | ✅ `tool_call: true` | ❌ Empty response |
    | qwen3-coder-480b | ✅ `tool_call: true` | ❌ Not offered |
    | qwen2.5-72b | ✅ Available with tools | ❌ Returns text |

    Source: https://models.dev/api.json (database of 186 models with tool calling)

    ## Root Cause Analysis
    See `WHY_OPENCODE_WORKS_MAPLE_DOESNT.md` for full analysis.

    **TL;DR:**
    - OpenCode Zen uses models from providers (Cerebras, xAI, Alibaba) with `tool_call: true`
    - Maple serves models through encrypted enclaves but tool calling isn't configured
    - This isn't a model capability issue - it's an API configuration issue

    ## Environment
    - Runtime: Bun 1.2.23
    - AI SDK: `ai` v4.x
    - Provider: `@ai-sdk/openai-compatible`
    - Maple SDK: Custom encrypted fetch (from opensecret-client)

    ## Request
    Please enable OpenAI-style tool calling for Maple models. The encryption/privacy features are great, but without tool calling, the models can't be used for agentic workflows.

    ## Contact
    Justin Moon ([@justinmoon](https://github.com/justinmoon))
    Project: yeet - minimal coding agent (https://github.com/justinmoon/yeet)
  2. justinmoon revised this gist Oct 27, 2025. 1 changed file with 145 additions and 0 deletions.
    145 changes: 145 additions & 0 deletions WHY_OPENCODE_WORKS_MAPLE_DOESNT.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,145 @@
    # Why OpenCode Zen Works But Maple Doesn't for Tool Calling

    ## The Question

    > "Why doesn't Qwen Coder work? Doesn't this same model work from OpenCode Zen? How does it work for one and not the other?"
    ## The Answer

    **They're NOT the same models.** OpenCode Zen and Maple offer DIFFERENT models with different capabilities.

    ## Evidence from models.dev/api.json

    ### OpenCode Zen Qwen Coder Models
    From `cerebras` provider (what OpenCode Zen uses):

    ```json
    {
    "qwen-3-coder-480b": {
    "id": "qwen-3-coder-480b",
    "name": "Qwen 3 Coder 480B",
    "tool_call": true, ✅
    "cost": {"input": 2, "output": 2}
    },
    "qwen-3-235b-a22b-instruct-2507": {
    "id": "qwen-3-235b-a22b-instruct-2507",
    "name": "Qwen 3 235B Instruct",
    "tool_call": true, ✅
    "cost": {"input": 0.6, "output": 1.2}
    }
    }
    ```

    From `alibaba` provider (Qwen's official provider):

    ```json
    {
    "qwen3-coder-flash": {
    "id": "qwen3-coder-flash",
    "name": "Qwen3 Coder Flash",
    "tool_call": true, ✅
    "cost": {"input": 0.3, "output": 1.5}
    },
    "qwen3-coder-30b-a3b-instruct": {
    "id": "qwen3-coder-30b-a3b-instruct",
    "name": "Qwen3-Coder 30B-A3B Instruct",
    "tool_call": true, ✅
    "cost": {"input": 0.45, "output": 2.25}
    },
    "qwen3-coder-480b-a35b-instruct": {
    "id": "qwen3-coder-480b-a35b-instruct",
    "name": "Qwen3-Coder 480B-A35B Instruct",
    "tool_call": true, ✅
    "cost": {"input": 1.5, "output": 7.5}
    },
    "qwen3-coder-plus": {
    "id": "qwen3-coder-plus",
    "name": "Qwen3 Coder Plus",
    "tool_call": true, ✅
    "cost": {"input": 1, "output": 5}
    }
    }
    ```

    ### Maple's Qwen Model
    From Maple's config:

    ```json
    {
    "model": "qwen2-5-72b",
    "tool_call": ??? ❌ (Not in models.dev database)
    }
    ```

    **Maple's model is NOT listed in the models.dev API database**, which tracks hundreds of models from 75+ providers.

    ## Why The Difference?

    ### 1. Model Serving Configuration
    - **OpenCode Zen (via Cerebras/Alibaba)**: Uses models explicitly configured with `"tool_call": true`
    - **Maple**: Serves models through their secure enclave, but tool calling may not be enabled/configured

    ### 2. Model Versions Matter
    Even "Qwen Coder" isn't one model:
    - `qwen2.5-coder-32b-instruct` (older, tested with tools via Vultr)
    - `qwen3-coder-30b` (newer, tool support via Alibaba)
    - `qwen3-coder-480b` (largest, tool support via multiple providers)
    - `qwen2-5-72b` (Maple's version - general instruct, not specifically "coder")

    ### 3. API Layer Differences
    **OpenCode Zen**:
    - Uses standard OpenAI SDK
    - Direct model access through providers (Cerebras, etc.)
    - Models pre-configured for tool calling

    **Maple**:
    - Encrypted secure enclave
    - Custom inference pipeline
    - Privacy-first architecture (may sacrifice some features)

    ## Test Results Explained

    ### ✅ OpenCode Zen: `grok-code`
    ```typescript
    // From xai provider in models.dev
    {
    "grok-code-fast-1": {
    "tool_call": true, ✅
    "cost": {"input": 0.2, "output": 1.5}
    }
    }
    ```
    **Works perfectly** - model explicitly supports tool calling

    ### ❌ Maple: `qwen2-5-72b`
    ```typescript
    // Not in models.dev database
    // Likely qwen2.5-72b-instruct from Alibaba, but served differently
    ```
    **Outputs tool syntax as text** - model not configured for tool calling through Maple's API

    ## Conclusion

    **It's not a yeet bug.** The models are different:

    1. **OpenCode Zen** offers models from providers who explicitly enable tool calling (`tool_call: true`)
    2. **Maple** prioritizes privacy and security, serving models through encrypted enclaves, but tool calling isn't configured
    3. **Same base model ≠ same capabilities** - it depends on how the provider serves it

    ## Solution

    **Use OpenCode Zen for yeet** - it's designed for coding agents with proper tool support.

    **Use Maple for chat** - it's designed for private, secure inference.

    ## Interesting Find

    From models.dev, **186 models** explicitly support `tool_call: true` across providers:
    - Alibaba: 31 models
    - Anthropic: 15 models
    - xAI (Grok): 17 models
    - Google (Gemini): 18 models
    - OpenAI: 12 models
    - And 20+ more providers

    Maple's models just aren't in this list (yet).
  3. justinmoon created this gist Oct 27, 2025.
    159 changes: 159 additions & 0 deletions maple-tool-calling-bug-repro.ts
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,159 @@
    #!/usr/bin/env bun
    /**
    * Bug Reproduction: Maple AI models don't use OpenAI tool calling format
    *
    * Testing: gpt-oss-120b (which works with tool calling on Cerebras)
    * Expected: Model should return tool_calls in response
    * Actual: Model returns empty response or text instead of tool calls
    *
    * Setup:
    * bun install ai @ai-sdk/openai-compatible
    * export MAPLE_API_KEY="your-key"
    * bun run maple-tool-calling-bug-repro.ts
    */

    import { streamText } from "ai"
    import { createOpenAICompatible } from "@ai-sdk/openai-compatible"
    import { createMapleFetch } from "./src/maple"

    const MAPLE_API_KEY = process.env.MAPLE_API_KEY || ""

    if (!MAPLE_API_KEY) {
    console.error("❌ Missing MAPLE_API_KEY environment variable")
    console.error(" Get one at: https://trymaple.ai")
    process.exit(1)
    }

    console.log("🐛 Maple AI Tool Calling Bug Reproduction\n")
    console.log("📋 Test Setup:")
    console.log(" Model: gpt-oss-120b")
    console.log(" API: https://enclave.trymaple.ai")
    console.log(" Task: Write 'Hello World' to /tmp/test.txt")
    console.log(" Expected: Model calls write() tool")
    console.log(" Actual: ???\n")

    // Create Maple's encrypted fetch
    const mapleFetch = await createMapleFetch({
    apiUrl: "https://enclave.trymaple.ai",
    apiKey: MAPLE_API_KEY,
    pcr0Values: [
    "79e7bd1e7df09fdb5b7098956a2268c278cc88be323c11975e2a2d080d65f30f8e0efe690edd450493c833b46f40ae1a",
    "ed9109c16f30a470cf0ea2251816789b4ffa510c990118323ce94a2364b9bf05bdb8777959cbac86f5cabc4852e0da71",
    "4f2bcdf16c38842e1a45defd944d24ea58bb5bcb76491843223022acfe9eb6f1ff79b2cb9a6b2a9219daf9c7bf40fa37",
    "b8ee4b511ef2c9c6ab3e5c0840c5df2218fbb4d9df88254ece7af9462677e55aa5a03838f3ae432d86ca1cb6f992eee7",
    ],
    })

    // Create provider with Maple's fetch
    const maple = createOpenAICompatible({
    name: "maple",
    baseURL: "https://enclave.trymaple.ai/v1",
    fetch: mapleFetch,
    })

    const tools = {
    write: {
    description: "Write content to a file on disk",
    parameters: {
    type: "object" as const,
    properties: {
    path: {
    type: "string" as const,
    description: "The file path to write to",
    },
    content: {
    type: "string" as const,
    description: "The content to write to the file",
    },
    },
    required: ["path", "content"],
    },
    execute: async ({ path, content }: { path: string; content: string }) => {
    try {
    await Bun.write(path, content)
    return `Written to ${path}`
    } catch (err: any) {
    return `Error: ${err.message}`
    }
    },
    },
    }

    try {
    console.log("📤 Sending request with tool definitions...\n")

    const result = streamText({
    model: maple("gpt-oss-120b"),
    messages: [
    {
    role: "user",
    content: "Write 'Hello World from Maple!' to /tmp/maple-test.txt",
    },
    ],
    tools,
    maxSteps: 3,
    })

    let hasToolCalls = false
    let hasText = false
    let textContent = ""
    let toolCallsFound: string[] = []

    for await (const chunk of result.fullStream) {
    if (chunk.type === "tool-call") {
    hasToolCalls = true
    toolCallsFound.push(chunk.toolName)
    console.log(`✅ Tool call: ${chunk.toolName}`)
    console.log(` Args: ${JSON.stringify(chunk.args)}`)
    } else if (chunk.type === "text-delta") {
    hasText = true
    const delta = String(chunk.textDelta || "")
    textContent += delta
    process.stdout.write(delta)
    } else if (chunk.type === "finish") {
    console.log("\n\n📊 Result:")
    console.log(` Tool calls: ${hasToolCalls ? "✅ YES" : "❌ NO"}`)
    console.log(` Text output: ${hasText ? "YES" : "NO"}`)
    console.log(` Text length: ${textContent.length} chars`)

    if (toolCallsFound.length > 0) {
    console.log(` Tools called: ${toolCallsFound.join(", ")}`)
    }

    if (hasText) {
    const sample = textContent.trim().substring(0, 200)
    console.log(` Text content: "${sample}${sample.length < textContent.trim().length ? '...' : ''}"`)
    }

    console.log("\n🔍 Diagnosis:")
    if (hasToolCalls) {
    console.log(" ✅ SUCCESS: Model correctly used tool calling!")
    } else if (hasText) {
    console.log(" ❌ BUG: Model returned text instead of calling tools")
    console.log(" Expected: model.tool_calls = [{ name: 'write', args: {...} }]")
    console.log(" Actual: model.content = text explaining what to do")
    } else {
    console.log(" ❌ BUG: Model returned empty response")
    console.log(" Expected: model.tool_calls = [{ name: 'write', args: {...} }]")
    console.log(" Actual: No tool calls, no text")
    }
    }
    }

    console.log("\n📝 Notes:")
    console.log(" • This same model (gpt-oss-120b) DOES support tool calling on Cerebras")
    console.log(" • See: https://inference-docs.cerebras.ai/models/openai-oss")
    console.log(" • OpenCode Zen models (grok-code, etc) work fine for tool calling")
    console.log(" • All Maple models tested show this issue (qwen, llama, deepseek, gpt-oss)")

    } catch (error: any) {
    console.error("\n❌ Error:", error.message)
    if (error.cause) {
    console.error(" Cause:", error.cause)
    }
    if (error.stack) {
    console.error("\n Stack trace:")
    console.error(error.stack)
    }
    process.exit(1)
    }