Skip to content

Instantly share code, notes, and snippets.

@data2json

data2json/LLAMA-3.1-8B-Transformer-Tool-calling.py

Created August 27, 2024 04:02
Show Gist options
  • Save data2json/09508bb910b9a4f9c02fb70c4c5ebb62 to your computer and use it in GitHub Desktop.
Save data2json/09508bb910b9a4f9c02fb70c4c5ebb62 to your computer and use it in GitHub Desktop.
Download ZIP
LLAMA-3.1-8B Template Bug
Raw
LLAMA-3.1-8B-Transformer-Tool-calling.py
essobi@thegreenbox:~/LLaMA-Factory$ more OKG.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
checkpoint = "meta-llama/Meta-Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(
checkpoint, torch_dtype=torch.bfloat16, device_map="auto"
)
# Define weather-related functions
def get_current_temperature(location: str, unit: str) -> float:
"""
Get the current temperature at a location.
Args:
location: The location to get the temperature for, in the format "City, Country"
unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
Returns:
The current temperature at the specified location in the specified units, as a float.
"""
return 22.0 # A real function should probably actually get the temperature!
def get_current_wind_speed(location: str) -> float:
"""
Get the current wind speed in km/h at a given location.
Args:
location: The location to get the temperature for, in the format "City, Country"
Returns:
The current wind speed at the given location in km/h, as a float.
"""
return 6.0 # A real function should probably actually get the wind speed!
# Set up tools and initial messages
tools = [get_current_temperature, get_current_wind_speed]
messages = [
{
"role": "system",
"content": "You are a bot that responds to weather queries. You should reply with the unit used in the queried location.",
},
{"role": "user", "content": "Hey, what's the temperature in Paris right now?"},
]
# Function to generate response
def generate_response(messages, tools):
inputs = tokenizer.apply_chat_template(
messages,
tools=tools,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
)
print(tokenizer.decode(inputs["input_ids"][0]))
# Generate initial response
print("Initial response:")
print(generate_response(messages, tools))
Raw
output.txt
essobi@thegreenbox:~/LLaMA-Factory$ python OKG.py
Loading checkpoint shards: 100%|███████████████████████████████| 4/4 [00:00<00:00, 12.50it/s]
Some parameters are on the meta device device because they were offloaded to the disk and cpu.
Initial response:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython
Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024
You are a bot that responds to weather queries. You should reply with the unit used in the queried location.<|eot_id|><|start_header_id|>user<|end_header_id|>
Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.
Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.Do not use variables.
{
"type": "function",
"function": {
"name": "get_current_temperature",
"description": "Get the current temperature at a location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The location to get the temperature for, in the format \"City, Country\""
},
"unit": {
"type": "string",
"enum": [
"celsius",
"fahrenheit"
],
"description": "The unit to return the temperature in."
}
},
"required": [
"location",
"unit"
]
},
"return": {
"type": "number",
"description": "The current temperature at the specified location in the specified units, as a float."
}
}
}
{
"type": "function",
"function": {
"name": "get_current_wind_speed",
"description": "Get the current wind speed in km/h at a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The location to get the temperature for, in the format \"City, Country\""
}
},
"required": [
"location"
]
},
"return": {
"type": "number",
"description": "The current wind speed at the given location in km/h, as a float."
}
}
}
Hey, what's the temperature in Paris right now?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment