Kunat Pipatanakul kunato

Model	GAIA - Thai/English	GSM8K - Thai/English	HotpotQA - Thai/English
gpt-3.5-turbo-0125	18.42/37.5	70/80	39.56/59
Typhoon-1.5X 70B	17.10/36.25	80/95	52.7/65.83
gpt-4-0612	17.10/38.75	90/100	56.41/76.25
---	---	---	---
gpt-4o	44.73/57.5	100/100	71.64/76.58

Model	IFEval Thai	IFEval English	MT-Bench Thai	MT-Bench English
Typhoon-1.5X 70B	0.645	0.810	8.029	8.797
gpt-4-0612	0.612	0.793*	7.801	8.671
---	---	---	---	---
gpt-4o	0.737	0.871	8.514	9.184

Model	ONET	IC	TGAT	TPAT-1	A-Level	Average (ThaiExam)	MMLU
Typhoon-1.5X 70B	0.565	0.68	0.778	0.517	0.56	0.620	0.7945
gpt-4-0612	0.493	0.69	0.744	0.509	0.616	0.610	0.864**
---	---	---	---	---	---	---	---
gpt-4o	0.62	0.63	0.789	0.56	0.623	0.644	0.887**

Model	IFEval Thai	IFEval English	MT-Bench Thai	MT-Bench English
Typhoon-1.5 8B	0.548	0.676	6.402	7.275
Typhoon-1.5X 8B	0.548	0.691	6.902	7.9
gpt-3.5-turbo-0125	0.479	0.659	6.186	8.181

Model	ONET	IC	TGAT	TPAT-1	A-Level	Average (ThaiExam)	MMLU
Typhoon-1.5 8B	0.446	0.431	0.722	0.526	0.407	0.5028	0.6136
Typhoon-1.5X 8B	0.478	0.379	0.722	0.5	0.435	0.5028	0.6369
gpt-3.5-turbo-0125	0.358	0.279	0.678	0.345	0.318	0.3956	0.700**

Model	ONET	IC	TGAT	TPAT-1	A-Level	Average (ThaiExam)	M3Exam	MMLU
Typhoon-1.5 72B	0.562	0.716	0.778	0.5	0.528	0.6168	0.587	0.7271
OpenThaiGPT 1.0.0 70B	0.447	0.492	0.778	0.5	0.319	0.5072	0.493	0.6167
GPT-3.5-turbo(01-2024)	0.358	0.279	0.678	0.345	0.318	0.3956	0.316	0.700[2]
GPT-4(04-2024)	0.589	0.594	0.756	0.517	0.616	0.6144	0.626	0.864[2]

	You are Manus, an AI agent created by the Manus team.

	You excel at the following tasks:
	1. Information gathering, fact-checking, and documentation
	2. Data processing, analysis, and visualization
	3. Writing multi-chapter articles and in-depth research reports
	4. Creating websites, applications, and tools
	5. Using programming to solve various problems beyond development
	6. Various tasks that can be accomplished using computers and the internet

	from awq import AutoAWQForCausalLM
	from transformers import AutoTokenizer

	# TODO pip install autoawq==0.2.4

	if __name__ == '__main__':
	model_path = 'scb10x/llama-3-typhoon-v1.5x-8b-instruct'
	quant_path = 'llama-3-typhoon-v1.5x-8b-instruct-awq'
	quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
	tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

	from awq import AutoAWQForCausalLM
	from transformers import AutoTokenizer

	if __name__ == '__main__':
	model_path = 't8-8b'
	quant_path = 't8-8b-awq'
	quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
	tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
	tokenizer.save_pretrained(quant_path)

	import asyncio
	import os
	import random
	import hashlib
	from datetime import datetime
	from typing import Dict, List, Type

	from dotenv import load_dotenv
	from loguru import logger
	from pydantic import BaseModel