Instructions to use heegyu/mandoo-9b-2407-sft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use heegyu/mandoo-9b-2407-sft with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="heegyu/mandoo-9b-2407-sft")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("heegyu/mandoo-9b-2407-sft")
model = AutoModelForCausalLM.from_pretrained("heegyu/mandoo-9b-2407-sft")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use heegyu/mandoo-9b-2407-sft with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "heegyu/mandoo-9b-2407-sft"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "heegyu/mandoo-9b-2407-sft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/heegyu/mandoo-9b-2407-sft

SGLang

How to use heegyu/mandoo-9b-2407-sft with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "heegyu/mandoo-9b-2407-sft" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "heegyu/mandoo-9b-2407-sft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "heegyu/mandoo-9b-2407-sft" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "heegyu/mandoo-9b-2407-sft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use heegyu/mandoo-9b-2407-sft with Docker Model Runner:
```
docker model run hf.co/heegyu/mandoo-9b-2407-sft
```

Mandoo is a LM assistant supporting English, Chinese and Korean.

Example

from transformers import pipeline

pipe = pipeline("text-generation", model="heegyu/mandoo-9b-2407", device_map="auto", torch_dtype="auto")

messages = [
    {"role": "user", "content": "I want to start saving some money by growing my own food. Can I do this during the winter with an indoor garden?"},
]
pipe(messages, max_new_tokens=128, do_sample=True)

Benchmark Result

Every generation of this model was sampled with temperature=0.7, top_p=0.9, top_k=50

Korean

Model	싱글턴
gemma-2-9b-it	7.45
mandoo-9b-2407-sft	6.50

I used sampling with temperature=0.7, max_new_tokens=2048 for generation.

# mandoo-9b-2407-sft
카테고리: 추론(Reasoning), 싱글 점수 평균: 6.86, 멀티 점수 평균: 3.86
카테고리: 수학(Math), 싱글 점수 평균: 5.14, 멀티 점수 평균: 3.71
카테고리: 글쓰기(Writing), 싱글 점수 평균: 7.29, 멀티 점수 평균: 7.00
카테고리: 코딩(Coding), 싱글 점수 평균: 8.29, 멀티 점수 평균: 8.14
카테고리: 이해(Understanding), 싱글 점수 평균: 9.29, 멀티 점수 평균: 8.57
카테고리: 문법(Grammar), 싱글 점수 평균: 6.43, 멀티 점수 평균: 3.43
전체 싱글 점수 평균: 7.21
전체 멀티 점수 평균: 5.79
전체 점수: 6.50

# gemma-2-9b-it
카테고리: 추론(Reasoning), 싱글 점수 평균: 9.43, 멀티 점수 평균: 6.71
카테고리: 수학(Math), 싱글 점수 평균: 6.14, 멀티 점수 평균: 8.57
카테고리: 글쓰기(Writing), 싱글 점수 평균: 8.71, 멀티 점수 평균: 8.86
카테고리: 코딩(Coding), 싱글 점수 평균: 7.43, 멀티 점수 평균: 6.86
카테고리: 이해(Understanding), 싱글 점수 평균: 8.29, 멀티 점수 평균: 8.29
카테고리: 문법(Grammar), 싱글 점수 평균: 6.29, 멀티 점수 평균: 3.86
전체 싱글 점수 평균: 7.71
전체 멀티 점수 평균: 7.19
전체 점수: 7.45

English

AlpacaEval

                            length_controlled_winrate  win_rate  standard_error  n_total  avg_length
gpt-4o-2024-05-13                               57.46     51.33            1.47      805        1873
gpt-4-turbo-2024-04-09                          55.02     46.12            1.47      805        1802
gpt4_1106_preview                               50.00     50.00            0.00      805        2049
claude-3-opus-20240229                          40.51     29.11            1.39      805        1388
claude-3-sonnet-20240229                        34.87     25.56            1.34      805        1420
Meta-Llama-3-70B-Instruct                       34.42     33.18            1.39      805        1919
gemini-pro                                      24.38     18.18            1.16      805        1456
Mixtral-8x7B-Instruct-v0.1                      23.69     18.26            1.19      805        1465
Meta-Llama-3-8B-Instruct                        22.92     22.57            1.26      805        1899
**heegyu/mandoo-9b-2407-sft**  <---             19.82     18.18            1.13      805        1847
Mistral-7B-Instruct-v0.2                        17.11     14.72            1.08      805        1676
alpaca-7b                                        5.88      2.59            0.49      805         396

IFEval

Model	싱글턴
gemma-2-9b-it	76.95
mandoo-9b-2407-sft	59.19

Strict Accuracy Scores: Avg 0.59191279139
prompt-level: 0.5471349353049908                  
instruction-level: 0.6366906474820144 

Loose Accuracy Scores:
prompt-level: 0.589648798521257                   
instruction-level: 0.6774580335731415

Downloads last month: 3

Safetensors

Model size

9B params

Tensor type

BF16

heegyu
/

mandoo-9b-2407-sft