Instructions to use NousResearch/Nous-Capybara-34B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NousResearch/Nous-Capybara-34B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NousResearch/Nous-Capybara-34B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NousResearch/Nous-Capybara-34B")
model = AutoModelForCausalLM.from_pretrained("NousResearch/Nous-Capybara-34B")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use NousResearch/Nous-Capybara-34B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NousResearch/Nous-Capybara-34B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Capybara-34B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/NousResearch/Nous-Capybara-34B

SGLang

How to use NousResearch/Nous-Capybara-34B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NousResearch/Nous-Capybara-34B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Capybara-34B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NousResearch/Nous-Capybara-34B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Capybara-34B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use NousResearch/Nous-Capybara-34B with Docker Model Runner:
```
docker model run hf.co/NousResearch/Nous-Capybara-34B
```

How did you train this without going OOM in RAM & VRAM?

#15

by vicplus - opened Mar 6, 2024

Discussion

vicplus

Mar 6, 2024

Hey Mr. Nous,

I tried finetuning this with a 4xA100 80GB setup + 700 GB of RAM. Loading up the model ate up half the memory for both. trainer.Evaluate made it go OOM due to lack of RAM.

Any idea what could have happened? I'm using transformers & peft to finetune it.

Sincerely,
Cat

bloc97

NousResearch org Mar 6, 2024

You probably need Flash Attention 2 and either Zero3 offloading or FSDP in order to train this model on a 4xA100 setup. If you are using LoRA, make sure Flash Attention 2 is enabled.

vicplus

Mar 6, 2024

Will it work as-is with a 8xA100 setup? I am using LoRA w/o FA2, so that might be it. Thank you, I will try that out!

teknium

NousResearch org Mar 6, 2024

Will it work as-is with a 8xA100 setup? I am using LoRA w/o FA2, so that might be it. Thank you, I will try that out!

For 8bit lora without flash attention it wont matter because it just does DDP. A full finetune will work on 8x 80gb or a qlora with flash attention should

also be sure to enable gradient checkpointing

teknium changed discussion status to closed Mar 6, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment