Instructions to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF",
	filename="OpenCodeInterpreter-DS-6.7B.IQ1_S.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S
# Run inference directly in the terminal:
llama-cli -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S
# Run inference directly in the terminal:
llama-cli -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S
# Run inference directly in the terminal:
./llama-cli -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S

Use Docker

docker model run hf.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S

LM Studio
Jan

vLLM

How to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S

Ollama
How to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with Ollama:
```
ollama run hf.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S
```

Unsloth Studio new

How to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF to start chatting

Docker Model Runner
How to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with Docker Model Runner:
```
docker model run hf.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S
```

Lemonade

How to use CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF:IQ1_S

Run and chat with the model

lemonade run user.OpenCodeInterpreter-DS-6.7B-SOTA-GGUF-IQ1_S

List all available models

lemonade list

CISCai commited on Feb 27, 2024

Commit

4c1ab44

verified ·

1 Parent(s): a24bb6a

Added IQ4_XS

Browse files

Files changed (3) hide show

.gitattributes +1 -0
OpenCodeInterpreter-DS-6.7B.IQ4_XS.gguf +3 -0
README.md +4 -2

.gitattributes CHANGED Viewed

@@ -44,3 +44,4 @@ OpenCodeInterpreter-DS-6.7B.IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
 OpenCodeInterpreter-DS-6.7B.IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
 OpenCodeInterpreter-DS-6.7B.IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
 OpenCodeInterpreter-DS-6.7B.imatrix-4096.dat filter=lfs diff=lfs merge=lfs -text

 OpenCodeInterpreter-DS-6.7B.IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
 OpenCodeInterpreter-DS-6.7B.IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
 OpenCodeInterpreter-DS-6.7B.imatrix-4096.dat filter=lfs diff=lfs merge=lfs -text
+OpenCodeInterpreter-DS-6.7B.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text

OpenCodeInterpreter-DS-6.7B.IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ce70c09cc1ab5d6415dd24ff452cb5b4e1b6a1f7fd5853ded6cbfc8006cf9b9e
+size 3621315744

README.md CHANGED Viewed

@@ -47,7 +47,7 @@ You are an AI programming assistant, utilizing the Deepseek Coder model, develop
 <!-- compatibility_gguf start -->
 ## Compatibility
-These quantised GGUFv3 files are compatible with llama.cpp from February 26th 2024 onwards, as of commit [a33e6a0](https://github.com/ggerganov/llama.cpp/commit/a33e6a0d2a66104ea9a906bdbf8a94d050189d91)
 They are also compatible with many third party UIs and libraries provided they are built using a recent llama.cpp.
@@ -67,6 +67,7 @@ The new methods available are:
 * GGML_TYPE_IQ3_XS - 3-bit quantization in super-blocks with an importance matrix applied, effectively using 3.3 bpw
 * GGML_TYPE_IQ3_S - 3-bit quantization in super-blocks with an importance matrix applied, effectively using 3.44 bpw
 * GGML_TYPE_IQ3_M - 3-bit quantization in super-blocks with an importance matrix applied, effectively using 3.66 bpw
 Refer to the Provided Files table below to see what files use which methods, and how.
 </details>
@@ -86,6 +87,7 @@ Refer to the Provided Files table below to see what files use which methods, and
 | [OpenCodeInterpreter-DS-6.7B.IQ3_XS.gguf](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.IQ3_XS.gguf) | IQ3_XS | 3 | 2.7 GB| 4.7 GB | small, substantial quality loss |
 | [OpenCodeInterpreter-DS-6.7B.IQ3_S.gguf](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.IQ3_S.gguf) | IQ3_S | 3 | 2.8 GB| 4.8 GB | small, greater quality loss |
 | [OpenCodeInterpreter-DS-6.7B.IQ3_M.gguf](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.IQ3_M.gguf) | IQ3_M | 3 | 3.0 GB| 5.0 GB | medium, balanced quality - recommended |
 Generated importance matrix file: [OpenCodeInterpreter-DS-6.7B.imatrix.dat](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.imatrix.dat)
 Generated importance matrix file (4K context): [OpenCodeInterpreter-DS-6.7B.imatrix-4096.dat](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.imatrix-4096.dat)
@@ -97,7 +99,7 @@ Generated importance matrix file (4K context): [OpenCodeInterpreter-DS-6.7B.imat
 <!-- README_GGUF.md-how-to-run start -->
 ## Example `llama.cpp` command
-Make sure you are using `llama.cpp` from commit [a33e6a0](https://github.com/ggerganov/llama.cpp/commit/a33e6a0d2a66104ea9a906bdbf8a94d050189d91) or later.
 ```shell
 ./main -ngl 33 -m OpenCodeInterpreter-DS-6.7B.IQ3_M.gguf --color -c 16384 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n### Instruction:\n{prompt}\n### Response:"

 <!-- compatibility_gguf start -->
 ## Compatibility
+These quantised GGUFv3 files are compatible with llama.cpp from February 26th 2024 onwards, as of commit [0becb22](https://github.com/ggerganov/llama.cpp/commit/0becb22ac05b6542bd9d5f2235691aa1d3d4d307)
 They are also compatible with many third party UIs and libraries provided they are built using a recent llama.cpp.
 * GGML_TYPE_IQ3_XS - 3-bit quantization in super-blocks with an importance matrix applied, effectively using 3.3 bpw
 * GGML_TYPE_IQ3_S - 3-bit quantization in super-blocks with an importance matrix applied, effectively using 3.44 bpw
 * GGML_TYPE_IQ3_M - 3-bit quantization in super-blocks with an importance matrix applied, effectively using 3.66 bpw
+* GGML_TYPE_IQ4_XS - 4-bit quantization in super-blocks with an importance matrix applied, effectively using 4.25 bpw
 Refer to the Provided Files table below to see what files use which methods, and how.
 </details>
 | [OpenCodeInterpreter-DS-6.7B.IQ3_XS.gguf](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.IQ3_XS.gguf) | IQ3_XS | 3 | 2.7 GB| 4.7 GB | small, substantial quality loss |
 | [OpenCodeInterpreter-DS-6.7B.IQ3_S.gguf](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.IQ3_S.gguf) | IQ3_S | 3 | 2.8 GB| 4.8 GB | small, greater quality loss |
 | [OpenCodeInterpreter-DS-6.7B.IQ3_M.gguf](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.IQ3_M.gguf) | IQ3_M | 3 | 3.0 GB| 5.0 GB | medium, balanced quality - recommended |
+| [OpenCodeInterpreter-DS-6.7B.IQ4_XS.gguf](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.IQ4_XS.gguf) | IQ4_XS | 4 | 3.4 GB| 5.4 GB | small, substantial quality loss |
 Generated importance matrix file: [OpenCodeInterpreter-DS-6.7B.imatrix.dat](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.imatrix.dat)
 Generated importance matrix file (4K context): [OpenCodeInterpreter-DS-6.7B.imatrix-4096.dat](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/OpenCodeInterpreter-DS-6.7B.imatrix-4096.dat)
 <!-- README_GGUF.md-how-to-run start -->
 ## Example `llama.cpp` command
+Make sure you are using `llama.cpp` from commit [0becb22](https://github.com/ggerganov/llama.cpp/commit/0becb22ac05b6542bd9d5f2235691aa1d3d4d307) or later.
 ```shell
 ./main -ngl 33 -m OpenCodeInterpreter-DS-6.7B.IQ3_M.gguf --color -c 16384 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n### Instruction:\n{prompt}\n### Response:"