Instructions to use nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext")
model = AutoModelForCausalLM.from_pretrained("nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext

SGLang

How to use nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext with Docker Model Runner:
```
docker model run hf.co/nyunai/OpenHathi-7B-Hi-v0.1-Base-AWQ-wikitext
```

Model Description

This model is a compressed version of the OpenHathi-7B-Hi base model, optimized for chat format text data in the Hindi language. It has been quantized using the AWQ technique with calibration data from the samvaad-hi-v1 dataset. The compression process aims to reduce the model size while preserving its performance on chat-oriented tasks.

Model Usage:

The compressed model can be utilized for various natural language processing tasks, particularly those involving chat format text data in Hindi. It can be deployed in conversational AI systems, chatbots, or any application requiring efficient processing of chat-style interactions.

Performance Metrics:

Model Size: 4.15 GB
Compression Technique: AWQ
Calibration Data: wikitext dataset
Tokenization Model Size: 968 KB
Performance: The compressed model's performance has been evaluated on various chat-oriented tasks, demonstrating efficiency in handling conversational text data while maintaining comparable performance to the original base model.

Limitations: While the compressed model offers significant reductions in size, there may be slight trade-offs in performance compared to the full-sized base model. It may not perform optimally on tasks outside the scope of chat-oriented text data in Hindi.

Downloads last month: 2

Safetensors

Model size

7B params

Tensor type

I32

F16