BUGFIX_HUGGINGFACE_404.md

Bug Fix: HuggingFace Spaces 404 Error for Embeddings

Issue Summary

Date: 2025-11-17 Environment: HuggingFace Spaces deployment Severity: Critical (blocks deployment) Status: ✅ Fixed

Error Log

2025-11-17 08:46:13,968 - rag.embeddings - ERROR - Error generating embedding: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}
2025-11-17 08:46:22,171 - __main__ - ERROR - Workflow error: RetryError[<Future at 0x7fc76c42fcd0 state=finished raised NotFoundError>]

Root Cause

The error occurred because the AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME environment variable was not set in HuggingFace Spaces, causing the Azure OpenAI API to return a 404 error when trying to generate embeddings.

Why This Happened

1. Inconsistent variable name in .env.example: The example file had the variable commented out and named differently:

   # .env.example (OLD - BROKEN)
   # AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-3-small  # Wrong name!
   

2. No validation on startup: The app did not validate that all required environment variables were set before attempting to use them.

3. Unclear error messages: The 404 error from Azure OpenAI didn't clearly indicate which deployment was missing.

The Fix

1. Fixed .env.example (lines 7-8)

Before:

# Optional: Embedding model deployment name (if different)
# AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-3-small

After:

# REQUIRED: Embedding model deployment name
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME=text-embedding-3-small

Changes:

• ✅ Uncommented the variable (it's required, not optional)
• ✅ Fixed variable name: AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME (matches code)
• ✅ Added AZURE_OPENAI_API_VERSION=2024-05-01-preview for completeness

2. Added Environment Validation in app.py (lines 43-75)

def validate_environment():
    """Validate that all required environment variables are set."""
    required_vars = [
        "AZURE_OPENAI_ENDPOINT",
        "AZURE_OPENAI_API_KEY",
        "AZURE_OPENAI_DEPLOYMENT_NAME",
        "AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME"  # Now validated!
    ]

    missing_vars = []
    for var in required_vars:
        value = os.getenv(var)
        if not value or value.strip() == "":
            missing_vars.append(var)

    if missing_vars:
        error_msg = (
            f"Missing required environment variables: {', '.join(missing_vars)}\n"
            f"Please set them in your .env file or HuggingFace Spaces secrets.\n"
            f"See .env.example for reference."
        )
        logger.error(error_msg)
        raise ValueError(error_msg)

    # Log configuration (masked)
    logger.info(f"Azure OpenAI Endpoint: {os.getenv('AZURE_OPENAI_ENDPOINT')}")
    logger.info(f"LLM Deployment: {os.getenv('AZURE_OPENAI_DEPLOYMENT_NAME')}")
    logger.info(f"Embedding Deployment: {os.getenv('AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME')}")
    logger.info(f"API Version: {os.getenv('AZURE_OPENAI_API_VERSION', '2024-02-01')}")

# Validate environment before importing other modules
validate_environment()

Benefits:

• ✅ Fails fast with clear error message at startup
• ✅ Shows which variables are missing
• ✅ Logs configuration for debugging
• ✅ Prevents cryptic 404 errors later in pipeline

3. Enhanced Error Messages in rag/embeddings.py (lines 37-64, 99-109, 164-174)

Added deployment name validation in __init__:

# Validate configuration
if not self.embedding_model:
    raise ValueError(
        "AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME environment variable is not set. "
        "This is required for generating embeddings. Please set it in your .env file."
    )

Added better error handling for 404 errors:

except Exception as e:
    error_msg = str(e)
    if "404" in error_msg or "Resource not found" in error_msg:
        logger.error(
            f"Embedding deployment '{self.embedding_model}' not found. "
            f"Please verify that this deployment exists in your Azure OpenAI resource. "
            f"Original error: {error_msg}"
        )
    else:
        logger.error(f"Error generating embedding: {error_msg}")
    raise

Benefits:

• ✅ Clear error message pointing to missing deployment
• ✅ Guides user to check Azure OpenAI resource
• ✅ Applied to both single and batch embedding methods

4. Updated HuggingFace Startup Script (lines 10-40)

# Check if required environment variables are set
echo ""
echo "🔍 Checking environment variables..."

required_vars=("AZURE_OPENAI_ENDPOINT" "AZURE_OPENAI_API_KEY" "AZURE_OPENAI_DEPLOYMENT_NAME" "AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME")
missing_vars=()

for var in "${required_vars[@]}"; do
    if [ -z "${!var}" ]; then
        missing_vars+=("$var")
        echo "❌ Missing: $var"
    else
        echo "✅ Found: $var"
    fi
done

if [ ${#missing_vars[@]} -ne 0 ]; then
    echo ""
    echo "⚠️  ERROR: Missing required environment variables!"
    echo "Please set the following in HuggingFace Spaces Settings > Repository secrets:"
    for var in "${missing_vars[@]}"; do
        echo "  - $var"
    done
    echo ""
    echo "See .env.example for the complete list of required variables."
    exit 1
fi

Benefits:

• ✅ Validates environment variables before starting Python app
• ✅ Shows clear ✅/❌ status for each variable
• ✅ Fails early with deployment instructions
• ✅ Prevents wasted time debugging Python errors

5. Created Comprehensive Deployment Guide

New file: HUGGINGFACE_DEPLOYMENT.md

Contents:

• Complete list of required environment variables
• Step-by-step deployment instructions
• Common issues and solutions (including this 404 error)
• Azure OpenAI deployment verification steps
• Performance and cost considerations
• Security best practices

6. Updated README.md (lines 662-685)

Added prominent link to deployment guide and highlighted the required embedding deployment variable:

**Required**: Add the following secrets in Space settings → Repository secrets:
- `AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME` (e.g., `text-embedding-3-small`) ⚠️ **Required!**

Testing

All fixes were tested locally:

1. ✅ Environment variable validation detects missing AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME
2. ✅ EmbeddingGenerator raises clear error when deployment name is missing
3. ✅ App startup logs show all configuration values
4. ✅ Startup script validates environment variables before running Python

How to Deploy the Fix to HuggingFace Spaces

Option 1: Automated Deployment (Recommended)

git add .
git commit -m "Fix: Add missing AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME validation"
git push origin main

The GitHub Actions workflow will automatically sync to HuggingFace Spaces.

Option 2: Manual Deployment

1. Push changes to your HuggingFace Space repository
2. Critical: Add the missing secret in HuggingFace Spaces:
   • Go to your Space → Settings → Repository secrets
   • Add new secret: AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME = text-embedding-3-small
   • (Or whatever your actual Azure deployment name is)
3. The Space will rebuild and start successfully

Verification

After deploying, you should see in the logs:

🔍 Checking environment variables...
✅ Found: AZURE_OPENAI_ENDPOINT
✅ Found: AZURE_OPENAI_API_KEY
✅ Found: AZURE_OPENAI_DEPLOYMENT_NAME
✅ Found: AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME
✅ All required environment variables are set!

🚀 Starting application...
2025-11-17 00:00:00,000 - app - INFO - Azure OpenAI Endpoint: https://your-resource.openai.azure.com/
2025-11-17 00:00:00,000 - app - INFO - LLM Deployment: gpt-4o-mini
2025-11-17 00:00:00,000 - app - INFO - Embedding Deployment: text-embedding-3-small
2025-11-17 00:00:00,000 - app - INFO - API Version: 2024-05-01-preview

Prevention Measures

This fix includes multiple layers of defense to prevent similar issues:

1. Example file accuracy: .env.example now matches actual required variables
2. Startup validation: App fails fast with clear error message
3. Component validation: EmbeddingGenerator validates its own requirements
4. Shell-level validation: Startup script checks before Python runs
5. Documentation: Comprehensive deployment guide with troubleshooting
6. Error messages: 404 errors now explain which deployment is missing

Files Modified

• ✅ .env.example - Fixed variable name and uncommented
• ✅ app.py - Added validate_environment() function
• ✅ rag/embeddings.py - Enhanced error messages and validation
• ✅ huggingface_startup.sh - Added environment variable checks
• ✅ README.md - Updated deployment section with required variables
• ✅ HUGGINGFACE_DEPLOYMENT.md - Created comprehensive guide (new file)
• ✅ BUGFIX_HUGGINGFACE_404.md - This document (new file)

Related Issues

• This bug only affected HuggingFace Spaces deployment
• Local development worked because .env had the correct variable set
• The issue would have been caught immediately with these validation layers

Lessons Learned

1. Always validate environment on startup - fail fast with clear errors
2. Keep .env.example in sync - it's the source of truth for deployments
3. Multi-layer validation - shell + Python + component level
4. Better error messages - 404 should explain what's missing
5. Comprehensive documentation - deployment guides prevent issues