Spaces:
Runtime error
Runtime error
🇨🇭 Complete Apertus Transparency Analysis Report
Generated from real A40 GPU analysis: September 7, 2025
🖥️ System Configuration
Model: swiss-ai/Apertus-8B-Instruct-2509
GPU: NVIDIA A40 (47.4 GB Memory)
Parameters: 8,053,338,176 (8.05 Billion)
Architecture: 32 layers × 32 attention heads × 4096 hidden dimensions
GPU Memory Usage: 15.0 GB
Processing Speed: 0.043s forward pass
🎯 Key Findings: Why Apertus Chooses "Unexpected" Words
📊 Sampling Parameters Revealed
🎛️ Default Settings:
Temperature: 0.7 (creativity control)
Top-P: 0.9 (nucleus sampling - 90% probability mass)
Top-K: 50 (candidate pool size)
🎲 Real Decision Process: "Die Schweizer KI-Forschung ist"
Step 1: "international" (rank 2 selected, not rank 1)
🌡️ Temperature Effect:
Without temp: Top-1 = 7.4% (fairly distributed)
With temp=0.7: Top-1 = 15.0% (more decisive)
🎯 Top Predictions:
1. ' in' → 15.0% (logit: +19.25) ✅
2. ' international' → 9.1% (logit: +18.88) ✅ ← SELECTED!
3. ' im' → 6.3% (logit: +18.62)
4. ' stark' → 4.9% (logit: +18.50)
5. ' gut' → 4.9% (logit: +18.50)
🔄 Filtering Process:
• Top-K: 131,072 → 50 candidates (99.96% reduction)
• Top-P: 50 → 27 tokens (kept 91.4% probability mass)
• Final sampling: ' international' had 10.9% chance
🎲 WHY RANK 2?
Temperature + Top-P sampling allows creative choices!
Model didn't just pick "in" (boring) but chose "international" (more interesting)
Step 2: "sehr" (rank 3 selected from very confident predictions)
🌡️ Temperature Effect:
Without temp: Top-1 = 27.5%
With temp=0.7: Top-1 = 50.4% (much more confident)
🎯 Top Predictions:
1. ' aner' → 50.4% (anerkannt = recognized) ← Expected top choice
2. ' gut' → 14.5% (good)
3. ' sehr' → 6.8% (very) ← SELECTED!
4. ' hoch' → 6.8% (high)
5. ' bekannt' → 6.0% (well-known)
🌀 Nucleus Sampling Effect:
• Only 6 tokens in nucleus (88.7% mass)
• Very focused distribution
• "sehr" still had 7.8% final probability
🎲 WHY RANK 3?
Even with high confidence, sampling diversity chose "sehr"
Creates more natural sentence flow: "international sehr angesehen"
⚖️ Native Weights Analysis: Layer 15 Attention
Query Projection (Q_proj):
📊 Shape: (4096, 4096) - Full attention dimension
📊 Parameters: 16,777,216 (16.8M - 20% of total model!)
📊 Memory: 64.0 MB
📈 Weight Health:
Mean: -0.000013 (perfectly centered!)
Std: 0.078517 (healthy spread)
Range: 2.289 (well-bounded: -1.17 to +1.12)
🕸️ Sparsity (dead weights):
|w| < 0.0001: 0.1% (almost no dead weights)
|w| < 0.01: 11.2% (mostly active weights)
|w| < 0.1: 81.4% (reasonable activation range)
🎯 Weight Distribution:
50th percentile: 0.049 (median weight)
99th percentile: 0.221 (strongest weights)
99.9th percentile: 0.340 (most critical weights)
Key vs Value Projections:
K_proj: (1024, 4096) - 4x dimensionality reduction
V_proj: (1024, 4096) - Same reduction
Key advantages: More compact, efficient
Query maintains: Full 4096 dimensions for rich queries
What this means: Apertus uses asymmetric attention - rich queries, compressed keys/values for efficiency!
🧠 Layer Evolution: From Syntax to Semantics
The Neural Journey Through 32 Layers:
Input → Layer 0: L2=4.8 (raw embeddings)
↓
Early → Layer 3: L2=18,634 (4000x increase! syntax processing)
↓
Mid → Layer 15: L2=19,863 (semantic understanding)
↓
Late → Layer 27: L2=32,627 (peak conceptual representation)
↓
Output→ Layer 30: L2=25,293 (output preparation, slight compression)
What Each Stage Does:
Layer 0 (Embeddings):
- 🔤 Raw token → vector conversion
- 📊 Sparsity: 21.6% (many inactive dimensions)
- 🎯 Focus: Technical terms ('-In', 'nov') get initial boost
Layers 3-9 (Syntax Processing):
- 🧠 Grammar and structure analysis
- 📈 Massive activation jump (4000x increase!)
- 🎯 Sentence boundaries ('.', '<s>') become dominant
- 🔍 Why: Model learns punctuation is structurally crucial
Layers 15-21 (Semantic Processing):
- 🧠 Meaning emerges beyond grammar
- 📊 Continued growth: 19K → 23K L2 norm
- 🎯 Content concepts: 'Sch' (Swiss), 'nov' (innovation)
- 🔍 Why: Model builds conceptual understanding
Layer 27 (Peak Understanding):
- 🧠 Full conceptual representation achieved
- 📊 Peak L2: 32,627 (maximum representation strength)
- 🎯 Identity focus: 'we' (Swiss context) highly active
- 🔍 Why: Complete semantic integration
Layer 30 (Output Ready):
- 🧠 Preparing for text generation
- 📉 Slight compression: 32K → 25K L2
- ⚖️ Mean goes negative: -5.16 (output pattern)
- 🎯 Structural prep: '<s>', 'K', '-In' for continuation
👁️ Real-Time Attention Patterns
Generation: "Apertus ist transparent." → "Im Interesse der"
Step 1: '.' attends to:
1. '\<s\>' (66.0%) - Strong sentence-level context
2. 'transparent' (10.5%) - Key concept
3. 'ist' (2.8%) - Grammatical anchor
→ Generates: ' Im'
Step 2: 'Im' attends to:
1. '\<s\>' (64.1%) - Maintains global context
2. '.' (4.0%) - Sentence boundary awareness
3. 'transparent' (2.5%) - Semantic connection
→ Generates: ' Interesse'
Step 3: 'Interesse' attends to:
1. '\<s\>' (63.3%) - Consistent global focus
2. 'Im' (3.3%) - Immediate context
3. '.' (3.0%) - Structural awareness
→ Generates: ' der'
Attention Insights:
- 🎯 Global Context Dominance: '<s>' gets 60-66% attention consistently
- 🔗 Semantic Connections: Strong links to key concepts ('transparent')
- 📝 Structural Awareness: Punctuation influences generation direction
- 🇩🇪 German Grammar: Perfect "Im Interesse der" construction
🔤 German Language Excellence: "Bundesgesundheitsamt"
Tokenization Comparison:
| Model | Tokens | Efficiency | Strategy |
|---|---|---|---|
| 🇨🇭 Apertus | 6 | 3.3 chars/token | Morphological awareness |
| 🤖 GPT-2 | 9 | 2.2 chars/token | Character-level splitting |
| 📚 BERT | 7 | 2.9 chars/token | Subword units |
Apertus Tokenization:
'Bundesgesundheitsamt' (20 chars) →
['B', 'undes', 'ges', 'und', 'heits', 'amt']
Morphological Analysis:
• 'B' + 'undes' = Bundes (Federal)
• 'ges' + 'und' + 'heits' = gesundheits (health)
• 'amt' = amt (office)
Vocabulary: 131,072 tokens (2.6x larger than GPT-2)
German Compound Performance:
Krankenversicherung → 5 tokens (3.8 chars/token) ✅
Rechtsschutzversicherung → 6 tokens (4.0 chars/token) ✅
Arbeitsplatzcomputer → 5 tokens (4.0 chars/token) ✅
Donaudampfschifffahrt → 9 tokens (2.3 chars/token) ⚠️ (very complex)
Why Apertus Wins at German:
- ✅ 50% more efficient than GPT-2 for compound words
- ✅ Morphological boundaries - splits at meaningful parts
- ✅ Swiss linguistic optimization - trained on German text
- ✅ Largest vocabulary - 131K vs 50K (GPT-2)
🎛️ Sampling Strategy Deep Dive
Why Models Don't Always Pick Top-1:
🌡️ Temperature = 0.7 Effect:
Original: [7.4%, 5.1%, 4.0%, 3.5%, 3.5%] (flat distribution)
With 0.7: [15.0%, 9.1%, 6.3%, 4.9%, 4.9%] (more decisive)
🌀 Top-P = 0.9 Effect:
Keeps tokens until 90% probability mass reached
Example: 131,072 total → 27 nucleus tokens (massive filtering!)
🔄 Top-K = 50 Effect:
Only considers 50 most likely tokens
Eliminates 131,022 impossible choices (99.96% reduction!)
Real Sampling Decisions:
Step 1: " international" selected from rank 2
- 🎯 Final probability: 10.9% (after filtering)
- 🎲 Why not rank 1? Creative diversity over predictability
- 🧠 Result: More interesting content than "Die Schweizer KI-Forschung ist in..."
Step 5: " ist" selected from rank 9
- 🎯 Final probability: ~2-3% (low but possible)
- 🎲 Why rank 9? High entropy (3.672) = many good options
- 🧠 Result: Grammatical continuation (though repetitive)
📊 Transparency vs Black-Box Comparison
What You See with Apertus (This Analysis):
- ✅ Every weight value in every layer
- ✅ Every attention score between every token pair
- ✅ Every probability for every possible next token
- ✅ Every sampling decision with full reasoning
- ✅ Every hidden state through all 32 layers
- ✅ Every parameter that influences decisions
What You See with ChatGPT/Claude:
- ❌ Just final output - no internal visibility
- ❌ No attention patterns - can't see focus
- ❌ No probability scores - don't know confidence
- ❌ No sampling details - don't know why choices made
- ❌ No weight access - can't inspect learned parameters
🇨🇭 Swiss AI Engineering Excellence
Model Quality Indicators:
✅ Perfect Weight Initialization:
- All layers show near-zero means (-0.000013 to +0.000024)
- Healthy standard deviations (0.073-0.079)
- No dead neurons or gradient flow problems
✅ Balanced Architecture:
- Query: Full 4096 dimensions (rich representations)
- Key/Value: Compressed 1024 dimensions (efficient computation)
- 3:1 Q:KV ratio optimizes speed vs quality
✅ Dynamic Attention Patterns:
- Consistent global context awareness (60%+ to '<s>')
- Adaptive semantic connections
- Proper German language structure handling
✅ Intelligent Sampling:
- Temperature creates controlled creativity
- Top-P ensures quality while allowing diversity
- Top-K eliminates nonsensical choices
🔍 Practical Implications
For Developers:
- 🎛️ Tune sampling params based on use case
- 📊 Monitor attention patterns for quality control
- ⚖️ Inspect weights for model health
- 🧠 Track layer evolution for optimization
For Researchers:
- 🔬 Study decision-making processes in detail
- 📈 Analyze representation learning across layers
- 🌍 Compare multilingual tokenization strategies
- 🎯 Understand sampling vs deterministic trade-offs
For End Users:
- 🤔 Understand why certain responses are generated
- 🎲 See confidence levels for each prediction
- 👁️ Know what the model is "paying attention to"
- 📊 Trust through transparency instead of blind faith
🎯 The "Rank 2/9 Selection" Phenomenon Explained
This is NOT a bug - it's a FEATURE:
Why Apertus chooses non-top-1:
- 🎨 Creative Diversity: Pure top-1 selection creates boring, repetitive text
- 🎲 Controlled Randomness: Temperature + Top-P balance quality with creativity
- 🧠 Human-like Choice: Humans don't always say the most obvious thing
- 📚 Rich Training: Model knows many valid continuations, not just one "correct" answer
- 🇩🇪 Linguistic Richness: German especially benefits from varied expression
Quality Metrics Prove It Works:
- Average confidence: 41.0% - Strong but not overconfident
- Generation quality: High - Despite not always picking rank 1
- Proper German grammar - All selections are linguistically correct
- Coherent meaning - "international sehr angesehen" makes perfect sense
🇨🇭 Conclusion: True AI Transparency
This analysis proves that Apertus delivers unprecedented transparency:
- 🔍 Complete Visibility: Every computation is accessible
- 📊 Real Data: All numbers come directly from model calculations
- 🧠 Understandable AI: Complex decisions broken down step-by-step
- 🎯 Swiss Precision: Detailed, accurate, reliable analysis
- 🌍 Language Excellence: Superior German and multilingual handling
The future of AI is transparent, and Apertus leads the way. 🇨🇭✨
This report contains 100% real data from swiss-ai/Apertus-8B-Instruct-2509 running on NVIDIA A40.