Update README.md

README.md

@@ -1,199 +1,183 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- audio
+- spatial
+- speech
+license: mit
+datasets:
+- agkphysics/AudioSet
+metrics:
+- accuracy
+pipeline_tag: feature-extraction
 ---
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
+GRAM (General Purpose Audio Representation Model) is trained on AudioSet with newly proposed naturalistic training methadology.
+GRAMs utilize MWMAE (Multi-window multi-head attention), and RIR augmentations to achieve state-of-the-art results on downstream tasks such as FSD50K, ESC50, VL even when conditions are very adverse. What is different about GRAMs is the spatial reasoning capabilities, and the robustness to noise and reverberation.
+GRAMs natively support binaural localization, and ambisonics localization tasks unlike other models in the literature. Therefore, GRAMs can be used for understanding spatial scenes as well as classifying sounds, or recognizing speech in very noisy and reverberant environments.
 
 
 ## Model Details
 
+The GRAM learns spatial audio representation by reconstructing multi-channel masked spectrogram
+patches. First, a patch extractor consisting of a single convolutional layer with 2D convolutional
+filters divides each multi-channel spectrogram into n non-overlapping patches. Nonmasked patch embeddings are input to the encoder, for which we selected the 12-layer ViT-Base
+(ViT-B) Transformer (Dosovitskiy et al., 2021) similar to Huang et al. (2022); Yadav et al. (2024).
+The encoder outputs patch representations that can be further used for fine-tuning.
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+GRAM is a self-supervised, multi-channel masked auto-encoder model that efficiently learns spatial general-purpose audio representations
+from simulated real-world sound scenes. To train GRAM, we developed a custom pipeline which
+makes use of the Soundspace 2.0 platform (Chen et al., 2022a) to simulate high-quality real-world
+sound scenes from AudioSet (Gemmeke et al., 2017), and of WHAMR! (Maciejewski et al., 2020)
+for adding background noise. We present two versions of GRAM to ensure flexible application across audio formats:
+GRAM-Binaural for two-channel audio clips, and GRAM-Ambisonics for four-channel audio clips in
+the first-order Ambisonics format.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
 
-### Model Sources [optional]
+- **Developed by:** Goksenin Yuksel, [email protected]
+- **Model type:** Transformers, Audio Foundation Models
+- **Language(s) (NLP):** GRAMs support all languages, but mainly English.
+- **License:** MIT
 
-<!-- Provide the basic links for the model. -->
+### Model Sources
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/labhamlet/GRAM-T
+- **Paper:** https://arxiv.org/abs/2506.00934
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+GRAMs can be used as a powerful feature extractor for downstream tasks such as enviromental sound classification, speech recognition, speaker counting, sound localization. 
+Later, training a linear head on top of these extracted features would yield a fine-tuned audio scene analysis model.
 
-[More Information Needed]
 
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
+## How to Get Started with the Model
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+GRAMs have three strategies to choose from; "raw", "mean" or "cls". We advise to use "raw" strategy as this prooduces embeddings over the time frames, and is more robust to silent parts.
 
-## How to Get Started with the Model
+Raw : Break audio clips into non-overlapping 2 second chunks, concatenating the features in time and finally taking a mean over the time axis to generate a fixed vector representation independent of the input audio duration
+Mean : Break audio clips into non-overlapping 2 second chunks, and taking a mean over all the patches to generate a fixed vector representation independent of the input audio duration
+Cls : Break audio clips into non-overlapping 2 second chunks, and return the "[CLS]" token representation.
+~~~python
+from transformers import AutoModel, AutoFeatureExtractor
 
-Use the code below to get started with the model.
+model = AutoModel.from_pretrained("labhamlet/gramt-mono", trust_remote_code=True)
+extractor = AutoFeatureExtractor.from_pretrained("labhamlet/gramt-mono", trust_remote_code=True)
 
-[More Information Needed]
+audio = torch.zeros([1,320000])
+extracted = extractor(audio, return_tensors="pt")
+log_mel = extracted['input_values']
+print(model(log_mel, strategy = "raw").shape)
+~~~
 
 ## Training Details
 
+
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The 85,000 naturalistic scenes were split into a train set of 70,000 scenes (corresponding to 70 Matterport3D houses), and a test set of 15,000 scenes (15
+Matterport3D houses) for down-stream evaluation (see Section 3.4). We used the 70,000 naturalistic
+scenes in the train set to generate naturalistic scenes for all audio clips in the unbalanced training
+set of AudioSet (10-second sound tracks of 1.74 million YouTube videos (Gemmeke et al., 2017)).
+Specifically, during training we randomly paired an AudioSet clip with a noise sound clip from the
+WHAMR! background noise database (Maciejewski et al., 2020). WHAMR! noise clips longer than
+10 s were trimmed to 10 s duration and a linear fade-in/fade-out of 200 ms was applied to every noise
+clip prior to mixing of the sound scene.
+To create a naturalistic sound scene, we then convolved the AudioSet clip either with BRIR(s, r, θ)
+for GRAM-Binaural, or with a ARIR(s, r, θ) for GRAM-Ambisonics, to obtain T. Similarly, we
+convolved the WHAMR! noise clip with the to obtain naturalistic scenes.
 
-[More Information Needed]
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+We transformed the channels of each sound scene (i.e., the waveforms) into logscale mel spectrograms using 128 mel filters in the frequency range of 50-16000 Hz with a 25 ms
+Hanning window and 10 ms hop length, resulting in spectrograms of dimension 1024 × 128. For
+GRAM-Ambisonic, we extracted normalized active Intensity Vectors (IVs) from the spectrograms
+as additional input features encoding spatial information. We concatenated mel
+spectrograms and intensity vectors, resulting in input x = [xmel, IV s] for each naturalistic scene
+generated from an AudioSet clip. In-batch sampling: As the online mixing of naturalistic acoustic scenes is computationally expensive
+due to multiple long convolutions, we used a random in-batch sampling procedure to increase the
+effective batch size in a computationally efficient manner. We randomly sampled 16 partially
+overlapping segments of 2 seconds to create 16 samples of dimension 200 × 128. This increases the
+original batch size of 96 to an effective batch size of 1536. 
 
-#### Preprocessing [optional]
+For pre-training, we divided the binaural spectrogram into 2×8×16,ambisonics spectrograms into 7×8×16 patches. We used an adapted version
+of the mask-based framework of MW-MAE (Yadav et al., 2024), randomly selecting a subset of
+n patches M1, . . . , Mn for i = 1, . . . , n for masking (masking ratio = 0.8) and replacing their
+embedding with a learnable mask token. Finally, we added fixed sinusoidal positional embeddings to
+all embedded patches.
 
-[More Information Needed]
 
+We trained all GRAMs for 500 K steps on an H100 92 GB GPU machine
+with 16 CPU cores. We used the AdamW optimizer (Loshchilov & Hutter, 2017) with weight decay
+rate of 0.01, gradient clipping, and a cosine learning rate scheduler with 10 K steps warm-up. The
+initial learning rate was set to 0.0002, and decayed to 0. We optimize the mean squared error (MSE)
+loss function between the predicted masked patches and their corresponding input spectrogram
+patches.
 
-#### Training Hyperparameters
+#### Preprocessing
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+Firstly, RMS Normalization was applied to audio clips to get all of them in the same loudness levels. Later, instance normalization was applied to the convolved scenes
 
-#### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Training Hyperparameters
 
-[More Information Needed]
+- **Training regime:**: GRAMs were trained with mixed precision, torch.compile and flash attention.
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
+We evaluate GRAM and other state-of-the-art models on the HEAR benchmark
+task suite, which presents a wide range of tasks to evaluate the downstream performance of audio
+representation models (Turian et al., 2022). We additionally evaluated performance on simulated real-worldsound scenes using Nat-HEAR.
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
+**HEAR**:  The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. 
+HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. 
+HEAR was launched as a NeurIPS 2021 shared challenge. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+**NatHEAR**: Orovides a naturalistic version of all selected datasets in the HEAR benchmark suite in
+two audio formats: a two-channel, binaural format and a four-channel, first-order Ambisonics
+format. Weincluded sound localization tasks for two different domains which we generated using
+HEAR benchmark datasets: A speech localization task based on SC-5, and an environmental sound
+localization task based on ESC-50. The localization tasks are modeled as a multi-output regression
+task in which model outputs represent the estimated 3D Cartesian coordinates [x, y, z] on the unit
+sphere (Adavanne et al., 2018). Finally, to assess the transferability of GRAM to real-world sound
+scenes, we evaluate also on the sound event detection and localization tasks in TUT Sound Events
+2018 REAL (Adavanne et al., 2019)
 
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
 
 ### Results
 
-[More Information Needed]
-
-#### Summary
-
-
 
-## Model Examination [optional]
+| Model | DCASE | FSD50K | LC | ESC-50 | CD | VL | SC-5 | NS | BO | Mri-S | Mri-T | s(m) |
+|-------|-------|--------|-------|--------|-------|-------|------|-----|-------|-------|-------|------|
+| HEAR-Naive | 8.8 | 13.2 | 43.5 ± 1.6 | 28.6 ± 3.1 | 38.0 ± 2.3 | 14.8 ± 3.0 | 13.3 | 87.6 | 98.7 ± 1.9 | 94.1 ± 0.5 | 87.6 ± 6.4 | 0.0 |
+| Wav2Vec 2.0 | 23.5 | 29.4 | 69.9 ± 2.1 | 46.4 ± 1.8 | 57.3 ± 1.1 | 34.9 ± 2.4 | 85.3 | 17.4 | 81.4 ± 4.8 | 90.7 ± 0.8 | 77.0 ± 0.9 | 31.5 |
+| HuBERT | 78.3 | 32.8 | 63.3 ± 1.2 | 58.6 ± 2.8 | 71.2 ± 1.2 | 65.2 ± 2.9 | 94.0 | 19.8 | 93.2 ± 5.9 | 94.6 ± 0.4 | 85.0 ± 2.5 | 44.7 |
+| WavLM | 27.0 | 25.7 | 61.3 ± 2.3 | 49.5 ± 3.8 | 64.3 ± 1.3 | 60.1 ± 3.2 | 93.8 | 18.2 | 84.3 ± 6.3 | 88.8 ± 1.0 | 76.8 ± 0.5 | 36.8 |
+| MAE | – | 33.4 | 62.3 ± 1.1 | 72.9 ± 2.1 | 60.8 ± 1.8 | 21.3 ± 5.8 | 66.6 | 63.6 | 94.5 ± 5.6 | 94.8 ± 0.6 | 85.1 ± 10.4 | 32.7 |
+| SSAST* | – | 21.4 | 57.8 ± 3.3 | 58.3 ± 2.6 | 48.0 ± 2.1 | 15.4 ± 2.6 | 22.0 | 64.2 | 95.8 ± 4.3 | 90.2 ± 5.9 | 89.1 ± 8.0 | 15.8 |
+| BEATs | – | 54.1 | 77.8 ± 1.2 | 85.8 ± 2.9 | 66.9 ± 2.5 | 39.7 ± 4.3 | 86.9 | 68.6 | 94.1 ± 3.5 | 95.5 ± 0.4 | 96.6 ± 0.5 | 61.4 |
+| MW-MAE | 94.2 | 51.8 | 80.3 ± 1.9 | 82.2 ± 3.2 | 74.4 ± 1.5 | 45.5 ± 1.7 | 91.6 | 69.4 | 95.8 ± 4.3 | 97.5 ± 0.4 | 97.6 ± 0.6 | 71.0 |
+| SSAM | 87.3 | 53.5 | 75.5 ± 1.4 | 82.9 ± 3.6 | 70.2 ± 0.4 | 56.4 ± 5.2 | 89.3 | 72.6 | 93.2 ± 3.5 | 97.8 ± 0.5 | 96.9 ± 0.5 | 71.2 |
+| **GRAM-Binaural** | 95.6 | 56.1 | 81.0 ± 1.1 | 86.7 ± 2.4 | 75.0 ± 1.4 | 53.2 ± 3.0 | 92.5 | 77.0 | 94.9 ± 3.2 | 97.3 ± 0.3 | 98.1 ± 0.2 | 74.6 |
+| **GRAM-Ambisonics** | 94.3 | 53.0 | 79.4 ± 1.5 | 85.9 ± 1.5 | 71.9 ± 1.9 | 53.7 ± 1.2 | 89.6 | 73.8 | 94.9 ± 4.9 | 97.6 ± 0.5 | 98.5 ± 0.4 | 73.4 |
+| **GRAM-Mono** | 95.3 | 56.8 | 81.3 ± 1.8 | 87.5 ± 2.3 | 75.1 ± 0.6 | 57.3 ± 3.4 | 93.5 | 75.8 | 95.8 ± 3.7 | 97.4 ± 0.3 | 98.0 ± 0.2 | 76.1 |
 
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
+#### Summary
 
-[More Information Needed]
+We present a General-purpose, Real-world Audio representation Model (GRAM), which learns spatial
+audio representations using a multi-channel masked auto-encoder approach. GRAM demonstrates
+remarkable performance in naturalistic sound scenes as well as clean sound scenes, surpassing all
+state-of-the-art self-supervised spectrogram-based audio foundation models while requiring only a
+fraction of the training data. Moreover, GRAM is the first audio foundation model that is available
+in both a two-channel, binaural format and a four-channel, first-order ambisonics format
 
 ## Model Card Contact
 
-[More Information Needed]
+Goksenin Yuksel; [email protected]