[GLJS]

This adds evaluation of audio-text (CLAP) models on MAEB, MSEB and HEAR with the existing CLI. --model_type clap loads LAION-CLAP models and --model_type clap2 loads NaFlex CLAP models through OpenCLIP. The benchmarks are the datasets maeb, mseb and hear, or single tasks such as maeb/<task>. Each runs in its own library (MTEB, MSEB, heareval), and the data is downloaded when missing. Image evaluation is unchanged, and the audio dependencies are in new audio and hear extras. Supersedes #148. The naflexclap_* pretrained tags need mlfoundations/open_clip#1230.

[GLJS]

@mehdidc can you review?

[mehdidc]

Thanks a lot @GLJS ! do you have any numbers with MAEB, MSEB and HEAR ? i'd like to try to reproduce them.
Otherwise, are there any extra changes you plan to make to the PR ?

[mehdidc]

Currently I get :

Repository Not Found for url: https://huggingface.co/laion/clap2-large-audioxl-s1B/resolve/main/open_clip_pytorch_model.bin.

with : clip_benchmark eval --model_type clap2 --model naflexclap_large --pretrained laion_audio_xl_s1b --dataset maeb

[GLJS]

Hey @mehdidc thanks for taking the time to review!
The error:

Repository Not Found for url: https://huggingface.co/laion/clap2-large-audioxl-s1B/resolve/main/open_clip_pytorch_model.bin.

should now be fixed. It was because the CLAP2 models were private hosted at first. They are now publicly released.

Here are the numbers for MAEB, MSEB and HEAR:

Model	MAEB-30	HEAR-17	MSEB
CLAP2-XXS	45.84	79.01	27.26
CLAP2-XS	48.75	81.10	28.49
CLAP2-Small	51.76	81.65	29.95
CLAP2-Medium	51.67	81.35	29.42
CLAP2-Large	53.27	81.75	30.41

Be aware, MAEB takes around 0.5 GPU h, HEAR 1.4 GPU h and MSEB 2.1 GPU h to run fully on GH200 (of course you can parallelize over multiple GPU's / nodes). The numbers in the paper are mostly the same, but sometimes differ 0.1-0.2% due to numeric differences, including an older version of scikit-learn' (its k-means for MSEB) thats incompatible with this repo (1.8.0 vs 1.9.1, thought having separate env looks ugly)

You can run with

for m in hf-hub:laion/clap2-{xxs,xs,small,medium,large}-audioxl-s1B; do
    clip_benchmark eval --model_type clap2 --model $m --dataset maeb hear mseb --batch_size 8 --output "results/${m##*/}/{dataset}.json"
done

I updated open_clip version in new commit, but apart from that, no more changes at this point :)
Let me know if you have any questions :)