3a4e65bafa33a98974647825c931fec373f52b7d
Transformer encoder models (MLM) transformer huggingface-transformers transformer-encoder mlm

 

challenge
GLUE-LM-GAP
submitter
kaczla
submitted
2023-02-06 13:06:05.255948 UTC
original repo
https://gitlab.com/kaczla/glue-lm-gap.git / branch encoder
publicly available at
git://gonito.net/glue-lm-gap / branch submission-07994
browsable at
https://gonito.net/gitlist/glue-lm-gap.git/submission-07994/
clone by
git clone --single-branch git://gonito.net/glue-lm-gap -b submission-07994
# model_name dev-0 PerplexityHashed test-A PerplexityHashed
150 MiniLM-L12-H384-XLMR-Large 870218.401911 876508.944709
148 MiniLM-L6-H384-RoBERTa-large 869793.785438 875278.906409
146 MiniLM-L12-H384-RoBERTa-large 869064.584648 875114.462710
144 MiniLM-L6-H768-BERT-large-uncased 869876.865963 873852.303343
142 MiniLM-L6-H768-BERT-base-uncased 871216.070328 870332.354133
140 MiniLM-L6-H384-BERT-large-uncased 869304.616131 869819.109384
138 MiniLM-L6-H384-BERT-base-uncased 872216.010612 869300.217146
136 MiniLM-L6-H768-RoBERTa-large 874622.829677 869124.251622
134 MiniLM-L6-H384-XLMR-Large 868924.084221 867987.178408
118 XLM-en 408239.470759 428281.101166
86 XLM-17-lang 40697.047197 52129.248038
81 German-BERT-base-cased 27421.852502 27859.712465
79 XLM-100-lang 13760.997821 15046.785077
57 CamemBERT-base 4236.441996 3469.326138
43 ALBERT-base 1678.076625 1575.853755
41 BERT-base-multilingual-uncased 1419.618051 1477.047924
39 DistilBERT-base-uncased 1278.244697 1303.640415
36 PolishRoBERT-base N/A 1024.000000
34 ALBERT-large 957.703457 933.385723
32 MobileBERT-uncased 852.327720 913.525374
30 BERT-base-uncased 756.596451 804.253307
27 ALBERT-xxlarge N/A 746.990635
25 ALBERT-xlarge 739.702271 730.841384
23 BERT-large-uncased 642.869447 670.671724
21 BERT-base-multilingual-cased 709.262387 635.838112
19 DistilBERT-base-cased 563.054777 494.413642
14 XLM-RoBERTa-base 254.756477 193.883054
11 DistilRoBERTa-base 231.609393 179.690259
9 BERT-base-cased 198.220916 172.101095
7 XLM-RoBERTa-large 179.035358 138.830588
5 BERT-large-cased 147.667554 125.733251
3 RoBERTa-base 114.419134 91.907166
1 RoBERTa-large 86.217101 69.393009
Parameter Value
method simple
token_length 1
top_k 15