0ad3e0e81bd485db7f026a608bfae782521214c4
Transformer encoder models (MLM) transformer huggingface-transformers transformer-encoder mlm

 

challenge
GLUE-LM-GAP
submitter
kaczla
submitted
2023-02-25 16:30:48.111204 UTC
original repo
https://gitlab.com/kaczla/glue-lm-gap.git / branch encoder
publicly available at
git://gonito.net/glue-lm-gap / branch submission-08008
browsable at
https://gonito.net/gitlist/glue-lm-gap.git/submission-08008/
clone by
git clone --single-branch git://gonito.net/glue-lm-gap -b submission-08008
# model_name dev-0 PerplexityHashed test-A PerplexityHashed
151 MiniLM-L12-H384-XLMR-Large 870218.401911 876508.944709
149 MiniLM-L6-H384-RoBERTa-large 869793.785438 875278.906409
147 MiniLM-L12-H384-RoBERTa-large 869064.584648 875114.462710
145 MiniLM-L6-H768-BERT-large-uncased 869876.865963 873852.303343
143 MiniLM-L6-H768-BERT-base-uncased 871216.070328 870332.354133
141 MiniLM-L6-H384-BERT-large-uncased 869304.616131 869819.109384
139 MiniLM-L6-H384-BERT-base-uncased 872216.010612 869300.217146
137 MiniLM-L6-H768-RoBERTa-large 874622.829677 869124.251622
135 MiniLM-L6-H384-XLMR-Large 868924.084221 867987.178408
119 XLM-en 408239.470759 428281.101166
87 XLM-17-lang 40697.047197 52129.248038
82 German-BERT-base-cased 27421.852502 27859.712465
80 XLM-100-lang 13760.997821 15046.785077
76 PolishRoBERT-base 13507.011385 12559.885478
69 BERT-tiny-uncased 6677.054334 7540.606730
64 ClinicalBERT 4364.651919 4443.596582
63 SportsBERT 4562.718171 4407.714425
59 FinBERT 3268.202377 3527.824565
58 CamemBERT-base 4236.441996 3469.326138
56 BERT-mini-uncased 2471.697715 2714.984099
52 SciBERT-cased 2132.405410 2072.331054
47 SciBERT-uncased 1695.065247 1681.679705
45 BERT-small-uncased 1438.074270 1577.662051
44 ALBERT-base 1678.076625 1575.853755
42 BERT-base-multilingual-uncased 1419.618051 1477.047924
40 DistilBERT-base-uncased 1278.244697 1303.640415
38 BERT-medium-uncased 1158.238019 1260.641963
37 CodeBERT-base 1482.468166 1174.661467
35 ALBERT-large 957.703457 933.385723
33 MobileBERT-uncased 852.327720 913.525374
31 BERT-base-uncased 756.596451 804.253307
28 ALBERT-xxlarge N/A 746.990635
26 ALBERT-xlarge 739.702271 730.841384
24 BERT-large-uncased 642.869447 670.671724
22 BERT-base-multilingual-cased 709.262387 635.838112
20 DistilBERT-base-cased 563.054777 494.413642
17 BioMed-RoBERTa-base 424.568655 312.169270
15 XLM-RoBERTa-base 254.756477 193.883054
12 DistilRoBERTa-base 231.609393 179.690259
10 BERT-base-cased 198.220916 172.101095
8 XLM-RoBERTa-large 179.035358 138.830588
6 BERT-large-cased 147.667554 125.733251
4 RoBERTa-base 114.419134 91.907166
2 RoBERTa-large 86.217101 69.393009
Parameter Value
method simple
token_length 1
top_k 15