Metric	Score
PerplexityHashed	447.01

Metric	Score
PerplexityHashed	399.61

worst items

note: the gold standard is taken from the submission itself, not from the challenge data!

#	input	expected output	actual output	test-A PerplexityHashed
1	02086dae55c9b8917b734d043ddf0e1d THE DICKINSON PRESS ChronAm 1892.4822404055353 46.879176 -102.789624 tariff, and demands Ibe enforcement of…	Neal	with:0.8911373019218445 and:0.015198012813925743 in:0.008982216008007526 for:0.007585650309920311 on:0.004248502664268017 of:0.0041757244616…	16638.89
2	3f439f01b13bddaec7d3ee4f4feecd9f RICHMOND ENQUIRER ChronAm 1860.2937158153713 37.538509 -77.43428 1 resume iny narrative at the point \\vl\n…	abored	of:0.8095766305923462 and:0.047601066529750824 to:0.03760799020528793 or:0.011091992259025574 in:0.008272645063698292 <unk>:0.00723240803927…	15159.47
3	3ff22426092d3a6869b2bd48364ba5f2 RAFTSMAN'S JOURNAL ChronAm 1858.9356164066464 41.02728 -78.439188 aware that the earliest and the best frui…	tages.	tage:0.6261821389198303 tages:0.24222512543201447 <unk>:0.03817729279398918 ':0.013502168469130993 a:0.0030028289183974266 e:0.0029705322813…	14819.10
4	26824b6b961482d5ad7e90c8fbcb575d THE JASPER WEEKLY COURIER ChronAm 1901.3164383244546 38.391448 -86.930874 The war department lias now sole\…	istrative	istration:0.7596465945243835 <unk>:0.1239887923002243 ':0.011399590410292149 istered:0.010142737999558449 ister:0.008653422817587852 and:0.0…	14176.47
5	06c470adf4a46c7204e72274d92deb7c THE DICKINSON PRESS ChronAm 1917.264383529934 46.879176 -102.789624 NOTICE IS HEREBY GIVEN That\nunder auth…	ter,	ter:0.7380045056343079 <unk>:0.07581037282943726 ters:0.06670046597719193 tered:0.01636340282857418 terly:0.014740810729563236 ':0.007201355…	13814.06
6	2b4578faaa5d040beefbcf8ece106e6e RICHMOND ENQUIRER ChronAm 1816.8483606241145 37.538509 -77.43428 The funds opened and shut todayatth\nsame …	tent	tem:0.7853129506111145 <unk>:0.05842067301273346 tems:0.042068079113960266 tern:0.01593857817351818 and:0.00433559063822031 i:0.001446146867…	11362.25
7	a46a5a890b809b63b0644f1f3f50d16e RICHMOND ENQUIRER ChronAm 1822.17397257103 37.538509 -77.43428 »v hat officers nave been specially appointe…	tion,	tion:0.8183668255805969 tions:0.05066613480448723 tional:0.0211029089987278 tors:0.005886088125407696 <unk>:0.004847696051001549 tlon:0.0030…	11220.46
8	483b31c888630a6712df48fe6cee758d RICHMOND ENQUIRER ChronAm 1831.2397259956874 37.538509 -77.43428 11. The said superior courts shall have th…	y	try:0.5433560609817505 ty:0.18314629793167114 ties:0.04875285178422928 tries:0.0441068634390831 cil:0.032415688037872314 <unk>:0.02775322087…	10701.88
9	d9284676477f5d3c6cf470e2aba86f6a RICHMOND ENQUIRER ChronAm 1818.1547944888382 37.538509 -77.43428 It has hcon admitted hy the exccnli e\nCou…	cil,	try:0.5433560609817505 ty:0.18314629793167114 ties:0.04875285178422928 tries:0.0441068634390831 cil:0.032415688037872314 <unk>:0.02775322087…	10701.88
10	58320d189e9d6557760c6de39f0382ff DEMOCRATIC MESSENGER ChronAm 1881.6726027080163 38.177063 -75.392696 Sec. 4. Aud be it enacted, That at\nth…	tion,	tion:0.7761617302894592 tions:0.08141963183879852 <unk>:0.006319105159491301 ted:0.006255040410906076 tive:0.005583527032285929 tlon:0.00431…	9147.57
11	0b4f47c072781044dc785153b75ff5b4 THE WYANDOT PIONEER ChronAm 1860.872950788049 40.827279 -83.281309 We are enabled here to offer the communi…	tion.	tion:0.5696092247962952 tional:0.18617555499076843 tions:0.05705728754401207 ents:0.02643231675028801 <unk>:0.022252194583415985 ency:0.0114…	8800.21
12	2c32368440ed329f43c57467551285aa RICHMOND ENQUIRER ChronAm 1848.7445354875026 37.538509 -77.43428 limns,replete with argument, wit and anec\…	ler	ter:0.477856308221817 <unk>:0.17049333453178406 teristic:0.1164984330534935 ters:0.08414627611637115 and:0.013898873701691628 ':0.0128286508…	8750.52
13	37ea1116c71eeaf35cf7aca41125dd8c THE INTERIOR JOURNAL ChronAm 1914.878082160071 37.53119 -84.661888 Miss Ola Williams and sister\nbft hist w…	ton,	ton:0.7695492506027222 and:0.030132602900266647 <unk>:0.02706936001777649 of:0.017626062035560608 ':0.015033715404570103 i:0.011859891936182…	8675.83
14	de111a1d125c8de6438046abae481278 RICHMOND ENQUIRER ChronAm 1825.9602739408929 37.538509 -77.43428 lo al irm the patriot”? With die opinions …	ng,	ing:0.8048835396766663 <unk>:0.022073639556765556 and:0.013779174536466599 ed:0.011065534316003323 ng:0.006845229305326939 the:0.00654057785…	8349.89
15	1229cf5042d1e798830034bd2b110f78 BRADFORD OPINION ChronAm 1877.9356164066464 43.994599 -72.127742 The total exports during the last fiscal y…	by	angeles:0.6712619662284851 augeles:0.0756048858165741 <unk>:0.07422136515378952 ange:0.013907477259635925 angelea:0.01339224074035883 the:0.…	8214.36
16	11a618922a619ac74b505d8e1e13f9c2 COURIER DEMOCRAT ChronAm 1911.5876712011668 48.76059 -98.367824 Order Hearing Petition for Sale of Land\nUp…	to.	to:0.7976786494255066 the:0.018697714433073997 and:0.017618244513869286 <unk>:0.010328685864806175 for:0.009677166119217873 a:0.009440399706…	8139.08
17	494648073f3e8d8b6a2581ae7d35e668 RICHMOND ENQUIRER ChronAm 1842.9493150367834 37.538509 -77.43428 Petitions were presented and referred:.By …	lolhu	to:0.7976786494255066 the:0.018697714433073997 and:0.017618244513869286 <unk>:0.010328685864806175 for:0.009677166119217873 a:0.009440399706…	8139.08
18	bcbcc8cb2d92e94cb04e79cb2a2c5271 RICHMOND ENQUIRER ChronAm 1834.582191749112 37.538509 -77.43428 ' V hen Mr. Calhoun’s dreaded rival, Mr Cra…	live	tive:0.5000432133674622 tion:0.27413588762283325 ted:0.02427772991359234 tors:0.02382214553654194 tions:0.020398208871483803 tives:0.0110284…	7849.29
19	c954306d673d88d08987736c007f2dd6 THE WYANDOT PIONEER ChronAm 1868.5396174547157 40.827279 -83.281309 warrant in the Constitution, and if\nsa…	of.	of:0.5232012271881104 to:0.2929660677909851 and:0.014834392815828323 for:0.008113366551697254 ot:0.007350444793701172 from:0.007075948640704…	7532.81
20	25a2c22ec0d42967dbd94992792887b5 THE DICKINSON PRESS ChronAm 1914.8315068176053 46.879176 -102.789624 Mr. Norton made his remarks very\ninte…	ture.	ture:0.5347313284873962 tures:0.25154998898506165 tural:0.059520866721868515 and:0.006254787556827068 tation:0.004766042344272137 for:0.0018…	7434.31
21	9a23949c6410e0a789c008623c370c36 DEMOCRAT AND SENTINEL ChronAm 1866.891780790208 40.485194 -78.724957 The present President of the United\nS…	States.	states:0.7321258187294006 <unk>:0.04590647295117378 slates:0.022088874131441116 state:0.019300131127238274 stales:0.01868295669555664 and:0.…	7424.78
22	0ba3ca99842e04d54ea7df3d3133e753 CLARKSVILLE CHRONICLE ChronAm 1861.6424657217149 36.527761 -87.35887 supposed that imprisonment for debt hu…	Mates,	states:0.7321258187294006 <unk>:0.04590647295117378 slates:0.022088874131441116 state:0.019300131127238274 stales:0.01868295669555664 and:0.…	7424.78
23	0b10beeb0214b3ac3f65555659106adf ARIZONA REPUBLICAN ChronAm 1910.6205479134958 33.448587 -112.077346 pounds',, against ,488,000,000 In 1909,…	States'	states:0.7321258187294006 <unk>:0.04590647295117378 slates:0.022088874131441116 state:0.019300131127238274 stales:0.01868295669555664 and:0.…	7424.78
24	14453ce06937921547bd704890587cbf EVENING CAPITAL ChronAm 1884.8702185476118 38.97864 -76.492786 Should your soil be new, perhaps a\npiece of…	ficient.	ficient:0.28876397013664246 frage:0.23191995918750763 fering:0.080691397190094 ficiently:0.06782117486000061 fered:0.06234200298786163 ferer…	7367.12
25	f1479cb9a6256b326867f7c18fb195e5 EVENING CAPITAL ChronAm 1923.3849314751394 38.97864 -76.492786 room dwelling. Numbered . ".5 and 37. wltl\n…	o	in:0.6063769459724426 on:0.07263907045125961 lying:0.05777180939912796 and:0.03801674768328667 at:0.03362217918038368 <unk>:0.02730324491858…	7117.95
26	1f61d0be115721333a36e20b0f91f639 THE MINERS ChronAm 1852.3811475093605 42.500622 -90.664797 The people of L.wa demanded a new set of\nlaws, …	od,	ods:0.531021773815155 od:0.1554173082113266 odist:0.1289641112089157 <unk>:0.01478900108486414 and:0.005647766869515181 of:0.005101770162582…	6668.29
27	1480d040bf88ccd74e068709857438bb GREAT FALLS DAILY TRIBUNE ChronAm 1920.8374316623658 47.504885 -111.291891 Bonds numbered 1 to 7, inclusive…	sive,	sive:0.7687647342681885 ding:0.056504588574171066 ':0.0038734630215913057 ded:0.003635011613368988 <unk>:0.003485962050035596 of:0.003249582…	6496.72
28	ee21a1f4d082e7ebf867d3506aeba084 MOUNTAIN HOME REPUBLICAN ChronAm 1925.7712328450025 43.13295 -115.691197 H. A. Beach and John W. Caldwell,\…	sive,	sive:0.7687647342681885 ding:0.056504588574171066 ':0.0038734630215913057 ded:0.003635011613368988 <unk>:0.003485962050035596 of:0.003249582…	6496.72
29	22a61f44162da23ff5d2177a9f1de2c8 ARIZONA REPUBLICAN ChronAm 1904.2827868536228 33.448587 -112.077346 General Thomas, Mr. jTeller saying he\n…	ing	ment:0.7530133128166199 ments:0.04985814541578293 and:0.009845630265772343 ed:0.007641562260687351 for:0.006329886615276337 the:0.0053061116…	6480.19
30	104c6456c01727cf0ac8932be19be006 EVENING CAPITAL ChronAm 1884.6352458700162 38.97864 -76.492786 Professor Voelker, the agricultural\nchemist…	ment,	ment:0.7530133128166199 ments:0.04985814541578293 and:0.009845630265772343 ed:0.007641562260687351 for:0.006329886615276337 the:0.0053061116…	6480.19
31	5338c42e45fa5a16f71c0bb9faddbc15 RICHMOND ENQUIRER ChronAm 1843.3027396943176 37.538509 -77.43428 At a meeting of a large and respectable po…	tion,	tionary:0.4714210331439972 tion:0.23712295293807983 tions:0.057110611349344254 tionists:0.025306696072220802 of:0.022996695712208748 <unk>:0…	6477.12
32	06f5c84e09b4a5d14a89751be629d857 ARIZONA REPUBLICAN ChronAm 1903.878082160071 33.448587 -112.077346 A wise programme, truly, and am-\nbitiou…	somewhat	to:0.7799972891807556 for:0.01980246976017952 and:0.016543468460440636 from:0.007651460822671652 by:0.005319654475897551 in:0.00492657860741…	6474.52
33	02e25922b083a8046b8e43e93d0d306c THE JASPER WEEKLY COURIER ChronAm 1908.8866119902348 38.391448 -86.930874 Close Inspection of tho election …	en	to:0.6464062929153442 the:0.061051663011312485 by:0.03687451034784317 a:0.03228673338890076 in:0.02226264961063862 for:0.015620782040059566 …	6429.46
34	531517feae16d174c438b774dddd97e2 RICHMOND ENQUIRER ChronAm 1820.1707649956993 37.538509 -77.43428 consideration of this stu ject, Mr.C. said…	r	erty:0.5479443073272705 erly:0.10661743581295013 <unk>:0.05476875230669975 osition:0.05420356243848801 er:0.037572070956230164 erties:0.0230…	6406.83
35	86e6a88018dda3cc324889dd32bb5411 RICHMOND ENQUIRER ChronAm 1846.5027396943176 37.538509 -77.43428 By way ol Brazos Santiago we received yes¬…	Antonio	':0.740774929523468 <unk>:0.029244089499115944 nelly:0.012158251367509365 the:0.01160260196775198 and:0.010706191882491112 who:0.00492379395…	5621.58
36	47760e14f5f772dc8bc8d84795614664 FREMONT WEEKLY FREEMAN ChronAm 1852.0068305694697 41.35033 -83.121863 unite with the Galena road, over whic…	nor	of:0.6610783338546753 and:0.06876680254936218 in:0.023063823580741882 <unk>:0.016289371997117996 for:0.01592222787439823 or:0.01336592156440…	5613.21
37	297520dc574a3fd9e38dccf1db713cb0 THE WICHITA CITY EAGLE ChronAm 1879.4643835299341 37.692236 -97.337545 It Ls not every man who will take th…	nance.	nance:0.7701074481010437 nances:0.014130106195807457 <unk>:0.008687980473041534 in:0.007901296950876713 and:0.004525747150182724 for:0.00259…	5457.47
38	330f9167c4ce9943c3d497c76488a1a1 CENTRE DEMOCRAT ChronAm 1880.0013660885954 40.913486 -77.773747 forming matters than almost any other\ndesc…	tion,	tion:0.514204204082489 tions:0.2047724723815918 <unk>:0.033725786954164505 tionate:0.024822842329740524 tional:0.014640451408922672 of:0.007…	5441.44
39	4d41aef961aa1d8bd1a90480f2a68d50 ARIZONA SILVER BELT ChronAm 1883.2260273655504 33.399478 -110.87095 Rfsolved, That whilst wo do not\nfor th…	That	that:0.4343715310096741 to:0.24751608073711395 by:0.049874015152454376 the:0.019418062642216682 and:0.016778605058789253 for:0.0153205869719…	5206.30
40	2bcecac3982dcdcce8877554a04cfed6 RICHMOND ENQUIRER ChronAm 1826.064383529934 37.538509 -77.43428 quire, the immense expense of yearly repair…	on	to:0.3955494165420532 by:0.28387540578842163 in:0.030016528442502022 and:0.022270428016781807 the:0.022246886044740677 a:0.01724611967802047…	4981.25