612e18aad25aedcd1791573dbb89b22be97cd114
mbart large huggingface-transformers mbart-large-50

 

challenge
CAICCAIC: Centre for Artificial Intelligence Challenge on Conversational AI Correctness
submitter
s444386
submitted
2023-06-11 15:44:34.219783 UTC
original repo
https://git.wmi.amu.edu.pl/s444386/cnlps-caiccaic.git / branch mbart-large
publicly available at
git://gonito.net/cnlps-caiccaic / branch submission-08850
browsable at
https://gonito.net/gitlist/cnlps-caiccaic.git/submission-08850/
clone by
git clone --single-branch git://gonito.net/cnlps-caiccaic -b submission-08850
file basename
out

test-A / fad6f139f1ee677b9d3008531749eda4cc3b93c4
Metric Score
Exact Match Accuracy 0.7537169337709982
Domain accuracy 0.962927206024329
Intent accuracy 0.9159104074145588
Slot WRR 0.8715758407116432
en-US es-ES pl-PL
Exact Match Accuracy 0.569377990430622 0.8835227272727273 0.7993703491700057
Domain accuracy 0.9078947368421053 0.9928977272727273 0.9854035489410418
Intent accuracy 0.8142942583732058 0.9747159090909091 0.953921007441328
Slot WRR 0.6580449826989619 0.9555016921247599 0.914692265795207

dev-A / 1ce7761af1a120b81f28bb182c506f3f448a6011
Metric Score
Exact Match Accuracy 0.8239205769410883
Domain accuracy 0.9855764727919828
Intent accuracy 0.9621616558958509
Slot WRR 0.9226995517420497
en-US es-ES pl-PL
Exact Match Accuracy 0.791907514450867 0.8877608573040046 0.7924528301886793
Domain accuracy 0.9779796311588219 0.9906937394247038 0.9882790165809033
Intent accuracy 0.9540324800440407 0.9737732656514383 0.9588336192109777
Slot WRR 0.8917974458351865 0.9577504196978176 0.904771559881049

worst items

note: the gold standard is taken from the submission itself, not from the challenge data!
# input expected output actual output test-A Exact Match Accuracy
1 5 en-US valid it is too cold in here Airconditioner ChangeTemperature {} Weather OpenWeather {} 0.0
2 9 en-US valid i feel that it is too dry in here Airconditioner GetHumidity {} Airconditioner GetTemperature {} 0.0
3 13 en-US valid check with the gorash thor mister says Airconditioner GetTemperatureFromDevice {} Twitter FollowUser {'username': 'gorash thor'} 0.0
4 19 en-US valid tell what the adam room temperature sensor says Airconditioner GetTemperatureFromDevice {'device_name': 'adam room'} Airconditioner GetTemperature {'av_alias': 'adam room'} 0.0
5 24 en-US valid show what's the temperature on my vac Airconditioner GetTemperature {'av_alias': 'vac'} Weather WeatherInLocation {'location': 'vac'} 0.0
6 32 en-US valid how many celsius degrees are on my ac Airconditioner GetTemperature {'av_alias': 'ac'} Airconditioner GetTemperature {} 0.0
7 34 en-US valid what temperature is it in here according to my ac Airconditioner GetTemperature {'av_alias': 'ac'} Airconditioner GetTemperature {} 0.0
8 36 en-US valid it is too chilly in here Airconditioner GetTemperature {} Weather OpenWeather {} 0.0
9 48 en-US valid i want different temperature on my library Airconditioner SetTemperatureOnDevice {'device_name': 'library'} Airconditioner SetTemperatureOnDevice {'location': 'library'} 0.0
10 52 en-US valid change temperature on my game room thermostat to 15 degrees celsius Airconditioner SetTemperatureToValueOnDevice {'device_name': 'game room', 'value': '15 degrees celsius'} Weather SetTemperatureToValueOnDevice {'device_name': 'game room', 'value': '15 degrees celsius'} 0.0
11 54 en-US valid change the temperature to 80 degrees fahrenheit on my living room thermostat Airconditioner SetTemperatureToValueOnDevice {'device_name': 'living room', 'value': '80 degrees fahrenheit'} Weather SetTemperatureToValueOnDevice {'device_name': 'living room', 'value': '80 degrees fahrenheit'} 0.0
12 55 en-US valid set temperature on my foyer thermostat to 29 degrees celsius Airconditioner SetTemperatureToValueOnDevice {'device_name': 'foyer', 'value': '29 degrees celsius'} Weather SetTemperatureToValueOnDevice {'device_name': 'foyer', 'value': '29 degrees celsius'} 0.0
13 63 en-US valid modify temperature on my master bedroom thermostat to 81 degrees Airconditioner SetTemperatureToValueOnDevice {'device_name': 'master bedroom', 'value': '81 degrees'} Weather SetTemperatureToValueOnDevice {'device_name': 'master bedroom', 'value': '81 degrees'} 0.0
14 65 en-US valid adjust the temperature to 25 degree on my adam room thermostat Airconditioner SetTemperatureToValueOnDevice {'device_name': 'adam room', 'value': '25 degree'} Weather SetTemperatureToValueOnDevice {'device_name': 'adam room', 'value': '25 degree'} 0.0
15 66 en-US valid adjust temperature on my enterance hall thermostat to 17 degrees celsius Airconditioner SetTemperatureToValueOnDevice {'device_name': 'enterance hall', 'value': '17 degrees celsius'} Weather SetTemperatureToValueOnDevice {'device_name': 'enterance hall', 'value': '17 degrees celsius'} 0.0
16 71 en-US valid set the temperature on my thermostat to 16 degrees celsius Airconditioner SetTemperatureToValue {'value': '16 degrees celsius'} Weather SetTemperatureToValue {'value': '16 degrees celsius'} 0.0
17 72 en-US valid set the temperature to 22 degrees celsius on my thermostat Airconditioner SetTemperatureToValue {'value': '22 degrees celsius'} Weather SetTemperatureToValue {'value': '22 degrees celsius'} 0.0
18 73 en-US valid set my thermostat to 20 degrees celsius Airconditioner SetTemperatureToValue {'value': '20 degrees celsius'} Weather SetTemperatureToValue {'value': '20 degrees celsius'} 0.0
19 76 en-US valid modify my thermostat to 17 degree Airconditioner SetTemperatureToValue {'value': '17 degree'} Weather SetTemperatureToValue {'value': '17 degree'} 0.0
20 78 en-US valid adjust the temperature to 29 degrees celsius on my thermostat Airconditioner SetTemperatureToValue {'value': '29 degrees celsius'} Weather SetTemperatureToValue {'value': '29 degrees celsius'} 0.0
21 79 en-US valid 20 degrees celsius would be ideal temperature because it is too cold in here Airconditioner SetTemperatureToValue {'value': '20 degrees celsius'} Weather SetTemperatureToValue {'value': '20 degrees celsius'} 0.0
22 81 en-US valid turn the ac off Airconditioner TurnOff {'av_alias': 'ac'} Speaker TurnOnOrOff {'power': 'ac'} 0.0
23 102 en-US valid there is lunch with isla paveglio May 20th and i won't be available Calendar AddEventOnDateWithName {'date': 'May 20th', 'event_name': 'lunch with isla paveglio'} Calendar AddEventOnDateWithName "{'date': ""May 20th and i won't be available"", 'event_name': 'lunch with isla paveglio'}" 0.0
24 103 en-US valid there is council meeting July 2nd so i will be available Calendar AddEventOnDateWithName {'date': 'July 2nd', 'event_name': 'council meeting'} Calendar CheckCalendarOnDate {'date': 'July 2nd'} 0.0
25 123 en-US valid check my meeting flight to carter Calendar CheckCalendarEventName {'event_name': 'flight to carter'} Calendar CheckCalendarOnDate {'date': 'flight to carter'} 0.0
26 126 en-US valid check details of my appointment meeting with susanna okimoto Calendar CheckCalendarEventName {'event_name': 'meeting with susanna okimoto'} Calendar CheckCalendarEventName {'event_name': 'meeting with susanna'} 0.0
27 128 en-US valid show me lunch with narcissa coppolino appointment to me Calendar CheckCalendarEventName {'event_name': 'lunch with narcissa coppolino'} Calendar AddEventWithName {'event_name': 'lunch with narcissa coppolino'} 0.0
28 132 en-US valid display details of lunch with jay nieratko appointment Calendar CheckCalendarEventName {'event_name': 'lunch with jay nieratko'} Calendar AddEventWithName {'event_name': 'lunch with jay nieratko'} 0.0
29 141 en-US valid tell me wits in myagend on ninth november twenty twenty three Calendar CheckCalendarOnDate {} Websearch SearchTextOnEngineOnEngine {} 0.0
30 147 en-US valid give me my appointments in 1 minute Calendar CheckCalendarOnDate {'date': 'in 1 minute'} Calendar CheckCalendarInLocation {'location': '1'} 0.0
31 149 en-US valid did i forget anything today Calendar CheckCalendarOnDate {} Calendar OpenCalendar {} 0.0
32 150 en-US valid give me information about incoming events Calendar CheckCalendarOnDate {} Calendar CheckCalendarEventName {} 0.0
33 151 en-US valid give me information about my events Calendar CheckCalendarOnDate {} Calendar OpenCalendar {} 0.0
34 153 en-US valid don't tell me when a meeting in the calendar in shelbyville begins Calendar NotNotifyOnEventInLocation {'location': 'shelbyville'} Calendar NotNotifyOnEventInLocation {'location': 'hellyville'} 0.0
35 164 en-US valid don't inform me when an appointment in the calendar in sittingbourne begins Calendar NotNotifyOnEventInLocation {'location': 'sittingbourne'} Calendar NotNotifyOnEventInLocation {'location': 'sitchingbourne'} 0.0
36 173 en-US valid shut up when a meeting in my calendar in spennymoor begins Calendar NotNotifyOnEventInLocation {'location': 'spennymoor'} Calendar NotNotifyOnEventInLocation {'location': ' spennymoor'} 0.0
37 202 en-US valid tell me when an event in the calendar in location nettie begins Calendar NotifyOnEventInLocation {'location': 'nettie'} Calendar NotifyOnEventInLocation {'location': 'nettine'} 0.0
38 203 en-US valid notify me when an appointment in my calendar in location molena begins Calendar NotifyOnEventInLocation {'location': 'molena'} Calendar NotifyOnEventInLocation {'location': 'molona'} 0.0
39 204 en-US valid notify me when an appointment in the calendar in sutter begins Calendar NotifyOnEventInLocation {'location': 'sutter'} Calendar NotifyOnEventInLocation {'location': 'suppter'} 0.0
40 219 en-US valid update me when an appointment in the calendar in location thornton begins Calendar NotifyOnEventInLocation {'location': 'thornton'} Calendar NotifyOnEventInLocation {'location': 'tornton'} 0.0

Compare with other submission