2f2fb969c0cd3b809c2ced15531a91127ac884c7
Flan-t5-small transformer small t5

 

challenge
CAICCAIC: Centre for Artificial Intelligence Challenge on Conversational AI Correctness
submitter
s444415
submitted
2023-06-03 18:15:07.852865 UTC
file basename
out

test-A / 75c301190b8fa8d5f4f30bb4bff27f85fba79bd2
Metric Score
Exact Match Accuracy 0.10648773894574243
Domain accuracy 0.7324773122224367
Intent accuracy 0.3582737980305078
Slot WRR 0.2869019412229289
en-US es-ES pl-PL
Exact Match Accuracy 5.651913875598086e-2 0.18693181818181817 7.326846021751574e-2
Domain accuracy 0.7090311004784688 0.846875 0.6396680022896394
Intent accuracy 0.3104066985645933 0.47244318181818185 0.28906697195191755
Slot WRR -9.359861591695506e-2 0.5074544955638891 0.2580337690631809

dev-A / 5d35c6d09ea2e197730fe360be636b6c7752fb16
Metric Score
Exact Match Accuracy 0.12597171490118947
Domain accuracy 0.7808373138522057
Intent accuracy 0.38175517467453407
Slot WRR 0.3966024740760692
en-US es-ES pl-PL
Exact Match Accuracy 0.11395540875309662 0.1897913141567964 7.375643224699828e-2
Domain accuracy 0.8329204514175612 0.853355893965031 0.6532304173813608
Intent accuracy 0.37462152491054224 0.46841511562323745 0.30131503716409375
Slot WRR 0.3797572908062774 0.5093266181682522 0.2511489591781563

worst items

note: the gold standard is taken from the submission itself, not from the challenge data!
# input expected output actual output test-A Exact Match Accuracy
1 2 en-US valid change the minimum and maximum temperatures on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice {'device_name': 'diamond'} 0.0
2 3 en-US valid adjust the minimum temperature on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureToValue {'value': '0'} 0.0
3 4 en-US valid adjust the minimum and maximum temperatures on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureToValue {'value': 'minium and maximum temperatures'} 0.0
4 5 en-US valid it is too cold in here Airconditioner ChangeTemperature {} Weather Weather's {WeatherTemperature } 0.0
5 6 en-US valid check the humidity level Airconditioner GetHumidity {} Airconditioner CheckTemperature {} 0.0
6 7 en-US valid tell me how humid is the air in the room Airconditioner GetHumidity {} Airconditioner SetTemperature {} 0.0
7 8 en-US valid probe the humidity Airconditioner GetHumidity {} Airconditioner SetTemperature {} 0.0
8 9 en-US valid i feel that it is too dry in here Airconditioner GetHumidity {} Websearch SearchImagesWithTextOnEngineWithWidthAndWithHeight {'img_query': 'too dry'} 0.0
9 10 en-US valid check the temperature on the keeping room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'keeping room'} Airconditioner SetTemperatureOnDevice {'device_name': 'keep room'} 0.0
10 11 en-US valid check the temperature on the dining room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'dining room'} Airconditioner CheckTemperatureOnDevice {'device': 'dinner room'} 0.0
11 12 en-US valid check what the home gym temperature sensor says Airconditioner GetTemperatureFromDevice {'device_name': 'home gym'} Airconditioner SetTemperature {} 0.0
12 13 en-US valid check with the gorash thor mister says Airconditioner GetTemperatureFromDevice {} Yelp SearchByCategory {'category': 'gorash thor mister', 'to': 'gorash thor'} 0.0
13 14 en-US valid show me the temperature on the master bedroom sensor Airconditioner GetTemperatureFromDevice {'device_name': 'master bedroom'} Airconditioner ShowTemperatureToDate {'date': 'master bedroom'} 0.0
14 15 en-US valid show me the temperature on the guest room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'guest room'} Airconditioner ShowTemperatureOnDevice {'device_name': 'guessroom', 'device_name': 'trg_guest room'} 0.0
15 16 en-US valid show me the temperature on the game room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'game room'} Airconditioner OpenTemperature {} 0.0
16 17 en-US valid show me what's on the music room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'music room'} Spotify PlaySong {'song': 'music room sensor'} 0.0
17 18 en-US valid show the temperature on the living room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'living room'} Airconditioner SetTemperatureToValue {'value_value': '0'} 0.0
18 19 en-US valid tell what the adam room temperature sensor says Airconditioner GetTemperatureFromDevice {'device_name': 'adam room'} Airconditioner SetTemperatureToValue {'value': 'adam'} 0.0
19 20 en-US valid give me the temperature of the family room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'family room'} Weather CheckTemperature {} 0.0
20 21 en-US valid check the temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner SetTemperatureToValue {} 0.0
21 22 en-US valid check the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner GetTemperatureOnEngine {'device_name': 'hvac'} 0.0
22 23 en-US valid show what's the temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner ShowTemperature {'value': 'lower'} 0.0
23 24 en-US valid show what's the temperature on my vac Airconditioner GetTemperature {'av_alias': 'vac'} Airconditioner ShowTemperatureOnDevice {'device_name': 'vacuum'} 0.0
24 25 en-US valid show me what's the temperature on my cooling system Airconditioner GetTemperature {'av_alias': 'cooling system'} Airconditioner SetTemperatureOn {} 0.0
25 26 en-US valid show me what's the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner ShowTemperatureOnEngine {} 0.0
26 27 en-US valid tell me the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner SetTemperatureOnDevice {'device_name': 'hvac'} 0.0
27 28 en-US valid measure temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner {} 0.0
28 29 en-US valid measure temperature on my cooling system Airconditioner GetTemperature {'av_alias': 'cooling system'} Airconditioner SetTemperatureOnDevice {'device_name': 'cooler system'} 0.0
29 30 en-US valid measure the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner SetTemperatureOnDevice {'device_name': 'hvac'} 0.0
30 31 en-US valid how many fahrenheits degrees are on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner SetTemperatureToValue {'value': 'air'} 0.0
31 32 en-US valid how many celsius degrees are on my ac Airconditioner GetTemperature {'av_alias': 'ac'} Websearch SearchTextOnEngine {'txt_query': 'cibernicius degrees'} 0.0
32 33 en-US valid how many kelvins degrees are on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner SetTemperatureToValue {'value': 'kelvin', 'value': 'hvac'} 0.0
33 34 en-US valid what temperature is it in here according to my ac Airconditioner GetTemperature {'av_alias': 'ac'} Airconditioner SetTemperatureOnDate {'date': 'it'} 0.0
34 35 en-US valid what's the temperature in here according to my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner FindTemperature {} 0.0
35 36 en-US valid it is too chilly in here Airconditioner GetTemperature {} Weather 'weather': {'height'} 0.0
36 37 en-US valid turn up the temperature above 23 degree but lower than and 29 degree on my thermostat Airconditioner SetTemperatureBetweenValues {} Airconditioner TurnOnTemperatureToValue {'value': '23 degree', 'value': 'and 29 degree'} 0.0
37 38 en-US valid change the temperature on my kids room thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'kids room'} Airconditioner SetTemperatureOnDate {'date': 'children'} 0.0
38 39 en-US valid change the temperature on my nursery thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'nursery'} Airconditioner ChangeTemperatureFromDevice {'device_name': 'canceled', 'device_name': 'cairry'} 0.0
39 40 en-US valid change the minimum and maximum temperatures on my home gym thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'home gym'} Airconditioner SetTemperatureOnDevice {'from': 'house'} 0.0
40 41 en-US valid set the temperature on my max room thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'max room'} Airconditioner SetTemperatureOnDestal {'device_name': 'max room'} 0.0

Compare with other submission