d82a256af4cbacf759fedb6317c8f004c41b32d8
flanT5-base transformer t5

 

challenge
CAICCAIC: Centre for Artificial Intelligence Challenge on Conversational AI Correctness
submitter
s444476
submitted
2023-06-06 12:55:10.017475 UTC
original repo
https://git.wmi.amu.edu.pl/s444476/cnlps-caiccaic.git / branch master
publicly available at
git://gonito.net/cnlps-caiccaic / branch submission-08727
browsable at
https://gonito.net/gitlist/cnlps-caiccaic.git/submission-08727/
clone by
git clone --single-branch git://gonito.net/cnlps-caiccaic -b submission-08727
file basename
out

test-A / 6cd29abf9cb2d9c23c534985be571e02ecb51d26
Metric Score
Exact Match Accuracy 0.48040162193473646
Domain accuracy 0.9051940529059664
Intent accuracy 0.7450279976829504
Slot WRR 0.6850189134139751
en-US es-ES pl-PL
Exact Match Accuracy 0.4019138755980861 0.6102272727272727 0.42472810532341154
Domain accuracy 0.8343301435406698 0.9647727272727272 0.9129937034917001
Intent accuracy 0.6369617224880383 0.8590909090909091 0.7335432169433315
Slot WRR 0.31496539792387546 0.8542943382420196 0.7242647058823529

dev-A / fc8e42f0bb63b77397d12f984c11d17fbf4fdffd
Metric Score
Exact Match Accuracy 0.5586775311417065
Domain accuracy 0.9483937435609253
Intent accuracy 0.7960101152008991
Slot WRR 0.7970911329575405
en-US es-ES pl-PL
Exact Match Accuracy 0.6215249105422516 0.6156232374506486 0.43567753001715265
Domain accuracy 0.9534819708230113 0.9636209813874789 0.9276729559748428
Intent accuracy 0.782824112303881 0.8544839255499154 0.7504288164665524
Slot WRR 0.7943325497172629 0.8539451594851707 0.7176263855095972

worst items

note: the gold standard is taken from the submission itself, not from the challenge data!
# input expected output actual output test-A Exact Match Accuracy
1 1 en-US valid change the maximum temperature on my thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice {'device_name': 'task'} 0.0
2 2 en-US valid change the minimum and maximum temperatures on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice {'device_query': 'tootle'} 0.0
3 3 en-US valid adjust the minimum temperature on the thermostat Airconditioner ChangeTemperature {} Airconditioner GetTemperature {'av_alias': 'the minimum temperature'} 0.0
4 4 en-US valid adjust the minimum and maximum temperatures on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice {'device_name': 'heter', 'device_name': 'thermostat'} 0.0
5 5 en-US valid it is too cold in here Airconditioner ChangeTemperature {} Weather WeatherTomorrow {} 0.0
6 9 en-US valid i feel that it is too dry in here Airconditioner GetHumidity {} Weather WeatherTomorrow {} 0.0
7 10 en-US valid check the temperature on the keeping room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'keeping room'} Airconditioner GetTemperature {} 0.0
8 11 en-US valid check the temperature on the dining room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'dining room'} Airconditioner SetTemperatureOnDevice {'device_name': 'dining room'} 0.0
9 13 en-US valid check with the gorash thor mister says Airconditioner GetTemperatureFromDevice {} Console ConsoleCD {'pathname': 'thor mister'} 0.0
10 15 en-US valid show me the temperature on the guest room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'guest room'} Airconditioner GetTemperature {'av_alias': 'guest room'} 0.0
11 17 en-US valid show me what's on the music room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'music room'} Speaker ShowScreen {} 0.0
12 19 en-US valid tell what the adam room temperature sensor says Airconditioner GetTemperatureFromDevice {'device_name': 'adam room'} Airconditioner GetTemperatureFromDevice {'device_name': 'adam'} 0.0
13 23 en-US valid show what's the temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature {} 0.0
14 24 en-US valid show what's the temperature on my vac Airconditioner GetTemperature {'av_alias': 'vac'} Airconditioner GetTemperature {} 0.0
15 25 en-US valid show me what's the temperature on my cooling system Airconditioner GetTemperature {'av_alias': 'cooling system'} Airconditioner GetTemperature {} 0.0
16 28 en-US valid measure temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature {} 0.0
17 29 en-US valid measure temperature on my cooling system Airconditioner GetTemperature {'av_alias': 'cooling system'} Airconditioner GetTemperatureFromDevice {'device_name': 'cooling system', 'device_type': 'height'} 0.0
18 30 en-US valid measure the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner GetTemperature {} 0.0
19 31 en-US valid how many fahrenheits degrees are on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature {} 0.0
20 32 en-US valid how many celsius degrees are on my ac Airconditioner GetTemperature {'av_alias': 'ac'} Airconditioner GetTemperatureFromDevice {'device_name': 'ac', 'device_type': 'ac'} 0.0
21 33 en-US valid how many kelvins degrees are on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner GetTemperatureFromDevice {'device_name': 'hvac'} 0.0
22 34 en-US valid what temperature is it in here according to my ac Airconditioner GetTemperature {'av_alias': 'ac'} Airconditioner GetTemperatureFromDevice {'device_name': 'ac'} 0.0
23 35 en-US valid what's the temperature in here according to my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature {} 0.0
24 38 en-US valid change the temperature on my kids room thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'kids room'} Airconditioner SetTemperatureToValue {'value': 'kids room'} 0.0
25 39 en-US valid change the temperature on my nursery thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'nursery'} Airconditioner GetTemperatureFromDevice {'device_name': 'nursery'} 0.0
26 42 en-US valid set the minimum and maximum temperatures on my foyer thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'foyer'} Airconditioner SetTemperatureOnDevice {'device_name': 'facility'} 0.0
27 44 en-US valid adjust minimum and maximum temperatures on my attic thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'attic'} Airconditioner SetTemperatureOnDevice {'device_name': 'attic thermostat'} 0.0
28 47 en-US valid i want different temperature on sun room Airconditioner SetTemperatureOnDevice {'device_name': 'sun room'} Airconditioner GetTemperature {'av_alias': 'sun room'} 0.0
29 48 en-US valid i want different temperature on my library Airconditioner SetTemperatureOnDevice {'device_name': 'library'} Airconditioner GetTemperature {'av_alias': 'different temperature'} 0.0
30 49 en-US valid i want different temperature on my bathroom Airconditioner SetTemperatureOnDevice {'device_name': 'bathroom'} Airconditioner GetTemperatureFromDevice {'device_name': 'bathroom'} 0.0
31 50 en-US valid i want different temperature on my powder room Airconditioner SetTemperatureOnDevice {'device_name': 'powder room'} Airconditioner SetTemperatureOnDevice {'device_name': 'porporous'} 0.0
32 51 en-US valid i want different temperature on my laundry room Airconditioner SetTemperatureOnDevice {'device_name': 'laundry room'} Airconditioner ChangeTemperature {} 0.0
33 52 en-US valid change temperature on my game room thermostat to 15 degrees celsius Airconditioner SetTemperatureToValueOnDevice {'device_name': 'game room', 'value': '15 degrees celsius'} Airconditioner SetTemperatureOnDevice {'device_name': 'game room', 'value': '15 degrees celsius'} 0.0
34 54 en-US valid change the temperature to 80 degrees fahrenheit on my living room thermostat Airconditioner SetTemperatureToValueOnDevice {'device_name': 'living room', 'value': '80 degrees fahrenheit'} Airconditioner SetTemperatureToValueOnDevice {'device_name': 'living room thermostat', 'value': '80 degrees fahrenheit'} 0.0
35 55 en-US valid set temperature on my foyer thermostat to 29 degrees celsius Airconditioner SetTemperatureToValueOnDevice {'device_name': 'foyer', 'value': '29 degrees celsius'} Airconditioner SetTemperatureToValue {'value': '29 degrees celsius'} 0.0
36 57 en-US valid set temperature on my dining room thermostat to 82 degrees fahrenheit Airconditioner SetTemperatureToValueOnDevice {'device_name': 'dining room', 'value': '82 degrees fahrenheit'} Airconditioner SetTemperatureToValue {'value': '82 degrees fahrenheit'} 0.0
37 58 en-US valid set the temperature to 16 degrees celsius on my reception thermostat Airconditioner SetTemperatureToValueOnDevice {'device_name': 'reception', 'value': '16 degrees celsius'} Airconditioner SetTemperatureToValue {'value': '16 degrees celsius'} 0.0
38 62 en-US valid set my master bedroom thermostat to 18 degrees celsius Airconditioner SetTemperatureToValueOnDevice {'device_name': 'master bedroom', 'value': '18 degrees celsius'} Airconditioner SetTemperatureToValue {'value': '18 degrees celsius'} 0.0
39 66 en-US valid adjust temperature on my enterance hall thermostat to 17 degrees celsius Airconditioner SetTemperatureToValueOnDevice {'device_name': 'enterance hall', 'value': '17 degrees celsius'} Airconditioner SetTemperatureToValueOnDevice {'device_name': 'entrance hall', 'value': '17 degrees celsius'} 0.0
40 68 en-US valid let parlor set temperature to 24 degree Airconditioner SetTemperatureToValueOnDevice {'device_name': 'parlor', 'value': '24 degree'} Airconditioner SetTemperatureToValue {'value': '24 degree'} 0.0

Compare with other submission