f36437a6227e2095b95d0e16e65db1d990f81d96
flanT5-base transformer t5

 

challenge
CAICCAIC: Centre for Artificial Intelligence Challenge on Conversational AI Correctness
submitter
s444476
submitted
2023-06-06 11:57:51.81867 UTC
file basename
out

test-A / 0e28298d3fb25ccd1df4b9ef161d8e42de9e2691
Metric Score
Exact Match Accuracy 0.0
Domain accuracy 0.0
Intent accuracy 0.0
Slot WRR 0.0
en-US es-ES pl-PL
Exact Match Accuracy 0.0 0.0 0.0
Domain accuracy 0.0 0.0 0.0
Intent accuracy 0.0 0.0 0.0
Slot WRR 0.0 0.0 0.0

dev-A / 74c5f12700ed4909b17685cd573124a9d95ee7df
Metric Score
Exact Match Accuracy 0.0
Domain accuracy 0.0
Intent accuracy 0.0
Slot WRR 0.0
en-US es-ES pl-PL
Exact Match Accuracy 0.0 0.0 0.0
Domain accuracy 0.0 0.0 0.0
Intent accuracy 0.0 0.0 0.0
Slot WRR 0.0 0.0 0.0

worst items

note: the gold standard is taken from the submission itself, not from the challenge data!
# input expected output actual output test-A Exact Match Accuracy
1 1 en-US valid change the maximum temperature on my thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice 'device_name' 0.0
2 2 en-US valid change the minimum and maximum temperatures on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice 'device_name' 0.0
3 3 en-US valid adjust the minimum temperature on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice 'device_name' 0.0
4 4 en-US valid adjust the minimum and maximum temperatures on the thermostat Airconditioner ChangeTemperature {} Airconditioner SetTemperatureOnDevice 'device_name' 0.0
5 5 en-US valid it is too cold in here Airconditioner ChangeTemperature {} Weather ForecastInLocation 'location': 'here' 0.0
6 6 en-US valid check the humidity level Airconditioner GetHumidity {} Airconditioner GetHumidity 0.0
7 7 en-US valid tell me how humid is the air in the room Airconditioner GetHumidity {} Airconditioner GetHumidity 0.0
8 8 en-US valid probe the humidity Airconditioner GetHumidity {} Airconditioner GetHumidity 0.0
9 9 en-US valid i feel that it is too dry in here Airconditioner GetHumidity {} Weather Clouds 0.0
10 10 en-US valid check the temperature on the keeping room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'keeping room'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
11 11 en-US valid check the temperature on the dining room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'dining room'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
12 12 en-US valid check what the home gym temperature sensor says Airconditioner GetTemperatureFromDevice {'device_name': 'home gym'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
13 13 en-US valid check with the gorash thor mister says Airconditioner GetTemperatureFromDevice {} Spotify PlaySongByArtist 'artist': 'thor 0.0
14 14 en-US valid show me the temperature on the master bedroom sensor Airconditioner GetTemperatureFromDevice {'device_name': 'master bedroom'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
15 15 en-US valid show me the temperature on the guest room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'guest room'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
16 16 en-US valid show me the temperature on the game room thermostat Airconditioner GetTemperatureFromDevice {'device_name': 'game room'} Airconditioner GetTemperature 'av_alias': ' 0.0
17 17 en-US valid show me what's on the music room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'music room'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
18 18 en-US valid show the temperature on the living room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'living room'} Airconditioner SetTemperatureOnDevice 'device_name' 0.0
19 19 en-US valid tell what the adam room temperature sensor says Airconditioner GetTemperatureFromDevice {'device_name': 'adam room'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
20 20 en-US valid give me the temperature of the family room sensor Airconditioner GetTemperatureFromDevice {'device_name': 'family room'} Fitbit ShowSteps 0.0
21 21 en-US valid check the temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature 'av_alias': ' 0.0
22 22 en-US valid check the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
23 23 en-US valid show what's the temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
24 24 en-US valid show what's the temperature on my vac Airconditioner GetTemperature {'av_alias': 'vac'} Airconditioner GetTemperature 0.0
25 25 en-US valid show me what's the temperature on my cooling system Airconditioner GetTemperature {'av_alias': 'cooling system'} Airconditioner GetTemperature 'av_alias': ' 0.0
26 26 en-US valid show me what's the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner GetTemperature 0.0
27 27 en-US valid tell me the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner SetTemperatureOnDevice 'device_name' 0.0
28 28 en-US valid measure temperature on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature 'av_alias': ' 0.0
29 29 en-US valid measure temperature on my cooling system Airconditioner GetTemperature {'av_alias': 'cooling system'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
30 30 en-US valid measure the temperature on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner GetTemperature 0.0
31 31 en-US valid how many fahrenheits degrees are on my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature 0.0
32 32 en-US valid how many celsius degrees are on my ac Airconditioner GetTemperature {'av_alias': 'ac'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
33 33 en-US valid how many kelvins degrees are on my hvac Airconditioner GetTemperature {'av_alias': 'hvac'} Airconditioner GetTemperature 'av_alias': ' 0.0
34 34 en-US valid what temperature is it in here according to my ac Airconditioner GetTemperature {'av_alias': 'ac'} Airconditioner GetTemperatureFromDevice 'device_name' 0.0
35 35 en-US valid what's the temperature in here according to my air conditioning Airconditioner GetTemperature {'av_alias': 'air conditioning'} Airconditioner GetTemperature 0.0
36 36 en-US valid it is too chilly in here Airconditioner GetTemperature {} Weather WeatherTomorrow 0.0
37 37 en-US valid turn up the temperature above 23 degree but lower than and 29 degree on my thermostat Airconditioner SetTemperatureBetweenValues {} Airconditioner SetTemperatureBetweenValues 0.0
38 38 en-US valid change the temperature on my kids room thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'kids room'} Airconditioner SetTemperatureToValueOnDevice 'dev 0.0
39 39 en-US valid change the temperature on my nursery thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'nursery'} Airconditioner SetTemperatureOnDevice 'device_name' 0.0
40 40 en-US valid change the minimum and maximum temperatures on my home gym thermostat Airconditioner SetTemperatureOnDevice {'device_name': 'home gym'} Airconditioner ChangeTemperature 'av_alias': ' 0.0

Compare with other submission