Proctor report

Checked on

Suite	Platform	Device	OS	Model version	Backend	Prompt	Availability	Run at
ticket-classifier	ios-simulator	iPhone19,2	Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0	iOS 27.0 sim on Darwin 27.0.0 (os-derived)	foundation-models	v1	available	2026-10-26T00:20:35.110628Z
ticket-classifier	ios-simulator	iPhone19,2	Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0	iOS 27.0 sim on Darwin 27.0.0 (os-derived)	foundation-models	v2	available	2026-10-26T00:21:07.182325Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-fast	mlkit-genai-prompt preview/fast	v1	available	2026-10-29T14:01:12.964471Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-fast	mlkit-genai-prompt preview/fast	v2	available	2026-10-29T14:01:54.380611Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-full	mlkit-genai-prompt preview/full	v1	available	2026-10-29T14:09:34.419629Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-full	mlkit-genai-prompt preview/full	v2	available	2026-10-29T14:10:21.130233Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-full	mlkit-genai-prompt stable/full	v1	available	2026-10-29T14:30:23.099710Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-full	mlkit-genai-prompt stable/full	v2	available	2026-10-29T14:31:04.289800Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-fast	mlkit-genai-prompt stable/fast	v1	available	2026-10-29T14:33:38.391955Z
ticket-classifier	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-fast	mlkit-genai-prompt stable/fast	v2	available	2026-10-29T14:34:19.541792Z
ticket-extraction	ios-simulator	iPhone19,2	Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0	iOS 27.0 sim on Darwin 27.0.0 (os-derived)	foundation-models	v1	available	2026-10-26T00:21:37.913947Z
ticket-extraction	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-fast	mlkit-genai-prompt preview/fast	v1	available	2026-10-29T14:02:35.633296Z
ticket-extraction	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-full	mlkit-genai-prompt preview/full	v1	available	2026-10-29T14:11:06.107860Z
ticket-extraction	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-full	mlkit-genai-prompt stable/full	v1	available	2026-10-29T14:31:45.662350Z
ticket-extraction	android	Google Pixel 11 Pro	Android 17 (API 37, build CD1A.261005.003)	nano-v4-fast	mlkit-genai-prompt stable/fast	v1	available	2026-10-29T14:35:00.740138Z

Results

Run	Cases	Passed	Failed	Mean score	Verdict
ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1	12	11	1	92%	pass
ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v2	12	11	1	97%	pass
ticket-classifier · android nano-v4-fast · prompt v1	12	11	1	97%	pass
ticket-classifier · android nano-v4-fast · prompt v2	12	12	0	100%	pass
ticket-classifier · android nano-v4-full · prompt v1	12	12	0	100%	pass
ticket-classifier · android nano-v4-full · prompt v2	12	12	0	100%	pass
ticket-classifier · android nano-v4-full · prompt v1	12	12	0	100%	pass
ticket-classifier · android nano-v4-full · prompt v2	12	12	0	100%	pass
ticket-classifier · android nano-v4-fast · prompt v1	12	12	0	100%	pass
ticket-classifier · android nano-v4-fast · prompt v2	12	12	0	100%	pass
ticket-extraction · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1	4	2	2	92%	pass
ticket-extraction · android nano-v4-fast · prompt v1	4	3	1	96%	pass
ticket-extraction · android nano-v4-full · prompt v1	4	3	1	99%	pass
ticket-extraction · android nano-v4-full · prompt v1	4	3	1	96%	pass
ticket-extraction · android nano-v4-fast · prompt v1	4	3	1	96%	pass

Not graded means no verdict, for example the quota ran out. It is not a failure.

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v2

Differs in prompt version: v1 to v2.

Score 92% to 97% (+6 pts). 0 regressed, 0 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	fail	Same

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-fast · prompt v1

Differs in platform: ios-simulator to android; device: iPhone19,2 to Google Pixel 11 Pro; OS version: Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0 to Android 17 (API 37, build CD1A.261005.003); model version: iOS 27.0 sim on Darwin 27.0.0 (os-derived) to nano-v4-fast; backend: foundation-models to mlkit-genai-prompt preview/fast.

Score 92% to 97% (+6 pts). 0 regressed, 0 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	fail	Same

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-fast · prompt v2

Score 92% to 100% (+8 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	pass	Improved

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-full · prompt v1

Differs in platform: ios-simulator to android; device: iPhone19,2 to Google Pixel 11 Pro; OS version: Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0 to Android 17 (API 37, build CD1A.261005.003); model version: iOS 27.0 sim on Darwin 27.0.0 (os-derived) to nano-v4-full; backend: foundation-models to mlkit-genai-prompt preview/full.

Score 92% to 100% (+8 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	pass	Improved

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-full · prompt v2

Differs in platform: ios-simulator to android; device: iPhone19,2 to Google Pixel 11 Pro; OS version: Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0 to Android 17 (API 37, build CD1A.261005.003); model version: iOS 27.0 sim on Darwin 27.0.0 (os-derived) to nano-v4-full; backend: foundation-models to mlkit-genai-prompt preview/full; prompt version: v1 to v2.

Score 92% to 100% (+8 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	pass	Improved

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-full · prompt v1

Differs in platform: ios-simulator to android; device: iPhone19,2 to Google Pixel 11 Pro; OS version: Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0 to Android 17 (API 37, build CD1A.261005.003); model version: iOS 27.0 sim on Darwin 27.0.0 (os-derived) to nano-v4-full; backend: foundation-models to mlkit-genai-prompt stable/full.

Score 92% to 100% (+8 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	pass	Improved

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-full · prompt v2

Differs in platform: ios-simulator to android; device: iPhone19,2 to Google Pixel 11 Pro; OS version: Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0 to Android 17 (API 37, build CD1A.261005.003); model version: iOS 27.0 sim on Darwin 27.0.0 (os-derived) to nano-v4-full; backend: foundation-models to mlkit-genai-prompt stable/full; prompt version: v1 to v2.

Score 92% to 100% (+8 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	pass	Improved

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-fast · prompt v1

Score 92% to 100% (+8 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	pass	Improved

Compare: ticket-classifier

ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-classifier · android nano-v4-fast · prompt v2

Score 92% to 100% (+8 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
double-charge	My card was charged twice for the same order	pass	pass	Same
refund	I need a refund for the annual plan I never used	pass	pass	Same
invoice-name	The invoice shows a company name I did not enter	pass	pass	Same
crash-camera	The app crashes as soon as I open the camera screen	pass	pass	Same
search-empty	Search returns nothing even though the item exists	pass	pass	Same
dark-mode-contrast	Dark mode turns the text black on a black background	pass	pass	Same
reset-email	I forgot my password and the reset email never arrives	pass	pass	Same
change-email	Please change the email address on my profile	pass	pass	Same
export-csv	Can you add an option to export my data as CSV?	pass	pass	Same
widget	It would be great to have a widget for the home screen	pass	pass	Same
thanks	Thanks for the quick help last week, just saying hi	pass	pass	Same
office	Where is your office located?	fail	pass	Improved

Compare: ticket-extraction

ticket-extraction · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-extraction · android nano-v4-fast · prompt v1

Score 92% to 96% (+4 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
blender	Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow.	pass	pass	Same
headphones	My headphones arrived scratched. Order A-77.	fail	pass	Improved
dashboard	I cannot log in to the web dashboard, blocking my whole team right now.	fail	fail	Same
shipping	Just wondering if you ship to Norway.	pass	pass	Same

Compare: ticket-extraction

ticket-extraction · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-extraction · android nano-v4-full · prompt v1

Differs in platform: ios-simulator to android; device: iPhone19,2 to Google Pixel 11 Pro; OS version: Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0 to Android 17 (API 37, build CD1A.261005.003); model version: iOS 27.0 sim on Darwin 27.0.0 (os-derived) to nano-v4-full; backend: foundation-models to mlkit-genai-prompt preview/full.

Score 92% to 99% (+7 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
blender	Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow.	pass	pass	Same
headphones	My headphones arrived scratched. Order A-77.	fail	pass	Improved
dashboard	I cannot log in to the web dashboard, blocking my whole team right now.	fail	fail	Same
shipping	Just wondering if you ship to Norway.	pass	pass	Same

Compare: ticket-extraction

ticket-extraction · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-extraction · android nano-v4-full · prompt v1

Differs in platform: ios-simulator to android; device: iPhone19,2 to Google Pixel 11 Pro; OS version: Version 27.0 (Build 24A434), simulator on host Darwin 27.0.0 to Android 17 (API 37, build CD1A.261005.003); model version: iOS 27.0 sim on Darwin 27.0.0 (os-derived) to nano-v4-full; backend: foundation-models to mlkit-genai-prompt stable/full.

Score 92% to 96% (+4 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
blender	Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow.	pass	pass	Same
headphones	My headphones arrived scratched. Order A-77.	fail	pass	Improved
dashboard	I cannot log in to the web dashboard, blocking my whole team right now.	fail	fail	Same
shipping	Just wondering if you ship to Norway.	pass	pass	Same

Compare: ticket-extraction

ticket-extraction · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1 against ticket-extraction · android nano-v4-fast · prompt v1

Score 92% to 96% (+4 pts). 0 regressed, 1 improved.

Case	Input	Base	Other	Change
blender	Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow.	pass	pass	Same
headphones	My headphones arrived scratched. Order A-77.	fail	pass	Improved
dashboard	I cannot log in to the web dashboard, blocking my whole team right now.	fail	fail	Same
shipping	Just wondering if you ship to Norway.	pass	pass	Same

Cases: ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

fail office Where is your office located? score 0%, spread 0.0

Expected Other

Attempt 1 fail

Account

fail exactMatch Expected "Other", got "Account"

Attempt 2 fail

Account

fail exactMatch Expected "Other", got "Account"

Attempt 3 fail

Account

fail exactMatch Expected "Other", got "Account"

Cases: ticket-classifier · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v2

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

fail office Where is your office located? score 67%, spread 0.47

Expected Other

Attempt 1 fail

Account

fail exactMatch Expected "Other", got "Account"

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-fast · prompt v1

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

fail office Where is your office located? score 67%, spread 0.47

Expected Other

Attempt 1 fail

Account

fail exactMatch Expected "Other", got "Account"

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-fast · prompt v2

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

pass office Where is your office located? score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-full · prompt v1

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

pass office Where is your office located? score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-full · prompt v2

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

pass office Where is your office located? score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-full · prompt v1

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

pass office Where is your office located? score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-full · prompt v2

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

pass office Where is your office located? score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-fast · prompt v1

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

pass office Where is your office located? score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-classifier · android nano-v4-fast · prompt v2

pass double-charge My card was charged twice for the same order score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass refund I need a refund for the annual plan I never used score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass invoice-name The invoice shows a company name I did not enter score 100%, spread 0.0

Expected Billing

Attempt 1 pass

Billing

pass exactMatch

Attempt 2 pass

Billing

pass exactMatch

Attempt 3 pass

Billing

pass exactMatch

pass crash-camera The app crashes as soon as I open the camera screen score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass search-empty Search returns nothing even though the item exists score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass dark-mode-contrast Dark mode turns the text black on a black background score 100%, spread 0.0

Expected Bug

Attempt 1 pass

Bug

pass exactMatch

Attempt 2 pass

Bug

pass exactMatch

Attempt 3 pass

Bug

pass exactMatch

pass reset-email I forgot my password and the reset email never arrives score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass change-email Please change the email address on my profile score 100%, spread 0.0

Expected Account

Attempt 1 pass

Account

pass exactMatch

Attempt 2 pass

Account

pass exactMatch

Attempt 3 pass

Account

pass exactMatch

pass export-csv Can you add an option to export my data as CSV? score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass widget It would be great to have a widget for the home screen score 100%, spread 0.0

Expected Feature

Attempt 1 pass

Feature

pass exactMatch

Attempt 2 pass

Feature

pass exactMatch

Attempt 3 pass

Feature

pass exactMatch

pass thanks Thanks for the quick help last week, just saying hi score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

pass office Where is your office located? score 100%, spread 0.0

Expected Other

Attempt 1 pass

Other

pass exactMatch

Attempt 2 pass

Other

pass exactMatch

Attempt 3 pass

Other

pass exactMatch

Cases: ticket-extraction · ios-simulator iOS 27.0 sim on Darwin 27.0.0 (os-derived) · prompt v1

pass blender Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow. score 100%, spread 0.0

Expected {"orderId":"48213","product":"blender","urgent":true}

Attempt 1 pass

{"orderId": "48213", "product": "blender", "urgent": true}

pass jsonSchema

pass detailsMatch

Attempt 2 pass

{"orderId": "48213", "product": "blender", "urgent": true}

pass jsonSchema

pass detailsMatch

Attempt 3 pass

{"orderId": "48213", "product": "blender", "urgent": true}

pass jsonSchema

pass detailsMatch

fail headphones My headphones arrived scratched. Order A-77. score 83%, spread 0.0

Expected {"orderId":"A-77","product":"headphones","urgent":false}

Attempt 1 fail

{"orderId": "A-77", "product": "headphones", "urgent": true}

pass jsonSchema

fail detailsMatch urgent true instead of false

Attempt 2 fail

{"orderId": "A-77", "product": "headphones", "urgent": true}

pass jsonSchema

fail detailsMatch urgent true instead of false

Attempt 3 fail

{"orderId": "A-77", "product": "headphones", "urgent": true}

pass jsonSchema

fail detailsMatch urgent true instead of false

fail dashboard I cannot log in to the web dashboard, blocking my whole team right now. score 83%, spread 0.0

Expected {"orderId":null,"product":"web dashboard","urgent":true}

Attempt 1 fail

{"orderId": null, "product": null, "urgent": true}

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 2 fail

{"orderId": null, "product": null, "urgent": true}

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 3 fail

{"orderId": null, "product": null, "urgent": true}

pass jsonSchema

fail detailsMatch product null instead of web dashboard

pass shipping Just wondering if you ship to Norway. score 100%, spread 0.0

Expected {"orderId":null,"product":null,"urgent":false}

Attempt 1 pass

{"orderId": null, "product": null, "urgent": false}

pass jsonSchema

pass detailsMatch

Attempt 2 pass

{"orderId": null, "product": null, "urgent": false}

pass jsonSchema

pass detailsMatch

Attempt 3 pass

{"orderId": null, "product": null, "urgent": false}

pass jsonSchema

pass detailsMatch

Cases: ticket-extraction · android nano-v4-fast · prompt v1

pass blender Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow. score 100%, spread 0.0

Expected {"orderId":"48213","product":"blender","urgent":true}

Attempt 1 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

pass headphones My headphones arrived scratched. Order A-77. score 100%, spread 0.0

Expected {"orderId":"A-77","product":"headphones","urgent":false}

Attempt 1 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

fail dashboard I cannot log in to the web dashboard, blocking my whole team right now. score 83%, spread 0.0

Expected {"orderId":null,"product":"web dashboard","urgent":true}

Attempt 1 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 2 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 3 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

pass shipping Just wondering if you ship to Norway. score 100%, spread 0.0

Expected {"orderId":null,"product":null,"urgent":false}

Attempt 1 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Cases: ticket-extraction · android nano-v4-full · prompt v1

pass blender Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow. score 100%, spread 0.0

Expected {"orderId":"48213","product":"blender","urgent":true}

Attempt 1 pass

{
"orderId": "48213",
"product": "blender",
"urgent": true
}

pass jsonSchema

pass detailsMatch

Attempt 2 pass

{"orderId": "48213", "product": "blender", "urgent": true}

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

pass headphones My headphones arrived scratched. Order A-77. score 100%, spread 0.0

Expected {"orderId":"A-77","product":"headphones","urgent":false}

Attempt 1 pass

{"orderId": "A-77", "product": "headphones", "urgent": false}

pass jsonSchema

pass detailsMatch

Attempt 2 pass

{"orderId": "A-77", "product": "headphones", "urgent": false}

pass jsonSchema

pass detailsMatch

Attempt 3 pass

{"orderId": "A-77", "product": "headphones", "urgent": false}

pass jsonSchema

pass detailsMatch

fail dashboard I cannot log in to the web dashboard, blocking my whole team right now. score 94%, spread 0.08

Expected {"orderId":null,"product":"web dashboard","urgent":true}

Attempt 1 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 2 pass

{
"orderId": null,
"product": "web dashboard",
"urgent": true
}

pass jsonSchema

pass detailsMatch

Attempt 3 pass

{"orderId": null, "product": "web dashboard", "urgent": true}

pass jsonSchema

pass detailsMatch

pass shipping Just wondering if you ship to Norway. score 100%, spread 0.0

Expected {"orderId":null,"product":null,"urgent":false}

Attempt 1 pass

{"orderId": null, "product": null, "urgent": false}

pass jsonSchema

pass detailsMatch

Attempt 2 pass

{"orderId": null, "product": null, "urgent": false}

pass jsonSchema

pass detailsMatch

Attempt 3 pass

{"orderId": null, "product": null, "urgent": false}

pass jsonSchema

pass detailsMatch

Cases: ticket-extraction · android nano-v4-full · prompt v1

pass blender Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow. score 100%, spread 0.0

Expected {"orderId":"48213","product":"blender","urgent":true}

Attempt 1 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

pass headphones My headphones arrived scratched. Order A-77. score 100%, spread 0.0

Expected {"orderId":"A-77","product":"headphones","urgent":false}

Attempt 1 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

fail dashboard I cannot log in to the web dashboard, blocking my whole team right now. score 83%, spread 0.0

Expected {"orderId":null,"product":"web dashboard","urgent":true}

Attempt 1 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 2 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 3 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

pass shipping Just wondering if you ship to Norway. score 100%, spread 0.0

Expected {"orderId":null,"product":null,"urgent":false}

Attempt 1 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Cases: ticket-extraction · android nano-v4-fast · prompt v1

pass blender Order 48213: the blender stopped working. This is urgent, I need it for an event tomorrow. score 100%, spread 0.0

Expected {"orderId":"48213","product":"blender","urgent":true}

Attempt 1 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": "48213",
  "product": "blender",
  "urgent": true
}
```

pass jsonSchema

pass detailsMatch

pass headphones My headphones arrived scratched. Order A-77. score 100%, spread 0.0

Expected {"orderId":"A-77","product":"headphones","urgent":false}

Attempt 1 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": "A-77",
  "product": "headphones",
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

fail dashboard I cannot log in to the web dashboard, blocking my whole team right now. score 83%, spread 0.0

Expected {"orderId":null,"product":"web dashboard","urgent":true}

Attempt 1 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 2 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

Attempt 3 fail

```json
{
  "orderId": null,
  "product": null,
  "urgent": true
}
```

pass jsonSchema

fail detailsMatch product null instead of web dashboard

pass shipping Just wondering if you ship to Norway. score 100%, spread 0.0

Expected {"orderId":null,"product":null,"urgent":false}

Attempt 1 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 2 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch

Attempt 3 pass

```json
{
  "orderId": null,
  "product": null,
  "urgent": false
}
```

pass jsonSchema

pass detailsMatch