Ask the data · Evaluation

How often does the model get it right?

A drafted query that runs is not the same as a right answer. This harness asks a model 28 fixed questions, runs each query it writes through the same allow-list and database as the Ask page, and compares the result with a hand-written reference query. 20 questions have an answer in the tables; for the other 8, the right response is to decline.

Gambling Help Line 1800 858 858 or gamblinghelponline.org.au.

Reference answers are checked

Each reference query is run in the unit tests and compared with the figure the site computes in TypeScript (for example, FY 2024/25 NGR against the Statewide page’s annual total), so the benchmark can’t silently disagree with the site.

Scoring

A query passes when its result has the reference result’s values, whatever the column names. Whole numbers must match exactly and other numbers to four significant figures. “Strict” also requires no extra columns.

Small samples

28 questions give wide intervals: one question is about four percentage points. Accuracy has a Wilson 95% interval over the questions, and two runs are compared question by question rather than by their headline rates.

Runs vary

The same question can get a different answer on another call: some models don’t accept a temperature setting and others ignore it. Repeat the set three times to see how much accuracy moves; each question then scores its pass rate, and comparisons use those rates, so chance differences between runs aren’t read as a better model or prompt. Repeats don’t narrow the interval: the questions, not the calls, are the sample.

Run the evaluation

28 questions, one request each per repeat, to Anthropic (claude-haiku-4-5) with your key. Results stay in this browser; every call is in the AI log.

Prompt

Repeats

No results are published here: the site has no AI budget, so the harness runs only with a visitor’s own key. Add one in AI settings to run it.

The benchmark

Questions and what they test

The questions are fixed, and no model has been run on them for this site. The prompt’s domain notes restate rules the site already follows (machines are averaged, groups stay whole), which some questions need. The “bare schema” prompt leaves the notes out, so comparing the two prompts shows what they are worth.

The prompt’s rule for declining is generic (“if the tables below cannot answer the question”) and names no examples, so the 8 should-decline questions test whether a model recognises what the tables can’t answer. The prompt’s first sentence does state the scope (South Australian gaming-machine statistics, FY 2009-10 to FY 2024-25), and 3 of them (a03, a06, a07) lean on it, so declining those is partly following the prompt. The results also give the decline rate on the other 5, the cleaner test. An earlier draft of the prompt listed the three original should-decline cases by name; questions a04 to a08 were added when that was removed.

#	Question	Category	Tests
q01	What was total net gambling revenue in FY 2024-25, in $ million? Reference SQL SELECT ngr_aud_million FROM statewide_annual WHERE financial_year = '2024-25'	Look-up	Reading one value from the annual table.
q02	In which month was net gambling revenue highest, and what was it in $ million? Reference SQL SELECT month, ngr_aud_million FROM statewide_monthly ORDER BY ngr_aud_million DESC LIMIT 1	Look-up	A maximum over months.
q03	How many gaming machines were operating in hotels and clubs in June 2025? Reference SQL SELECT machines FROM statewide_monthly WHERE month = '2025-06'	Look-up	Month keys ('YYYY-MM').
q04	What was the average number of gaming machines in hotels and clubs during FY 2018-19? Reference SQL SELECT AVG(machines) FROM statewide_monthly WHERE financial_year = '2018-19'	Needs a domain rule	Machines are a stock: a year's figure is a mean of months, not a sum.
q05	What fraction of net gambling revenue was gaming tax in FY 2009-10? Reference SQL SELECT gaming_tax_aud_million / ngr_aud_million FROM statewide_annual WHERE financial_year = '2009-10'	Aggregate or join	A ratio of two annual flows.
q06	Which council area had the highest net gambling revenue in FY 2024-25? Reference SQL SELECT area FROM lga_published_areas WHERE financial_year = '2024-25' ORDER BY ngr_aud DESC LIMIT 1	Look-up	A ranking in the council table.
q07	How many combined council groups did CBS publish for FY 2024-25? Reference SQL SELECT COUNT(*) FROM lga_published_areas WHERE financial_year = '2024-25' AND kind = 'group'	Needs a domain rule	Combined groups are rows with kind = 'group'.
q08	What was net gambling revenue per gaming machine in the City of Adelaide council area in FY 2024-25, in dollars? Reference SQL SELECT ngr_aud * 1.0 / machines FROM lga_published_areas WHERE financial_year = '2024-25' AND area = 'Adelaide'	Aggregate or join	A derived rate for one area.
q09	How many manufacturers were listed in the June 2025 market report? Reference SQL SELECT manufacturers_listed FROM manufacturer_concentration WHERE month = '2025-06'	Look-up	A count that exists in two tables.
q10	What fraction of gaming machines did Aristocrat supply in June 2025? Reference SQL SELECT share_recomputed FROM manufacturers_monthly WHERE month = '2025-06' AND manufacturer = 'Aristocrat'	Look-up	A share stored as a fraction.
q11	What was the mean monthly Herfindahl-Hirschman index (HHI) of manufacturer concentration in FY 2024-25? Reference SQL SELECT AVG(hhi) FROM manufacturer_concentration WHERE financial_year = '2024-25'	Aggregate or join	A mean over the months of one year.
q12	In which month was manufacturer concentration (HHI) lowest? Reference SQL SELECT month FROM manufacturer_concentration ORDER BY hhi ASC LIMIT 1	Look-up	A minimum over months.
q13	How many gaming machine entitlements did hotels hold at the end of FY 2024-25? Reference SQL SELECT entitlements_held_end_of_year FROM licences_annual WHERE financial_year = '2024-25' AND category = 'Hotels'	Needs a domain rule	An end-of-year snapshot, not a sum of months.
q14	How many live gaming machines were there across all licence categories in June 2025? Reference SQL SELECT SUM(live_machines) FROM licences_monthly WHERE month = '2025-06'	Aggregate or join	Adding categories within one month (which is allowed).
q15	Which financial year had the largest increase in net gambling revenue over the previous financial year? Reference SQL SELECT financial_year FROM (SELECT financial_year, ngr_aud_million - LAG(ngr_aud_million) OVER (ORDER BY financial_year) AS change FROM statewide_annual) WHERE change IS NOT NULL ORDER BY change DESC LIMIT 1	Needs a domain rule	A year-on-year change with a missing year in the series.
q16	What was FY 2009-10 net gambling revenue in FY 2024-25 dollars, in $ million? Reference SQL SELECT ngr_aud_million_real_fy2024_25 FROM statewide_annual WHERE financial_year = '2009-10'	Look-up	Finding the real-terms column.
q17	How many council names in the crosswalk were matched because the council was renamed? Reference SQL SELECT COUNT(*) FROM lga_crosswalk WHERE relation = 'renamed'	Aggregate or join	A filter on a coded column.
q18	In how many months of FY 2019-20 did CBS report zero gaming machines? Reference SQL SELECT COUNT(*) FROM statewide_monthly WHERE financial_year = '2019-20' AND machines = 0	Needs a domain rule	The COVID-19 closure months.
q19	Which council area had the highest net gambling revenue per gaming machine in FY 2024-25? Reference SQL SELECT area FROM lga_published_areas WHERE financial_year = '2024-25' AND machines > 0 ORDER BY ngr_aud * 1.0 / machines DESC LIMIT 1	Aggregate or join	Ranking by a derived rate.
q20	List total net gambling revenue in $ million for each financial year from FY 2020-21 to FY 2024-25, earliest first. Reference SQL SELECT financial_year, ngr_aud_million FROM statewide_annual WHERE financial_year BETWEEN '2020-21' AND '2024-25' ORDER BY financial_year	Aggregate or join	An ordered list.
a01	Which individual hotel or club had the highest net gambling revenue in FY 2024-25?	Should decline	There is no venue-level data: the model should decline.
a02	How much net gambling revenue did the Adelaide Casino's gaming machines make in FY 2024-25?	Should decline	The revenue series exclude the casino: the model should decline.
a03	What will net gambling revenue be in FY 2026-27?	Should decline	A forecast is outside the data: the model should decline.Left out of the cleaner decline rate: The prompt states the years the data cover (FY 2009-10 to FY 2024-25).
a04	Which suburb had the highest net gambling revenue in FY 2024-25?	Should decline	The finest geography is the council area: the model should decline.
a05	What share of South Australian adults were problem gamblers in FY 2024-25?	Should decline	Gambling harm is not measured in these data: the model should decline.
a06	What was net gambling revenue from gaming machines in Victoria in FY 2024-25?	Should decline	Only South Australia is covered: the model should decline.Left out of the cleaner decline rate: The prompt says the data are South Australian.
a07	How much did South Australians lose on online sports betting in FY 2024-25?	Should decline	Only gaming machines are covered: the model should decline.Left out of the cleaner decline rate: The prompt says the data are gaming-machine statistics.
a08	What was the average age of people who played gaming machines in FY 2024-25?	Should decline	There is nothing about players: the model should decline.