Evidence at scale: how AI changed what we decide to build

We were deciding what to build from what we could scan by hand, one page at a time, so system decisions rested on small samples and our own reading of dense standards.

A laptop showing the card analysis of 3,328 cards from 24 old sites: a 500-character cap would mean rewriting 3.5% of cards, a 300 cap 8.7% (289 cards), and a 200 cap 17.8% (594 cards).

At a glance

What I did. As the Design Engineer on MSU's web team, I used AI as a sourced analyst for four jobs: scanning old site content for patterns, checking findings against peer design systems, reading the WCAG standard, and extending sets by precedent. It works only from material I give it, and I keep every decision and every test.

Role
Design Engineer
Team
7-person web team at MSU Web & Digital Communications
Scale
3 campuses, 1,800 users, built to serve 800+ sites
Standard
WCAG 2.2 AA
Verification
axe-core on the new CMS, WAVE on old and new sites
Timeline
2024 to present. Core build largely complete, first pages live, migration not started.

Why it matters: system decisions repeat everywhere

The old CMS leaned on editors to remember their accessibility training. The new system takes that burden off them, which only works if the components are right from the start.

That raises the stakes on every decision. An accordion with a broken keyboard pattern, a set that shouldn't exist, or a character limit set too loosely isn't one bad page, it's every page built on the system.

Before: one WAVE scan at a time

Before AI, we went through the old sites' content by hand and ran WAVE to find heading order violations, misuse of snippets and odd behavior. We read the WCAG standard ourselves and interpreted it ourselves.

It worked, but it was slow, and it only covered what we had time to open.

How I used AI: four jobs, all from sources

The rule for every job is the same: AI works from material I give it or can cite, with no training data and no opinions.

  1. Scanning old content for patterns. My boss helped me export the content of 24 old sites so AI could scan it for patterns and violations. We used it to validate character limits and to decide which sets belong in the system: how often a pattern is actually used, whether it exists at all, and whether it solves a problem editors were working around.

  2. Checking findings against peers. AI finds design systems from peer institutions and government agencies, so our conclusions are validated against how others solved the same problem.

  3. Reading WCAG. AI reads the dense standard and translates the technical language, so we can check whether we are building correctly instead of interpreting it alone.

  4. Extending sets by precedent. I make the call by hand on one set, with the evidence and criteria written down. AI then writes the files for other sets by following that pattern.

What I own

AI never decides what goes in the system. I set the criteria, make the first judgment by hand, and AI applies it at a scale we couldn't reach manually, with sources I can check. Everything it helps build is still tested with axe-core and WAVE.

The catch: it invents sources and misses context

AI sometimes made up sources, and sometimes misread a real one, taking a line without the information around it.

So every claim now has to come with an inline link. I open it, read the surrounding text and decide what it means myself. AI is good at sniffing out where the information lives. It is not reliable at what that information means in context, so that part stays with me.

What improved: limits and sets from real content

The clearest change is the evidence behind each decision. The card analysis shows what that looks like: it parsed 13,235 files from 24 old sites, counted 3,328 card items, and reported how much editors actually write.

  • Character limits. The median card is 87 characters. A 500-character cap would leave 96.5% of existing cards untouched. A 300 cap would force edits on 289 cards, and below 200 the cap turns into a migration project, with 594 cards to rewrite. A 50-character heading cap fits 98% of existing headings.

  • Which sets should exist. Nearly a third of cards are pure navigation, five words or fewer, an image and a label, which may be better served by a link grid than a card set. Half of all cards have no image, so image-optional has to be a real layout. Over 90% contain a link, so a clickable card is the main case, not an edge case.

  • What isn't a card. The multi-column snippet has a median of 7 words but a 90th percentile of 80. Editors used it for date labels, people directories and full prose, so it was never one component and needs case-by-case triage.

  • A correction. MSU card copy averages about 6.8 characters per word, not the usual 5.8. Converting a word guideline into a character limit with the generic number would have set limits too low.

What I can't prove yet:

  • It's a sample. The content comes from 24 old sites, and the system is built to serve 800+.

  • It says what fits, not what's best. The data shows what existing content would survive a limit, so the final limits are still a design judgment.

  • It's untested in the wild. Migration hasn't started, so we don't yet know whether editors hit these limits in practice.

Glossary

  • Set: A pre-designed page section an editor adds from a picker, such as an accordion or card grid, with fixed fields and limits.

  • WCAG 2.2 AA: The web accessibility standard, at the level most institutions target.

  • Semantic HTML: Using the HTML element that matches a thing's purpose, such as a button for a button, so assistive technology understands it.

  • ARIA: Attributes that add accessibility information to HTML when native elements aren't enough. Wrong ARIA can make a page worse than none.

  • axe-core: An automated accessibility testing engine that flags issues in rendered pages.

  • WAVE: A browser tool that shows accessibility errors, contrast problems and alerts on a page.