By Nora Guo Zhan
Large language models have made it easier than ever to draft polished text. But for public policy work, polish is not enough. A strong public comment must engage the specific rule, explain its reasoning, use evidence, identify real-world impacts, and give agencies recommendations they can act on.
That is why one of the central questions for policy technology today is not simply whether AI can write. It is whether a model trained for a specific field can outperform a general-purpose model when the task requires domain knowledge, structure, and practical judgment.
At Georgetown’s Tech & Public Policy program, we evaluated that question through a study of AI-generated public regulatory comments. We compared comments produced by a specialized policy-comment model from Statt (a policy intelligence firm that uses fine-tuned AI models for policy tasks) against comments generated with Google Gemini, and then benchmarked both against human-written public comments from Regulations.gov. Statt did not fund the study, and Georgetown controlled the evaluation design, scoring pipeline, and interpretation independently; Statt’s role was limited to providing access to its specialized policy-comment model and, for the first-round baseline, a reference set of human comments.
The result was clear: specialization matters. In this setting, fine-tuning worked.
What We Tested
Our project focused on public comments across three regulatory contexts: EPA, FDA, and FMCSA dockets. For each context, we evaluated three groups of comments: human-authored submissions from Regulations.gov, comments generated by a specialized Statt model, and comments generated by Google Gemini under comparable persona-based conditions.
We used two human baselines. The first round drew on a smaller set of human comments that were comparatively high quality: more structured, more complete, and more similar to the kind of polished submission that an evaluator might reward. The second round expanded the human corpus with roughly 3,000 additional comments downloaded from Regulations.gov, giving us a broader and more realistic view of what public comment dockets often look like in practice.
How We Generated the AI Comments
The AI-generated comments were an original part of the study. Human comments can be downloaded from public dockets, but the Gemini- and Statt-generated comments had to be created under controlled conditions so that we could compare models on the same kinds of policy-writing tasks.
For each agency context, we developed a set of stakeholder personas and policy stances. The personas included trade associations, advocacy organizations, companies, researchers, small businesses, state agencies, and individual stakeholders such as patients, residents, and drivers. Each persona was paired with a stance such as supportive, concerned, opposed, or neutral, along with information about the organization or person and the kinds of impacts they were likely to emphasize.
We then used those persona inputs to generate parallel AI comment sets. One set was generated with Google Gemini as the general-purpose model comparison. The other was generated with the specialized Statt policy-comment model. This matched design matters: rather than comparing unrelated AI outputs, we asked both systems to respond to similar regulatory contexts through comparable stakeholder roles and policy positions.
That generated corpus is where much of the project’s value comes from. It allowed us to test not just whether AI could produce fluent public comments, but whether a specialized policy-writing system produced more relevant, actionable, and structured comments than a general-purpose model when both were given similar policy personas.
Each comment was scored on a 1-to-5 scale across six dimensions that matter for public participation in agency rulemaking:
- Relevance: whether the comment addressed the specific docket topic.
- Reasoning: whether the argument was coherent and logically developed.
- Evidence: whether the comment used facts, examples, citations, or other support.
- Impact: whether it identified plausible consequences for people, organizations, or communities.
- Actionability: whether it gave the agency concrete recommendations.
- Structure and formatting: whether the comment was readable, organized, and professionally presented.
We also calculated an overall score for each comment. This allowed us to compare not only whether the models sounded fluent, but whether they produced comments that looked useful in the context of real regulatory decision-making.
Methods note: The 1-to-5 scores were assigned by an automated judge using the OpenAI API (gpt-5.4-mini, with temperature 0.2 and a 1,000-token output cap). The prompt supplied the policy ID, optional policy context, and comment text, and requested JSON scores and rationales for each rubric dimension; it did not send source labels, filenames, display names, or whether the comment was human, Gemini, or Statt. In Round 1, the primary corpus included about 298 human comments, 317 Statt-generated comments, and 316 Gemini-generated comments across EPA, FDA, and FMCSA topics. In Round 2, the human corpus was expanded to about 3,295 successfully scored Regulations.gov comments; the Statt and Gemini sets were unchanged. The evaluator notebook capped downloaded human comments at up to 1,000 per policy before scoring.
What We Found
In the primary analysis using the first-round human comments, the specialized model outperformed both Gemini and the human reference set across every criterion. On the overall score, Statt-generated comments averaged 4.18 out of 5, compared with 3.20 for the first-round human-written comments and 2.35 for Gemini-generated comments. That comparison is meaningful, but it should be read with an important caveat: the first-round human comments were relatively high quality and unusually complete compared with many real-world public submissions.
The advantage was especially visible in actionability and structure. The specialized model produced comments that were easier to follow and more likely to make concrete requests of agencies. It also performed better on evidence and impact, two areas where generic AI-generated comments often stayed vague.
| Criterion | Human | Statt specialized model | Google Gemini |
|---|---|---|---|
| Relevance | 4.13 | 4.54 | 3.55 |
| Reasoning | 3.46 | 3.99 | 2.40 |
| Evidence | 2.12 | 3.30 | 1.30 |
| Impact | 3.11 | 3.97 | 2.29 |
| Actionability | 3.35 | 4.82 | 2.52 |
| Structure/formatting | 3.37 | 4.89 | 3.07 |
| Overall | 3.20 | 4.18 | 2.35 |
Table 1. Mean evaluation scores by source in the primary corpus, on a 1-to-5 scale.
Figure 1. Overall score distribution by source. Statt comments are concentrated at higher scores, while Gemini comments cluster lower.
Figure 2. Criterion-level score distributions by source. Statt performs especially strongly on actionability, structure, evidence, and impact.
These results do not mean that every specialized model will outperform every general model on every task. They show that, for this policy-specific writing task and under the criteria we proposed, a model designed around the norms and needs of public commenting produced stronger results than a general-purpose chatbot.
Why the Human Baseline Matters
The second round changed the interpretation of the human comparison. When we expanded the human corpus to include a much larger set of comments from Regulations.gov, the average human score dropped substantially. That does not mean human participation is less valuable. It means real public dockets contain a wide mix of submissions: long and short, polished and informal, highly substantive and very brief.
This broader round is useful because it is closer to the real-world environment agencies actually face. It suggests that the gap between specialized AI comments and typical public comments may be larger than the first round alone implies. At the same time, it also reminds us that “quality” is not a settled concept. A short, personal, or less polished comment may still matter in democratic participation, even if it scores lower on our structured rubric.
What Topic Modeling Added
We also used BERTopic to analyze themes in the evaluator’s rationales and overall summaries. This did not replace the score comparison, but it helped explain what kinds of strengths and weaknesses the evaluation pipeline repeatedly noticed across sources.
- For Gemini-generated comments, the topic clusters often centered on broad relevance, safety or compliance language, and recurring gaps in specificity, concrete evidence, or operational detail.
- For human comments, topics clustered around real stakeholder concerns such as transparency, safety, authority, and requests to agencies. This supports the idea that human comments often carry substantive lived or organizational concerns even when the writing is less polished.
- For Statt-generated comments, the dominant topics clustered around clear claims, concrete recommendations, impacts, relief, reporting, and implementation. That pattern aligns with the higher actionability and structure scores.
The topic modeling therefore reinforces the central finding: the specialized model’s advantage was not just that it sounded cleaner. The evaluator repeatedly recognized the same policy-relevant features that strong public comments need: specific claims, concrete recommendations, and a clearer connection between impacts and agency action.
Why Fine-Tuning Helped
General-purpose models are impressive because they are flexible. They can summarize, brainstorm, draft, and answer questions across many fields. But that flexibility can become a weakness when a task has a specific institutional setting. Public comments are not just essays. They are part of a legal and administrative process, and the most useful comments tend to follow a recognizable pattern: identify the rule, explain a stakeholder position, connect claims to evidence, and ask for specific agency action.
Fine-tuning helps because it can teach a model the shape of the task. In our evaluation, Statt’s specialized model, which was based on a proprietary dataset of more than 100 million indexed policy documents across the U.S. and international contexts, was more likely to produce comments that were organized around regulatory relevance and concrete recommendations. Gemini often produced fluent text, but it was more likely to rely on generic language, broad claims, or placeholder-like phrasing. Human comments varied widely: some were substantive and persuasive, while others were short, informal, or only loosely connected to the docket.
That pattern matters for public policy. If agencies receive growing numbers of AI-assisted comments, the difference between generic volume and useful participation will become increasingly important. Better tools could help people write clearer comments, but they could also make it harder for agencies to distinguish genuine public input from mass-produced submissions. The design of these tools therefore affects not only writing quality, but the health of public participation itself.
What This Means for Policy and AI
The main lesson from this project is that AI performance should be evaluated in context. A model that performs well in a general chat setting may not be the best tool for a specialized policy workflow. Conversely, a smaller or narrower system may perform better when it has been tuned to the task, the audience, and the standards of quality that matter in that field.
For civic technology, this suggests a more practical way to think about AI adoption. The question should not be whether organizations should use general models or specialized models in the abstract. The better question is: what work needs to be done, what counts as quality, and what kind of model is most likely to meet that standard?
In our case, the answer leaned strongly toward specialization under the evaluation framework we built. The fine-tuned model produced comments that were more relevant, more actionable, and more professionally structured than the general-purpose comparison model. It also exceeded the first-round human baseline and scored far above the broader second-round human baseline. But those results should be understood as a prompt for further inquiry, not a final answer about what agencies do or should value most.
A Note of Caution
This study should be read as evidence about one defined task, not as a universal verdict on AI writing or public-comment quality. The evaluation used an automated scoring pipeline and a set of criteria we proposed: relevance, reasoning, evidence, impact, actionability, and structure. Those criteria are reasonable for assessing public comments, but we do not know whether they match the criteria government officials actually use, formally or informally, when reviewing comments in real-world rulemaking.
For that reason, the project is best understood as an experiment that can spark discussion. It shows that specialized models can perform very well when judged against a structured policy-writing rubric. It also raises the next set of questions: what kinds of comments agencies find most useful, how AI-generated comments should be evaluated, and whether future models can be designed around standards that reflect real institutional needs rather than only our research assumptions.