Can Language Models Distinguish Scientific Epistemic Acts?

Abstract

Scientific writing encodes not only propositions but the author’s epistemic relationship to them: the same content may be observed, asserted, inferred, hypothesized, reported from prior work, or challenged. Any system that compiles literature into structured form must preserve these distinctions, yet it is unknown how reliably current language models can recover them. This study proposes a controlled evaluation: a corpus of scientific passages annotated with a small six-act epistemic taxonomy, against which multiple language models are evaluated for agreement with human reference annotation, inter-model agreement, consistency under repetition, and calibrated abstention. The working hypothesis is strong performance on explicitly marked cases and degradation where the act must be inferred from discourse context. The expected outputs — an annotated corpus, a reproducible benchmark, a model comparison, and an error taxonomy — will determine whether grounding classifications in exact textual evidence should be the next step toward reliable machine compilation of scientific knowledge.

01Background and motivation

A machine-readable representation of scientific literature is trustworthy only if it preserves what authors actually did with their propositions. Authors may observe a result, assert a conclusion, infer an explanation from evidence, hypothesize a possible mechanism, report a claim made elsewhere, or challenge an existing claim. A compiler that extracts the proposition but assigns the wrong epistemic status produces representations that are syntactically correct and scientifically misleading.

The failure mode is easy to state: a sentence such as “X may explain Y” must not silently become “X explains Y” in a structured representation. In modal terms,

Preserving this distinction is a small requirement, but a foundational one: every later stage of compilation — grounded extraction, verification, knowledge-graph construction — inherits whatever epistemic errors are made here. That is why this study comes first in the research program.

02Research question

Formally, given a proposition $s$ and its surrounding context $c$, a model is asked to produce a classification and, where supported, a confidence:

where $\mathcal{E}$ is the closed set of epistemic acts defined in the proposed taxonomy below, and $\gamma$ additionally admits abstention — a model should be able to say that a passage is ambiguous rather than being forced into a category.

03Hypothesis

H1. Current language models will perform well on explicit linguistic cases — for example, propositions introduced with phrases such as we hypothesize, we observed, or previous studies suggest — but performance will decline when the epistemic act must be inferred from broader discourse context.

The most interesting failures may therefore occur not in proposition extraction itself, but in determining what status the proposition has within the author’s argument.

04Proposed taxonomy

The study begins from a deliberately small operational taxonomy of six epistemic acts. It is not intended as a complete theory of scientific discourse, and it may be revised as annotation exposes ambiguities.

Observation

The authors report something measured, detected, or directly observed.

Assertion

The authors present a proposition as something they take to be true.

Inference

The authors derive or interpret a proposition from observations, results, or other evidence.

Hypothesis

The authors propose a possibility that remains uncertain or requires further testing.

Reported claim

The authors describe a proposition attributed to another source without necessarily adopting it themselves.

Challenge

The authors dispute, weaken, or question an existing proposition.

05Study design and methods

Corpus construction

A corpus of scientific passages will be constructed by sampling published papers across fields. Each example consists of:

• the relevant scientific text,
• the proposition being evaluated,
• a human-assigned epistemic-act label, and
• surrounding context, where the sentence alone is insufficient to decide.

Model evaluation

Multiple language models classify the same examples under a fixed schema, with controlled variation in model choice, prompting, terminology, and the amount of surrounding context provided. Where a model supports it, confidence is recorded and abstention is permitted rather than forcing every sentence into a category.

Repetition and disagreement

Each evaluation is repeated to measure consistency, and disagreement between independent models is retained as a signal: cases where models diverge are candidates for genuinely ambiguous language, and are examined against human judgment rather than discarded.

06Measures

The primary outcome is agreement between model classifications and the reference annotation, reported with chance correction as Cohen’s $\kappa$:

Secondary measures characterize how models fail rather than only how often:

• Inter-model agreement — whether independent models converge on the same labels.
• Consistency — stability of a model’s answers across repeated evaluations.
• Confusion structure — which categories are mistaken for which (e.g. inference read as assertion).
• Context sensitivity — how classifications change as surrounding context is added.
• Calibration and abstention — whether reported confidence tracks correctness, and whether models abstain on cases humans also find ambiguous.

07Expected contributions

• An annotated evaluation corpus for scientific epistemic acts.
• A reproducible classification benchmark.
• A comparison of current language models under controlled conditions.
• An error taxonomy: which distinctions are reliable, which are consistently ambiguous — even for humans.
• Observations about which epistemic distinctions are suitable for automated compilation.

08Future work

The results will determine the next research question: whether grounding each classification in exact textual evidence can make epistemic extraction more reliable.

That question — grounded extraction — is the second problem area of the Machine Compilation of Scientific Knowledge program, followed by verified compilation, selective compilation, and the design of a scientific intermediate representation.