# Named-entities page sources The /named-entities page reports one pre-registered check of an open general model, `numind/NuNER_Zero-span` (NuNER Zero-span, MIT licence, revision `063b4dec`, threshold 0.5), served by the open-source `sie-server`. It is compared with the ways a team extracts its own entity types today: AWS Comprehend, spaCy, and an LLM prompt with a JSON schema. - Plan, published before any output: [superlinked/sie-task-evidence @b3056a1a](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/b3056a1a4a2038a93a9d07b7a9388fc287b1add1/ner-custom-types-general-prereg/2026-10-09) (`CUSTOM-TYPES-PLAN.md`, sha256 `70fba5ab…`). - Results, rows and scores: [@8474bd33](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/8474bd33ca56242a4a297804499c094145397297/ner-custom-types-general-results/2026-10-09). - Display pick rule and the selected sentence: [@6f15fb48](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/6f15fb489237fcc9309c36f6d889e1d4ff808201/ner-custom-types-general-results/2026-10-09/display). - Harness: `tools/run_named_entities.py` and `tools/run_named_entities_modal.py` in the benchmark repository at commit `bc16e467c789`, unchanged. - Cost packet for the same 652 sentences: [@2b106bfa](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/2b106bfac795edf931c1fc63976378d0b2e06661/ner-custom-types-general-cost/2026-10-10) (`COSTS.json`, sha256 `cc744c9e…`). The proof's "Run this eval" link opens it. Every quality figure on the page comes from the published results, and every cost figure from the cost packet's `COSTS.json` after its digest is checked. The site's importers write both, and the page build fails if the Comprehend or spaCy claim stops holding, or if the cost packet's SIE rate stops matching the rate the site publishes. ## The test The standard zero-shot, out-of-domain NER benchmark: the five CrossNER domains (AI, literature, music, politics, science; github.com/zliucr/CrossNER, MIT licence) and MIT Movie and MIT Restaurant (groups.csail.mit.edu/sls). Each set has its own entity types, 8 to 17 of them, such as "programming language", "political party", "chemical compound" or "restaurant name". SIE and the LLMs are asked for exactly the set's types, as the same English label strings. Comprehend and spaCy cannot be asked for types; their own types are mapped to the set's, as described under Models and settings. The sample is drawn from each set's test split with a fixed seed, sentence by sentence, until the set has at least 350 gold entities: ai 87 sentences, 350 entities; literature 72, 359; music 51, 352; politics 60, 355; science 57, 351; MIT Movie 158, 350; MIT Restaurant 167, 351. That is 652 sentences and 2,468 gold entities in all. The metric is micro F1 over exact matches of start, end and type, pooled over all seven sets. A span with the right text and the wrong type is a miss. LLMs return strings, not positions, so each returned string is matched to every place it occurs in the sentence on word boundaries; a string that occurs nowhere counts as a false positive (the "ungrounded" column). Intervals are paired 95% sentence-cluster bootstrap intervals, 2,000 resamples, seed 20260930. ## How the model was chosen On the sets' development splits, never on test. The plan reuses the recorded development screen of the original study (`e0_results.json` at `bc16e467`, sha256 `037cf88d…`): the CrossNER development files plus 500 MIT training sentences per set, the same label strings, one sentence per input item, a threshold grid from 0.2 to 0.8 plus each model's default. The candidates are the general-purpose GLiNER-family models that the public SIE catalog serves; domain-specific, PII, relation and decision models are excluded. The rule, unchanged from the original study: the highest tuned development macro F1 wins, and among models within 1 point of the best, the cheapest wins. `gliner-community/gliner_large-v2.5` scored 64.03 at threshold 0.8 and NuNER Zero-span 63.70 at 0.5. Neither had a rate row, so the rule priced each at its nearest catalog sibling: $0.05 per million input tokens for NuNER Zero-span against more than that for gliner_large-v2.5. NuNER Zero-span was picked at 0.5, which is also its default. Before the test run, the same harness reproduced its development score on `sie-server` 0.9.0 (PyPI) with the public catalog configurations at superlinked/sie v0.9.0: macro F1 0.6370 against the recorded 0.6370 on 3,121 sentences, with every span slicing back to its offsets. ## Models and settings - **SIE NuNER Zero-span**: one sentence per input item, the set's labels, threshold 0.5, `sie-server` 0.9.0, run on 9 October 2026. - **GPT-6 Luna, GPT-6 Sol** (OpenAI) and **Claude Haiku 4.5** (Anthropic), recorded on 30 September 2026 and re-scored from their rows, with no new calls: reasoning off where offered, temperature 0, a strict JSON schema whose type field is an enum of the set's labels, and this system prompt: > Extract every entity of these types from the text: {labels}. Give each > entity exactly one type from the list. Copy each entity exactly as it > appears in the text. Return an empty list if there are none. - **AWS Comprehend** `DetectEntities`, built-in model, us-east-1, recorded on 30 September 2026. - **spaCy** `en_core_web_trf` 3.8.0, recorded on 30 September 2026. Comprehend and spaCy cannot be asked for a type; they return their own (nine and eighteen). So each of their types is mapped to one of the set's types, or to none, by whichever choice gives that product the highest F1 on the sets' development splits. The maps were fitted and frozen, with their hashes, before either product saw a test sentence. This is as generous to them as the data allows. The [published NuNER rows](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/8474bd33ca56242a4a297804499c094145397297/ner-custom-types-general-results/2026-10-09/rows/main) contain 652 sentence results in 44 consecutive groups with identical recorded per-item latency, consistent with batches of up to 16 sentences. These records do not establish the number of individual requests, retries or billable events. The plan's one-sentence-per-request wording does not describe this grouping. ## Results | Arm | F1 | Precision | Recall | Macro F1 | SIE minus it (95% interval), points | Ungrounded strings | | --- | --- | --- | --- | --- | --- | --- | | SIE NuNER Zero-span | 63.0% | 70.2% | 57.2% | 62.7% | | 0 | | GPT-6 Luna, no reasoning | 66.1% | 66.2% | 66.0% | 65.8% | −3.1 (−5.7 to −0.3) | 16 | | Claude Haiku 4.5 | 63.3% | 66.5% | 60.4% | 62.4% | −0.2 (−2.8 to +2.2) | 145 | | GPT-6 Sol | 74.3% | 76.3% | 72.4% | 74.0% | −11.2 (−13.5 to −8.8) | 10 | | AWS Comprehend | 39.0% | 45.0% | 34.4% | 37.5% | +24.0 (+21.0 to +27.0) | 0 | | spaCy en_core_web_trf | 35.0% | 44.2% | 29.0% | 33.9% | +28.1 (+24.6 to +31.3) | 0 | Per set (exact-span F1; GPT-6 Luna with no reasoning, as the study ran it): | Set | SIE NuNER Zero-span | GPT-6 Luna | Claude Haiku 4.5 | GPT-6 Sol | AWS Comprehend | spaCy en_core_web_trf | | --- | --- | --- | --- | --- | --- | --- | | ai | 60.8% | 53.1% | 63.3% | 64.2% | 29.7% | 28.7% | | literature | 68.2% | 71.8% | 64.5% | 75.1% | 54.6% | 51.6% | | music | 70.8% | 74.0% | 71.8% | 85.2% | 36.4% | 34.2% | | politics | 72.1% | 73.9% | 76.7% | 79.4% | 52.7% | 38.9% | | science | 66.9% | 68.3% | 69.6% | 74.3% | 37.6% | 30.7% | | mit movie | 55.2% | 63.2% | 44.5% | 75.6% | 28.7% | 27.8% | | mit restaurant | 44.9% | 56.7% | 46.4% | 63.9% | 22.7% | 25.5% | ## Current API models (rerun, 11 October 2026) The current Anthropic and OpenAI models ran on the same 652 sentences in a separate pre-registered rerun, written before any test sentence was sent: [`current-comparators/2026-10-11/named-entities`](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/fd38d4b9f9137e6dd1dde42cdf6c5db515b9b2ee/current-comparators/2026-10-11/named-entities/results) at revision `fd38d4b9f9137e6dd1dde42cdf6c5db515b9b2ee`. Scoring used the rules of the public `examples/custom-entity-types/score.py`, unchanged. Re-scoring the recorded arms with it reproduces the figures above exactly. No request failed, was refused or was cut at its cap. The site's importer checks `results.json` against its pinned SHA-256 and recomputes each arm's cost from its recorded tokens. | Arm | Sentences | F1 | Precision | Recall | SIE minus it (95% interval), points | Ungrounded strings | $ per million sentences | | --- | --- | --- | --- | --- | --- | --- | --- | | Claude Haiku 5.5, no thinking | 652 | 71.4% | 75.6% | 67.7% | −8.3 (−10.7 to −5.8) | 57 | $86.78 | | GPT-6.1 Sol, low reasoning | 652 | 78.1% | 78.1% | 78.1% | −15.1 (−17.4 to −12.8) | 10 | $1,047.41 | | GPT-6.1 Sol, high reasoning | 652 | 79.2% | 79.0% | 79.4% | −16.1 (−18.5 to −13.7) | 11 | $2,160.58 | | Claude Sonnet 5.5, no thinking | 652 | 79.6% | 79.6% | 79.7% | −16.6 (−18.7 to −14.4) | 13 | $1,804.32 | | Claude Sonnet 5.5, adaptive thinking, high effort | 140 | 78.9% | 77.7% | 80.2% | −13.3 (−17.8 to −9.3) | 3 | $2,066.20 | - **Settings.** "No thinking" is each Claude model's lowest setting: `disabled` for Claude Haiku 5.5 and `between_tools` for Claude Sonnet 5.5, which rejects `disabled`. GPT-6.1 Sol ran at `low`, its lowest accepted setting, and at `high`. Claude Sonnet 5.5 with adaptive thinking ran on a 140-sentence subset in this rerun (SIE 65.6% on those 140); the follow-up below completed it on all 652. - **Forced deviations.** All three models reject temperature 0, so each ran at its API default. - **Prices.** Standard real-time list prices read on 11 October 2026, per 1M input and output tokens: GPT-6.1 Sol $2 and $10, Claude Sonnet 5.5 $2 and $10, Claude Haiku 5.5 $0.10 and $0.50. Reasoning tokens bill as output. Anthropic's structured-output requests count about 550 input tokens per sentence because the schema is billed, against about 200 on OpenAI. - **Latency** came from a laptop client with four requests in flight, so it is not compared with SIE's recorded time from inside the serving cloud. ## Each LLM at its strong setting (follow-up, 13 October 2026) The chart shows each LLM at its best-scoring setting, priced at that setting. After the rerun, GPT-6 Luna and Claude Haiku 5.5 had one setting each on all 652 sentences (reasoning and thinking off), and Claude Sonnet 5.5's adaptive setting had 140 sentences. A follow-up added the missing strong setting of each, in its own unit written before any test sentence was sent to these arms: [`current-comparators/2026-10-11/named-entities/best-setting-20261013-v1`](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/e8f82fe7a11a352fb55ef39742f50f20e72556ed/current-comparators/2026-10-11/named-entities/best-setting-20261013-v1/results) at revision `e8f82fe7a11a352fb55ef39742f50f20e72556ed` (pre-registration at `76a4a40e13232c9f27cba50d5b1802ba9471e23b`). Same sentences, prompt, strict schema, runner and scorer as the rerun; the SIE arm is the recorded rows. No request failed, was refused, was cut at its cap or needed a retry. The site's importer checks this `results.json` against its pinned SHA-256 too, and checks that it re-scores the rerun's arms to the rerun's own figures. | Arm | Sentences | F1 | Precision | Recall | SIE minus it (95% interval), points | Ungrounded strings | $ per million sentences | | --- | --- | --- | --- | --- | --- | --- | --- | | GPT-6 Luna, high reasoning | 652 | 69.4% | 68.1% | 70.7% | −6.3 (−8.6 to −4.0) | 14 | $159.25 | | Claude Haiku 5.5, adaptive thinking, high effort | 652 | 73.7% | 77.7% | 70.1% | −10.7 (−12.8 to −8.3) | 9 | $183.78 | | Claude Sonnet 5.5, adaptive thinking, high effort | 652 | 79.5% | 78.6% | 80.5% | −16.5 (−18.7 to −14.3) | 14 | $1,859.41 | - **Settings.** GPT-6 Luna with `reasoning_effort: "high"`; Claude Haiku 5.5 and Claude Sonnet 5.5 with `thinking: {type: "adaptive"}` and effort `high`; a 16,000-token output cap, because reasoning counts against it. `xhigh` and `max` were not run for any model. - **Claude Sonnet 5.5 on all 652.** The 140 subset rows of 11 October were kept and not sent again; the other 512 sentences ran on 13 October with the same settings. The arm is scored as one. - **Prices.** Standard real-time list prices read on 12 October 2026, per 1M input and output tokens: GPT-6 Luna $0.10 and $0.50, Claude Haiku 5.5 $0.10 and $0.50 (prompts up to 100,000 tokens), Claude Sonnet 5.5 $2 and $10, GPT-6.1 Sol $2 and $10: the prices the earlier runs used ([developers.openai.com/api/docs/pricing](https://developers.openai.com/api/docs/pricing), [platform.claude.com/docs/en/about-claude/pricing](https://platform.claude.com/docs/en/about-claude/pricing)). Reasoning and thinking tokens bill as output: GPT-6 Luna used 146,351 reasoning tokens over the 652 sentences, which is most of its cost at this setting. No request read or wrote a prompt cache. - **Latency** came from the same laptop client with four requests in flight per arm; the page states none. ### Which setting the chart shows The rule was written into the follow-up's plan before it ran: each model is charted at the setting with the highest F1 among its settings measured on all 652 sentences, the cheaper one if two agree to the 0.1 point shown, and it is priced at that setting. The 14 October page audit shows one setting per model and leads with the AWS quality and price comparison. Highest F1 is selected using the full recorded score; cost breaks only exact ties. This does not change the chosen settings here. The former extra cheapest-Luna point and price-multiple heading are retired; their historical [addendum](https://huggingface.co/datasets/superlinked/sie-task-evidence/blob/de76ec5ee9e1daa87f1eab0d6fd730ccbeacad5f/current-comparators/2026-10-11/named-entities/best-setting-20261013-v1/results/ADDENDUM.md) and all measured settings remain available. | Model | Setting | F1 | $ per million sentences | On the page | | --- | --- | --- | --- | --- | | GPT-6 Luna | no reasoning (30 September) | 66.1% | $42.77 | retained historical setting | | GPT-6 Luna | high reasoning | 69.4% | $159.25 | **charted**: its best-scoring setting | | Claude Haiku 5.5 | no thinking | 71.4% | $86.78 | | | Claude Haiku 5.5 | adaptive thinking, high effort | 73.7% | $183.78 | **charted** | | GPT-6.1 Sol | low reasoning | 78.1% | $1,047.41 | | | GPT-6.1 Sol | high reasoning | 79.2% | $2,160.58 | **charted** | | Claude Sonnet 5.5 | no thinking | 79.6% | $1,804.32 | **charted** | | Claude Sonnet 5.5 | adaptive thinking, high effort | 79.5% | $1,859.41 | | The page rounds each rival's price down. ## What the page claims, and why - **"61% higher F1 than AWS Comprehend."** The printed primary scores are 63.0% and 39.0%; floor((63.0 / 39.0 - 1) × 100) is 61%. The registered exact-span-and-type F1 difference is +24.0 points (paired 95% interval +21.0 to +27.0), so the pre-registered positive-lower-bound gate passes. The earlier recall-led claim is replaced by this primary-metric claim. - **"99% cheaper" than AWS Comprehend.** The figure is the list-price projection for 652 sentences. The printed bars round SIE up to $1.38 and Comprehend down to $307 per million sentences: floor((1 - 1.38 / 307) × 100) = 99%. - **The workload.** 652 sentences, 83,356 characters, a mean of 128 characters a sentence. Comprehend was called once per sentence; the SIE result rows show the grouping described above. - **Comprehend's price.** `DetectEntities` at its first tier, $0.0001 per 100-character unit ([aws.amazon.com/comprehend/pricing](https://aws.amazon.com/comprehend/pricing/), read 30 September 2026), the tier a million sentences a month falls in, with its 3-unit minimum on each request: 2,008 billable units. - **SIE's price.** 17,946 input tokens at $0.05 per million. SIE bills input tokens, so another text's figure depends on its tokens per character. - **How much the figure depends on the minimum charge and the tier.** The 3-unit minimum is most of the gap on sentences this short. At Comprehend's cheapest public tier ($0.000025 per unit, above 50 million units a month) Comprehend's same 652 requests cost $0.0502, and SIE's token-price projection is 98% less. On long text, where the minimum never applies, that tier is $0.25 per million characters against SIE's $0.0108 on these characters: 95% less, the least favourable case retained here. - **Every current LLM finds more entities than SIE.** The chart shows each current model at its best-scoring setting. The full table above retains each other setting. At their best-scoring settings they are ahead of SIE by 6.3 to 16.6 F1 points, at 115 to 1,569 times SIE's price per sentence. At their other settings they are ahead by 3.1 to 16.5 points, at 31 to 1,351 times. The page claims no quality win and no match against any LLM. Claude Haiku 4.5 and GPT-6 Sol ran in the original study and stay in the tables above. Claude Haiku 5.5 and GPT-6.1 Sol have replaced them, so neither old figure stands for its current model. - **Exact positions.** Every span SIE returned slices back to its offsets. The LLMs return strings, and between 9 and 145 of theirs occur nowhere in the text. - The result is pooled over seven domains, not a guarantee in each. Comprehend is behind on every set; GPT-6 Luna with no reasoning is ahead on six of the seven. ## Disclosures - **The test sample is not fresh.** It was used on 30 September 2026 to confirm a domain-specific model that this page does not use, and that result was known when the plan limited the candidates to general models. The limit is a product rule (an industry model belongs on a solutions page), and the pick came from the development screen, not from test. - The sets are public and older than every model tested, so some models may have seen them in training. That would favour the largest training corpora, the LLMs'. - After the test run finished and all 652 rows were written, the local Modal client reported a dropped connection. No request was repeated. ## The example sentence The hero, the AWS Comprehend comparison and the API call all use one test sentence, picked from the recorded rows by a rule fixed before looking at candidates: SIE matches every gold span with no false span, Comprehend's mapped output finds none of them, and the sentence has at least three gold spans of at least two types; CrossNER sets first (science, ai, music, politics, literature), then MIT Restaurant and MIT Movie; within a set, most gold spans first, then the lowest sentence id. The pick is `science-main-14` (CrossNER science, MIT licence). The example sentence is shown exactly as recorded: CrossNER text is pre-tokenized, so punctuation is separated by spaces ("Heidelberg , the"). The page shows Comprehend's verbatim `DetectEntities` types and spaCy `en_core_web_trf`'s verbatim types for that sentence, both recorded on 30 September 2026 (spaCy's row is `custom-entity-types/rows/spacy-trf/science.jsonl` at [@6f15fb48](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/6f15fb489237fcc9309c36f6d889e1d4ff808201/custom-entity-types/rows/spacy-trf)), and SIE's recorded spans from the 9 October run, each grouped by the type it returned. The source marks the requested type with the most gold spans in the sentence (`university`, three spans). The search row is caller-side logic applied to the three recorded replies, not a reported incident: keeping spans of type `university` keeps 0 of 3 from Comprehend and 0 of 3 from spaCy, which returned them as ORGANIZATION and ORG, and 3 of 3 from SIE. ## The API call and price The API chapter shows the recorded request and reply for the example sentence: `numind/NuNER_Zero-span`, the science set's 17 labels, threshold 0.5, recorded on 9 October 2026 on `sie-server` 0.9.0 in the same run as the test rows. Copy and Run send the same request to the SIE API. The model's public configuration is in [superlinked/sie v0.9.0](https://github.com/superlinked/sie/blob/v0.9.0/packages/sie_server/models/numind__NuNER_Zero-span.yaml). NuNER Zero-span's rate is in the published rate book `2026-10-13-production-bootstrap-v36` (SHA-256 `3bf8dbe74220efe78b14b573715e68cc41a17581b80e74dca70d962fcd3bef68`): `extract:numind/NuNER_Zero-span:default:input_tokens:us`, $0.05 per million input tokens, the same as NuNER Zero. The Managed Cloud card prices its credit workload from that row, and the proof's cost lane uses the same rate. ## Cost per million sentences The proof's second lane is the list-price projection for the same 652 saved sentences (seven sets, 83,356 input characters), from the cost packet, scaled to a million sentences like these (× 1,000,000 / 652). It adds no inference and no quality scoring. Lower is better. The packet prices the chart's SIE, AWS Comprehend and GPT-6 Luna no-reasoning points; the superseded arms are listed for the record. The other charted LLM points are priced in the two sections above, each from its own run's recorded tokens. | Arm | List-price projection for 652 sentences | Per million sentences | Shown as | | --- | --- | --- | --- | | SIE NuNER Zero-span | $0.0008973 | $1.3762 | $1.38 | | GPT-6 Luna, no reasoning | $0.027886 | $42.770 | $42 | | AWS Comprehend | $0.2008 | $307.98 | $307 | | Claude Haiku 4.5 (superseded, not charted) | $0.457201 | $701.2 | | | GPT-6 Sol (superseded, not charted) | $0.54621 | $837.7 | | - **SIE NuNER Zero-span:** 17,946 input tokens, counted by the model's public SIE processor and tokenizer on CPU, times $0.05 per million input tokens: the v36 row `extract:numind/NuNER_Zero-span:default:input_tokens:us`. The token count is a deterministic projection; the 9 October run did not record returned usage. The page build checks that the packet's rate is the site's published rate and that tokens times rate equals the total. This projection excludes terminal-event rounding; the retained rows do not establish the event grouping or settled charge. The CPU recount does not establish the original run's full tokenizer/runtime identity or numerical equivalence under a different batch configuration. - **GPT-6 Luna, GPT-6 Sol, Claude Haiku 4.5:** the input and output tokens each provider returned on the saved 30 September rows, at Standard list prices verified on 10 October 2026: Luna $0.10 / $0.50, Sol $2 / $10, Haiku $1 / $5 per million input / output tokens. Batch discounts are not used; the observed requests were Standard. - **AWS Comprehend:** 2,008 billable 100-character units at $0.0001, its standard tier. That includes the 3-unit minimum on each of the 652 requests. The original study's report had omitted that minimum; the packet corrects it. - **spaCy** runs on your own hardware and has no served-API price, so it has no cost row. No compute price is invented for it. - Rounding goes against SIE: its cost is rounded up to the cent and every rival's down to the dollar. - These are list-price projections, not invoices. The chart makes no claim that SIE matches the LLMs' quality or speed: every LLM arm, at every setting, is above SIE on F1. `recompute_prices.py` in the packet recomputes the arithmetic offline, and `count_native_inputs.py` recounts the SIE input tokens with the public meter on CPU. ## Latency Not on the page. The plan measured quality only. ## The pain - The example sentence above: AWS Comprehend and spaCy answer in their own fixed types, so a search for the requested type finds none of the three universities. - AWS's own route to custom types, read at the sources on 11 October 2026: - [Amazon Comprehend Developer Guide, "Guidelines and quotas"](https://docs.aws.amazon.com/comprehend/latest/dg/guidelines-and-limits.html#limits-custom-entity-recognition): a custom entity recognizer takes 1 to 25 entity types. It is trained either from annotated documents, with a minimum of 25 annotations per entity type for plain text (PDF training needs 100 per type and 250 to 10,000 documents), or from an entity list plus unannotated documents, with at least 25 matches per entity type ([entity lists](https://docs.aws.amazon.com/comprehend/latest/dg/cer-entity-list.html)). - [Amazon Comprehend pricing](https://aws.amazon.com/comprehend/pricing/): training at $3 an hour, $0.50 a month model management, and a real-time endpoint at $0.0005 a second for each inference unit (100 characters a second). One unit kept running for 30 days is $1,296. - Amazon Comprehend Developer Guide, "Entities" (the nine built-in types): https://docs.aws.amazon.com/comprehend/latest/dg/how-entities.html ## Not measured - AWS Comprehend Custom Entity Recognition, trained on each set's training split: the study account could not train models. - Reasoning settings above `high` (`xhigh`, `max`) for any LLM, and GPT-6.1 Sol and GPT-6 Luna at their `medium` default. - Google Cloud Natural Language `analyzeEntities`: no key. ## Limits - One run per arm. Hosted models can change behind the same name. - Sentences, not whole documents. The sets are sentence-level. - An ordinary-names comparison against GPT-6 Luna on fresh documents is planned separately; it is not on this page. The 30 September study this sample comes from, with the recorded rows of the LLM, Comprehend and spaCy arms above, is at [superlinked/sie-task-evidence @8736554b](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/8736554b7afd71f05a9e53daef6be72fb8fe176f/custom-entity-types). Its runnable example, [`examples/custom-entity-types`](https://github.com/superlinked/sie/tree/7860d1f28e17c899db8f011ae8b4d5ccb76ab8c1/examples/custom-entity-types), fetches those rows and reproduces their scores with no key. ## Page audit and additional pain selection, 14 October 2026 Before inspecting additional per-sentence outputs, freeze this display rule: use GPT-6.1 Sol, the strongest measured OpenAI model, and choose the first sentence by dataset name and sentence ID with an incorrect entity text or wrong entity type that persists at both its recorded low and high settings. The visible source sentence and requested types must make the error clear. Exclude whitespace-only differences, valid alternate span boundaries and ambiguous entity typing. This is an illustration selected from existing outputs, not a new independent quality estimate. If no case meets the rule, report the gap instead of selecting a weaker setting. Preserve the current AWS Comprehend example and all historical results. The Sol audit found no qualifying factual error: invented text did not persist at both settings, and the remaining shared type mismatches involved alternate valid types or questionable dataset labels. Examples excluded from display include MATLAB as a product or programming language, ACM and AAAI as organizations or conferences, and mythology names labelled as astronomical objects. A benchmark mismatch alone does not establish a factual mistake. Display amendment, recorded before opening Haiku's per-sentence rows: apply the same rule to Claude Haiku 5.5 across every recorded setting (no thinking and adaptive thinking at high effort). Keep the same dataset and sentence-ID order, factual-error requirement and exclusions. This amendment selects an illustration from the completed study, whose aggregate results were already known; it is not a prospective measurement plan. If no case qualifies, keep the AWS Comprehend and spaCy limitations and report the missing LLM example. The Haiku audit also found no qualifying persistent factual error under this rule. No invented string persisted at both settings after harmless formatting was normalized. Persistent type disagreements included field/task categories, product/programming-language overlap and rating/review distinctions. The page keeps its recorded AWS Comprehend and spaCy type limitations; it does not present a current LLM mistake that this audit could not substantiate.