Measurement note 01 · 2026-08-06
Cross-platform character consistency
One character, seven arms across five independent base models, the same five scenes each, five metrics — plus three further rounds reported separately because their conditions differ. Every number came from one day's work, and the limits of that are stated before the results.
The useful part of this note is not the ranking. It is the measurement setup — and the three places where our own earlier conclusions turned out to be wrong.
Method
Identity similarity alone is easy to game. A model that ignores the prompt and reproduces the reference framing every time will score extremely well on any embedding metric, because the output resembles the reference — that is the whole trick. The run below contains a clean example of exactly that.
So every arm is reported on three axes that can disagree:
| Axis | Instrument | Reads |
|---|---|---|
| Identityis it the same person | SFace & ArcFace, on the largest detected face | Same-person thresholds 0.363 / 0.30 |
| Whole subjectdoes it look like the character | Unicom ViT-B/16, background removed | Normalised between floor and ceiling |
| Prompt adherencedid it do what was asked | CLIP-T-C, image against its own scene text | Compared against a deliberately mismatched baseline |
| Rigiditydid it just repeat itself | Within-set similarity across the five outputs | High values mean the composition collapsed |
Normalisation. Raw cosine values are meaningless on their own, so each is placed between two measured endpoints. The ceiling is how similar the character's own reference images are to each other (0.7236 here) — the practical upper bound. The floor is how similar this character is to unrelated characters (0.1951). Prompt adherence gets the same treatment: the same images scored against the wrong scene text give a baseline of ≈0.199, which is what "ignored the prompt" actually looks like.
Two rules that came out of getting this wrong before. Photoreal humans are judged on face metrics, not whole-image embeddings — the latter are dominated by pose, lighting and framing. Non-human and stylised characters have no usable face, so they are judged on Unicom with the background removed and replaced by a uniform grey; without that step the score tracks the background rather than the character.
Results
Photoreal human character. Same five scene prompts, same reference set, same ceiling and floor. Reference-image count differs per arm because it is each platform's native limit — flattening that would test something no one actually does.
| Arm | Base model | Unicom | SFace ≥.363 | ArcFace ≥.30 | Prompt | Rigidity |
|---|---|---|---|---|---|---|
| GPT Image 2reference · 3 img | OpenAI | 83.3% | 0.5152 | 0.5290 | 0.2360 | 0.6395 |
| Flux LoRA + anchorfine-tune · 5 img | Black Forest Labs | 90.3% | 0.4583 | 0.4723 | 0.2432 | 0.6110 |
| Seedream 4reference · 3 img | ByteDance | 87.2% | 0.4865 | 0.4523 | 0.2506 | 0.6177 |
| Character packmultimodal · 5 img | Google Gemini | 61.4% | 0.4173 | 0.4369 | 0.2447 | 0.4242 |
| Runway Gen-4reference · 3 img | Runway | 85.5% | 0.3396 | 0.3972 | 0.2401 | 0.6478 |
| Flux LoRA alonefine-tune · no text | Black Forest Labs | 44.6% | 0.3601 | 0.3506 | 0.2675 | 0.2815 |
| Flux 1.1 proreference · 1 img | Black Forest Labs | 96.9% | 0.3251 | 0.2581 | 0.2149 | 0.8731 |
Ceiling 0.7236 · floor 0.1951 · prompt-adherence mismatch baseline ≈0.199. Tick marks on each gauge show the same-person threshold for that instrument. Prompt and rigidity columns are raw values, comparable within this table only.
Base-model overlap must be disclosed. Platforms are not independent evidence just because they are different products. Higgsfield's character system runs on Google's Nano Banana Pro — the same family as the Gemini arm. Three of the seven arms above sit on Flux. Counting brands instead of base models inflates apparent coverage.
Findings
It has the highest whole-subject score in the table (96.9%) and the lowest identity score (0.2581, below threshold). Its rigidity value of 0.8731 is higher than the character's own reference set — the five outputs resemble each other more than the references do. Visual inspection matches: all five are the same half-body studio portrait, and three of the five scene descriptions were simply not executed. The high score is a by-product of composition collapse, not consistency.
A LoRA trained on the character's five reference images, prompted with only a trigger word and a scene, failed the identity threshold (0.3601) and produced a long-haired woman for one of the five scenes. Nothing in the prompt stated the character's gender; the weights alone did not hold against the scene text.
Adding the character pack's written anchor to the same LoRA moved it to 0.4583 and 90.3%. The conclusion is not "don't fine-tune" — it is that the written specification is doing work the weights do not do, and the two compose.
Sweeping Midjourney's Omni Reference weight across five points, identity and prompt adherence both rise together (0.3448→0.4834 and 0.2032→0.2260). At the low end, prompt adherence sits at the mismatched-text baseline — the output has no measurable relationship to the scene description at all. Turning the dial down buys nothing.
None of the five sweep outputs produced the requested full-body street shot. Swapping in a full-body reference image made the framing tighter, not wider. Removing the facial micro-details from the anchor — implant line, heterochromia, temple etching — while keeping silhouette and garment widened the shot immediately.
The exchange rate is poor: prompt adherence rose 7% while ArcFace identity fell 67%. Wide framing is bought with facial identity.
Two platforms, same result. Training an anime character into Higgsfield's Soul ID completed successfully and retained 0% of its declared identity markers; attaching the same images as plain references retained 100%. Binding the same character into Kling's Element system produced no measurable gain over not binding it.
The mechanism explains it: these systems lock the face, and a stylised character's identity mostly lives elsewhere — hair colour, a mark, a garment palette.
Three arms scored high on whole-subject similarity while failing the face threshold: Runway (85.5% / 0.3396), Flux (96.9% / 0.2581), and a full-body-reference variant (99.2% / 0.2418). In each case inspection showed the same styling on a different face. For photoreal humans, whole-image embeddings are not a substitute for a face metric.
Field notes
--ow 0–1000 parameter. Typing --ow 200 leaves the slider
untouched.custom_reference_id.
Passing --soul-id raises no error and quietly returns an image with no
character applied — it scored at the floor.completedsuccess. A poller waiting on the wrong string never exits.
And a completed training is not a usable one: verify by generating.Corrections
These are included because a benchmark with no failed hypotheses has not been audited.
Reference strength trades character fidelity against compositional
freedom, so it should be tuned down for scene-heavy shots.
Both rise together across the full range. There is no trade to make; the dial should sit at maximum.
Composition is locked to the reference image's framing and cannot be
released.
It is not the reference — a full-body reference produced a tighter crop. It is the proportion of facial detail in the prompt, and it can be released by rewriting the anchor.
Kling exposes a three-step reference-strength control that would let us
test whether the trade-off generalises.
No such control exists in the product. The claim came from a third-party article and was not verified before planning around it.
Limits
Every image, prompt, and score in this note came from scripts that take the character specification as input and write the measurement table as output. The setup is designed to be re-run against new platforms as they appear, which is the only way a note like this stays true for longer than a quarter.