Using a Model to Score Signup Risk from Device and Email Signals

No single attribute proves fraud, and a fingerprint is best read as linked evidence rather than a verdict (SEON). Group signals so the model can reason about combinations, and keep the groups small enough that every feature earns its place.

Two design rules follow from how fingerprinting behaves. Prefer signals that persist: hardware and rendering attributes survive browser changes and private browsing, while cookie identifiers do not (Plaid). And cap the list. Every feature you add is one more the model must weigh consistently, and a list of forty invites an inconsistency no calibration curve repairs.

A worked example of why conflicts matter. Signup A: new device, residential IP in the same metro as the billing address, email domain registered nine years ago with valid SPF and DMARC. Signup B: new device, residential IP, email on a domain registered two days ago with no MX record. Both are "new device, residential IP." A rule engine scores them the same. A model given the email features should separate them by a wide margin, and the reasons field should say "email domain age 2 days, no MX record" so a reviewer checks the logic in one glance.

The prompt has three jobs: fix the feature set, fix the output shape, and anchor the scale so a score means the same thing across signups. Do all three in the system prompt, and put only the feature object in the user message.

System:
You are a signup risk scorer. You do not decide whether to block.
Input: a JSON object of features with exactly the keys listed below.
Output: one JSON object, nothing else, matching:
{
  "score": <integer 0..100>,
  "band": "low" | "medium" | "high",
  "reasons": [<exactly 3 short strings, each naming a feature and its value>],
  "missing": [<feature keys that were null>]
}
Score anchors:
0-20: all signals consistent with a normal new user.
21-50: one mild anomaly (new email domain, VPN) with an otherwise clean device.
51-75: two or more anomalies, or one strong device-integrity flag.
76-100: device-integrity flag plus an email or network anomaly, or device linked to 3+ prior accounts.
Never output text outside the JSON. Score only on non-null features and list nulls in "missing".

User:
{"device_new": true, "device_accounts_seen": 0, "emulator_flag": false,
 "ip_type": "residential", "geo_tz_mismatch": false,
 "email_domain_age_days": 2, "email_disposable": false,
 "email_has_mx": false, "email_plus_tag": false, "form_fill_seconds": 41}

Few-shot anchors do calibration that prose cannot. Put three or four labeled examples in the system prompt, each a full feature object with its expected output, chosen to bracket the scale: one clean signup that should land near 10, one ambiguous signup (new email domain, VPN, otherwise clean) near 45, one obvious ring member (emulator plus three prior accounts on the device) near 90. Keep anchors fixed. Rotate them and scores drift, and your threshold stops meaning what it meant last month.

Validate output in code, not by hope. Reject anything that is not parseable JSON, reject scores outside 0 to 100, reject reason arrays without exactly three entries, and treat a rejection as a retry with the same input. Log the feature object, the raw model output and the validation result for every signup. That log is both your calibration dataset and your audit trail.

A model that outputs 70 is not telling you a 70% fraud probability. It is telling you "70 on a scale I was taught with anchors." To make that operational you need labels and a calibration step.

Labels are the hard part, and the bias is the trap. Good labels come from things that happen after signup: chargebacks, spam complaints, takedown notices, referral reward reversals, manual review verdicts. Note that you only get review labels for signups you chose to review, so labeled data is not a random sample of all signups. Review a small random slice of signups in every band, not just the suspicious ones, or your calibration is measured on a skewed slice.

The procedure, in steps:

1. Hold out a random slice of labeled signups, ideally the most recent by signup time so it reflects current attacker behavior.
2. Score the holdout with the exact prompt, anchors and feature code you will run in production.
3. Bucket by score (0-20, 21-40, 41-60, 61-80, 81-100) and compute, per bucket, the fraction of signups that turned out bad.
4. Check that the bad fraction rises monotonically with the bucket. If it does not, the anchors or features are wrong and no threshold saves you.
5. Set the block threshold as the lowest score where the bad fraction justifies a block and the false positive rate (good signups you would turn away) sits below the ceiling your team accepts.

A worked example you can substitute. Suppose your holdout has 10,000 labeled signups and the 81-100 bucket holds 300 of them, of which 240 were bad. That is 80% precision in the bucket: blocking it loses 60 good signups and stops 240 bad ones. If the 61-80 bucket holds 1,200 signups with 300 bad, precision is 25%, far too low to block, and that is exactly the bucket for a challenge. The rule that falls out of the data is not "block above 80" but "block above the score where measured precision justifies the loss, challenge the bucket below, and leave the rest alone."

Rebuild this table on a fixed schedule and after any prompt change. Scores compare to a threshold only while the prompt, anchors and feature set are frozen between calibrations.

The block decision is an economics question wearing a security costume. Every threshold trades a false negative (a bad signup you let through) against a false positive (a good signup you turn away). The right threshold is where the cost of the bad signups you stop exceeds the cost of the good signups you lose, plus the cost of the challenge you would otherwise have spent.

Two refinements. Make the block threshold product-specific at minimum: a free-tier signup with no payment method can tolerate a lower bar than a trial with a card on file. And never let the model's band field be the production trigger. The band keeps the reasoning legible; the trigger should be your code comparing the numeric score to a threshold you measured. When you change the model, you re-measure rather than re-trust.

One failure mode deserves a name: the high score from a single feature. A VPN is common among privacy-conscious users, and a fresh email domain is common among people who register a domain to try a product. The model should give those a moderate score, not a high one, and the anchors are how you enforce that. If calibration shows the 21-50 band full of legitimate VPN users, adjust the anchors before you touch the threshold.

A score is useful only while you know it means what it meant at calibration. Three habits keep it that way.

Log everything and label continuously. Store the feature object, the score, the reasons, the action taken and the eventual outcome for every signup. Outcomes arrive late, since a chargeback can land weeks after signup, so join them back to the log on a schedule. Without that join, your next calibration runs on stale labels.

Watch the score distribution, not only the fraud rate. If the median score on clean signups drifts upward, either the anchors stopped controlling the model or the feature pipeline changed (a field now returning null, a VPN flag firing on mobile carriers). Track the missing-field rate straight from the model's own array. A spike there is a pipeline bug, not a fraud wave.

Treat explanations as a grader. The reasons field is not decoration. Spot-check a sample of blocks by hand each week and confirm the three named reasons actually justify the score. When they do not, you have found a prompt or feature problem before it found you. This is the discipline fraud teams apply when they insist on explainable findings rather than a bare score (TigerGraph).

If you run several model families, a small fast model for the first pass and a larger one to draft review packets, route both through one inference layer so the prompt, schema and logging stay identical. Quanta Inference puts text, image, audio and embedding models behind one OpenAI-compatible API key with prepaid pricing in US dollars (Quanta Inference), which makes a model swap a config change instead of a rewrite.