This repository provides the FraudFeat benchmark, a separately released dataset, and the evaluation suite, together with EvoFeat, our proposed self-evolving multi-agent method for automated fraud-graph feature discovery.
The Explorer in EvoFeat discovers fraud patterns and evidence chains, the Synthesizer implements executable feature plans with validation and repair, and the Evaluator predicts feature utility and selects candidates for measured IV. Episode-wise feedback updates the feature pool, evaluator memory, and reusable playbooks. See the EvoFeat guide for reproduction instructions.
FraudFeat pairs typed Parquet node and edge tables with an English data dictionary, account-level fraud labels, and a Python package for evaluating graph-derived features. Its evaluation package measures effectiveness, structural diversity, semantic diversity, and interpretability. Features can be submitted as DuckDB SQL queries or constrained Gremlin plans.
The released dataset is de-identified.
.
├── EvoFeat/ # Proposed self-evolving agent method
│ ├── src/ # Framework, experiments, and baselines
│ ├── config/ # Model, execution, and graph configuration
│ ├── scripts/ # Reproduction entry points
│ └── examples/ # Illustrative playbook entries and feature plans
├── data/ # FraudFeat dataset (Release asset; not versioned)
│ ├── schema.json
│ ├── labels.parquet
│ ├── splits/ # Official fixed splits (shipped with the dataset)
│ │ ├── target_streams_s{42,43,44}.json
│ │ ├── offline_panel.parquet
│ │ └── evaluation_panel.parquet
│ ├── nodes/*.parquet
│ └── edges/rel_*.parquet
├── evaluation/ # FraudFeat evaluation package
│ ├── fraudfeat/
│ ├── examples/ # SQL and Gremlin feature examples
│ ├── tests/
│ ├── pyproject.toml
│ └── README.md
├── LICENSE
└── README.md
The dataset is distributed only through versioned GitHub Release assets; it is not stored in Git history. Code releases and data releases are versioned independently.
- Data release:
data-v1.0.0 - Archive:
fraudfeat-data-v1.0.0.tar.gz - Checksum manifest:
SHA256SUMS
The archive contains one top-level data/ directory. From a clone with the GitHub CLI:
gh release download data-v1.0.0 \
--pattern 'fraudfeat-data-v1.0.0.tar.gz' \
--pattern 'SHA256SUMS'
sha256sum -c SHA256SUMS
tar -xzf fraudfeat-data-v1.0.0.tar.gzOn Windows, tar -xzf fraudfeat-data-v1.0.0.tar.gz also works in PowerShell; compare Get-FileHash .\fraudfeat-data-v1.0.0.tar.gz -Algorithm SHA256 with SHA256SUMS. Alternatively, download both files from the release page, verify the checksum, and extract the archive at the repository root.
To keep the dataset elsewhere, supply its location through --data and --schema. No dataset files need to be committed.
| Component | Scale |
|---|---|
| Entity types | 8 |
| Nodes | 12,394,721 |
| Relation types | 18 |
| Directed edges | 18,065,212 |
| Transaction edges | 5,319,577 across 10 relation types |
| Non-transaction edges | 12,745,635 across 8 relation types |
| Business attributes | 21 |
Labeled wechat_pay_account nodes |
9,004,117 |
Accounts marked as fraud (label = 1) |
276,702 |
Accounts marked as non-fraud (label = 0) |
8,727,415 |
Entity type (entity_ty
7438
pe) |
Nodes |
|---|---|
wechat_pay_account |
9,004,117 |
merchant_pay_account |
536,580 |
individual_customer |
1,191,147 |
natural_person |
1,162,192 |
bank_card |
366,090 |
phone_number |
35,120 |
wechat_pay_bill_deletion_event |
57,892 |
wechat_pay_card_back_information_query_event |
41,583 |
- Nodes:
unique_id,entity_type, and entity-specific attributes.entity_typevalues matchentities[].entity_typein the schema. - Edges:
src,dst, andrelation. Transaction edges also carrytotal_amount,trade_count, andtop_k. - Labels:
unique_id,label;1marks an account as fraud and0marks it as non-fraud. - Schema:
schema.jsonis authoritative for entity types, fields, enum values, relation names, stored directions, coverage,top_k, missing values, and file locations.
The official splits are already prepared and shipped under data/splits/: target_streams_s42.json, target_streams_s43.json, target_streams_s44.json, offline_panel.parquet, and evaluation_panel.parquet. Use them as shipped; no split generation is required.
Transaction amounts use RMB fen (100 fen = 1 yuan). Relation identifiers follow the names in relations[].en; traversal follows the stored src/dst direction. Six personal/merchant transaction relations are bounded top-eight counterparty lists, while four bank-card transaction relations are per-account-and-card aggregates without top-eight truncation. Degrees from these six relations must therefore not be interpreted as total real-world counterparty degrees. See coverage and the relation-specific top_k definitions in the schema.
Use Python 3.9 or newer. From the FraudFeat repository root, install the benchmark evaluation package first and then the EvoFeat method:
python -m pip install -e ./evaluation
python -m pip install -e ./EvoFeatThis installs the fraudfeat benchmark package and the evofeat method distribution with their command-line entry points. EvoFeat requires fraudfeat==1.0.0, provided by the local evaluation/ package. Benchmark commands use the fraudfeat-* prefix; the method's main command is evofeat.
evaluation/fraudfeat/ is the standalone, authoritative benchmark protocol for external SQL and constrained-Gremlin feature submissions. EvoFeat/src/experiments/ contains the experiment harness: run-directory adapters, fixed-panel checks, episode curves, efficiency and cost metrics, and multi-seed aggregation.
Both layers use the same underlying IV, structural-diversity, semantic-diversity, and interpretability implementations. In particular, EvoFeat's Gremlin evaluation delegates to the benchmark's Gremlin engine, AutobinSplit, typed-graph/GED implementation, embedding protocol, and judge rubric/domain brief. SQL support is an additional standalone-submission interface, not a separate Gremlin metric definition.
For comparable reports, use the same fixed panel, feature-pool prefix, IV results, encoder/model, judges, and rubric fingerprint. EvoFeat-only episode trends, multi-judge analysis, efficiency, and cost reports extend the shared metrics rather than redefining them.
After extracting FraudFeat under data/, run the complete EvoFeat pipeline from the repository root with Git Bash or WSL:
export DATA="/absolute/path/to/FraudFeat/data"
bash EvoFeat/scripts/run_public.sh "$DATA" evofeat_public_s42The script loads the dataset's official splits (discovery targets and fixed panels), runs EvoFeat with seed 42, and evaluates the generated feature pool. Configure the model services described in the EvoFeat guide first. Experiment outputs are written to EvoFeat/runs/<run-id>/.
The commands below run from the repository root.
The official final evaluation panel is provided at data/splits/evaluation_panel.parquet.
DuckDB executes each SQL feature as a set-based query over the evaluation panel:
python -m fraudfeat.sql_effectiveness \
--data data \
--features evaluation/examples/features.sql.example.jsonl \
--eval-panel data/splits/evaluation_panel.parquet \
--out-dir results/effectiveness \
--pool-limit 100SQL features use the evaluation_accounts, nodes, and edges views and return unique_id,value. Feature values are scored with the shared AutobinSplit information-value estimator.
For constrained Gremlin feature plans, use python -m fraudfeat.effectiveness with evaluation/examples/features.example.jsonl. The same command accepts EvoFeat/examples/features.json; its example guide includes a complete IV command. For standard one-relation aggregates, use python -m fraudfeat.direct_iv.
python -m fraudfeat.diversity \
--features evaluation/examples/features.sql.example.jsonl \
--schema data/schema.json \
--iv-jsonl results/effectiveness/effectiveness.jsonl \
--out-dir results/diversity \
--pool-limit 100Structural similarity combines anchored graph edit distance and statistic signatures. SQL submissions describe their structure through graph_pattern; Gremlin plans are parsed directly.
Configure OpenAI-compatible model services:
export FRAUDFEAT_EMBED_BASE_URL="https://api.openai.com/v1"
export FRAUDFEAT_EMBED_API_KEY="<api-key>"
export FRAUDFEAT_EMBED_MODEL="text-embedding-3-small"
export FRAUDFEAT_JUDGE_BASE_URL="https://api.openai.com/v1"
export FRAUDFEAT_JUDGE_API_KEY="<api-key>"
export FRAUDFEAT_JUDGE_MODEL="gpt-4o-mini"python -m fraudfeat.embedding_diversity \
--features evaluation/examples/features.sql.example.jsonl \
--iv-jsonl results/effectiveness/effectiveness.jsonl \
--out-dir results/embedding \
--pool-limit 100
python -m fraudfeat.interpretability \
--features evaluation/examples/features.sql.example.jsonl \
--schema data/schema.json \
--iv-jsonl results/effectiveness/effectiveness.jsonl \
--out-dir results/interpretability \
--pool-limit 100Semantic diversity reports pairwise cosine similarity and nearest-neighbor distance. Interpretability uses blind 1–5 scoring of feature meaning, computation logic, risk relevance, and actionability.
All four evaluators use the same feature-pool contract: --pool-limit 100 selects the first 100 input entries, then IV ranks that pool for top-10/30/50/100 summaries. --pool-limit 0 evaluates the full input. Use the same feature file, panel, and IV results across evaluators.
| Dimension | Main outputs |
|---|---|
| Effectiveness | effectiveness.jsonl, effectiveness_summary.json, effectiveness_report.txt |
| Structural diversity | diversity_summary.json, diversity_report.txt |
| Semantic diversity | embedding_diversity.json, embedding_diversity_report.txt |
| Interpretability | judge_scores.jsonl, interpretability_summary.json, interpretability_report.txt |
See the evaluation guide for feature formats, Python APIs, local embedding models, judge configuration, and detailed metric definitions.
After installation, run the FraudFeat evaluation test suite from the repository root:
python -m unittest discover -s evaluation/tests -t evaluation -vThe FraudFeat tests cover query execution, IV estimation, fixed-panel loading, feature-pool selection, structural similarity, and evaluator reports using synthetic fixtures.
The dataset and accompanying documentation use CC BY 4.0. The fraudfeat evaluation package uses Apache-2.0, as declared in its package metadata. The EvoFeat source code is released under the MIT License.