Repository navigation
Data Quality Improvements for Species Catalog #253
Description
Description
Improve species data quality by auto-fixing common issues, validating scientific names, and detecting duplicates.
Background
Current issues found in the species catalog:
- Capitalization errors: "Danio Kerri" should be "Danio kerri" (genus capitalized, species lowercase)
- Duplicate species: See `SPECIES_DUPLICATES_REVIEW.md` for 41 potential duplicates
- Invalid names: Species like "sp." or common names instead of scientific names
- Taxonomic drift: Species names change over time (synonyms)
Proposed Improvements
1. Auto-Fix Scientific Name Capitalization
Current Issues:
-- Examples from database:
"Danio Kerri" → should be "Danio kerri"
"neocaridina Davidi" → should be "Neocaridina davidi"Solution:
- Create migration script to fix capitalization
- Add validation on species creation/editing
- Auto-correct on external data sync
2. Validate Against Authoritative Sources
Use external APIs to validate species names:
- GBIF Species Matching API (already integrated)
- WoRMS (World Register of Marine Species)
- Catalog of Life
Implementation:
// scripts/validate-species-names.ts
// - Check each species against GBIF
// - Flag low-confidence matches (<80%)
// - Suggest corrections
// - Generate report3. Detect and Merge Duplicates
Current State: `SPECIES_DUPLICATES_REVIEW.md` lists 41 potential duplicates
Tasks:
- Review duplicate list manually (admin decision required)
- Create merge script for confirmed duplicates
- Preserve submission history during merge
- Update all foreign key references
Merge Script:
scripts/merge-duplicate-species.ts --keep-id=123 --merge-id=4564. Detect Invalid Names
Flag species that are likely invalid:
- Contains "sp." (unidentified species)
- Contains common names only
- Missing genus or species epithet
- Contains special characters
Tasks
- Create script to fix scientific name capitalization
- Add validation middleware for species forms
- Create species name validation script (vs. GBIF/WoRMS)
- Review `SPECIES_DUPLICATES_REVIEW.md` manually
- Create merge script for duplicate species
- Test merge preserves all submissions and history
- Create invalid name detection script
- Generate quality report for admins
Acceptance Criteria
- All scientific names follow proper capitalization
- Validation prevents future capitalization errors
- Species validation script identifies issues
- Duplicate merge script works without data loss
- Quality report available for admins
- Documentation updated
Priority
Medium - Improves data integrity but not urgent.
Related Files
- `SPECIES_DUPLICATES_REVIEW.md` - List of 41 potential duplicates
- `SPECIES_DUPLICATES.csv` - Raw duplicate data
- `scripts/find-duplicate-species.ts` - Duplicate detection script
Related to #240 (External Data Sources Integration)
Activity