https://github.com/facebookresearch/fairseq/blob/main/examples/translation/README.md
pdf-ingest X:\yourfiles
The tool automatically detects the language of processed documents and tracks processing status through JSON metadata files.
Each processed file has a corresponding .json metadata file that tracks processing status:
{"language": ""}- File was found during scanning but hasn't been processed yet
- Processing is still pending
{
"language": "en",
"language_detection_reliable": true,
"should_translate": true
}- File was successfully converted to text
- Language was detected (e.g., "en" for English)
language_detection_reliableindicates successful completionshould_translateindicates if file should be processed further
Successfully processed files generate:
- Text file:
document-EN.txt(with language code suffix) - JSON metadata:
document.json(with processing status)
- Scanning: Tool finds PDF/DJVU/EPUB/FB2 files and creates empty JSON metadata
- Text Extraction: Converts document to text using appropriate parser
- Language Detection: Analyzes text to determine language
- File Naming: Adds language code to output filename (e.g.,
-EN.txt) - Metadata Update: Updates JSON with language info and completion status
If you see many files with empty "language": "" fields:
- Files were found but processing failed or was interrupted
- Check for missing dependencies (pdftotext, tesseract, etc.)
- Check for file access permissions
- Review error logs for specific parsing failures
- How to use GPU in paddleocr
- PaddlePaddle/PaddleOCR#10429
[ ] fb2 [ ] epub
MA: I was wondering though if the filename could have a pre-extension based on language like *-EN.txt MA: Or *-RUS.txt, etc. MA: Like, if it's easy for your program to realize what language it is
ME: that's trivial ME: but how is your AI going to make sense out of different languages?
MA: We are just gonna archive non-English for now, and only process English