Repository navigation
bug(parser): DOCParser ignores remove_toc and remove_header_footer configuration #21987
Description
Self Checks
- I have searched for existing issues, including closed ones.
- I confirm that I am using English to submit this report (Language Policy).
- Non-English title submissions will be closed directly.
- Please do not modify this template :) and fill in all the required fields.
RAGFlow workspace code commit ID
5b54cb0dad44c7bed82d5d9786dd87c50ba7ba8c (branch main)
Actual behavior
In the Go ingestion pipeline, defaultSetups() (internal/ingestion/component/parser.go:315-319) declares remove_toc: false and remove_header_footer: false for the doc file family:
"doc": {
"remove_toc": false,
"remove_header_footer": false,
"suffix": []string{"doc"},
},Additionally, pipeline templates (e.g. ingestion_pipeline_book.json) and the web client (web/src/pages/agent/utils.ts:300-305) configure and send remove_toc and remove_header_footer for FileType.Doc.
However, DOCParser (internal/parser/parser/doc_parser.go):
- Does not implement
parserSetupConfigurer(ConfigureFromSetup(map[string]any)), causingconfigureParserFromSetups(internal/ingestion/component/parser_dispatch.go:48-59) to silently skip passing thedocfamily setup to the parser. - Lacks
RemoveTOCandRemoveHeaderFooterfields on theDOCParserstruct. - In
ParseWithResult, never checks or applies TOC removal (removeTOCWordwithextractDOCXOutlines(irJSON)on structured sections, orremoveContentsTableon fallback plain text).
As a result, configuring remove_toc: true on .doc documents is inert, and table-of-contents content is not removed.
Expected behavior
DOCParsershould implementConfigureFromSetup(map[string]any), readingremove_tocandremove_header_footerintoDOCParser.RemoveTOCandDOCParser.RemoveHeaderFooter.- When
RemoveTOCis enabled,ParseWithResultshould remove TOC entries usingremoveTOCWord(sections, extractDOCXOutlines(irJSON), isEnglishItems(sections))on structured sections, andremoveContentsTable(items, isEnglishItems(items))on plain-text items. - When
RemoveHeaderFooteris enabled,ParseWithResultshould filter headers and footers usingremoveDOCXHeaderFooterSections(sections, extractDOCXHeaderFooterTexts(data)).
Steps to reproduce
- In Go, attempt to configure
DOCParserfrom setup:
p := NewDOCParser()
p.ConfigureFromSetup(map[string]any{"remove_toc": true})- Observe that
DOCParserdoes not implementparserSetupConfigurerand lacksConfigureFromSetup, soconfigureParserFromSetupssilently skips it andParseWithResultnever removes TOC.
Reacted by dosubot
Activity