spacy - removing a token from doc while preserving span attributes
There is no way to change the text of a spaCy Doc after it's created. That's a design decision to avoid data loss.
If you need to modify the contents of a Doc after creating it, what you can do is create a new Doc and pass it to a different pipeline. If you pass a Doc rather than a string as input to nlp then tokenization will be skipped.
I would like to add information about paragraphes and headings on my spacy document.
For this I have added beacon in between the paragraphs text and heading (such as < p_start > for the beginning of a paragraph).
For this I have placed the custom functions that detect these beacon and tagged the spans as paragraphs and/or headings, after the tokenizer, but before the tok2vec component. Thus the pipeline tokenize the text, the tag the spans, then apply the regular pipeline components.
Now I have a problem as I don't want these beacons to be processed in my final doc. However I couldn't find a way, either to remove such token during the pipeline processing, or even substitute them into whitespace.
So is there a way to change these tagged documents, in order to remove only the non relevant tokens ?