LayoutTokensUtil { public static List<LayoutToken> subListByOffset(List<LayoutToken> token, int startIncluded) { return subListByOffset(token, startIncluded, Integer.MAX_VALUE); }  static List<LayoutToken> enrichWithNewLineInfo(List<LayoutToken> toks); static String normalizeText(String text); static String normalizeText(List<LayoutToken> tokens); static String normalizeDehyphenizeText(List<LayoutToken> tokens); static String toText(List<LayoutToken> tokens); static boolean noCoords(LayoutToken t); static boolean spaceyToken(String tok); static boolean newLineToken(String tok); static boolean containsToken(List<LayoutToken> toks, String text); static int tokenPos(List<LayoutToken> toks, String text); static int tokenPos(List<LayoutToken> toks, Pattern p); static List<List<LayoutToken>> split(List<LayoutToken> toks, Pattern p, boolean preserveSeparator); static List<List<LayoutToken>> split(List<LayoutToken> toks, Pattern p, boolean preserveSeparator, boolean preserveLeftOvers); static boolean tooFarAwayVertically(List<BoundingBox> boxes, double distance); static String getCoordsString(List<LayoutToken> toks); static String getCoordsStringForOneBox(List<LayoutToken> toks); static List<LayoutToken> dehyphenize(List<LayoutToken> tokens); static List<LayoutToken> subListByOffset(List<LayoutToken> token, int startIncluded); static List<LayoutToken> subListByOffset(List<LayoutToken> token, int startIncluded, int endExcluded); static List<LayoutToken> getLayoutTokensForTokenizedText(List<String> tokens); static final Function<LayoutToken, String> TO_TEXT_FUNCTION; }
@Test public void testSubList() throws Exception { String text = "This is a simple text that I'm making up just for fun... or well for the sake of the test!"; List<LayoutToken> layoutTokens = GrobidAnalyzer.getInstance().tokenizeWithLayoutToken(text); layoutTokens.stream().forEach(layoutToken -> layoutToken.setOffset(layoutToken.getOffset() + 1000)); List<LayoutToken> result = LayoutTokensUtil.subListByOffset(layoutTokens, 1005, 1008); assertThat(result, hasSize(2)); assertThat(result.get(0).getText(), is("is")); assertThat(result.get(1).getText(), is(" ")); }
@Test public void testSubList_noEnd() throws Exception { String text = "This is a simple text that I'm making up just for fun... or well for the sake of the test!"; List<LayoutToken> layoutTokens = GrobidAnalyzer.getInstance().tokenizeWithLayoutToken(text); layoutTokens.stream().forEach(layoutToken -> layoutToken.setOffset(layoutToken.getOffset() + 1000)); List<LayoutToken> result = LayoutTokensUtil.subListByOffset(layoutTokens, 1005); assertThat(result, hasSize(43)); assertThat(result.get(0).getText(), is("is")); assertThat(Iterables.getLast(result).getText(), is("!")); }