macro for custom indexing with conditionals and text recognition

I was unable to solve this by myself and by searching this wonderful site. The problem is with indexing. I still consider myself as a LaTeX newbie :)

The situation: - I'm working on an english book about programming (300+ pages). For this I'm using makeidx and varindex packages; they are awesome and work well. - In this book there are number of macros, like \type{std:integer}, \var{variable}, \const{SIZE}, \kword{for}, etc. They just provide correct text formatting, nothing magic.

What I want - These macros I would like to use to index all those words for one- or two-level indexing. Some questions are at the end of this post.

What I did: - I changed all macros (\type{}, \const{}, etc.) to the versions, which include also \index{#1}, which was quite simple. Here is the example:

\newcommand{\type}[1]{
\mbox{\texttt{#1}\index{#1}}
}


This gives me a simple index, and it is working well. However, in the Index I have very long lists of many variables, constants and types, all beginning with the same prefix, i.e. cl::Source, cl::Program, or CL_xxx, CL_yyy, etc.

My attempts: I would like to make an index with main single word - cl:: and all words beginning with cl:: put as subitems under it. This can be done by hand, and much of editing, but I figured out the xstring package, and \StrBehind function.

So, I defined the macro in the following way:

 \newcommand{\typei}[1]{%
\IfBeginWith{#1}{cl}{%
\IfBeginWith{#1}{cl::}
{\mbox{\texttt{#1}}\index{cl::!\StrBehind{#1}{cl::}}}
{\mbox{\texttt{#1}}\index{cl!#1}}
}{%
\IfBeginWith{#1}{CL}{%
\IfBeginWith{#1}{CL\_}
{\mbox{\texttt{#1}}\index{CL\_!\StrBehind{#1}{CL\_}}}
{\mbox{\texttt{#1}}\index{CL!#1}}
}%
{%
\mbox{\texttt{#1}}\I{#1}%
}%
}%
}


This (probably) is causing LaTeX error:

! TeX capacity exceeded, sorry [input stack size=5000].
\reserved@a ->\def \reserved@a
*{\let \@xs@assign \@xs@expand@and@detokenize...
l.1127 ...nstructor of the type \typei{cl::Source}
and then the source is pa...

!  ==> Fatal error occurred, no output PDF file produced!


So, I have a rather general questions:

1. How to define a macro which can index any word with given starting characters, which separates this start from the rest for two-level index? I mean the following: \type{cl::Type} will generate the effect like \texttt{cl::Type}\index{cl::!Type}. And preferably the macro will by default separate cl:: from the rest, and should be working for any other word just like normal.

2. How to make such macros for different types of usage? I'd like rather to avoid typing all very similar macros for \kword, \var, \const, \type, \func, etc. - they are all quite similar, and differ only in typography. Maybe it is a way to write something like a macro for generating another macros?

3. The ideal solution will allow to index words with some predefined prefixes, like std::, cl::, or CL_... this would be very helpful.

Indexing such a book is huge task and I'm a bit overhelmed. Using only \index{} command is messing the normal text also. The usage of varindex package greatly improves the indexing by defining circular references and cross-references, which are very customizable. The whole package is a bit difficult to understand but has a great number of possibilities. Maybe there is some hacky way to made this simple?

Sorry for such long post, I tried to make things as clear as possible. I count on your help, *TeX hackers! :) You are my only HOPE! :)

-
Code snippets are useful to understand what you are doing but it is always best to compose a MWE that illustrates the problem including the \documentclass so that those trying to help don't have to recreate it. –  Peter Grill Nov 4 '11 at 21:46
Try adding a % at the end of the lines of the definition of \type. See the difference between the output of \type{a}\type{b} with and without the %. As I had a similar error before with tex capacity exceeded if remove % after use of macro, so perhaps that is one cause of you problems. –  Peter Grill Nov 4 '11 at 21:52
Thank you, Peter, I did not thought about it. Surely this will help, I will remember next time! –  piotao Nov 5 '11 at 20:44

An implementation for your \type macro can be the following:

\documentclass[a4paper]{book}
\usepackage[T1]{fontenc}
\usepackage{xstring,imakeidx}
\makeindex
\makeatletter
\def\pi@tryprefix#1{%
\if@tempswa\else
\IfBeginWith{\pi@temp}{#1}{\@tempswatrue\def\pi@prefix{#1}}{}%
\fi}
\def\pi@protect{\noexpand\string\noexpand\detokenize}
\newcommand{\type}[1]{%
% normalize string
\edef\pi@temp{\detokenize{#1}}\def\pi@prefix{}%
% print the string
\texttt{\pi@temp}%
% look for a prefix
\@tempswafalse
\expandafter\pi@tryprefix\expandafter{\detokenize{cl::}}%
\expandafter\pi@tryprefix\expandafter{\detokenize{CL_}}%
%%%
%%%
\if@tempswa
\StrBehind{\pi@temp}{\pi@prefix}[\pi@temp]%
\begingroup\edef\x{\endgroup
\noexpand\index{\pi@prefix @\pi@protect{\pi@prefix}!\pi@temp @\pi@protect{\pi@temp}}}\x
\else
\begingroup\edef\x{\endgroup
\noexpand\index{\pi@temp @\pi@protect{\pi@temp}}}\x
\fi
}
\makeatother
\begin{document}
\type{cl::Type}

\type{CL_Type}

\type{CL_xyz}

\type{cc_yy}
\printindex
\end{document}


Here is the contents of the .idx file:

\indexentry{cl::@\detokenize{cl::}!Type@\detokenize{Type}}{1}
\indexentry{CL_@\detokenize{CL_}!Type@\detokenize{Type}}{1}
\indexentry{CL_@\detokenize{CL_}!xyz@\detokenize{xyz}}{1}
\indexentry{cc_yy@\detokenize{cc_yy}}{1}


All these \detokenize are just for protecting the _ so that you don't have to type it as \_.

EDIT: in order to support also \type{CL\_Type}, modify the macros as

\makeatletter
\def\pi@tryprefix#1{%
\if@tempswa\else
\IfBeginWith{\pi@temp}{#1}{\@tempswatrue\def\pi@prefix{#1}\let\pi@prefixp\pi@prefix}{}%
\expandafter\IfEndWith\expandafter\pi@prefix\expandafter{\detokenize{\_}}%
{\StrBefore{\pi@prefix}{\string\_}[\pi@prefixp]\edef\pi@prefixp{\pi@prefixp\string_}}{}%
\fi}
\def\pi@protect{\noexpand\string\noexpand\detokenize}
\newcommand{\type}[1]{%
% normalize string
\edef\pi@temp{\detokenize{#1}}\def\pi@prefix{}%
% print the string
\texttt{\pi@temp}%
% look for a prefix
\@tempswafalse
\expandafter\pi@tryprefix\expandafter{\detokenize{cl::}}%
\expandafter\pi@tryprefix\expandafter{\detokenize{CL_}}%
\expandafter\pi@tryprefix\expandafter{\detokenize{CL\_}}%
%%%
%%%
\if@tempswa
\StrBehind{\pi@temp}{\pi@prefix}[\pi@temp]%
\begingroup\edef\x{\endgroup
\noexpand\index{\pi@prefixp @\pi@protect{\pi@prefixp}!\pi@temp @\pi@protect{\pi@temp}}}\x
\else

\begingroup\edef\x{\endgroup
\noexpand\index{\pi@temp @\pi@protect{\pi@temp}}}\x
\fi
}
\makeatother


EDIT2: another way to do for \_ strings

\makeatletter
\def\pi@tryprefix#1{%
\if@tempswa\else
\IfBeginWith{\pi@temp}{#1}{\@tempswatrue\def\pi@prefix{#1}}{}%
\fi}
\def\pi@protect{\noexpand\string\noexpand\detokenize}
\newcommand{\type}[1]{%
% normalize string
\begingroup\edef\_{\string_}\edef\x{\endgroup\def\noexpand\pi@temp{#1}}\x
\edef\pi@temp{\detokenize\expandafter{\pi@temp}}\def\pi@prefix{}%
% print the string
\texttt{\pi@temp}%
% look for a prefix
\@tempswafalse
\expandafter\pi@tryprefix\expandafter{\detokenize{cl::}}%
\expandafter\pi@tryprefix\expandafter{\detokenize{CL_}}%
%%%
%%%
\if@tempswa
\StrBehind{\pi@temp}{\pi@prefix}[\pi@temp]%
\begingroup\edef\x{\endgroup
\noexpand\index{\pi@prefix @\pi@protect{\pi@prefix}!\pi@temp @\pi@protect{\pi@temp}}}\x
\else
\begingroup\edef\x{\endgroup
\noexpand\index{\pi@temp @\pi@protect{\pi@temp}}}\x
\fi
}
\makeatother


Now all \_ will be transformed into _ before processing, so there's no need to look for a CL\_ prefix, but only for CL_. However \_{} must be changed into \_. Multiple _ are supported, but only strings with alphanumeric characters and _ or \_ are allowed in the argument to \type.

-
WOW, man, this is really a piece of TeX-aRt! It's brilliant, outstanding, awesome! I suspected that they may be an answer from some tex-hackers, and now I got it. It works like a charm. For me, it's like chinese, or japanese :) hahaha, but nevertheless, I carefully rewrote it char by char, manually, by hand, instead copy/paste, and I was even able to add few things to it. This piece is really helpful, so THANK YOU VERY MUCH! :) This can push my work to diffrent level :) I'm not sure if I can ask for such a 'generator' of \type-like macros, but with variable text formatting of \pi@temp? –  piotao Nov 5 '11 at 21:28
One more thing. It's pity to say, but your macro is too good for this book :( It produces required output brilliantly, but due to the _ hack I have backslashes prepended in the whole index. This book was written with thousands of index references and nearly each contain this ugly _ characters. If I have your macro before... SIGH... Could you please provide some version which accepts quoted underscores and don't make such nice magick in the output? I'm really sad now because this solution with avoided escaping is sooo cool. I don't have time to edit entire book again and delete all _. –  piotao Nov 5 '11 at 22:16
@piotao Search and replace? :) I'll try and think about it. –  egreg Nov 5 '11 at 22:25
@piotao Look at edited answer –  egreg Nov 5 '11 at 22:58
OMG this is un-be-li-ve-la-ble! :) This looks even more cryptic. I incorporated new macro into document, make a few of them for diffrenet types of words and it seems they works partially. Some _ dissapeared but some still remain. I'm checking now if some authors did not type smth like \typei{cl_{}somethig_{}somethig} because this may be the cause and really should be replaced. (I'm working with vim so there is a chance it may be done in hour or so). BTW thank you very much, those macros are really magickal and are very helpful. –  piotao Nov 5 '11 at 23:19

Not sure to understand what you want. Pershaps this MWE may help you:

\documentclass{article}
\usepackage{xstring}
\usepackage{makeidx}
\makeindex
\newcommand\typei[1]{%
\def\printentry{#1}%
\IfBeginWith{#1}{cl}
{\IfBeginWith{#1}{cl::}
{\StrSubstitute[1]{#1}{cl::}{cl::!}[\newentry]}
{\StrSubstitute[1]{#1}{cl}{cl!}[\newentry]}%
}
{\IfBeginWith{#1}{CL}
{\IfBeginWith{#1}{CL_}
{\StrSubstitute[1]{#1}{CL_}{CL\_!}[\newentry]%
\StrSubstitute{#1}{_}{\_}[\printentry]%
}
{\StrSubstitute[1]{#1}{CL}{CL!}[\newentry]}%
}%
{\def\newentry{#1}}%
}%
\texttt{\printentry}%
\expandafter\index\expandafter{\newentry}%
}
\begin{document}
\noexpandarg
First is \typei{clfoo}, then \typei{cl::bar}.
After, there is \typei{CLone} and \typei{CL_two}!

Finaly: \typei{foobar}

\printindex
\end{document}


EDIT : Here is a more complete solution in which you can define a comma separated list of prefixes like x[y] where x is the mandatory part and y the optional part:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{xstring}
\usepackage{makeidx}
\makeindex

\def\prefixlist{cl[::],foo[_],a[//],zoo[]}

\makeatletter
\newcommand\Idx[2][\texttt]{%
\def\argument@{#2}\let\prefix\@empty \let\sublevel\@empty \let\suffix\@empty
\saveexpandmode \expandarg
\expandafter\Idx@i\prefixlist,\@nil
#1{\prefix\suffix}% prints the entry with the macro #1
\index{\prefix\sublevel\suffix}%
\restoreexpandmode
}

\def\Idx@i#1[#2],#3\@nil{%
\IfBeginWith\argument@{\@empty#1}
{\IfBeginWith\argument@{\@empty#1#2}
{\def\sublevel{!}%
\def\prefix{\protect\detokenize{#1#2}}%
\StrBehind\argument@{\@empty#1#2}[\suffix]%
}
{\def\prefix{\protect\detokenize{#1}}%
\StrBehind\argument@{\@empty#1}[\suffix]%
}%
\ifx\suffix\@empty
\let\sublevel\@empty
\else
\edef\suffix{\noexpand\protect\noexpand\detokenize{\suffix}}%
\fi
}
{\ifx\@empty#3\@empty\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{\edef\prefix{\noexpand\protect\noexpand\detokenize{\argument@}}}
{\Idx@i#3\@nil}%
}%
}
\makeatother

\newcommand\type{\Idx}

\newcommand\const{\Idx[\fbox]}

\newcommand\word{\Idx[\quoteword]}

\newcommand\quoteword[1]{"\textit{\bfseries#1}"}
\begin{document}
\fboxsep=1pt
\type{cltest}

\type{cl::one}

\type{foobar}

\type{foo_zero}

\const{a1234}

\const{a//test}

\word{helloworld}

\word{zoo}

\type{zoologist}

\printindex
\end{document}


And the content of the .idx file will be:

\indexentry{\detokenize {cl}\detokenize {test}}{1}
\indexentry{\detokenize {cl::}!\detokenize {one}}{1}
\indexentry{\detokenize {foo}\detokenize {bar}}{1}
\indexentry{\detokenize {foo_}!\detokenize {zero}}{1}
\indexentry{\detokenize {a}\detokenize {1234}}{1}
\indexentry{\detokenize {a//}!\detokenize {test}}{1}
\indexentry{\detokenize {helloworld}}{1}
\indexentry{\detokenize {zoo}}{1}
\indexentry{\detokenize {zoo}!\detokenize {logist}}{1}
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.8938426971435547, "perplexity": 3096.3366119044026}, "config": {"markdown_headings": false, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2015-32/segments/1438042988930.94/warc/CC-MAIN-20150728002308-00255-ip-10-236-191-2.ec2.internal.warc.gz"}