diff options
| -rw-r--r-- | .gitignore | 1 | ||||
| -rwxr-xr-x | appunti/leTOUCAN.txt | 66 | ||||
| -rw-r--r-- | data/calc/co2_emissions.ods | bin | 24825 -> 28606 bytes | |||
| -rw-r--r-- | data/calc/electricity_consumption.ods | bin | 0 -> 16961 bytes | |||
| -rw-r--r-- | data/calc/electricity_production.ods | bin | 23362 -> 22717 bytes | |||
| -rw-r--r-- | data/pdf/co2_emissions.pdf | bin | 0 -> 65958 bytes | |||
| -rw-r--r-- | doc/pdi.bib | 9 | ||||
| -rw-r--r-- | doc/pdi.tex | 2 | ||||
| -rw-r--r-- | doc/tex/politica.tex | 60 | ||||
| -rw-r--r-- | doc/tex/produzione.tex | 4 | ||||
| -rwxr-xr-x | tools/QuickReference.pdf | bin | 0 -> 115139 bytes | |||
| -rwxr-xr-x | tools/QuickReference.tex | 49 | ||||
| -rwxr-xr-x | tools/README | 9 | ||||
| -rwxr-xr-x | tools/TeXcount.pdf | bin | 0 -> 247259 bytes | |||
| -rwxr-xr-x | tools/TeXcount.tex | 727 | ||||
| -rw-r--r-- | tools/TeXcount_3_0_0_24.zip | bin | 0 -> 641833 bytes | |||
| -rwxr-xr-x | tools/TechDoc.pdf | bin | 0 -> 235583 bytes | |||
| -rwxr-xr-x | tools/TechDoc.tex | 620 | ||||
| -rwxr-xr-x | tools/macros.tex | 86 | ||||
| -rwxr-xr-x | tools/sub_addrules.tex | 29 | ||||
| -rwxr-xr-x | tools/sub_options.tex | 143 | ||||
| -rwxr-xr-x | tools/sub_tc_other.tex | 28 | ||||
| -rwxr-xr-x | tools/texcount.pl | 3798 |
23 files changed, 5601 insertions, 30 deletions
@@ -1,2 +1,3 @@ doc/build/** **/*~ +**/*.swp diff --git a/appunti/leTOUCAN.txt b/appunti/leTOUCAN.txt new file mode 100755 index 0000000..f232bb2 --- /dev/null +++ b/appunti/leTOUCAN.txt @@ -0,0 +1,66 @@ +Canada e il consumo
+
+La produzione canadese di beni energetici, come petrolio, carbone e prodotti
+derivati dal petrolio sono aumentati. Di conseguenza il consumo canadese è
+aumentato, nel 2010, del 2.2% portando il consumo a 7622 petajoule. Sempre nel
+2010 l'energia maggiormente utilizzata è stata i prodotti rifiniti a base di
+petrolio 41% seguito dal molto utilzzato gas naturale 31% e poi da elettricità 24%
+Un utilizzo elevato di prodotti rifiniti a base di petrolio è dovuto dal consumo
+del settore dei trasporti, settore che ha consumato, sempre nel 2010, il 34%
+dell'energia totale consumata in Canada seguito vicino dal settore agricolo con
+il 20%. Il seguente grafico mostra la domanda di energia canadese nel 2012. Esso
+è separato in due gruppi:
+
+L'uso primario di energia
+
+L'uso primario di energia conta tutta la domanda di energia totale, esso include
+l'energia secondaria dunque e include pure la perdita di energia dovuta a
+conversioni e al trasporto di energia o altro al consumatore.
+Nel 2012 il totale ammontare di energia consumata era di 12'394.3 PJ
+
+L'uso secondario di energia
+
+L'uso secondario di energia conta l'energia utilizzata dai consumatori effettivi
+nei vari settori dell'economia. Inoltre include l'energia richiesta per far
+andare i veicoli e macchinari, nel settore del trasporto, agricolo e industriale
+l'energia usata per scaldare o raffreddare edifici nel settore residenziale,
+commerciale e instituzionale.
+Nel 2012 il totale ammontare di energia consumata era di 8'734.5 PJ
+
+[inserire grafico qui]
+
+Consumo residenziale
+
+Il consumo residenziale è legato al consumo energetico usato per scaldare o
+raffreddare edifici e acqua di abitazione domestiche, escludendo fabbriche e
+simili. Il settore residenziale, con una superficie totale di 133 milioni di m^2
+di spazio vivibile, occupa il 17% dell'energia utilizzata in Canada cioè 1'194 PJ
+divisi tra elettricità e gas naturale. In media in una casa canadese ci sono 2.6
+persone, 21 apparecchi elettrodomestici, il 48% dello spazio occupato è raffreddato
+e spendono 4'497. Tutti questi dati sono riferiti al 2012.
+
+[inserire grafico qui]
+
+Consumo commerciale e instituzionale
+
+Il consumo commerciale e instituzionale comprende tutto il consumo legato a
+edifici non residenziali; come scuole, teatri, uffici, ecc. ma anche i servizi
+come luci stradali o generatori ausiliari. Questo settore conta più di 13 milioni
+di dipendenti, con una superficie totale di 741 milioni di m^2 e occupa il 12%
+dell'energia utilizzata in canada, cioè 1'069 PJ per la maggior parte divisi tra
+elettricità e gas naturale.
+
+[inserire grafico qui]
+
+Consumo industriale
+
+Il consumo industriale comprende il consumo effettuato dal settore industriale
+che è tutto ciò che produce beni includendo agricoltura, construzioni, scienze
+forestali ecc. Il settore è migliorato dal 1990 al 2012 in efficienza del 10%,
+rispamiando 3.3 miliardi di dollari. Nel 2012 il settore industriale ha speso
+42.4 miliardi di dollari per energia di cui 714 PJ erano di elettricità
+
+[inserire grafico qui]
+
+linkz:
+http://www.statcan.gc.ca/pub/11-402-x/2012000/chap/ener/ener-eng.htm
diff --git a/data/calc/co2_emissions.ods b/data/calc/co2_emissions.ods Binary files differindex 76785ec..d37d665 100644 --- a/data/calc/co2_emissions.ods +++ b/data/calc/co2_emissions.ods diff --git a/data/calc/electricity_consumption.ods b/data/calc/electricity_consumption.ods Binary files differnew file mode 100644 index 0000000..f2d5608 --- /dev/null +++ b/data/calc/electricity_consumption.ods diff --git a/data/calc/electricity_production.ods b/data/calc/electricity_production.ods Binary files differindex 1a528af..dc5b550 100644 --- a/data/calc/electricity_production.ods +++ b/data/calc/electricity_production.ods diff --git a/data/pdf/co2_emissions.pdf b/data/pdf/co2_emissions.pdf Binary files differnew file mode 100644 index 0000000..3e0b29d --- /dev/null +++ b/data/pdf/co2_emissions.pdf diff --git a/doc/pdi.bib b/doc/pdi.bib index f66cd2c..c425208 100644 --- a/doc/pdi.bib +++ b/doc/pdi.bib @@ -146,6 +146,15 @@ $$ }}} keywords = "gov", url = "http://www5.statcan.gc.ca/cansim/a26?lang=eng&retrLang=eng&id=1530145&pattern=households+heating&tabMode=dataTable&srchLan=-1&p1=1&p2=9" } + +% file: data/raw/households/households_consumptions_joule_2011_2013.csv +@online{cansim:householdsconsumption, + author = "Statistics Canada CANSIM", + title = "Household energy consumption, Canada and provinces", + keywords = "gov", + url = "http://www5.statcan.gc.ca/cansim/a26?lang=eng&retrLang=eng&id=1530161&pattern=households+gigajoules&tabMode=dataTable&srchLan=-1&p1=1&p2=-1" +} + %% }}} %% - STATISTICS CANADA {{{ ----------------------------------------------------- diff --git a/doc/pdi.tex b/doc/pdi.tex index 11ad86f..d20e09b 100644 --- a/doc/pdi.tex +++ b/doc/pdi.tex @@ -31,7 +31,7 @@ % figures / resources \usepackage{graphicx} \usepackage{wrapfig} -\usepackage{caption} +\usepackage[width=.45\textwidth]{caption} \usepackage{subcaption} \usepackage{float} \usepackage{csvsimple} diff --git a/doc/tex/politica.tex b/doc/tex/politica.tex index 7a78b64..f9b8441 100644 --- a/doc/tex/politica.tex +++ b/doc/tex/politica.tex @@ -1,4 +1,4 @@ -\chapter{Politiche Ecologiche} \label{politics} +\chapter{Politiche Ambientali} \label{politics} Il Canada da un punto di vista diplomatico sembra sia una nazione che si impegna @@ -59,30 +59,36 @@ unit\`a) o in $\frac{kg}{kWh}$, nel primo caso non \`e necessario alcun passaggio intermediario mentre per il secondo \`e necessario conoscere il potere calorico del materiale. -\begin{table}[H] - \centering - \begin{tabular}{ | l r | } - \hline - \bfseries Fuel & \bfseries $k_{CO_2} ~ \Big [\frac{kg}{kg} \Big ]$ - \csvreader[head to column names]{res/data/co2_coeff.csv}{} - { \\ \hline \fuel & \csvcolii} \\ - \hline - \end{tabular} - \caption{Coefficenti di emissione dei combustibili utilizzati in Canada.} -\end{table} - -\begin{table}[H] - \centering - \begin{tabular}{ | l r r | } - \hline - \bfseries Fuel & \bfseries Mass $[10^3~t]$ & \bfseries CO\textsubscript{2} $[10^3~t]$ - \csvreader[head to column names]{res/data/co2_emissions.csv}{} - { \\ \hline \fuel & \csvcoliv & \csvcolv } \\ - \hline - \end{tabular} - \caption{Combustibili utilizzati dalle centrali elettriche per produrre - elettricit\`a nel 2015 \cite{cansim:electricityfuel} - (dati in migliaia di tonnellate).} +\begin{table} +\begin{minipage}[b]{.45\linewidth}\centering + \begin{table}[H] + \centering + \begin{tabular}{ | l r | } + \hline + \bfseries Fuel & \bfseries $k_{CO_2} ~ \frac{kg}{kg}$ + \csvreader[head to column names]{res/data/co2_coeff.csv}{} + { \\ \hline \fuel & \texttt{\csvcolii}} \\ + \hline + \end{tabular} + \caption{Coefficenti di emissione dei combustibili utilizzati in Canada.} + \end{table} +\end{minipage} +\hspace{5mm} +\begin{minipage}[b]{.45\linewidth}\centering + \begin{table}[H] + \centering + \begin{tabular}{ | l r r | } + \hline + \bfseries Fuel & \bfseries Mass in $10^3~t$ & \bfseries CO\textsubscript{2} in $10^3~t$ + \csvreader[head to column names]{res/data/co2_emissions.csv}{} + { \\ \hline \fuel & \texttt{\csvcoliv} & \texttt{\csvcolv} } \\ + \hline + \end{tabular} + \caption{Combustibili utilizzati dalle centrali elettriche per produrre + elettricit\`a nel 2015 \cite{cansim:electricityfuel} + (dati in migliaia di tonnellate).} + \end{table} +\end{minipage} \end{table} Conoscendo i coefficenti calorici e i @@ -93,8 +99,8 @@ seguente formula. m_{CO_2} = m\cdot\Delta_c\cdot k_{CO_2} \] \[ - [kg] = [kg]\cdot \cancel{\Bigg [\frac{kg}{MJ}\Bigg ]} - \cdot \cancel{\Bigg [\frac{MJ}{kg}\Bigg ]} + kg = kg\cdot \cancel{\frac{kg}{MJ}} + \cdot \cancel{\frac{MJ}{kg}} \] Infine conoscendo la produzione annua di elettricit\`a possiamo (indicata diff --git a/doc/tex/produzione.tex b/doc/tex/produzione.tex index 6d1d1cb..854fcb5 100644 --- a/doc/tex/produzione.tex +++ b/doc/tex/produzione.tex @@ -71,7 +71,7 @@ elettrica \cite{cansim:electricity}. \[ \frac{12.82\cdot 10^6 ~kWh}{365~gg\cdot35.85\cdot 10^6~persone} - = 0.98\cdot 10^{-3} ~ \frac{kW}{gg\cdot persona} + = 0.98\cdot 10^{-3} ~ \frac{kWh}{gg\cdot persona} \] Ci sarebbero luoghi migliori, quali la spiaggia di Fundy che se sfruttata porterebbe potenzialmente 50000 MW di energia ho anche la spiaggia Cobequid dove @@ -446,5 +446,5 @@ dietro al idroelettrico (e i combustibili fossili). Il prodotto elettrico dalle turbine alimentate da impianti nucleari nel 2015 \`e stato di 95.68 miliardi di kWh. \[ -\frac{95.68\cdot 10^9~kWh}{ + \frac{95.68\cdot 10^9~kWh}{} \] diff --git a/tools/QuickReference.pdf b/tools/QuickReference.pdf Binary files differnew file mode 100755 index 0000000..9694477 --- /dev/null +++ b/tools/QuickReference.pdf diff --git a/tools/QuickReference.tex b/tools/QuickReference.tex new file mode 100755 index 0000000..7115a1b --- /dev/null +++ b/tools/QuickReference.tex @@ -0,0 +1,49 @@ +\documentclass{article} +\usepackage[T1]{fontenc} +\usepackage{a4wide} +\usepackage{times} + +\include{macros} + + +\title{% +\TeXcount{} Quick Reference Guide\\ +Version \version\copyrightfootnote +} + +\begin{document} + +\maketitle + +\section{Command line options} + +Syntax for running \TeXcount{}: +\codeline{texcount \textit{[options] [files]}} +where \code{texcount} refers to the TeXcount Perl-script, and the options may be amongst the following: + +\input{sub_options} + + +\section{\TeXcount{} instructions embedded in \LaTeX{} documents} + +Instructions to \TeXcount{} can be given from within the +\LaTeX{} document using \LaTeX{} comments on the format +\codeline{\%TC:\textit{instruction [name] parameters}} +where the name is use for instructions providing macro handling rules to give the name of the macro or group for which the rule applies. +% +\input{sub_tc_other} + +Instructions for adding macro handling rules all take the format +\codeline{\%TC:\textit{instruction name parameters}} +where the name indicates the macro (with backslash) or group name for which the rule applies: +% +\input{sub_addrules} + +The available parser rules for environment contents and macro parameters are \code{word}/\code{text}, \code{headerword}, \code{otherword}, \code{header}, \code{float}, \code{inlinemath}, \code{displaymath}, \code{ignore}, \code{xx} (strong exclude), \code{xxx} (stronger exclude), \code{xall} (exclude all) or any of their aliases. + +The available counters are \code{word}/\code{text}, \code{headerword}, \code{otherword}, \code{header}, \code{float}, \code{inlinemath}, \code{displaymath} or any of their aliases. + +Available file specifications contain one or more of \code{input} (for \code{\bs{input}}), \code{file} (file path), \code{texfile} (use with \code{\bs{include}}), \code{<bbl>} (to include the bibliography file), \code{dir} and \code{subdir}. The \code{dir} and \code{subdir} are used to modify the search path within the included document (used with the \code{import} package). + + +\end{document} diff --git a/tools/README b/tools/README new file mode 100755 index 0000000..2ca33b6 --- /dev/null +++ b/tools/README @@ -0,0 +1,9 @@ +TeXcount is a Perl script that counts the number of words in the +text of LaTeX files. It has rules for handling most of the common +macros and provides colour coded output indicating which parts have +been counted. Go to + http://app.uio.no/ifi/texcount/ +for more information or to access the script online as a web service. + +The package, i.e. the script and all accompanying files, is +distributed under the LaTeX Project Public License. diff --git a/tools/TeXcount.pdf b/tools/TeXcount.pdf Binary files differnew file mode 100755 index 0000000..054b09c --- /dev/null +++ b/tools/TeXcount.pdf diff --git a/tools/TeXcount.tex b/tools/TeXcount.tex new file mode 100755 index 0000000..16f90ed --- /dev/null +++ b/tools/TeXcount.tex @@ -0,0 +1,727 @@ +\documentclass{article} +\usepackage[T1]{fontenc} +\usepackage{a4wide} +\usepackage{times} +\usepackage{listings} +\usepackage{url} + +%% TeXcount parsing rules +%TC:envir lstlisting [] xall +%TC:macro url [ignore] + +\include{macros} + +%\parindent=0pt\parskip=8pt +\lstset{basicstyle=\ttfamily\scriptsize,xleftmargin=2cm,xrightmargin=2cm} +\lstset{basicstyle=\ttfamily\scriptsize,xleftmargin=2cm,xrightmargin=2cm} + +\title{% +\LARGE +\TeXcount\\ +\Large +Perl script for counting words in \LaTeX{} documents\\ +Version \version\copyrightfootnote +} +\author{Einar Andreas R{\o}dland} + +\begin{document} + +\maketitle + +{\abstract% +\TeXcount{} is a Perl script for counting words in \LaTeX{} documents. It recognises most of the common macros, and has rules for which parameters to count and not to count; the main text is counted separately from the words in headers and in captions of figures and tables. Finally, it produces a colour coded version of the parsed document, either as a text document or as HTML to be viewed in a browser, indicating which parts of the document have been included in the count. +% +} + +{\scriptsize\tableofcontents} + +\pagebreak + + +% --------------------------------------------------------------------------- + +\section{What \TeXcount{} does} + +\TeXcount{} is a Perl script made for counting the words in a \LaTeX{} document. Since \LaTeX{} documents are formated using lots of macro instructions and often contain both mathematical formulae and floating tables and figures, this is no trivial task. + +Simple solutions to counting the words consists of detexing the documents, which often merely consisty of ignoring the \TeX{} and \LaTeX{} instructions. This is a bad solution since it will usually result in over-estimating the number of words as mathematical formulae, citations, labels and references are counted. + +A perfect solution, if such exists, needs to take into account how \LaTeX{} interprets each macro instruction. The simplest approach to taking this into account consisty of making the count based on the typeset document, but this too tends to over-estimate the word count as mathematical formulae, table contents and even page numbers may get counted. + +A simple but robust approach, which is the one I have taken with \TeXcount{}, is to parse the \LaTeX{} document using simple rules for how to interpret the different \TeX{} and \LaTeX{} instructions. Rules for most of the common macro instructions are included in \TeXcount{}, and it is possible to specify new rules in the \TeX{} document. + +The primary focus of \TeXcount{} is to: +\begin{itemize} + +\item provide an accurate count of the number of words in \LaTeX documents; + +\item exclude or count separately document elements which are not part of the main text such as figure captions; + +\item enable the user to, with relative ease, check how \TeXcount{} has parsed the document and which elements have been counted and which have not. + +\end{itemize} +The last point on this list is one of the most important. Having an accurate word count is of little value unless you know that it is accurate; conversly, trusting an inaccurate word count can be potentially harmful, e.g. if you are submitting a paper or a report which has a strict word limit. + +\TeXcount{} handles complete \LaTeX{} documents, i.e. that start with \code{\bs{documentclass}} and has the text between \code{\bs{begin}\{document\}} and \code{\bs{end}\{document\}}, as well as partial documents made to be included in another \LaTeX{} document. However, in either case, it requires that all groups are closed: \code{\{\ldots\}} and \code{\bs{begin}\ldots\bs{end}}. + +Automatic parsing of included documents is possible, but is by default turned off. There are two options for turning this on: \code{-inc} and \code{-merge}. Turning it on using \code{-merge} will merge the included files into the main document. By using \code{-inc}, however, the included files are parsed separately rather than include the text into the appropriate location: this will perform a separate word count of the included document which is then later included in the total sum. + +Since \TeXcount{} relies on a relatively simple rules for handling the different macros and only performs limited checks on the validity of the \LaTeX{} document, it is your responsibility to make sure the document actually typesets in \LaTeX{} before running it through \TeXcount{}. Also, \TeXcount{} relies on handling macros taking parameters enclosed with \{ and \}, and on ignoring options enclosed by [ and ]: macros with significantly different syntax such as \code{\bs{vskip}} cannot be handled. There are also limitations on what may be contained in macro options enclosed in [], although this restriction may be relaxed by specifying the command line option \code{-relaxed}. + + +\subsection{What \TeXcount{} counts} + +Basically, \TeXcount{} has seven different counts plus an additional file count for use with total counts over a set of files. These and their indices (numbers used to identify them) are: +\begin{description} +\item[0. Number of files:] When multiple files are included, this is counted. +\item[1. Text words:] Words that occur in the main text. +\item[2. Header words:] Words that occur in headers, e.g. \code{\bs{title}} and \code{\bs{section}}. +\item[3. Caption words:] Words that occur in figure and table captions. +\item[4. Header count:] This counts the number of headers, i.e. each \code{\bs{section}} counts as 1. +\item[5. Figure/float count:] This counts the number of floats and figures, e.g. \code{table} and \code{figure} environments. +\item[6. Inline formulae:] This counts the number of inline formulae, i.e. \code{\$\ldots\$}. +\item[7. Displayed formulae:] This counts the number of displayed formulae, e.g. \code{\bs{[}\ldots\bs{]}} or \code{equation} environments. +\end{description} +These are stored in an array and sometimes referenced by their index: e.g. in the option \code{-sum=} which takes parameter values corresponding to counts 1 to 7. In other contexts, however, like in the \code{-tempate=} or when incrementing specific counters through the \code{\%TC:macrocount} instruction, the counters may be referred to by keywords rather than the indices 0 to 7. + +The primary role is to count the words. It is not entirely clear what should be considered words, so I have had to make some decisions. A sequence of letters is certainly a word. \TeXcount{} also counts acronyms like \textit{e.g.}, dashed words like \textit{over-all}, and \textit{it's} as one word. It also counts numbers as words unless they are placed in a math group. If \TeXcount{} breaks words that contain special characters, you may try the option \code{-relaxed} which extends the range of characters allowed as part of words. + +Alternatively, \TeXcount{} may be asked to count the number of letters/characters (not including spaces). It may also be set to count Chinese or Japanese characters. + +Mathematical formulae are not counted as words: it would be difficult to define a sensible rule for this. Instead, \TeXcount{} counts the number of inline formulae and displayed formulae separately. You may then decide on how to combine these counts with the word counts, e.g. using the \code{-sum} option. + +Text in headers (\code{\bs{title}}, \code{\bs{section}}, etc.) are counted separately: \TeXcount{} counts the number of headers as well as the number of words in headers. It may also provide subcounts for each of these by specifying the \code{-sub} option. + +Floating environments (or potentially floating environments) such as tables and figures are not counted as text, even if the cells of a table may containt text. However, if they have captions, these will be counted separately much like headers were. Footnotes are included in this count. By default, environments do not modify the parsing state: i.e. environments within the text are counted as text, etc. Rules for the most common environments, at least those that require non-default treatment, should be predefined, but you may have to add more rules if you use environments defined in packages or by yourself. If you wish to be warned against any environments names you use that lack a defined rule, set the option \code{-strict}. + +Some macros are words by themselves: e.g. \code{\bs{LaTeX}}. These are counted as words provided the macro word rule has been defined for them, but you cannot expect \TeXcount{} to count something like \code{\bs{LaTeX}-word} or \code{\{\bs{TeX}\}count} as one word although the above explanation inicates that it should: \TeXcount{} will in both cases evaluate the macro and the following text separately and thus count them as separate entities. Since \TeXcount{} recognises \code{\bs{LaTeX}} and \code{\bs{TeX}} as single words, each of the two examples would end up being counted as two words. + + +\subsection{What \TeXcount{} does not do} + +While an ideal solution should be able to expand the macro instructions, thus being able to handle new macros, that would at it's worst require reimplementing much of \TeX{}, something that is clearly unrealistic. Instead, I have opted for a simpler solution: to define rules stating which paramters to count and which to ignore and allowing for such rules to be added easily. Thus, \TeXcount{} cannot handle macros that may take a variable number of parameters. Nor can it handle macros that takes parameters on forms other than \code{\{parameter\}}. However, support has now been added for macro options on the form \code{[\ldots]} to be parsed. + +In general, while \TeXcount{} does the parsing in some detail, it does not do it exacly as \TeX{} does it. In some respects there may therefore be critical differences: e.g. while \TeX{} reads one character at a time, \TeXcount{} reads one word at a time, so while \LaTeX{} would interpret \code{\bs{cite} me} as \code{\bs{}cite\{m\}e}, \TeXcount{} would interpret it like \code{\bs{cite}\{me\}}. + +Another issue is that, since \TeXcount{} does not know how to expand macros, it cannot handle macros like \code{\bs{maketitle}} that add text to the document. With respect to \code{\bs{maketitle}}, I have instead set the rule for \code{\bs{title}\{title text\}} to count this as a header although it does not itself produce any text. + + +\subsection{Problems to be aware of} + +In most large documents, there will be cases where \TeXcount{} does not give an exact count. Reasons may be macros \TeXcount{} does not recognise, words that \TeXcount{} split in two (or more) because of special characters not recognised as letters, or options and parameters not counted which actually produce text. Some problems may also arise because it is not always clear what should be counted and \TeXcount{} implements one particular choice: counting numbers as letters/words, not counting formulae as words, not to count tables as text, etc. However, hopefully these should either consist of individual, infrequent errors which should have limited effect on the total count, or entire regions that are included or excluded for which the user may change the parsing rule to produce the desired count. + +There are, however, problems that may arise which are more fundamental and result in counts which are simply wrong rather than just inaccurate, or even make \TeXcount{} fail entirely. + +If \TeXcount{} fails to detect environment endings properly, either closing \code{\{} or \code{\bs{end}}, it may end up ignoring major parts of the document. This should normally produce errors of some kind, although there may be cases when no errors are produced. However, by looking at the verbose output, it will be very clear that entire parts of the document has been excluded. Such problems may be cause by macros that allow unmatched group delimiters, and some effort has been made to minimise the risk of this at the cost of risking other but less critical errors: e.g. there are limits to what is permitted as macro options in order to ensure that a single unmatched \code{[} does not cause large parts of the document to be interpreted as a big option. + +For users of languages containing letters other than the Latin letters A to Z, there is a risk that \TeXcount{} may have difficulty identifying words correctly. The script relies on Perl to recognise words as sequence of letters, and must therefore know which characters are considered to be letters. Words containing letters not recognised by \TeXcount{} will tend to be split into two or more words, which can dramatically inflate the word count. The first step is to ensure that the file is read using the correct encoding: I generally suggest using the UTF-8 Unicode encoding, and from version 2.3. this is the default encoding used by \TeXcount{}, although other encodings may also be used. Unicode has good annotation of which characters are letters, and starting with version 2.3, \TeXcount{} uses Unicode internally to represent the text. + +While non-Latin letters like \code{\aa} and \code{\"a} should be recognised as letters, \TeX/\LaTeX codes using macros or special characters, such as \code{\bs{aa}} and \code{\bs{"}a}, are not immediately understood as letters. I have added patterns aimed at recognising these as well, but depending on the code you are writing, these patterns may either not be flexible enough to recognise all letter codes, or may be too flexible and recognise things it should not. I have added a relaxed mode (\code{-relaxed}) and a more restricted mode (\code{-restricted}) in which these patterns are more general or more constrained, but you should check how this performs on you actual texts by viewing the verbose output. + + + +% --------------------------------------------------------------------------- + +\section{Syntax and options} + + +\subsection{Running \TeXcount{}} + +The command to run \TeXcount{} may vary slightly depending on the operating system and the local settings. You may also wish to rename it or define an alias. + +Under Windows, running \code{texcount.pl} from the command line suffices if \code{texcount.pl} is in the path and pl-files are defined to run as Perl scripts. + +Under Linux/Unix, it should be sufficient to run \code{texcount.pl} provided it is in the PATH and has been made executable (\code{chmod u+x texcount.pl}). The first line of the file contains the line \code{\#!/usr/bin/env perl} which should find the correct location for \code{perl} (provided the program \code{/usr/bin/env} is available). If not, run \code{which perl} to locate Perl and replace the first line of the script with \code{\#!\textit{path}}. + +Alternatively, if the above methods do not work, you may have to run \TeXcount{} exclicitly in Perl by executing \code{perl texcount.pl}. You then need to have the \code{perl} executable in the path or give the explicit path. + +For simplicity, I will simply write \code{texcount.pl} in this manual for the code to execute the script. The syntax then becomes +\codeline{texcount.pl \textit{[options] [files]}} +where the options may be amongst the following: + +\input{sub_options} + +If more than one file is given, \TeXcount{} will perform the count on each of them printing the results for each, then print the total sum at the end. Note that files are parsed one by one in order of appearance and counts made per file; only afterwards are the totals computed. + + +\subsection{File encoding} + +If your \TeX/\LaTeX{} document consists entirely of ASCII characters, there should be no problems with file encoding. However, if it contains non-ASCII characters, e.g. non-Latin letters such as \o{}, there are different ways in which these may be encoded in the files. + +The main encoding supported by \TeXcount{} is UTF-8 (Unicode), and this is used to represent text internally in \TeXcount{}. In older versions of \TeXcount{}, Latin-1 (ISO-8859-1) was the default encoding, but this may cause problems when using non-Latin characters.% +\footnote{% +In Perl, which \TeXcount{} is written in, Latin-1 is the default. However, starting with version 2.3, \TeXcount{} has switched to using UTF-8 (Unicode) internally and will convert text to Unicode before processing: in older version, internal representation was UTF-8 or Latin-1 depending on the options used.} +Both of these are compatible with ASCII: i.e. both are extensions of ASCII, so ASCII characters will be treated correctly by both encodings, but non-ASCII characters will be treated differently. + +From version 2.3 of \TeXcount{}, it is possible to specify other encodings using the \code{-encoding=} option. If no encoding is specified, \TeXcount{} will guess which encoding is used. By default, this guessing is limited to ASCII, UTF-8 and Latin-1. If other encodings are used, the automatic guessing is likely to pick Latin-1 since most files would result in valid Latin-1 code. If the \code{-chinese} or \code{-japanese} option is set, it will guess at other encodings, but still with UTF-8 as the first choice. + +I generally recommend using UTF-8 Unicode: this is increasingly being the new standard. Basically, Unicode contains the characters needed for all existing languages, enumerated from 0 and upwards (beyond 100000), which resolves to problem of requiring different character sets. Since there are more than 256 characters in Unicode, Unicode cannot be represented using one byte per character: UTF-8 is a way to encode the Unicode characters into a list of bytes so that ASCII characters (no. 0--127) are represented by one byte (same as in ASCII), while non-ASCII characters are represented using two or more bytes. Unicode may also be encoded using two bytes to represent each of Unicode characters 0--65535, which covers most of practical use, but this is less commonly used as a file format: it is, however, common for internal representation of strings in memory, as done by e.g. Java, so Perl is the odd one out in using UTF-8 for internal string representation. + +If an encoding is specified using the \code{-encoding=} option, the input will be decoded from the specified encoding into UTF-8. If HTML output is specified, the output will be UTF-8. This ensures that all HTML produced is UTF-8, which is also the encoding specified in the HTML header. If text output is used, the specified encoding is used for the output. E.g. if you specify \code{-encoding=latin1}, \TeXcount{} will assume that all files are encoded in Latin-1, and will also produce the detailed output using Latin-1. For piping, i.e. option \code{-}, this is useful as it ensures the output has the same encoding as the input. + +For convenience, if no encoding is specified, \TeXcount{} will try to guess which encoding is the appropriate one. This is done simply by checking a specified list of encodings one by one until one is found that fits the text. The default is to check ASCII, then UTF-8, and finally Latin-1. If none fits, \TeXcount{} should try to decode the ASCII part of the text replacing non-ASCII characters with a wildcard character, although there may be cases when the decoding exits upon hitting an error. If Chinese or Japanese languages are specified, UTF-8 is tried first, then other encodings are checked depending on the language. + +Note that if no encoding is specified and \TeXcount{} left to guess the appropriate encoding, all output will be UTF-8. Thus, letting \TeXcount{} guess the encoding may not be suitable when using \TeXcount{} in a pipe since the UTF-8 output may not be compatible with the encoding of the input. If multiple files are parsed, \TeXcount{} will guess the encoding separately for each file even if they are included (\code{-inc} or \code{-merge}) in a file with an identified encoding, and may thus end up selecting different encodings for different files. + + +\subsection{Language scripts, alphabets and character sets} + +In additional to the traditional Latin letters, A-Z, a number of letters are recognised by Unicode as part of the extension of the Latin letters. Some languages, however, use entirely different character sets. + +By default, \TeXcount{} has been set up to recognise all alphabets. However, there is a distinction between alphabets like the Latin, Greek, Cyrillic, etc. in which words consists of multiple letters, and languages like Chinese in which each character should be counted as a word. For simplicity, we refer to these as \emph{alphabetic} characters and \emph{logograms}.\footnote{% +Actually, these names are not completely accurate. A logogram is a script which represents a word or `meaningful unit', but e.g. the Japanese kana and Korean hangul are counted as words although they represent sound or syllables rather than meanings.} +The options \code{-alphabets=} and \code{-logograms=} (or \code{-alpha=} and \code{-logo=} for short) allows you to specify which characters to use as either alphabetic letters or whole word characters. These take values that consist of Unicode properties separated by \code{,} or \code{+}. The default setting corresponds to +\codeline{-alphabets=Digit,alphabetic} +in which \code{alphabetic} is defined by \TeXcount{} as the Unicode \code{Alphabetic} class minus logographic script classes, and +\codeline{-logograms=Ideographic,Hiragana,Katakana,Thai,Lao} +which should cover Chinese characters (\code{Han}) as well as the Japanese characters (\code{Han} for the kanji, \code{Hiragana} and \code{Katakana} for the kana). Both options remove previous script settings, unless the list is prefixed by \code{+} in which case the scripts are added: e.g. \code{-logograms=+cjkpunctuation} will add the CJK punctuation characters (defined by \TeXcount) to the set of counted characters. + +Applicable Unicode properties/scripts include \code{Digit}, \code{Latin}, \code{Greek}, \code{Cyrillic}, \code{Hebrew}, \code{Arabic}, \code{Han}, \code{Katakana}, \code{Hiragana}, and more.\footnote{A more complete overview is available at Wikipedia: \url{http://en.wikipedia.org/wiki/Script_(Unicode)}.} + +In addition to the Unicode properties, \TeXcount{} has added a few additional character groups. The properties \code{alphabetic}, \code{digit} and \code{alphanumeric} are more restrictive than their Unicode name-sakes: \code{alphabetic} excludes the default logographic character sets, and \code{digit} consists only of 0--9 unlike Unicode \code{Digit} which includes numerals from other scripts. There is also \code{cjkpunctuation} which is intended to identify Chinese/Japanese/Korean punctuation. + +Note that the Unicode properties are case sensitive. The native Unicode properties start with capital letters, whereas the properties defined by \TeXcount{} are all lower case. Invalid properties will be ignored. + +The options \code{-chinese} and \code{-japanese} still exist and simply restrict the logographic character sets. In addition, \code{-chinese-only} and \code{-japanese-only} will exclude alphabetic words from the counting, equivalent to \code{-alphabets=} with no script properties given. In addition, these options will change the lists of file encodings \TeXcount{} will try if no encoding is given. + +The option \code{-stat} has been added to produce overall word counts per script type. This uses the character classes specified in the \code{-alphabets=} and \code{-logograms=} options, so the default will be able to count which words are purely alphabetic and which contain numbers (or a combination of both), but will not distinguish between e.g. Latin and Greek. To do that, you would have to specify the script classes: e.g. +\codeline{-alphabets=digit,Latin,Greek,Cyrillic} +will count words containing the numbers 0--9, Latin letters (including the extended Latin character set), Greek letters and Cyrillic letters. Words may contain any combination of these: \TeXcount{} does not require that a word consist of only one type of script. Also, note that if \code{digit} had not been included, numbers would not be allowed to be part of or counted as words. The output statistics will then give the number of words containing each of these script classes (or combination). + + +\subsection{Parsing details} + +By selecting one of the \code{-v} options, you can choose how much detail is printed. This is useful for checking what \TeXcount{} counts. Alternatives \code{-v0} to \code{-v4} control the amount of detail, with \code{-v} equal to \code{-v3}. The option \code{-showstate} shows the internal state and is for debugging purposes only: \code{-v4} switches this on. + +The output is colour coded with counted text coloured blue, other colours for other contexts. The colour coding is made using ANSI colour codes. These should work when printed directly to Linux xterm window, but need not work if piped through \code{more} or \code{less}: with \code{less} you need to use the option \code{-r} for the colours to be shown correcly. + +Under Windows or other operating systems, regular ANSI colour don't work, but there is a fix in place which adapts it for Windows, although this may not function exactly as desired. + +In general, however, I recommend using HTML output which can be viewed in a browser: in particular if the text output does not produce suitable colour coding. + +To print the details encoded as HTML document, use the option \code{-html}. Alternatively, \code{-htmlcore} only outputs the HTML body. I suggest using the options \code{-html -v} to get full detail, save this to a HTML file, e.g. using +\codeline{texcount.pl -html -v -sum \textit{files} > details.html} +where \code{-sum} computes the total count of words and formulae (or \code{-sum=1,1,1} to only count words) and adds the cumulative count at the end of each line of the parsing details, and \code{-sub} is on by default which produces subcounts per section. + +\subsubsection{Control of details in verbose output} + +The verbosity option, \code{-v=\parm{styles-list}} or \code{-v\alt{0-4}\parm{styles-list}}, can be used to select exactly which elements to include or exclude from the verbose output. The styles list consists of a list of styles or style categories with \code{+\parm{style}} or \code{-\parm{style}} used to indicate if they should be added or removed. If the first style in the list is one of the categories 0 to 4, the \code{=} can be dropped. The option \code{-help-style} returns an overview of the available styles and style categories, while \code{-help-style=\parm{style}} may be used to get a description of a particular style or style category. + +Each token in the verbose output has a defined style: e.g. \code{word}, \code{hword} (header word), \code{ignore}, \code{option}, etc. If the style is included in the styles list, it will be printed in the verbose output; if not included in the styles list, it will not be printed. Thus, by setting which styles are included in the styles list, you can specify in detail which tokens are included in the verbose output. The included styles correspond to the list of colour codes listed at the start of the output when \code{-codes} is set. + +The style categories, which include \code{0} to \code{4}, are groups of related styles: e.g. \code{Words}, \code{Macros}, \code{Options}, etc. Note that apart from \code{0} to \code{4}, the style categories have capital initials, while the styles themselves are all lower case. + +For example, if you only want to output words (including those in headers and other contexts), you can set the option \code{-v=Words}; using \code{-v=Words+math}, the equation contents will be included (but not the enclosing \code{\$\ldots\$}). + + +\subsection{Summary information} + +By default, \TeXcount{} outputs counts of text words, header words, caption words, number of headers, number of floats/figures, number of inlined formulae, and number of displayed formulae, and lists each of these counts. To shorten this to a one-line format per file, specify \code{-brief}. + +To get \TeXcount{} to produce a total count, specify \code{-sum}: this will compute the sum of all words plus the number of formulae. A customized sum may be computed by speficying \code{-sum=n,n,\ldots} with up to seven numbers separated by commas giving the weight (0=don't count, 1=count once) of each of the seven counts: e.g. the default is equivalent to \code{-sum=1,1,1,0,0,1,1}. To count words only, use \code{-sum=1,1,1}. Higher weights may also be used, e.g. to count displayed formulae or floats/figures as a given number of words. + +Specifying \code{-sum} has two main effects: the cumulative sum is added to the output in verbose formats, and the sum is added to the summary. If combined with \code{-brief}, the option \code{-total} is automatically set, resulting in a one line output containing only the total sum. + +For adding subcounts e.g. by sections, the option \code{-sub} (or \code{-subcount}) may be used. By default, this produces subcounts by part, chapter, section and subsection which are listed in a brief format. One may, however, specify \code{-sub=} followed by \code{part}, \code{chapter}, \code{section}, or \code{subsection} (default when given without value). Break points which initiate a new subcount may also be specified within the \LaTeX{} document using \code{\%TC:break name}. + +If included files are included in the count (\code{-inc}), counts per file will be produced followed by a total count. Note that the counts for the included files are not included in the counts for the main document, and in particular is not included in the subcounts (e.g. per section). To suppress per file counts, the option \code{-total} may be used. + +By adding the option \code{-freq}, \TeXcount{} will output the word frequencies in order of descending frequency: this is only done for the total count, not per file. You may restrict the frequency table to words occurring at least $n$ times by specifying \code{-freq=\it{n}}. \TeXcount{} will count words irrespective of case, but the output will retain upper case where this is consistently used. Note that \TeXcount{} may not recognise that words are the same if they are written differently in the code, e.g. \code{{U}pper} and \code{Upper}. + +A frequency table for each script type (alphabetic, Han, etc. or script classes like Greek, Hebrew etc. if specified in \code{-alphabets=}) is produced by the option \code{-stat}. + + +\subsection{Parsing options} + +\TeXcount{} uses regular expressions to identify words and macro options. By default, these have been set so as to fit most common usages. However, some users may find the default to be too strict, e.g. not recognise options that are long and contain less common symbols. More permissive patterns may be selected by using the option \code{-relaxed}. This allows more general document elements to be identified as words or macro options, which may sometimes be desired, but may also have undesirable effects, so check the verbose output to verify that \TeXcount{} has counted the appropriate elements. Conversely, if the default settings tends to combine words that should be counted as separate words, you may try the option \code{-restricted}. + +Macro options, i.e. \code{[\ldots]} after macros and macro parameters are ignored. Since \TeXcount{} has no specific knowledge of which macros take options, this is a general rule applied to all macros that take parameters\footnote{For macros that take no parameters, \code{[\ldots]} is not interpreted as a macro option. While slightly inconsisten, this avoids e.g. \code{\{\bs{bf}[bold text]\}} to be gobbled up as a macro option and ignored}. In order to avoid that uses of [\ldots] that are not macro options are mistaken as such, \TeXcount{} makes some restrictions on what may be contained in such an option. By default, this restriction is relatively strict under the assumption that it is better to count a few macro options as words than risk large fragments of text to be ignored. However, if your document contains macro options with more complicated values (e.g. certain special characters or macros), using \code{-relaxed} may help handle these correctly. + +By default, \TeXcount{} does not allow special characters or macros to be part of words. This may cause problems if character modifiers or some special characters are used which are entered as macros. The \code{-relaxed} option makes the word recognition regular expression somewhat more general. + + +\subsection{File inclusion} + +By specifying \code{-inc} or \code{-merge}, \TeXcount{} will automatically count documents that are included using \code{\bs{input}} or \code{\bs{include}}. The difference between the two is that \code{-inc} analyses the included files separately, while \code{-merge} merges the included documents into the parent document. Thus, \code{-inc} will result in one count per file and a total sum at the end, while \code{-merge} will treat the merged document as if it was one file. + +The default option is \code{-noinc} indicating that included documents are not counted. + +Paths can absolute or relative. Relative paths are by default relative to the working directory, although e.g. the \code{import} package can cause files to be included from other directories. The working directory is by default set to be the current directory: i.e. the directory from which \TeXcount{} is executed. This default behaviour corresponds to the option \code{-dir=.}. + +The working directory can be specified explicitly by the \code{-dir=\parm{path}} option. The file names on the command line should still be relative to the current directory, i.e. the one from which \TeXcount{} is executed, while files included within these will be relative to the specified working directory. + +Alternatively, if \code{-dir} is used without setting a path, the working directory is determined by the directory containing the top level \LaTeX{} documents, i.e. the document specified on the command line; if several files are provided on the command line, these may result in different working directories. Note that \code{-dir} and \code{-dir=} are fundamentally different: the first indicates that the working directory is determined by the top level \LaTeX{} documents, while the second fixes the working directory to be the current directory. + +Note that when included documents are parsed as separate files, i.e. using \code{-inc}, the text of included documents is not included where the \code{\bs{input}} or \code{\bs{include}} is located. This has two consequences. First, since word counts are produced per file, subcounts, e.g. by chapter, will only include the text in the same file, not that of the included file. Secondly, if TC-instructions to \TeXcount{} are embedded in the \LaTeX{} document, e.g. defining additional macro handling rules, these take effect in the order they are parsed by \TeXcount{}. Since included documents are parsed after the parent document, definitions in the parent document will be in effect for the included documents; definitions made in the included documents will only be in effect for subsequently included documents, not in the parent or previously included documents. + +In addition to the \code{-dir} option for setting the working directory, there is a similar option \code{-auxdir} for setting the path to the auxilary directory where e.g. the bibliography \code{.bbl} file should be read from. The default setting is \code{-auxdir} which means that working directory is used. However, \code{-auxdir=\parm{path}} can be used to overrule this and set an alternative path. If \code{-dir=\parm{path}} is used, the auxilary path should be relative to the current directory, not to the working directory specified with \code{-dir=\parm{path}}; if \code{-dir} is used, as is the default, the working directory will be the directory containing the top level \LaTeX{} dodcuments (the ones specified on the command line), and the auxilary path will be relative to this, unless an absolute path is specified. + + + +% --------------------------------------------------------------------------- + +\section{Macro handling rules} + +A few special macro handling rules are hard-coded into the \TeXcount{} script: i.e. the handling of those can only be changed by editing the script. However, \TeXcount{} primarily relies on a few general rules and macro and environment handling rules that follow a specific pattern. + + +\subsection{General macro handling rules} + +The general macro handling rules fall into a few general categories: + +\begin{description} + +\item[Macro]In its simplest form, this type of rule just tells how many parameters to ignore following the macro. More generally, one may specify the number of parameters a macro takes and how each of these should be handled. Options enclosed in \code{[]} before, between and after parameters are also ignored; this also applies to macros not specified here, so for a macro with no rule, immediately following \code{[]}-options will be ignored. (This type of rule was called an exclude rule in older versions of \TeXcount{}, the reason being that the rule originally only gave the number of +parameters to ignore following a given macro.) + +\item[Environment]For environments enclosed by \code{\bs{begin}\{\textit{name}\}} and \code{\bs{end}\{\textit{name}\}}, there are rules specifying how the contents should be interpreted. A macro rule is added for \code{begin\textit{name}} (without the backslash!) which is \TeXcount{}'s internal representation of \code{\bs{begin}\{\textit{name}\}}. Note that special characters like \code{*} may be part of the environment name, e.g. as in \code{equation*} and rules for these need be specified\footnote{Previously, trailing \code{*} was supposed to be ignored so the same rule would apply to environment \code{equation*} as to \code{equation}. However, due to a bug in a regular expression, this did not work as intended and I have decided not to follow that strategy and instead speficy these rules explicitly.}. \emph{Previously, environment rules were referred to as 'group rules', but I have now renamed this both in the \TeXcount{} script and documentation, and replaced \code{group} by \code{envir} wherever appropriate.} + +\item[Macroword]This type of rule indicates that the macro itself represents one or more words. Initially, \code{\bs{LaTeX}} and \code{\bs{TeX}} are defined with values 1 indicating that each represents one word. + +\item[Preamble]A few macros should be counted even if they are in the preamble. In particular, \code{\bs{title}\{title text\}} is counted as a header assuming it will later be used to produce a title. + +\item[Float inclusion]Within floats (environments with the \code{float} parsing rule) there may be texts that should still be counted: in particular captions. These are specified with the float inclusion rule. + +\end{description} + +Previously, there was also a separate header handling rule, but this is now incorporated into the more general macro handling rules. + +A macro parameter is normally on the form \code{\{\textit{something}\}}; more generally it may be anything \TeXcount{} parses as a single unit (or token), e.g. a macro, but since \TeXcount{} parses word by word rather than character by character this may not always be correct if parameters are not \code{\{\}}-enclosed or macros. In addition, some macros take optional parameters which are usually on the form \code{[\textit{option}]}, and \TeXcount{} can also (from version 4) count these. + + +\subsection{Special macro handling rules} + +Some macros do not follow the pattern used by \TeXcount{} to represent macro handling rules. For some of these, special handling rules have been hard-coded into the \TeXcount{} script. For some, the macro syntax differs from the general rule, while in other cases the macros may trigger special processing. + +\begin{description} + +\item[file include]If \code{-inc} is specified, included files will also be parsed and the total presented at the end. Initially, \code{\bs{input}} and \code{\bs{include}} trigger file inclusion, but more file inclusion macros may be added to the \code{\%TeXfileinclude} hash. In addition to potentially triggering file inclusion, the syntax may differ in that \code{input} does not require the file name to be enclosed in \code{\{\ldots\}}. + +\item[package include]When packages are included using \code{\bs{usepackage}{name}}, \TeXcount{} will check for package specific macro handling rules to include. Initially, only \code{\bs{usepackage}} triggers package inclusion, but more macros may be added to the \code{\%TeXpackageinc} hash. + +\end{description} + +Complete \LaTeX{} documents should start with a \code{\bs{documentclass}} specification, then a preamble region which should not contain typeset text, before the main document starts with \code{\bs{begin}\{document\}}. However, \LaTeX{} files which are ment to be included into a document will not contain \code{\bs{documentclass}} and \code{\bs{begin}\{document\}}. A rule to recognise the preamble region is hard-coded into \TeXcount{}. + +Rules for identifying \code{\$\ldots\$}, \code{\$\$\ldots\$\$}, \code{\bs(\ldots\bs)}, and \code{\bs[\ldots\bs]} as formulae are hard-coded and basically parse until the closing token is encountered. + +The macros \code{\bs{def}} and \code{\bs{verb}} have hard-coded rules since these do not follow the pattern for macro handling rules, but may contain \LaTeX{} code which could seriously disrupt the parsing, e.g. by containing unclosed \code{\bs{begin}}. Macros like \code{\bs{newcommand}}, however, are handled by ordinary macro rules. + +The macro \code{\bs{biblography}} is handled to check if the bibliography file should be parsed. The \code{thebibliography} environment is also handled differently, one difference being that a bibliography header is added to the count. + + +\subsection{Package specific macro handling rules} + +Starting with version 2.3, \TeXcount{} can handle different sets of macro handling rules for different packages. When a package is included in the \LaTeX{} code or through the \code{-incpackage} option, rules defined for the given package is added. + +Note that \TeXcount{} is still doing the analyses sequentially. It is therefore critical that the package inclusion takes place before any use of the package which may make a difference if you are analysing several files. E.g. if the main file contains \code{\bs{input} setup}, any packages included in \code{setup.tex} will not apply to the main file since this is parsed before \TeXcount{} parses \code{setup.tex}. + +As of now, the package support is sparse since most macro handling rules have been included in the main set of rules. + + +\subsection{Bibliography handling} + +By default, the bibliography is not included in the word count. If the \code{-incbib} option is specified, however, bibliography parsing is turned on. If the bibliography is included from the \code{bbl} file using the \code{\bs{bibliography}} macro, this will be parsed as if included with the \code{-inc} option. If \code{-merge} is specified together with \code{-incbib}, the bibliography will be merged into the document. + +Note that bibliography parsing may be non-trivial and depend on the bibliography style used, so the verbose output should be checked: some styles perform considerable formatting which may confuse \TeXcount{}. In addition, initials, page numbers, etc. will all be counted as words, which may result in a word count which is higher than intendet. + + +\subsection{Adding or modifying macro handling rules} + +There are basically two different ways in which you can add additional macro handling rules, e.g. for your own macros, or modify existing rules: by modifying the \TeXcount{} script, or by adding the rules through \TeXcount{} instructions embedded in the \LaTeX{} code. + +The simplest method is to use \TeXcount{} instructions which are embedded in your \LaTeX{} document as \LaTeX{} comments on the format \code{\%TC:\textit{instruction}}. This approach is described in some detail in section \ref{subsec:TC_addrule}. + +It is also possible to modify the \TeXcount{} code. The macro handling rules are mostly defined in the hash tables named \code{TeXmacro}, \code{TeXenvir}, etc., and editing these definitions is simple and does not required in-depth knowledge of Perl. A brief overview of the \TeXcount{} code is provided in section \ref{sec:code}. + + +\subsection{Cautions!} + +Since the rules are of a relatively general nature, macros that have a great deal of flexibility are hard to deal with. In particular this applies to macros with a variable number of parameters or where the handling of the parameters are not constant. + +By default, \TeXcount{} assumes that macro options, i.e. parameters on the form \code{[\ldots]}, should not be counted. From version 4.0, \TeXcount{} allows rules for optional parameters, but in most cases where optional parameters have not been specified in the macro handling rules, they will simply be ignored. There is some risk of misinterpreting text as an option: e.g. \code{\bs{bf}[text]}. This is not likely to be a frequent problem. However, if something like \code{\bs{bf}[a lot of text]} gets ignored because it is considered an option, it can influence the word count substantially. I have therefore been somewhat restrictive with what (and how much) may go into an option. The default restriction on what may be allowed as an option may sometimes be too restrictive, causing \TeXcount{} to interpret options as text or macro parameters; you may use the command line option \code{-relaxed} to relax this restriction and allow more general options. + +More advanced macros are not supported and can potentially confuse \TeXcount{}. In partcular, if you define macros that contain unbalanced \code{\bs{begin}}--\code{\bs{end}}, this will cause problems as \TeXcount{} needs to keep track of these to know where environments start and end. + + + +% --------------------------------------------------------------------------- + +\section{Output from \TeXcount{}} + +\TeXcount{} will by default provide a summary of the word and element counts. This may, however, be modified either by specifying \code{-brief} which reduces it to a one line summaryper file, \code{-total} to suppress per file summaries, or by providing an alternative template. + +If there are parsing errors, \TeXcount{} will print warnings about these. You may turn off this by specifying \code{-quiet} (\code{-q} for short), but there will still be an added comment about the number of errors in the final statistics to warn you of any errors. + + +\subsection{Count statistics} + +The summary output will by default provide a summary of all counts: i.e. word counts for text, header and captions, and the number of floats/tables, headers, inlined and displayed formulae. You may combine these into a summary count by using the \code{-sum} option which by default gives the total number of words and formulae. You may choose briefer output formats by using the \code{-brief} option which produces a one-line summary of the counts. The option \code{-1} is the same as specifying \code{-brief -total} and will give only one line of output for the total only. Combining \code{-brief} with \code{-sum} will cause only the sum to be printed rather than the full set of counts. + +If multiple files are processed in one run, \TeXcount{} will by default provide summary statistics per file. If files are included (using the \code{-inc} option), summaries of all files are provided as well as the total. If there is more than one file, i.e. main \LaTeX{} documents provided in the command line, it will also write a total summary. + +In order to only write the total summary, use the option \code{-total}. If there is only one file processed, the result will be similar except that subcounts (counts per section etc.) are not provided with the total count. + + +\subsection{Customising the summary output} + +You may specify an output template to use instead of the default output formats. This will replace the output per file or for the total with output produced using this template. + +The template is a string with codes for inserting the count values and titles. To specify it, use the option \code{-template="\textit{template}"}. The encapsulating \code{"\ldots"} are required if the template contains spaces. You may insert line shifts by using \code{\bs{n}}. + +The counts may be included by using the counter keywords: \code{word}, \code{headerword}, etc. Other codes that may be inserted are: \code{\{SUM\}} to insert the count as specified by the \code{-sum} option, \code{\{TITLE\}} for the title (e.g. section name) and a header (same as title unless \TeXcount{} has replaced it), \code{\{ERROR\}} for the number of parsing errors, \code{\{WARNINGS\}} for the number of distinct warnings, or \code{\{NWARNINGS\}} for the total number of warnings. Some of these also have shortened forms\footnote{Previously, one-letter versions of some of these codes were permitted, but that is no longer the case.} like \code{\{ERR\}} and \code{\{WARN\}}. + +Conditional inclusion may be performed using the format \code{\{\textit{label}?\ldots?\textit{label}\}} where \code{\textit{label}} is one of the counter keywords, \code{SUM}, \code{ERROR} or \code{TITLE} (or their alternative forms). The enclosed text will then be included only if the corresponding value exists and is non-zero. If you wish to include an alternative text when the value is non-existant or zero, use the format \code{\{\textit{label}?\textit{if non-zero}|\textit{if zero}?\textit{label}\}}. + +Subcounts, e.g. per section, may be included by using \code{\{SUB|\textit{template}|SUB\}} with a separate template text specified for the subcounts. This will only be included if there is more than one subcount, and in order to conditionally include prefix and suffix you may use \code{\{SUB?\textit{prefix}|\textit{template}|\textit{suffix}?SUB\}}. + +Note that you have to insert line shifts yourself. \TeXcount{} will only insert one line shift after each file count, and not after the total count: if you process only one file and want only to output the total sum without a line shift at the end, use \code{-sum -total -template="\{SUM\}"}, which should give the same output as \code{-1 -sum} when there are no parsing errors. + + + +% --------------------------------------------------------------------------- + +\section{\TeXcount{} instructions in the \LaTeX{} document} + +It is possible to give some instructions to \TeXcount{} from within the +\LaTeX{} document. These can be used to control the parsing of the document and add custom made macro and environment handling rules directly from the \LaTeX{} document. The general format of these instructions is +\codeline{\%TC:\textit{instruction \alt{parameters}}} +which \LaTeX{} will interpret as a comment but \TeXcount{} will detect. + +Adding your own macro handling rules is relatively simple. While it is fairly easy to edit the script to add more rules, this has the disadvantage that the modifications will be lost if updating to a new version of \TeXcount. A better and more flexible solution is to include instructions to \TeXcount{} in the \LaTeX{} documents, alternatively to make a definition file in which new macro handling rules are defined. The \TeXcount{} instructions for doing this take the form +\codeline{\%TC:\textit{instruction name parameters \alt{option}}} +where \code{name} is either the macro name (including backslash) or environment name: +% +\input{sub_addrules} + +Note that macro handling rules are added successively throughout the session: i.e. if more files are parsed, handling rules from previously parsed files still apply. This has advantages as well as disadvantages. If you give a list of files with the rules specified in the first file, these rules will be applied to all the documents. However, if you use the \code{-inc} option, included files will be parsed only after \TeXcount{} has finished parsing the file in which they are included, so any rules specified in these will not apply to the initial document. + +A few additional \TeXcount{} instructions exist to control the overall parsing and counting: + +\input{sub_tc_other} + + +\subsection{Parameter and content handling rules}\label{subsec:TC_addrule} + +There are a set of alternative rules that may be used for parsing macro parameters and environment contents. These rules, or \emph{parser states}, are identified by keywords: +% +\begin{description} +\def\option#1[#2]#3{\item[#1:] (key: \code{#3} formerly \code{#2})} +\option{Text}[1]{text, word, wd, w} + Count as text (i.e. count words). +\option{Header text}[2]{headertext, headerword, hword, hwd, hw} + Count as header text. +\option{Other text}[3]{otherword, other, oword, owd, ow} + Count as float/caption text. +\option{Displaymath}[7]{displaymath, dsmath, dmath, ds} + Count as displayed math formulae. +\option{Inline math}[6]{inlinemath, inline, imath, eq} + Count as inlined math formulae. +\option{To header}[4]{header, heading, head} + Count header, then count text as \code{headertext} (transition state). +\option{To float}[5]{float, table, figure} + Count float, then parse contents as \code{isfloat} (transition state). +\option{Preamble}[-9]{} + Parse as preamble, i.e. ignore text but look for \code{preambleinclude} macros. +\option{Ignore}[0]{ignore} + Ignore text, i.e. do not count, but will still parse the code. +\option{Float}[-1]{isfloat} + Float contents, ignore text but look for \code{floatinclude} macros. +\option{Strong exclude}[-2]{xx} + Strong ignore which ignore environments, e.g. to use in macro definitions where + \code{\bs{begin}}--\code{\bs{end}} need not be balanced. +\option{Stronger exclude}[-3]{xxx} + Stronger ignore, handles macros as isolated tokens without handling their parameters, + to use with macro definitions like \code{\bs{newcommand}} and \code{\bs{def}}. +\option{Exclude all}[-4]{xall} + Ignore all, including unbalanced braces (e.g. used by \code{\%TC:ignore} and the \code{verbatim} environment). This rule may be used for environment contents, but not for macro or environment parameters or options since the exclusion causes \{ and [ to be ignored. +\end{description} + +The keys are used to identify the rule. Environment content rules are simply specified by giving the desired key. Parameter rules are on the form \code{[\textit{rule},\textit{rule},\ldots]} with one rule provided per parameter, where each rule is either one of the above keywords or \code{option:\textit{rule}} to indicate an optional parameter on the form \code{[\ldots]}; alternatively, a single integer can be provided (not enclosed in \code{[]} to indicate that the indicated number of parameters should be ignored. + +The list of parser states is in order of increasing priority: i.e. when a parser state is specified as a rule for parsing a parameter or environment content, this will only take effect if it has higher priority than the current state. Thus, text within an ignored or excluded region will not be counted. + +The formerly used numberic codes are listed at the end of the keyword list for each state. Before version 4, these numeric codes were used to specify parsing rules, but although these should still work, using the named keywords is highly recommended. + +The transitional states indicates an incrementation of one of the counters and then change to another state: e.g. if the \code{header} rule is specified, this will first cause the header counter to be incremented, and then a change to the \code{headerword} state in which word counts are added to the header word counter. + +\subsubsection{Adding a macro handling rule} + +The \TeXcount{} instruction for adding (or changing) a rule for how \TeXcount{} handles a specific macro takes the form +\codeline{\%TC:macro \textit{macro-name parameter-rules}} +where the macro name includes the backslash and the parameter rules can be an integer or a \code{[]}-enclosed list as explained above. + +If a list, \code{[\textit{rule},\textit{rule},\ldots]}, of parsing rules is provided, the macro will be assumed to take this number of parameters. Each rule is either a keyword signifying the rule, or parser state, with which the parameter will be parsed, or \code{option:\textit{key}} for optional \code{[]}-enclosed parameters. Additional \code{[]}-enclosed options, between or after the macro and the parameters, will be ignored. + +Macro handling rules specified for macros \code{\bs{\textit{name}}} automatically apply to \code{\bs{\textit{name}*}}: i.e. a \code{*} is automatically gobbled up as a macro modifier. + +Here are some examples together with corresponding macro definitions: + +\begin{lstlisting} +%TC:macro \refnote[text,othertext] +\newcommand\refnote[2]{\textit{#1}\footnote{#2}} + +%TC:macro \newsection [header,ignore] +\newcommand\newsection[2]{\section{#1}\label{sec:#2}} + +%TC:macro \NB 1 +\newcommand\NB[1]{\marginpar{#1}} +\end{lstlisting} + +The predefined rules can easily be read off the script file: they are hash maps defined at the beginning of the script with names \code{TeXmacro}, \code{TeXenvir}, etc. + +\subsubsection{Adding an environment handling rule} + +Rules for environments may be added on the format +\codeline{\%TC:envir \parm{name} \parm{parameter-rules} \parm{content-rule}} +for parsing \code{\bs{begin}\{name\}\ldots\bs{end}\{name\}}. The parameter rules are specified as for the \code{macro} rule and is used to process the parameters that follow \code{\bs{begin}\{name\}}. The content rule is a single parsing rule to use on the environment content. + +\begin{lstlisting} +%TC:envir theorem [] text +\newtheorem{theorem}{Theorem} +\end{lstlisting} + +\subsubsection{Adding rules that apply to the preamble and float contents} + +Within the preamble (from \code{\bs{documentclass}} to \code{\bs{begin}\{document\}}) and within floating objects (tables, figures, etc. parsed using the \code{float}/\code{isfloat} states), texts and macros are generally ignored. However, it is possible to specify particular macro handling rules that apply within these regions by using the \code{preambleinclude} and \code{floatinclude} \TeXcount{} instructions. These take the same format as the \code{macro} instruction: +\codeline{\%TC:preambleinclude \textit{macro-name parameter-rules}} +\codeline{\%TC:floatinclude \textit{macro-name parameter-rules}} +It is possible for the same macro to specify different rules for preamble, floats and general use, although for most uses these should be expected to be the same. + +Preamble inclusion is typically used for macros like \code{\bs{title}} that define text that should be counted although it may be placed in the preamble. Another use is that macros that may occur in the preamble, like \code{\bs{newcommand}} and may contained unbalanced \code{\bs{begin}}--\code{\bs{end}} pairs, require a stronger exclusion than the regular \code{ignore} rule even in the preamble to ensure \TeXcount{} is not confused by these. + +Float inclusion is used e.g. for captions, and the parsing rules should normally be to count texts using the \code{otherword} parsing rule. + + +\subsection{Count macro, either as words or in other counters} + +Some macros, e.g. \code{\bs{LaTeX}}, generate words and should be counted as words. Other macros can generate other elements, e.g. headers or figures. Rules for counting macros can be specified as +\codeline{\%TC:macroword \textit{macro} \textit{number}} +where the parameter is the number of words produced by the macro, or +\codeline{\%TC:macroword \textit{macro} [\textit{countername},\ldots]} +which causes each of the counters in the list to be incremented by one (or more if listed multiple times). + +The counters for counting the number of files, text words, etc. are stored in an array. In some cases, e.g. when \code{-sum=} is specified, the order of the counters in this array is used to specify the rule. However, in most cases the counters should be specified by keywords. The counters, their index number and keywords are: +% +\begin{description} +\def\option[#1]#2{\item[#1.] (keys: \code{#2})} +\option[0]{file} Number of files. +\option[1]{text, word, wd, w} Number of words in text. +\option[2]{headerword, hword, hwd, hw} Number of words in headers. +\option[3]{otherword, oword, owd, ow} Words outside text, e.g. in floats/tables/figures. +\option[4]{header, heading, head} Number of headers. +\option[5]{float, table, figure} Number of floating environments, e.g. tables and figures. +\option[6]{inlinemath, inline, imath, eq} Number of inlined mathematics formulae. +\option[7]{displaymath, dsmath, dmath, ds} Number of displayed equations. +\end{description} + +Examples of uses: + +\begin{lstlisting} +%TC:macroword \TeXcount 1 +\newcommand\TeXcount{{\TeX}count} + +%TC:macroword acknowledge [header,hword] +\newcommand\acknowledge{\section*{Acknowledgements}} +\end{lstlisting} + + +\subsection{Specifying file inclusion macros} + +In addition to \code{\bs{input}} and \code{\bs{include}}, which are the standard \LaTeX{} macros for file inclusion, there are packages such as \code{import} intended to enable organising files into subfolders. \TeXcount{}, from version~3, adds some support for macros that change the path from which files are included. If the user needs to add additional file inclusion macros, the format is +\codeline{\%TC:fileinclusion \parm{macro} \parm{file-parameters}} +where the file parameters are a comma separated list of keywords, each corresponding to a macro parameter. Available parameters are: +% +\begin{description} + +\item[\code{input}:] This is a special keyword to use with \code{\bs{input}}. The handling of the parameter values is as \code{file}, but the parameter itself is not required to be enclosed in \code{\{\}}. + +\item[\code{file}:] This parameter simply gives the name of or path to a file. If the file is not found, \TeXcount{} will append \code{.tex} and try again. + +\item[\code{texfile}:] This parameter gives the name of or path to a file, but \code{.tex} will be appended, and is the rule used by \code{\bs{include}}. + +\item[\code{dir}:] This parameter provides the path of a directory relative to the \code{\$workdir}, and adds this to the search path before including any files. This is used with the \code{\bs{import}} macro of the \code{import} package. + +\item[\code{subdir}:] This parameter provides the path of a directory relative to the current directory, and adds this to the search path before including any files. This is used with the \code{\bs{subimport}} macro of the \code{import} package. + +\item[\code{<bbl>}:] This is a special keyword to use with \code{\bs{bibliography}} to specify inclusion of the bibliography file. It is different from the other keywords in that it does not take a macro parameter. + +\end{description} + +Examples showing how some existing macros, from basic \LaTeX{} and from the \code{import} package, are defined: + +\begin{lstlisting} +%TC:fileinclude \input input +\input macros.tex +%TC:fileinclude \include texfile +\include{intro} +%TC:fileinclude \import dir,file +\import{supplements/}{overview.tex} +%TC:fileinclude \subimport subdir,file +\subimport{tables/}{data.tex} +\end{lstlisting} + + +\subsection{Adding subcount break points} + +By specifying \code{-sub}, \TeXcount{} can produce subcounts, e.g. per section. Alternatively, or in addition, explicit break points can be entered in the \LaTeX{} document using the TC-instruction \code{break}. These take the form: +\codeline{\%TC:break \textit{title}} +A title (or name) may be given to identify the break point. + +If you define new section macros or macros you wish to cause a break point, these may be specified using the TC-instruction \code{breakmacro}: +\codeline{\%TC:breakmacro \textit{macro} \textit{label}} +This defines the given macro to cause a break point, and uses the given label to indicate the type of break (e.g. Section, Chapter, etc.). + + +\subsection{Ignoring segments of the file} + +The TC-instruction \code{ignore}, later canceled by \code{endignore}, may be used to turn of all counting in a segment of the \LaTeX{} file. The ignored segment should thus be started by +\codeline{\%TC:ignore} +and ended by +\codeline{\%TC:endignore} +causing all text inbetween to be ignored.\footnote{In older versions, \TeXcount{} would still parse this text and might thus be affected by unbalanced braces. As of version 2.3, however, this should be fixed to make the ignore instruction more robust.} + + +\subsection{Bibliography inclusion} + +In order to include the bibliography in the word counts, you can either specify \code{-incbib} on the command line, or use \TeXcount{} instruction +\codeline{\%TC:incbib} +which has the same effect: it specifies handling rules for the \code{\bs{bibliography}} macro and \code{thebibliography} environment that causes the bibliography to be included in the count, and if necessary the \code{.bbl} file to be included (without requiring \code{-inc} or \code{-merge}). + + +\subsection{Text substitution prior to parsing} + +There are cases where a macro needs to be substituted with a text prior to parsing. One such case is when a macro contains a file path which is later used by a file inclusion macro. Since \TeXcount{} does not actually expand the macros, it will not be able to generate the file path from the macro. Instead, one may perform an explicit substitution +\codeline{\%TC:subst \parm{macro} \parm{text}} +which will then cause all occurrences of the macro to be substituted by the provided text prior to parsing. Note that this substitution will therefore also be found in the verbose output. + +\begin{lstlisting} +\newcommand\chappath{chapters} +%TC:subst \chappath chapters +\input \chappath/chapter1 +\end{lstlisting} + +Note that the substitution is placed \emph{after} the \code{\bs{newcommand}} definition. Otherwise, the substitution would have taken effect, changing that line to \code{\bs{newcommand} chapters/chapters}. + + + +\subsection{Adding a new counter} + +Initially, \TeXcount{} has eight different counters: file, text words, header words, other words, number of headers, number of floating objects, number of inlined formulae, and number of displayed formulae. However, it is possible to add more counters, e.g. to count footnotes separately. The syntax is +\codeline{\%TC:newcounter \parm{name} \opt{description}} +where the given name is used as keyword to refer to this counter. If no description is provided, the name will be used as description. A new counter is then added, and a parsing rule (parser state) with the same name is added which may be used in specifying macro and environment handling rules. + +The following example shows how two different counters are added: one to count the number of footnotes, and another to count the words in footnote. + +\begin{lstlisting} +%TC:newcounter fwords Words in footnotes +%TC:newcounter footnote Number of footnotes +%TC:macro \footnote [fwords] +%TC:macroword \footnote [footnote] +Each footnote\footnote{Words in footnotes will be counted separately.} will be counted. +\end{lstlisting} + +Note that we have to specify one rule for counting the words in footnotes, and another rule for counting the footnotes. Unlike headers and floating bodies, there are no transition states available that can do both. + + +% --------------------------------------------------------------------------- + +\section{Using an option file} + +If you have a lot of settings, e.g. output template and TC commands for specifying parsing rules, you may place these into a file and include this using \code{-opt=\textit{file}}. + +The format of this file is quite simple: each line is read as one option, so different options should not be placed on the same line. If some options are so long you need to break the line, e.g. for specifying an output template, you can do so by placing \code{\bs{}} at the start of lines that continue the previous line. + +You may enter TC commands just as in the \LaTeX{} code by starting the line with \code{\%} instead of \code{TC:}. Using these, you may include specifications of parsing rules. + +Blank lines and lines starting with \code{\#} are ignored and may thus be used to add comments to the option file. So are leading spaces, which allows lines to be indented. Line breaks may be inserted by \code{\bs{n}}. + +Here is an example which sets the total sum to be the number of words (not including formulae), subcounts by section, parses included files, and adds an output template. + +\begin{lstlisting}[frame=single] +### Options to use with TeXcount + +# Counting options +-sum=1,1,1 +-sub=section +-inc + +# Macro rules +%macro \url 1 +%envir sourcecode 0 0 +%macroword \TeXcount 1 + +# Path used in file inclusion (\chapterpath filename) +%subst \chapterpath chap/ + +# Output template +-template= + \::: {title} :::\n + \Words: {sum}\n + \Formulae: {6} + {7}\n + \{5?Number of floats: {5}\n?5} + \{SUB? - {sum} words in {title}\n?SUB} +\end{lstlisting} + + + +% --------------------------------------------------------------------------- + +\section{Customising \TeXcount{}} + +\TeXcount{} is a self-contained Perl script: no external packages or resources required except that you need to have Perl installed to run it. Unfortunately, as with much of Perl code since Perl does not itself encourage structured programing, after expanding somewhat in size, it is not the most readable of codes. However, there may still be cases where you might yourself want to modify the code. + +There are some things that may be modified quite easily even without knowing Perl. + +\begin{description} + +\item[Preset startup options]On one of the first lines of the code, the list \code{@StartupOptions} is defined. A list is simply a sequence of values (an array) on the form \code{(\textit{value},\textit{value},\ldots)}. As it stands, this list is empty, but you may add startup options to be included prior to command line options when you run \TeXcount{}. E.g. if you change this to \code{("-inc")} it will automatically add the \code{-inc} option so you don't have to do that yourself every time you run \TeXcount{}. + +\item[Adding macro handling rules]While you may add macro handling rules using \code{\%TC:} commands either in the document or in a separate option file, this is inconvenient for large numbers of macros or if you want these rules always to be included. Also, you might want to add such rules for specific packages. In either case, it might be practical to add these directly to the \TeXcount{} code. \TeXcount{} stores the rules in hashes (maps from a key to a value) named \code{\%TeXmacro}, \code{\%TeXenvir}, etc. There is more documentation on each of these in the code itself, and you may also inspect how rules have been defined for other macros and environments. + +\item[Output style]The ANSI colour codes for different levels of verbosity are encoded in the \code{\%STYLES} hashes and may be changed. The HTML style is encoded in the method \code{html_head()} and is easily modified. + +\item[Character and word definitions]\TeXcount{} identifies words as those that match one of a given set of regular expressions (defined in \code{@WordPatterns}). Note that \code{@WordPatterns} is changed by options \code{-chinese}, \code{-japanese} and \code{-letters}. The pattern that is used within the word patterns to recognise letters is stored in \code{\$LetterPattern}. This is replaced if the \code{-relaxed} or \code{-restricted} option is set. Changing these definitions may be useful if you have special characters or wish to define words differently. + +\end{description} + + +% --------------------------------------------------------------------------- + +\section{Modifying the \TeXcount{} script}\label{sec:code} + +\TeXcount{} is written in Perl, and although hardly the best structured and documented code ever seen, I have tried to structure and document it somewhat. In particular, some parts of the code should be easily modifiable even without in-depth knowledge of Perl or the \TeXcount{} script: e.g. the macro handling rules. + +For more aid on how the \TeXcount{} script is coded and organised, please consult the Technical Documentation. However, here is a very brief overview: + +\begin{description} + +\item[Header and imports:] The shebang (\code{\#!}) and package imports (\code{use \parm{package}}). + +\item[Global variables (and some methods related to these):] This defines and initialised global variables related to option settings, state variables used in parsing and counting (including functions for unterpreting these), variables and hashes for storing macro handling rules, and character class definitions (must be defined before use). + +\item[Main program:] This simply contains a call to the \code{MAIN} routine with the command line arguments. + +\item[Routines/functions/procedures:] The first procedure defined is \code{MAIN} which contains the program flow, then follows other subroutines. Routines with capitalised initial letters indicate high-level routines, while routines starting with underscores (\code{_}) are low-level routines. + +\item[Text data:] At the end of the file is a \code{__DATA__} region containing text data used by the help routines. + +\end{description} + +Perl will first process the setup section which defines global variables, arrays and hashes. It then executes the main section (consisting of the call to \code{MAIN}), whereafter it exits. The subroutines and text data follow after the \code{exit}. + + + +% --------------------------------------------------------------------------- + +\section{License} + +The \TeXcount{} package---script and accompanying documents---is distributed +under the \LaTeX{} Project Public License (LPPL) +\codeline{\url{http://www.latex-project.org/lppl.txt}} +which grants you, the user, the right to use, modify and distribute +the script. However, if the script is modified, you must change its +name or use other technical means to avoid confusion with the original script. + + +\end{document} diff --git a/tools/TeXcount_3_0_0_24.zip b/tools/TeXcount_3_0_0_24.zip Binary files differnew file mode 100644 index 0000000..6929874 --- /dev/null +++ b/tools/TeXcount_3_0_0_24.zip diff --git a/tools/TechDoc.pdf b/tools/TechDoc.pdf Binary files differnew file mode 100755 index 0000000..b6a5bbe --- /dev/null +++ b/tools/TechDoc.pdf diff --git a/tools/TechDoc.tex b/tools/TechDoc.tex new file mode 100755 index 0000000..af38a49 --- /dev/null +++ b/tools/TechDoc.tex @@ -0,0 +1,620 @@ +\documentclass{article} +\usepackage[T1]{fontenc} +\usepackage{a4wide} +\usepackage{times} +\usepackage{listings} +\usepackage{url} +\usepackage{color} + +\include{macros} +\newcommand\Obj[1]{\textsl{#1}} +\newcommand\wild{\ldots} +\newcommand\eqsim{$=\sim$} + +% CGI exclusion macro +%\def\CGI#1\CGIend{} % To exclude CGI text +\def\CGI#1\CGIend{#1} % To include CGI text +\def\CGIend{} + +%%% Title +\title{% +\LARGE \TeXcount\\ +\Large Technical documentation\\ +\Large Version \version\copyrightfootnote +} +\author{Einar Andreas R{\o}dland} + +\sloppy + +\begin{document} + +\maketitle + +{\abstract% +The aim of this document is to explain the implementation details of \TeXcount{} with the aim of aiding anyone who wishes to modify the Perl code (including the author). To be of practical use, it will require some knowledge of Perl and familiarity with \TeXcount{}: while full fledged development of \TeXcount{} requires a working knowledge of Perl, code modification may often be done with only limited experience with Perl. +} + +{\scriptsize\tableofcontents} + +\pagebreak + + + +\section{Introduction} + + +\subsection{\TeXcount{} versioning} + +The version number is on the form \code{\textit{major}.\textit{version}.\textit{subversion}.\textit{build}}. Main releases contain only the first to terms, implying that subversion and build number are both zero. Minor releases only contain the first three terms. Main as well as minor releases should be functional, tested versions. The subversion number can also be \code{alpha} ($\alpha=-2$) or \code{beta} ($\beta=-1$) for which the testing has been limited. The build number is used to keep track of changes and versions during development: they may be made available, but are purely for testing. + + +\subsection{Some things you need to know about Perl} + +\TeXcount{} is written in Perl. The entire script, including macro rules and help texts, is contained in one file. This makes the file somewhat big, and modularisation of the code is therefore not strictly enforced. I have however tried to structure the code somewhat. + +Perl has a few build-in data structures which are referenced in somewhat different manners. In this document, it will be important to recognize the difference between three different types of data: regular variables, arrays and hash maps. + +\begin{description} + +\item[\code{\$\textit{name}=\textit{value}}:] The \code{\$} at the start indicates that it is a Perl variable. The value can be numbers or strings, or it can be a reference which points to another data object (e.g. array or hash map). + +\item[\code{@\textit{name}=(\textit{value},\ldots)}:] The \code{@} at the start indicates that this is an array. The positions are indexed from 0 to $\textrm{length}-1$. + +\item[\code{\%\textit{name}=(\textit{key}=>\textit{value},\ldots)}:] The \code{\%} at the start indicates that this is a hash map that maps keys to values. The key will usually be a string, but can also be a number. + +\item[\code{sub \textit{name}} \{\ldots\}:] This defines a subroutine: function or procedure. Normally, these can be defined anywhere in the script, and I have generally placed them after the main program. + +\end{description} + +Note that \code{(\textit{value},\ldots)} is a list of values, not an array or a hash: it simply produces a list of values that are used to fill the defined array or hash. Arrays and hashes can also be produced directly by \code{[\textit{value},\ldots]} or \code{\{\textit{key}=>\textit{value},\ldots\}} respectively. Both these return a reference to the array/hash rather than the array/hash itself. + +In much of the code, hashes are passed by reference: e.g. if \code{\%hash} is a hash, \code{\$href=\bs{\%hash}} stores a reference to the hash, where the leading \code{\bs{}} causes a reference to be returned rather than the hash itself. Retreiving a value from the hash is done by \code{\$hash\{\textit{key}\}} or \code{\$href->\{\textit{key}\}} if the hash is accessed by reference. Note that individual values in array and hashes are prefixed by \code{\$}: i.e. \code{\$\textit{array}[\textit{index}]} and \code{\$\textit{hash}\{\textit{key}\}}. + +\TeXcount{} makes extensive use of regular expressions (regex): expressions on the form \code{\$\textit{string}\eqsim/\textit{pattern}/} and \code{\$\textit{string}\eqsim s/\textit{pattern}/\textit{replace}/}. In \TeXcount{}, the main use is to recognize (and remove) tokens (words, macros, spaces, etc.) at the start of a string of \LaTeX{} code. Some of these may be fairly simple to understand, while others may be more complex. + + + +\section{Overview} + + +\subsection{Code structure} + +\TeXcount{} is written in Perl, and although hardly the best structured and documented code ever seen, I have tried to structure and document it somewhat. In particular, some parts of the code have been written with modifications in mind so that users can make their own changes without in-depth knowledge of Perl or the \TeXcount{} script. + +Here's a quick walk-through of the code structure and comments on how easily the code may be modified. Some parts of the code are marked as \emph{CMD specific}. There are two version of the script: the CMD version intended for command line use, and the CGI version used with the web interface. The one you have is the CMD version. + +\begin{description} + +\item[\em HEADER AND IMPORTS:] \textit{The shebang (\code{\#!}) and \code{use} imports.} + +\item[\em INITIAL SETUP:] \textit{These set up global variables prior to execution.} + +\item[Settings and setup variables:] The start of the script sets of initial settings and variables. Many of these may be modified by command line options, but if you want to change the default behaviour these may be changed. However, note that there is a list \code{@StartupOptions} intended for this: initially, it is empty, but this is probably the simplest place the change startup options. + +\item[Internal states:] As of version 2.3, internal state identifiers (which are numerical codes) have been defined as \code{STATE}, \code{TOKEN} and \code{CNT} variables, and these are also defined here. A few subroutines for interpreting these states have been included here, although most subroutines are defined after the main code, since they are intimately tied to the state's numerical values. None of these are intended to be modified. + +\item[Styles:] The style definitions basically define which elements to print for each of the verbosity levels. These map element names to ANSI colour codes. When used with HTML, the element names are used as tag classes. If you wish to change the ANSI colour scheme, or change which elements are written in each verbosity option, these may be changed. + +\item[Word pattern definitions:] This section contains regular expression patterns for identifying words and macro options. In addition, the additional character classes defined by \TeXcount{} are defined here. If you have special needs or wishes, modifying these definitions may be an option. + +\item[\TeXcount{} parsing rules:] This is the section in which the main rules for interpreting the \LaTeX{} code is specified: the exception is a few hard-coded rules that do not follow these general patterns. These are hashes that map the macro or environment name to the macro handling rules. First, the default rules are defined, then packages specific rules are defined. + +\item[\em MAIN:] \textit{This is the top-level code which gets executed. All else is done through calls to subroutines.} + +\item[Main \TeXcount{} code:] This is the main code that is run. It is very simple: just a call to the method \code{MAIN} passing the command line options. + +\item[\em SUBROUTINES:] \textit{The subroutines are organised into blocks. Subroutines names use capital letters or initials if they are main routines (like public in other languages) to be used at the top-level, lower case if they may be used throughout but are considered to be lower-level subroutines, prefixed by one or two underscores (_) if used only within the block.} + +\item[Main routines:] The \code{MAIN} routine gives the general processing flow. This in turn calls routines to parse to command line options, process/apply the options, parse the \TeX/\LaTeX{} files, and finally summarise the final results. The main routines are CMD specific. + +\item[CMD specific subroutines:] These are subroutine versions that are CMD specific, e.g. file inclusion and ANSI colours. Their location is somewhat illogical: logically, they might belong later together with related subroutines, but have been placed this early because they are specific to the CMD (or CGI) version. + +\item[Option handling:] After parsing the options, the option values are processed using these subroutines. Some of the option handling operations call on global variables, whereas some are more hard-coded. Like the global variables, if you have special wishes or needs, there may be parts here that can be modified quite easily to change default settings or effects of specific options. + +\item[\TeX{} object:] The main role of the \code{TeX} object (which is technically not an object in the ordinary sense but just a hash) is to be a container object which links to the \TeX/\LaTeX{} code, the word count object, etc. The \code{TeX} object pertaining to any parsed \TeX/\LaTeX{} file is passed along from subroutine to subroutine, usually called \code{\$tex}. The \code{Main} object produced by \code{getMain} is a simple substitute for the \code{TeX} object for use when none is available, e.g. to catch errors not specific to any particular \code{TeX} object. + +\item[File reading routines:] These are used to read files and STDIN. + +\item[Parsing routines:] These contain the main routines for parsing the \TeX/\LaTeX{} code. The main worker method is the \code{_{}parse_{}unit} which parses a block of code: the \emph{unit}. A unit of code may be the contents of an environment, a \code{\{\ldots\}} group, a macro option or parameter, etc. The parsing of one unit is determined by the parsing state, which is passed to the parsing method, and the end marker which indicates which token marks the end of the unit. Different subroutines are then used to process the different types of code: macros, environments, TC instructions, etc. Amongst these routines are also routines for converting the parsed code into tokens, which is done one token at the time which is then removed from the start of the code. + +\item[Count object and routines:] The count object contains the counters as an array, plus titles and labels; in addition it can contain a list of subcounts which are themselves count objects. The count object is used for each file, but also to summarise multiple files, and region counts within files (e.g. per section). The \code{TeX} object contains an active count object to which newly counted words, equations, etc. get added. However, each \code{TeX} object also has a summary count object which will contain the final sum. + +\item[Output routines:] First, there are some routines for general output, i.e. independent of specific \code{TeX} objects. There are then some routines for formatting output, e.g. for the verbose output. There are also routines for printing count summaries in various formats. A special set of routines exist for printing the verbose output itself, and some of these are also involved in the parsing. + +\item[Help functions:] These routines are used to print help. + +\item[HTML functions:] These are routines for producing HTML output. In particular, the HTML style is defined here and may be easily modified. + +\item[Text data:] Some texts are not hard-coded into the script, but added as text data at the end. There are some routines defined to handle the text data, and then the text data itself. + +\end{description} + +Perl will first process the setup section which defines global variables, arrays and hashes. It then executes the main section (consisting of the call to \code{MAIN}), whereafter it exits. The subroutines and text data follow after the \code{exit}. + +\CGI +There is a separate CGI version of the \TeXcount{} script for the web service. While this is mostly the same as the regular command line version, there are some differences in how options are set and \LaTeX{} documents are read. Occasional differences in the CGI version will be commented on, but the main emphasis will be on the command line version of \TeXcount{}. +\CGIend + + +\subsection{Global variables} + +A number of globally defined variables, including constants, arrays and hashes, are defined at the start of the program. These fall into a few different categories. + +There are a number of variables defined for storing options and settings, many of which can be modified by command line options. In addition, there are few variables for global summaries and statistics, as well as a few for internal states during parsing. + +Global constants are defined to represent different states and counters. One set of constants,\code{\$CNT_\wild}, specify the position of the different counters in the counting array; parser states are defined as \code{\$STATE_\wild}; token types are named \code{\$TOKEN_\wild}. For example, the parsing state \code{\$STATE_TEXT} indicates that a block of \LaTeX{} code should be parsed and have words counted as text words. The constants simply take numerical values, but help make the code more readable. Together with some of these are defined functions for interpreting or transforming these + +Alternative settings for different options are defined in a number of hashes, e.g. \code{\%STYLES} indicating which tokens to print at different levels of verbosity, and \code{\%NamedLetterPattern} which stores alternative regex rules which may be used to recognize letters. + +A special set of global settings are the macro handling rules that are stored in a number of hashes: \code{\%TeXmacro}, \code{\%TeXenvir}, etc. as well as similar sets of hashes for package specific rules. + + +\subsection{\TeXcount{} objects} + +First, note that what is referred to as objects here are just hash maps with a predefined set of values. However, these serve the same purposes as objects. There are no explicit class specifications defining these, just a set of functions returning hashes that contain the required keys, some of which may even be optional. Still, it is useful to think of them as objects, and their main purpose is to encapsulate data so that they can conveniently be passed around. + +\begin{description} + +\item[The \Obj{Main} object] +Each \TeXcount{} session instantiates a singleton \Obj{Main} object. This is used as a replacement when no \Obj{TeXcode} object is available for capturing (counting and storing) error messages and warnings. + +\item[The \Obj{TeXcode} object] +The \Obj{TeXcode} object encapsulates the \LaTeX{} code that is to be parsed as well as counts and lists of reported errors. In the code, it is generally referred to using the \code{\$tex} variable. + +\item[The \Obj{count} object] +The \Obj{count} object is primarily a container for the array of counts: i.e. the array containing word counts and counts of headers, equations, etc. However, it also keeps track of subcounts from contained files, sections, etc. + +\end{description} + +A more detailed explanation of the different objects is provided in section \ref{sec:objects} + + +\subsection{Main program flow} + +The main program consists of a single call to the procedure \code{MAIN}. This does the following: + +\begin{description} + +\item[\code{Initialise}:] +Most of the initialisation is done when defining the global variables, but some initalisation required code execution: e.g. OS specific initialisation. + +\item[\code{Check_Arguments}:] +Runs an initial check of the command line arguments passed to \TeXcount{}, e.g. for \code{-help}, and may exit \TeXcount{}. + +\item[\code{Parse_Arguments}:] +Parses the command line arguments, setting option variables, and returns the list of \LaTeX{} files to be parsed. + +\item[\code{Apply_Options}:] +This applies the options set either in the initial setup or initialisation, or when parsing the arguments. While most options are set directly during the argument parsing, settings that may depend on multiple options or options that should be applied only once, e.g. initialising the output and writing the HTML header, are applied here. + +\item[Parse files (or write help or error message):] +The file parsing calls \code{Parse_file_list} with the list of files to be parsed, and this returns the total count object. Apart from this, help, summary output and error reports are produced if required. + +\item[\code{Close_Output}:] This just makes sure the output channel is properly closed, e.g. writing closing HTML code. + +\end{description} + +\CGI +In the CGI version of \TeXcount{}, \code{Initialise}, \code{Check_Arguments} and \code{Parse_Arguments} are replaced by a single call to \code{Set_Options}. Also, since the CGI version only processes one file, alternatives for parsing and reporting on multiple files are not required and is instead replaced by a single call of \code{parse}. +\CGIend + + +\subsection{How \TeXcount{} processes \LaTeX{} documents} + +The \code{parse} routine is the entry point for parsing \LaTeX{} code of a single file. It takes a \Obj{TeXcode} object, the container object of a \LaTeX{} document and its corresponding \Obj{count} object, performs the parsing of the entire document. The counts are stored in the counter in the \Obj{TeXcode} object. + +The hierarchy of delegation from \code{MAIN} down to \code{parse} is as follows: + +\begin{description} + +\item[\code{MAIN}] calls \code{Parse_file_list} with a list of files which return the total count (a count object) for \code{MAIN} to report. + +\item[\code{Parse_file_list}] calls \code{parse_file} for each file in the provided file list, and for STDIN (identified by \code{\$_STDIN_}) if the option to parse standard input has been set. It then aggregates the counts returned by \code{parse_file} into a total count which it returns. + +\item[\code{parse_file}] calls \code{_add_file}, first for the main file, and then again for each included file if file inclusion (\code{-inc}) has been set. The aggregation of counts is done by \code{_add_file} into a total count object provided by \code{parse_file}, and this total count object is then returned by \code{parse_file} upon completing the parsing of the main file as well as all included files. + +\item[\code{_add_file}] reads the file into memory, creates a \Obj{TeXcode} object which encapsulates the \LaTeX{} code and the counts, and calls \code{parse} to perform the parsing of this \Obj{TeXcode} object. The counts are added directly into the \Obj{TeXcode} object, so only the \Obj{TeXcode} object reference is being passed around. + +\end{description} + +\CGI +In the CGI version, \code{parse} is called directly from \code{MAIN} since only one document can be parsed and no file inclusion is possible. +\CGIend + + +\subsection{\LaTeX{} code parsing by \code{parse}} + +The \code{parse} routine takes a \Obj{TeXcode} object and parses this to the end. It is, however, only the entry point for parsing the \LaTeX{} code: other routines do the main parsing with \code{_parse_unit} being the main work horse. In fact, \code{parse} only initiates the parsing, calling \code{_parse_unit} repeatedly until the end of the file. + +The \code{_parse_unit} routine is used to parse one unit or block of \LaTeX{} code: a unit/block can be the a part of the document enclosed in e.g. \{\ldots\} or \code{\bs{begin}\ldots\bs{end}}, or based on context enclosed by e.g. \code{[\ldots]}, or the document at the top level. It is passed the \Obj{TeXcode} object to parse, a parsing state instructing it how the block should be parsed, and optinally a block-end token which tells \code{_parse_unit} when the block ends. The \code{_parse_unit} routine is then called recursively whenever a unit/block is encountered that requires a separate parsing state or closing token. + +The parsing state indicates if the block is part of the main text in which words should be counted, a header, equation contents, should be excluded, etc. and is the only state variable of the parser. In addition to the regular states with which the \LaTeX{} code is parsed, there are transition states. E.g. \code{\$STATE_TO_HEADER} indicates that the block should be counted as a header and the contents should then be parsed using \code{\$STATE_TEXT_HEADER} as specified in \code{\%transition2state}. + +The document is tokenized, and \code{_parse_unit} retrieves one token at the time by calling \code{next_token}. Depending on the active parsing state and token, different rules (most with their own subroutines) are applied. These rules add to the \Obj{count} object of the \Obj{TeXcode} object by calling \code{_inc_count} and set the presentation style of the verbose output included which tokens to print. The active token and its style is by default stored in the \Obj{TeXcode} object and printed to the verbose output by \code{next_token} upon retrieving the next token, although this is occasionally overrun by calls to e.g. \code{flush_next}. + +When \code{_parse_unit} encounters a new block/unit, it will determine the state with which this unit should be parsed based the present state and the context that defines the unit. + + +\subsection{Summary statistics} + +The counts are stored in the \Obj{TeXcode} object: subroutines performing the actual parsing increments the appropriate counter upon processing the parsed tokens. The \Obj{TeXcode} object contains a main \Obj{count} object from which summary output is generated. The \Obj{count} object can also contain a list of subcounts, themselves \Obj{count} objects, which may also be presented in the summary. + +Depending on options set and the number of files parsed, summary output can range from a single number of the total word count, to an extensive summary for each spcified file with separate summaries for each included file, as well as a total summary. + + + +\section{Global constants and variables} + +There are a number of global variables defined at the start of the script for storing options and settings as well as global counters. + +In addition, there are sets of global constants, as well as other globally defined variables, hashes and arrays. Here, we outline the main groups. + + +\subsection{Global constants} + +There are a few sets of global constants. The use of global constants makes the code more readable. The sets of global constants are: + +\begin{description} + +\item[\code{\$STATE_\wild}:] Parsing states, e.g. \code{\$STATE_TEXT} for parsing \LaTeX{} code as regular text and \code{\$STATE_IGNORE} for regious that should not be counted. + +\item[\code{\$CNT_\wild}:] Index pointing to the location in the counter array used for a specific count, e.g. \code{\$CNT_WORDS_TEXT=1} indicating that words in text are counted in position 1 of the array. + +\item[\code{\$TOKEN_\wild}:] Token types, e.g. \code{\$TOKEN_WORD} and \code{\$TOKEN_MACRO}. When a token is parsed, the \Obj{TeXcode} object stores the token type as well as the token, which can then be used to determine how the token should be interpreted. + +\end{description} + +\subsubsection{Counter indices: \code{\$CNT_\wild}} + +The \Obj{count} object contains an array with the following counts: number of files, number of words in text, number of words in headers, number of words in captions, number of headers, number of floating objects/tables/figures, number of inline equations, number of displayed equations. Storing these are the main purpose of the \Obj{count} object. + +Each count has a fixed position in the array, and the \code{\$CNT_\wild} constants provide the positions of each count: e.g. \code{\$CNT_WORDS_TEXT=1} indicates that the counter for words in the text is stored in position 1 of the array. Originally, these positions were hard-coded and directly related to the parsing states, but by using these constants, and keeping the counter indices distinct from the parsing states, the code becomes both more readable and more flexible in case of future changes. + +\subsubsection{Parsing states: \code{\$STATE_\wild}} + +The parsing states fall into two categories. + +First there are parsing states used during the parsing of a unit/block: e.g. \code{\$STATE_TEXT}, \code{\$STATE_MATH}, \code{\$STATE_IGNORE}, ldots. In some of the states, words are counted either as text words, header words or captions words; in other states, words are ignored and the state primarily influences how the parsed \LaTeX{} code is styled in the verbose output. + +Secondly, there are transitional states: e.g. \code{\$STATE_TO_HEADER} which indicates the start of a header which should first cause the header count to be incremented and then the contained text to be parsed as header text using the parsing state \code{\$STATE_TEXT_HEADER}. The handling of the transitional states are encoded in \code{\%transition2state} and performed by the \code{transition_to_content_state} routine which is called by \code{_parse_unit}. + +Macro handling rules specify how many parameters the macro takes and which parsing states are used to parse each parameter; for environments, it additionally specifies a parsing state for the contents of the environment. + +Originally, before implementing the \code{\$STATE_\wild} constants, fixed numerical values were hard coded into the Perl code, and these numerical codes were required for adding new rules. For the macro rules specified within the Perl code of \TeXcount{}, the original numerical codes remain in the initial rule specification. However, from version 2.3 of \TeXcount{}, the intention is that users should no longer use these numerical codes to specify new macro handling rules, but instead use a set of keywords: e.g. \code{text}, \code{header}, \code{ignore}, etc. For this purpose, a hash \code{\%key2state} is defined which maps keywords to parsing states. The original numerical codes are included in this map in part for backward compatibility, but also because this key-to-state map is applied to the macro handling rule hashes \code{\%TeXmacro}, \code{\%TeXenvir}, etc. The \code{\%key2state} has is set up e.g. with +\codeline{add_keys_to_hash(\bs{\%key2state},\$STATE_TEXT,1,'word','w','wd');} +which maps the keys \code{1}, \code{'word'}, \code{'w'} and \code{'wd'} all to the value \code{\$STATE_TEXT} (which need not be 1!). However, this specification, which is used to convert keywords to states during initialisation of the macro handling rules and later if adding new rules, ensures that the original numberical codes will be handled as before: \TeXcount{} will be backward compatibile with respect to using the numberical codes to add new macro handling rules through \code{\%TC} commands. + +Although in theory the parsing state numerical codes could be changed without any effect to the code, there are still a few places where the actual numerical values are used: e.g. the routine \code{state_to_text}. + +\subsubsection{Token types: \code{\$TOKEN_\wild}} + +When the \LaTeX{} code is tokenized, i.e. the string containing the \LaTeX{} code is converted to tokens like words or macros, not only is the token stored in the \Obj{TeXcode} object, but a token type is stored as well indicating if the object is a word, macro, space, symbol, bracket, etc. To make the Perl code more readable, these token type, although just integer values, are represented by constants \code{\$TOKEN_\wild}. + +When \code{_parse_unit} parses the \LaTeX{} code, it frequently uses the token type stored in the \Obj{TeXcode} object rather than the token itself to determine how to interpret the parsed tokens. + + +\subsection{Option alternatives} + +Some options result in choosing between a number of alternatives for parsing, counting or presentation. These alternatives tend to be defined in arrays or hashes. When an alternative is selected, the corresponding value(s) are copied to a variable, array or hash which may then later be applied or further processed. + +\begin{description} + +\item[\code{\%BreakPointOptions}:] For keywords like \code{section} or \code{chapter}, this defines which macros indicate a new break point (i.e. initiates a new subcount). + +\item[\code{\%STYLES}, \code{\%STYLE}:] The \code{\%STYLES} hash contains different sets of style definitions, used to define the style with which tokens are printed, and are used to set the \code{\%STYLE} hash by \code{Apply_Options} after the options have been processed. Each value of the \code{\%STYLES} is a hash mapping style name to ANSI colour styles. For a given style, only style names defined in the style are printed in the verbose output. If ANSI colour coded output is used, these are the colour codes; otherwise, the ANSI colour styles are not themselves used, but the style name must still be included in the hash to enable the token to be printed. + +\item[\code{\%NamedLetterPattern}, \code{\$LetterPattern}:] Named regex patterns are defined in \code{\%NamedLetterPattern} where the selected pattern is stored in \code{\$LetterPattern}. This regex pattern defines what is recognized as letters when parsing \LaTeX{} code. + +\item[\code{\%NamedWordPattern}, \code{@WordPatterns}, \code{\$WordPattern}:] Named word patterns are defined in \code{\%NamedWordPattern}. The selected patterns are stored in the array \code{@WordPatterns}. Letters are indicated by a special character, and when the options are applied replaced by \code{\$LetterPattern} and merged into a single regex stored in \code{\$WordPattern}. + +\item[\code{\%NamedMacroOptionPattern}, \code{\$MacroOptionPattern}:] Named regex patterns are stored in \code{\%NamedMacroOptionPattern}, and the selected pattern copied to \code{\$MacroOptionPattern}. This pattern is used to recognize macro options which should be excluded from word counts. + +\item[\code{\%NamedEncodingGuessOrder}:] For each named language, this gives an array of encodings to try if none is given. + +\end{description} + + + +\section{Details of the \TeXcount{} objects}\label{sec:objects} + +These objects are simply hashes that are created with a given set of keys. Some keys may, however, be optional. + + +\subsection{The \Obj{Main} object} + +The \Obj{Main} object is used instead of the \Obj{TeXcode} object to capture errors and warnings. It is created by the \code{getMain} routine. The values (keys) it contains are: + +\begin{description} + +\item[\code{errorcount}:] Numerical value, initialised to 0, used to count the number of errors reported. + +\item[\code{errorbuffer}:] Array, initialised to an empty array, used to buffer error messages reported before output is available: e.g. before the header or HTML header has been printed. + +\item[\code{warnings}:] Hash, initially empty, used to store warnings. + +\end{description} + +When errors are reported through calls to \code{error}, they will be stored in the \code{errorbuffer} if this exists, otherwise printed immediately. This is used to store errors reported before e.g. the HTML header has been written. After the appropriate headers have been written and the output channel is ready for writing, a call to \code{flush_errorbuffer} is made which prints all the errors in the errorbuffer and then deletes it so further errors will be printed immediately rather than buffered. + + +\subsection{The \Obj{TeXcode} object} + +The \Obj{TeXcode} object is used to encapsulate the \LaTeX{} code and corresponding counts. It is created by the \code{TeXcode} routine. The values it contains are: + +\begin{description} + +\item[\code{filename}, \code{filepath}:] The name and path of the parsed \LaTeX{} file. + +\item[\code{PATH}:] An array containing the paths to search for included documents. At creation, this is empty, but calls to \code{_add_file} will set it; the top level files, initiated from \code{parse_file}, will have this set to \code{\$workdir}. + +\item[\code{texcode}:] Initialised with the \LaTeX{} document as a single string. If included files are to be inserted into the document, they will be inserted into the \code{texcode} string. + +\item[\code{texlength}:] Counts the total length (in characters) of \LaTeX{} code. Initialised with the length of the \LaTeX{} document. If included documents are inserted, their length is added to \code{texlength}. + +\item[\code{line}:] Initialised to an empty string. During parsing, segment by segment (one paragraph at a time) is moved from \code{texcode} to \code{line}. Tokens are then read and subsequently removed from \code{line}. + +\item[\code{next}:] Initialised to \code{undef}, this stored the next token to be processed. Upon tokenization, the token is identified and removed from the start of \code{line} and moved to \code{next}. + +\item[\code{type}:] Initialised to \code{undef}, this contains the token type (\code{\$TOKEN_\wild}) of the \code{next} token. + +\item[\code{style}:] Initialised to \code{undef}, this is used to set the style with which the \code{next} token should be presented in the verbose output. + +\item[\code{printstate}:] Initialised to \code{undef}, this is used output the active parsing state for use with verbose output (if \code{\$showstates} is set). + +\item[\code{eof}:] Initialised to 0, this is set to 1 once the end of the document is reached. + +\item[\code{countsum}:] The contains the main \Obj{count} object. + +\item[\code{subcount}:] This contains the present subcount which is also a \Obj{count} object. These subcount are used to count e.g. section and chapters of the document. + +\item[\code{errorcount}:] Initialised to 0, used to count the number of errors reported during the parsing. + +\item[\code{errorbuffer}:] Undefined at initiation, indicating that errors should be printed instantly rather than stored for later printing. Can be defined as an array which is then used to store error messages so they can be printed later. + +\item[\code{warnings}:] Hash used to store warnings. + +\end{description} + +When the \Obj{TeXcode} object is initialised, the \LaTeX{} document is placed as a single big string in \code{texcode}. During parsing, \code{next_token} is called on to return the next token, which in turn it delegates to \code{_get_next_token}. Instead of operating on the whole document, which was done in older version of \TeXcount{} and was quite slow on large document, \code{more_texcode} is called on to move segments (i.e. paragraphs) of \LaTeX{} code from \code{texcode} to \code{line}, and then it grabs one token at a time from \code{line}. This is when \code{next} and \code{type} are set. + +When the tokens are interpreted and counted, \code{inc_count} is called which increments the appopriate counter in \code{subcount}. If a new subcount is initiated, a call to \code{next_subcount} adds \code{subcount} to \code{sumcount}, including appending the \code{subcount} object to the list of subcounts stored with \code{sumcount}, and then replaces \code{subcount} with a new \Obj{count} object. + + +\subsection{The \Obj{count} object} + +The \Obj{count} object is used to store the word and text element counters. It is created by \code{new_count}. The values it contains are: + +\begin{description} + +\item[\code{title}:] A string set upon creating to contain a descriptive title of the count. + +\item[\code{counts}:] An array, initialized with 0s, which is used to store the counts. The size of the array is determined by \code{\$SIZE_CNT} and should reflect the number of \code{\$CNT_\wild} indices defined. + +\item[\code{subcounts}:] This is an array, initialised to an empty array, used to store the subcounts. + +\end{description} + +In addition to the default fields, when used as the \code{sumcount} field of a \Obj{TeXcode} object, a few additional fields are added: + +\begin{description} + +\item[\code{TeXcode}:] This is a reference pointing back to the \Obj{TeXcode} object in which it is contained. + +\end{description} + + + +\section{\LaTeX{} code parsing and interpreting} + +The entry point for parsing a \LaTeX{} document is the \code{parse} routine. This simply calls \code{_parse_unit} repeatedly using parsing state \code{\$STATE_TEXT} until the end of the document is reached. Thus, \code{_parse_unit} is the main routine for performing the actual parsing. + +The \code{_parse_unit} routine is called with a \Obj{TeXcode} object, a parsing state, and optionally an unit-ending token as arguments. It then calls \code{next_token} on the \Obj{TeXcode} object until the unit-ending token is reached: if the file ends before this is found, an error is reported. If no unit-ending token is provided, only one unit will be parsed. If the unit-ending token is set to \code{\$_PARAM_}, indicating that the unit to be parsed is a macro parameter, the \code{\$simple_token} flag is set and passed to \code{next_token} to avoid combining letters into words, and only one token is parsed before returning. + +For each token, depending on the token, token type, and active parsing state, \code{_parse_unit} decides how the token should be interpreted. In some cases, the interpretation is done within \code{_parse_unit}, but in many cases the interpretation is delegated to subroutines like \code{_parse_macro}, \code{_parse_math}, etc. If new groups (\code{\{\ldots\}} or \code{\bs{begin}\ldots\bs{end}}) are encountered, this causes \code{_parse_unit} to be cause recursively with an unit-ending token passed to \code{_parse_unit} to identify the group end. + +Note that by default, even blocks that are to be ignored are parsed and required balanced units. Different exclude states exist to deal with cases in which the unit should not be completely parsed. + +Upon interpreting the parsed tokens, \code{_parse_unit} or the subroutines to which it delegates the interpretation control the counter incrementation as well as how the tokens are presented in the verbose output. The counter incrementation is done through calls to \code{inc_count} passing as arguments the \Obj{TeXcode} object, the appropriate count reference (\code{\$CNT_\wild}), and optionally a number if the counter should be increased by a number different from 1. Specifying how the token should be presented in the verbose output is done by deciding on the style, usually set using \code{set_style}: the styles are represented by strings that give the style name, which are the same as used as keys in \code{\%STYLE} and as styles in the HTML output. + +If a style for presenting a token is selected which is not in the \code{\%STYLE} hash, the token is not printed. Thus, the \code{\%STYLE} hash also filters which tokens are printed to the verbose output. + + +\subsection{Tokenization and token handling} + +The routine for retrieving the next token is \code{next_token}. This first makes sure that the previous token gets printed to the verbose output with the style specified by \code{set_style}. It then calls \code{_get_next_token} to retrive the next token: this will process comments and line breaks itself until a token is retrieved that it returns. + +The \code{_get_next_token} routine checks the \code{line} field of the \Obj{TeXcode} object to determine which is the next token in \code{line}. If the \code{line} field is empty, it calls \code{more_texcode} to move the next segment of \LaTeX{} code from the \code{texcode} field of the \Obj{TeXcode} object to \code{line}. When it has decided on the approriate kind of token, removing it from the start of the \code{line} field in the process, it sets the \code{next} and \code{type} fields of the \Obj{TeXcode} object through calls to \code{__set_token} or \code{__get_token} (for single character tokens). + +If the optional \code{\$simple_token} flag is set, only simple tokens will be returned: i.e. letters will not be combined into words. This is used for parsing macro parameters. + + +\subsection{Processing parameters and options} + +In \code{_parse_unit}, based on the parsing state and parsed token, it is decided how to interpret and process the token. In some cases, this processing is restricted to the parsed token itself: counting or ignoring it as well as deciding on the style with which it should be presented in the verbose output. + +In some cases, the token influences the parsing of subsequent text: e.g. macros can take parameters and options. Special subroutines exist to handle parsing of macro parameters, gobble up spaces or macro parameters, or handle ignored regions. + + +\subsection{Verbose output} + +By default, all parsed code is processed for printing to the verbose output. If it actually gets printed or not depends on whether the set style is included in the \code{\%STYLE} hash or not. + +Upon parsing a token, it is stored in the \code{next} field of the \Obj{TeXcode} object. If \code{set_style} is called during processing, this will set the \code{style} field of the \Obj{TeXcode} object, but will not itself print the token. The \code{flush_next} routine is used to print the \code{next} token using the style set in the \code{style} field, or provided in the call; this in turn calls \code{print_style} which is responsible for the printing. There is an automatic call to \code{flush_next} when the next token is retrieved, ensuring that all tokens are sent off for printing. When \code{flush_next} is called, the \code{style} field is set to \code{\$STYLE_BLOCK='-'} which blocks further printing (or change in style) of the token; the \code{style} field is then set to \code{undef} by \code{next_token} upon reading the next token. + +The tokens are passed to \code{print_style}, either directly from the parsing or via \code{next_token}, which looks up the style in the \code{\%STYLE} hash. Only tokens whose style is defined in the \code{\%STYLE} hash get printed. If colour coded output to text is set, the values \code{\%STYLE} are used with the \code{ansiprint} function to print the token using ANSI colour codes. If output to HTML is chosen, the token will be printed enclosed in a \code{<span>} tag using the style as class; the HTML style definitions are then used to determine how these elements will be displayed. + +Special style values are \code{\$STYLE_EMPTY=' '} which is used for spaces and must be defined in the \code{\%STYLE} for spaces to be printed, and the \code{\$STYLE_BLOCK='-'} style value which is not actually a style but a value used to mark that the token has already been printed and block further printing of it. + +In addition to the \code{\%STYLE} hash which specifies which tokens get printed, there is a global variable \code{\$printlevel} the value of which is taken from the \code{\%STYLE} which is used to control if verbose output is on ($1$ or $2$) or off ($0$ or $-1$). The $-1$ values indicates the quiet mode in which errors should not be printed; the value $1$, as opposed to $2$, indicates that multiple ignored lines should be collapsed to make the verbose output more compact, although this is only partially done. + +The routines for handling tokens, styles and verbose printing remain from the earliest version of \TeXcount{} and has not undergone much improvements or cleaning up and remains somewhat unstructured. Hence, there may be stray calls to e.g. \code{set_style} that no longer have any effect. + + + +\section{Regex patters: letters, words, macro options} + +One of the most important regex definitions in \TeXcount{} is that used to recognize words. This is done in two steps: first a regex for letters is produced, and then this is combined with patterns for words to generate one big pattern. + +Another regex defined is the one used to recognize macro options, i.e. \code{[\ldots]}, that appear together with macros and which should be ignored. + +One reason behind the desire to generate one big pattern rather than loop through alternative patterns is to enable Perl to compile each pattern just once. The pattern compilation typically takes longer than the pattern matching, so this can make a big difference. + + +\subsection{The word regex} + +First note that \TeXcount{} distinguishes between alphabetic words, i.e. words composed of letters, and logograms (e.g. Chinese characters) which are counted per character. When words (or letters) are counted, these are made from characters defined as alphabetic; characters defined as logographic are counted separately character by character. + +The regex pattern recognizing a letter is placed in \code{\$LetterPattern}. This is usually taken from one of the optional patterns in \code{\%NamedLetterPattern}, but can be modified elsewhere or replaced by \code{undef} to signify that no words or letters should be counted. + +A number of regex patterns which should be recognized as words are place in the array \code{\@WordPatterns}. This is usually set by using one of the named lists of word patterns defined in \code{\%NamedWordPattern}, but can be redefine or modified by options. In the word patterns, the character \code{\@} is used to represent a letter, and this is later replaced by \code{\$LetterPattern} when the options are applied. + +After parsing the command line arguments, the options and settings are applied. At this point, through \code{apply_language_options}, \code{\$LetterPattern} is applied to \code{\@WordPatterns}, which are then combined into a single regex: \code{\$WordPattern}. At this point, patterns for recognizing logograms are also added. + + +\subsection{The macro option regex} + +After macros and macro parameters, macro options on the form \code{[\ldots]} will be ignored. There is a single regex used to recognize and remove these macro options. + +For most uses, macro options tend to be short codes which are easily recognized. However, there are also cases where the macro options can be more complex. On the other hand, there are also cases where brackets are used without being macro options, and it is vital that these cases should not be mistaken for macro options: in particular if they contain text that should be counted. + +In order to capture most macro options as options without running a risk of ignoring actual text enclosed in brackets, restrictions are placed on what can go inside macro options. The default rule is moderately strict, but can be relaxed to allow more extensive and general macro options. + +The different macro option regex patterns are named in \code{\%NamedMacroOptionPattern} and copied to \code{\$MacroOptionPattern} when initialised or changed by options. + + +\subsection{Unicode character classes} + +The user can specify which character classes should be considered alphabetic (i.e. letters) and which should be considered logographic (i.e. counted as indicidual characters). Typical alphabetic characters are the Latin letters. Typical logograms are the Chinese characters. If any of the language options are used, these character classes will automatically be set. + +Specifications of alphabets and logograms are done by options \code{-alpha=} and \code{-logo=} using Unicode character classes. Unicode classes include Latin, Digit, Ideographic, Han, etc. Note that all Unicode character classes start with capital letters. + + +\subsection{Custom made character classes} + +Some of the Unicode character classes are not defined quite as desired by \TeXcount{}. In particular, the \code{Alphabetic} character class includes \code{Ideographic}, which would cause e.g. Chinese characters to be allowed as parts of words together with Latin characters rather than force them to be counted as individual characters. To resolve this problem, new character classes are defined in \TeXcount{} that fit our need. + +New character classes can be defined within \TeXcount{} through subroutines named \code{Is_\textit{name}}. Most notable is the \code{Is_alphabetic} character class from which the logographic characters have been excluded. This is now used as the default alphabetic character class. + +Presently defined characters classe are named \code{digit}, \code{alphabetic}, \code{alphanumeric}, \code{punctuation}, \code{cjk}, \code{cjkpunctuation}. Note that these are all lower case, and have the prefix \code{Is_} added when referred to in the code. + +When adding character classes to the set of alphabetic or logographic characters using \code{-alpha=} or \code{-logo=}, the names without the prefix \code{Is_} may be used: for character classes starting with a lower case letter, the prefix is added automatically. + +Note that the subroutines specifying the character classes must be defined prior in the code to any use: this is unlike other subroutines which may be defined anywhere in the code. Also, to be permitted as character classes by Perl, the subroutines must start with \code{Is_} (or \code{In_} although that is not used by \TeXcount{}), although different versions of Perl need not enforce this. + + + +\section{Macro handling rules} + +While some rules for handling macros are hard-coded into \TeXcount{}, most of the rules are stored in a number of hashes which \TeXcount{} look up whenever a macro is encountered. The general rule is that the keys are either macros (e.g. \code{'\bs{section}'}) or environment names (e.g. \code{'quote'}). + +\begin{description} + +\item[\code{\%TeXmacro}:] The keys are macros, or \code{'begin\textit{name}'} where name is an environment name, and the values specify how many parameters the macro (or environmemt) takes and how these should be processed. See the section on parameter handling rules further down. + +\item[\code{\%TeXenvir}:] The keys are environment names, and values are the parsing state with which the contents of the environment should be parsed. + +\item[\code{\%TeXpreamble}:] These are macro handling rules to be applied in the preamble, i.e. after \code{\bs{documentclass}} but before \code{\bs{begin}\{document\}}. The rules are specified as for \code{\%TeXmacro}. + +\item[\code{\%TeXfloatinc}:] These are macro handling rules to be applied within floating bodies, i.e. tables and figures. + +\item[\code{\%TeXmacroword}:] The keys are macros, and the values are numbers representing how many words the macro generates. This is used for macros like \code{\%LaTeX} which generates text. + +\item[\code{\%TeXpackageinc}:] The keys are macros used to include packages. Although included in \code{\%TeXmacro}, the processing of package inclusion is actually performed by \code{_parse_include_package} independent of the hash value. The value should therefore be \code{1} or \code{[\$STATE_IGNORE]} since this is how it will be processed by \code{_parse_include_package}. + +\item[\code{\%TeXfileinclude}:] The keys are macros used to include \LaTeX{} files into the document, the value a keyword or list of keywords telling how file names and paths should be interpreted. Processing of these macros is done by \code{_parse_include_file}. + +\end{description} + +Note that the definition of \code{\%TeXmacro} starts by including \code{\%TeXpreamble}, \code{\%TeXfloatinc} and \code{\%TeXpackageinc}. After that, the values of \code{\%TeXpackageinc} are never used. For \code{\%TeXpreamble} and \code{\%TeXfloatinc}, however, it is in principle possible to rules within the preamble and floats, respectively, that are different from those defined in \code{\%TeXmacro} and applied elsewhere in the document. + + +\subsection{Parameter handling rules} + +A macro can be specified to take a given number of parameters: this will typically be \code{\{\ldots\}} blocks following the macro. For each of these parameters, a separate parsing state can be specified. This is represented by an array with one element for each parameter, the elements being the parsing state (\code{\$STATE_\wild}) with which that parameter should be parsed. + +In addition to the \code{\$STATE_\wild} rules are some modifier/option states, \code{\$_STATE_\wild}. The \code{\$STATE_OPTION} states indicates that the next rule in the list is an optional parameter enclosed in \code{[]}. By default \code{[]} options are ignored, which can be swithed off by \code{\$STATE_NOOPTION} or on by \code{\$STATE_AUTOOPTION}. + +An alternative specification of a parameter handling rule is to give the number of parameters to ignore. \TeXcount{} will check if the specified rule is an array (as described above) or a number and interpret the rule accordingly. + +The hashes \code{\%TeXmacro}, \code{\%TeXpreamble} and \code{\%TeXfloatinc} all take values that are this kind of parameter handling rules, as are q\code{\%TeXpackageinc} since they are included in \code{\%TeXmacro}. + +Throughout the script, parsing states are referred to using the \code{\$STATE_\wild} constants. In previous versions, however, these codes were hard-coded into the script and used both to set up the hashes and to specify new rules through \%TC instructions. For backward compatibility, the old numerical state codes remain in the conversions from keywords to \code{\$STATE_\wild} constants as stored in \code{\%key2state} and applied through calls to \code{convert_hash} accompanied by \code{keyarray_to_state} or \code{key_to_state}. + + +\subsection{File inclusion and the \code{\%TeXfileinclude} hash} + +The main \LaTeX{} commands for file inclusion are \code{\bs{input}} and \code{\bs{include}}, while \code{\bs{bibliography}} includes the \code{.bbl} bibliography file. However, additional packages exist that can also modify the file search path, of which \TeXcount{} has support for the \code{import} package. + +File inclusion macro rule are stored in the \code{\%TeXfileinclude} hash. The values are strings which contain one or more keywords (separated by space or comma): +% +\begin{description} + +\item[\code{input}:] This is a special keyword to use with \code{\bs{input}}. The handling of the parameter values is as \code{file}, but the parameter itself is not required to be enclosed in \code{\{\}}. + +\item[\code{file}:] This parameter simply gives the name of or path to a file. If the file is not found, \TeXcount{} will append \code{.tex} and try again. + +\item[\code{texfile}:] This parameter gives the name of or path to a file, but \code{.tex} will be appended, and is the rule used by \code{\bs{include}}. + +\item[\code{dir}:] This parameter provides the path of a directory relative to the \code{\$workdir}, and adds this to the search path before including any files. This is used with the \code{\bs{import}} macro of the \code{import} package. + +\item[\code{subdir}:] This parameter provides the path of a directory relative to the current directory, and adds this to the search path before including any files. This is used with the \code{\bs{subimport}} macro of the \code{import} package. + +\item[\code{<bbl>}:] This is a special keyword to use with \code{\bs{bibliography}} to specify inclusion of the bibliography file. + +\end{description} + +The parsing of the macros and parameters is done by \code{_parse_include_file}. For each keyword it parses a parameter, unless the parameter is on the form \code{<\textit{keyword}>}. The parsing of the \code{input} parameter is handled differently from the rest since it need not be enclosed by \code{\{\}}. It then delegates the processing of macro inclusion rules to \code{include_file}. + +In \code{include_file}, the file is located (based on search path) and either appends the file to the \code{@filelist} array of files to be include, or merged immediately into the document by a calls to \code{read_binary} and \code{prepend_code}. + +The \code{@filelist} array contains elements which are themselves arrays on the form \code{[file,path,\ldots]} where the first element is the path to the file to be included, and the remaining elements are the search paths used to set the \code{PATH} values of the \Obj{TeXcode} object. For the top level files, i.e. the ones specified on the command line, the search path will contain only \code{\$workdir}: the directory from which \TeXcount{} is executed unless \code{-dir} is used to specify otherwise. If more directories are added to the path, \code{\$workdir} will remain the last directory of the search path, while the first directory of the search path will be considered the current directory. + +File inclusion macros can also take parameters that should be parsed using regular macro parsing rules. The \code{TeXmacro} hash will be checked for \code{@pre\bs{macroname}} and \code{@post\bs{macroname}} entries which will be applied before and after the file handling rules. + + +\subsection{Package and document class specific rules} + +Whenever \TeXcount{} encounters a package inclusion, it will check for package specific rules. These are defined in hashes names \code{\%PackageTeXmacro} etc. which maps the package name to the hash map of rules to be added to \code{\%TeXmacro} etc. There is an additional \code{\%PackageSubpackage} which for each package name in the set of keys maps to a list of packages whose rules should automatically be included. + +Similarly, rules specific to particular document classes may be implemented by using the key \code{class\%\parm{name}} instead of the package name, and these will then be added to the set of parsing rules if \code{\bs{documentclass}\{\parm{name}\}} is encountered. + +Note that rules for including the bibliography is also stored in these hashes under the key \code{\%incbib}. + + + +\section{Presentation of summary statistics} + +The counts (words, headers, etc.) from a \LaTeX{} document are stored as a \Obj{count} objects. The main routine for printing the summary statistics from a \Obj{count} object is \code{print_count}: the routine \code{conditional_print_total} which is called from \code{MAIN} delegates printing to \code{print_count} except if the brief output format is selected. The \code{print_count} routine then delegates the printing to one of a number of subroutines depending on the settings. + +Word frequencies are store globally in \code{\%WordFreq}. This gets incremented each time \code{_process_word} is called. Summary of word frequencies are produced and printed by \code{print_word_freq} which tries to combine words that differ only by capitalization, and also produces subcounts per character class. + +A global count of the number of errors reported is stored in \code{\$errorcount}, while warnings are stored globally in the \code{\%warnings} hash mapping when added through the \code{warning} routine with the warning as key and the number of occurrences as value to ensure each warning is only listed once no matter how many times it is reported. Both warnings and errors are also stored in their respective \Obj{Main} or \Obj{TeXcode} objects when reported through calls to \code{error} or \code{warning}. + +In \code{MAIN}, after processing of the \LaTeX{} documents, \code{Report_Errors} is called to give a total report on errors and warnings. The exact output depends on the settings. + +NB: Processing of errors and warnings requires some improvement. Now, parts of the code handle errors per file, others do so globally. + + + +\section{Encodings} + +The preferred encoding is Unicode UTF-8. From version 2.3 of \TeXcount{}, this is used internally to represent the \LaTeX{} code, and Unicode is relied upon to handle different character sets and classes. + +When files are read into \TeXcount{}, they may have to be decoded from whatever encoding they are saved in into UTF-8. The file encoding may be specified explicitly using the \code{-enc=} option, otherwise \TeXcount{} will try to guess the appropriate encoding. + +The output from \TeXcount{} is be default UTF-8. However, if a file encoding is specified using \code{-enc=} and output is text, not HTML, this encoding will also be applied to the output. This may be useful when using \TeXcount{} in a pipe, otherwise the documents will be converted to UTF-8. + + + +\section{Help routines and text data} + +A hash, \code{\%GLOBALDATA}, and hash reference \code{\$STRINGDATA} are is defined for storing strings used for various outputs. The \code{\%GLOBALDATA} is set up containing string constants for version number, maintainer name, etc., while \code{\$STRINGDATA} is initially undefined. + +The \code{\$STRINGDATA} hash is accessed through calls to \code{StringData} which initialises the hash if undefined. Initialisation, which is done by \code{STRINGDATA}, reads through the \code{__DATA__} section at the end of the script, identifies headers which are used as keys in the hash which maps to and array containing the subsequent text lines. References in the read text on the form \code{\$\{keyword\}} are replaced by the corresponding string in \code{\%GLOBALDATA}: this allows e.g. version information to be inserted into the text. + +Headers in the text data consists of three or more colons followed by space(s) and a keyword. Lines containing three or more colons but no keyword have no effect. + +Lines starting with \code{\@} are used to format output printed by \code{wprintlines}. The two characters \code{'-'} and \code{':'} can then be used to indicate indentation tabulators, and subsequent lines will be indented and wrapped: this is used for printing help on command line options. The \code{wprintlines} also wraps text: the page with is set by \code{\$Text::Wrap::columns}. + + +\end{document} diff --git a/tools/macros.tex b/tools/macros.tex new file mode 100755 index 0000000..a79e8f0 --- /dev/null +++ b/tools/macros.tex @@ -0,0 +1,86 @@ +%% LaTeX macros +\usepackage{relsize} +\usepackage[dvipsnames]{xcolor} + +% Define underscore to be a regular character in text mode +\begingroup + \catcode`\_=\active + \gdef_#1{\ensuremath{\sb{#1}}} +\endgroup +\mathcode`\_=\string"8000 +\catcode`\_=12 + +% Set version number +\newcommand\version{$3.0$} + +\newcommand\copyrightfootnote{ +\footnotetext{Copyright (2008-2013) of Einar Andreas R{\o}dland, distributed +under the \LaTeX{} Project Public License (LPPL).} +} + +%TC:macroword \TeXcount 1 +\newcommand\TeXcount{{\TeX}count} + +% Text formats +\newcommand\codestyle[1]{\textsf{\color{Blue}#1}} +\newcommand\code[1]{{\smaller\codestyle{#1}}} +\newcommand\bigcode[1]{\codestyle{#1}} +\newcommand\codeline[1]{\begin{quote}\code{#1}\end{quote}} +\newcommand\bs[1]{\textbackslash#1} +\newcommand\URL[1]{\textsf{\small #1}} + +% Description items: options, parameters, optional parameters +% These are also interpreted by dos2html.pl +\def\option[#1]{\item[\bigcode{#1}]\hskip 0pt plus 10pt} +\def\parm#1{\textit{\color{OliveGreen}#1}} +\def\opt#1{\parm{[#1]}} +\def\alt#1{[#1]} +\def\optiontext#1{\textrm{\bfseries\color{black}#1}} + +% Mark off notification in contents for good visibility +\newcommand\ContentsNote[1]{\addtocontents{toc}{\string\marginpar{\textsf{\color{red}\tiny #1}}}} + +% Notabene: margin note +%TC:macro NB 1 +\newcommand\NB[1]{\ContentsNote{NB}\marginpar{\textsf{\tiny#1}}} + +% Mark text as a notification +%TC:macro NOTE [text] +\newcommand\NOTE[1]{\textit{\color{red}#1}} + +% Mark text for update +%TC:macro UPDATE [text] +\newcommand\UPDATE[1]{\ContentsNote{UPDATE}\textit{\textbf{\color{red}This needs to be updated:} {\color{blue}#1}}} + +% Mark text as a notification +%TC:macro TODO [text] +\newcommand\TODO[1]{\ContentsNote{TODO}\textit{\textbf{\color{red}#1}}} + +% BUG +%TC:macro BUG [text] +\newcommand\BUG[1]{{\color{red}#1}} + +\makeatletter + +\renewcommand\@maketitle{% +\newpage\null\vskip 2em% +\begin{center}% +\let\footnote\thanks +{\LARGE \@title \par}% +\end{center}% +\par +\vskip 1.5em +} + +\renewcommand\abstractname{Abstract} +\renewenvironment{abstract}{% + \begin{center}% + {\slshape\bfseries\large\abstractname\vspace{-.5em}\vspace{\z@}}% + \end{center}% + \vskip 4pt + \slshape +}{ +\vskip 0.5em +} + +\makeatother diff --git a/tools/sub_addrules.tex b/tools/sub_addrules.tex new file mode 100755 index 0000000..fb1d3ad --- /dev/null +++ b/tools/sub_addrules.tex @@ -0,0 +1,29 @@ +% Subsection: TC-commands for adding macro rules + +\begin{description} +\sloppy + +\option[macro \parm{macroname} \parm{parameter-rules}] +Defines macro handling rule for the specified macro. The parameter is on the form \code{[\textit{rule},\ldots]} where each rule is either a keyword indicating the parsing rule for a macro parameter or \code{option:\textit{rule}} for an optional \code{[]}-enclosed parameter. Alternatively, an integer value $n$ indicates that the $n$ first parameters to the macro should be ignored, equivalent to giving a list of $n$ \code{ignore} rules. + +\option[envir \parm{envirname} \parm{parameter-rules} \parm{content-rule}] +(The previously used command, \code{group}, remains an alias for \code{envir}, but the name \code{envir} is more appropriate and therefore recommended.) +This specifies the handling of environments with the given name. The parameter handling rule, applied to parameters following \code{\bs{begin}\{\textit{name}\}}, are specified as in the \code{macro} instruction. The second parameter specifies the rule, i.e. parser state, with which the contents should be parsed. + +\option[macrocount \parm{macroname} \opt{count-spec.}] +(An alias for \code{macrocount} is \code{macroword}; the preferred name was changed to reflect that this can count any element, not just words.) +If a number is provided as the count parameter, this defines the given macro to be counted as the specified number of words; if no count is specified, it is assumed to be 1. Alternatively, a \code{[]}-enclosed list of counters can be specified (using the counter keywords), causing each of them to be incremented: counter are \code{word}/\code{text}, \code{headerword}, \code{otherword}, \code{header}, \code{float}, \code{inlinemath}, \code{displaymath} plus a number of aliases. + +\option[breakmacro \parm{macroname}] +Specify that the given macro should cause a break point. + +\option[floatinclude \parm{macroname} \parm{parameter-rules}] +Specify macro handling rules used within float groups. The handling rules are specified as for \code{macro}. Most commonly, the parameter rule will be the \code{otherword}/\code{oword} to specify that words should be counted as \textit{other words}. + +\option[preambleinclude \parm{macroname} \parm{parameter-rules}] +Specifies macro handling rules to be used in the preamble: the text between \code{\bs{documentclass}} and \code{\bs{begin}\{document\}}. The rule is specified like the \code{macro} rules. + +\option[fileinclude \parm{macroname} \parm{file-path-spec.}] +Specifies macros that cause files to be included when \TeXcount{} is run with the \code{-inc} option. The parameters specify the format on which the file path is specified, and can also be used to modify the search path used within the included document. + +\end{description} diff --git a/tools/sub_options.tex b/tools/sub_options.tex new file mode 100755 index 0000000..8ed0b1a --- /dev/null +++ b/tools/sub_options.tex @@ -0,0 +1,143 @@ +% Subsection: command line options + +\begin{description} +\sloppy +\def\option[#1]{\item[\quad\code{#1}]\hskip 0pt plus 10pt} +\def\alt#1{[#1]} +\def\alts#1{[[#1]]} + +\option[-v]Verbose (same as \code{-v3}). + +\option[-v0]No details (default). + +\option[-v1]Prints counted text, marks formulae. + +\option[-v2]Also prints ignored text. + +\option[-v3]Also includes comments and options. + +\option[-v4]Same as \code{-v3 -showstate}. + +\option[-v=\ldots, -v\alts{0-4}\ldots]Allows detailed control of which elements are included in the verbose output. The provided values is a list of styles or style categories separated by \code{+} or \code{-} to indicate if they should be added or removed from the list of included styles. Style categories start with capital letter and include \code{Words}, \code{Macros}, \code{Options}; the individual styles are in all lower case and include \code{word}, \code{hword}, \code{option}, \code{ignore}. + +\option[-showstate]Show internal states (with verbose). + +\option[-brief]Only prints a one line summary of the counts for each file. + +\option[-q, -quiet]Quiet mode, does not print error messages. Use is discouraged, but it may be useful when piping the output into another application. + +\option[-strict]Prints a warning of begin-end groups for which no specific rule is defined. + +\option[-total]Only give total sum, no per file sums. + +\option[-1]Same as specifying \code{-brief} and \code{-total}, and ensures there will only be one line of output. If used with \code{-sum}, the output will only be the total number. + +\option[-0]Same as \code{-1}, i.e. \code{-brief} and \code{-total}, but does not put a line shift at the end. This may be useful when the one line output is to be used by another application, e.g. Emacs, for which the line shift would otherwise need to be stripped away. + +\option[-template="\ldots"]Specify an output template which is used to generate the summary output for each file and for the total count. Codes \code{\{\textit{label}\}} is used to include values, where \code{\textit{label}} is one of \code{0} to \code{7} (for the counts), \code{SUM}, \code{ERROR} or \code{TITLE} (first character of label is sufficient). Conditional inclusion is done using \code{\{\textit{label}?\textit{text}?\textit{label}\}} or \code{\{\textit{label}?\textit{if non-zero}|\textit{if zero}?\textit{label}\}}. If the count contains at least two subcounts, use \code{\{SUB|\textit{template}|SUB\}} with a separate template for the subcounts, or \code{\{SUB?\textit{prefix}|\textit{template}|\textit{suffix}?SUB\}}. + +\option[-sub\alt{=\ldots}, -subcount\alt{=\ldots}]Generate subcounts. Valid option values are \code{none}, \code{part}, \code{chapter}, \code{section} and \code{subsection} (default), indicating at which level subcounts are generated. (On by default.) + +\option[-nosub]Do not generate subcounts. + +\option[-sum\alt{=n,n,\ldots}]Produces total sum, default being all words and formulae, but customisable to any weighted sum of the seven counts (list of weights for text words, header words, caption words, headers, floats, inlined formulae, displayed formulae). + +\option[-nosum]Do not generate total sum. (Default choice.) + +\option[-col]Use ANSI colour codes in verbose output. This requires ANSI colours which is used on Linux, but may not be available under Windows. On by default on non-Windows systems. + +\option[-nc, -nocol]No colours (colours require ANSI). Default under Windows. + +\option[-nosep, -noseparator]No separating character/string added after each word in the verbose output (default). + +\option[-sep=, -separator=]Separating character or string to be added after each word in the verbose output. + +\option[-relaxed]Relaxes the rules for matching words and macro options. + +\option[-restricted]Restricts the rules for matching words and macro options. + +\option[-]Read \LaTeX{} code from STDIN. + +\option[-inc]Parse included files (as separate files). + +\option[-merge]Merge included files into document (in place). + +\option[-noinc]Do not parse or merge in included files (default). + +\option[-incbib]Include bibliography in count, include bbl file if needed. + +\option[-nobib]Do not include bibliography in count (default). + +\option[-incpackage=]Include rules for a given package. + +\option[-dir\alt{=\ldots}]Specify working directory which will serve as root for all include files. The default (\code{-dir=.}) is to use the current directory, i.e. from which \TeXcount{} is executed: the path can be absolute or relative to the current directory. Use \code{-dir} to use the path of the main \LaTeX{} document as working directory. + +\option[-auxdir\alt{=\ldots}]Specify the directory of the auxiliary files, e.g. the bibliography (\code{.bbl}) file. The default setting (\code{-auxdir} only) indicates that auxiliary files are in the working directory (from the \code{-dir} or \code{-dir=} option). If \code{-auxdir=} is used to provide a path and \code{-dir=} is used to specify the working directory, the path to the auxiliary directory is take to be relative to the current folder (from which \TeXcount{} is executed); if used with \code{-dir}, the working directory is taken from the path of the parsed file, and the auxiliary directory is taken to be relative to this (unless an absolute path is provided). + +\option[-enc=, -encoding=]Specify encoding to use in input (and text output). + +\option[-utf8, -unicode]Use UTF-8 (Unicode) encoding. Same as \code{-encoding=utf8}. + +\option[-alpha=, -alphabets=]List of Unicode character groups (or digit, alphabetic) permitted as letters. Names are separated by \code{,} or \code{+}. If list starts with \code{+}, the alphabets will be added to those already included. The default is Digit+alphabetic. + +\option[-logo=, -logograms=]List of Unicode character groups interpreted as whole word characters, e.g. Han for Chinese characters. Names are separated by \code{,} or \code{+}. If list starts with \code{+}, the alphabets will be added to those already included. By default, this is set to include Ideographic, Katakana, Hiragana, Thai and Lao. + +\option[-ch, -chinese, -zhongwen]Turn on Chinese mode in which Chinese characters are counted. I recommend using UTF-8, although \TeXcount{} will also test other encodings (GB2312, Big5, Hz) if UTF-8 fails, and other encodings may be specified by \code{-encoding=}. + +\option[-jp, -japanese]Turn on Japanese mode in which Japanese characters (Kanji and Kana) are counted. I recommend using UTF-8, although \TeXcount{} will also test other encodings (e.g. EUC-JP) if UTF-8 fails, and other encodings may be specified by \code{-encoding=}. + +\option[-kr, -korean]Turn on Korean mode in which Korean characters (Hangul and Han) are counted. I recommend using UTF-8, although \TeXcount{} will also test other encodings (e.g. EUC-KR) if UTF-8 fails, and other encodings may be specified by \code{-encoding=}. + +\option[-kr-words, -korean-words]Korean mode in which Hangul words are counted (i.e. as words separated by spaces) rather than characters. Han characters are still counted as characters. See also \code{-korean}. + +\option[-chinese-only, ..., -korean-words-only]As options \code{-chinese}, ..., \code{-korean-words}, but also excludes other alphabets (e.g. letter-based words) and logographic characters. + +\option[-char, -letter]Count letters instead of words. This count does not include spaces. + +\option[-out=]Send output to file. Takes file name as value. + +\option[-html]Output in HTML format. + +\option[-htmlcore]Only HTML body contents. + +\option[-htmlfile=]File containing a template HTML document with \code{<!-- TeXcount -->} included somewhere to indicate the location where the TeXcount output from the parsing should be inserted. + +\option[-tex]Encode \TeX{} special characters for output into \TeX{} code. + +\option[-css=]Reference to CSS to be included in the HTML output instead of including the style definition directly in the output. + +\option[-cssfile=, -css=file:]File containing style definitions to be included into the HTML output instead of the default styles. + +\option[-freq\alt{=\#}]Count individual word frequencies. Optionally, give minimal frequency required to be included in output. + +\option[-stat]Produce statistics on language usage, i.e. based on the alphabets and logograms included. + +\option[-macrostat, -macrofreq]Produce statistics on package, environment and macro usage. + +\option[-codes]Display an overview of the colour codes. Can be used as a separate option to only display the colour codes, or together with files to parse. + +\option[-nocodes]Do not display overview of colour codes. + +\option[-opt=, -optionfile=]Reads options (command line parameters) from a specified text file. Should use one option per line. May also include TC options in the same format as specified in \LaTeX{} documents, but prefixed by \code{\%} rather than \code{\%TC:}. Blank lines and lines starting with \code{\#} are ignored; lines starting with \code{\bs{}} are considered to be continuations of the previous line. + +\option[-split, -nosplit]The \code{-split} option, which is on by default, speeds up handling of large files by splitting the file into paragraphs. To turn it off, use the \code{-nosplit} option. + +\option[-showver, -nover]Include version number in output with \code{-showver}; use \code{-nover} not to show it (default). + +\option[-h, -?, --help, /?]Help. + +\option[-h=, -?=, --help=, /?=]Help on particular macro or group name: gives the parsing rule for that macro or group if defined. If the the macro or environment is package specific, use \code{-h=\parm{package}:\parm{name}}; replace \code{\parm{package}} with \code{class\%\parm{name}} if it is specific to a document class. + +\option[-help-options, -h-opt]Lists all TeXcount options and help on them. + +\option[--help-option=, -h-opt=]Lists all TeXcount options containing the provided string: e.g. \code{-h-opt=inc} lists all options containing \code{inc}, while \code{-h-opt=-v} lists all options starting with \code{v}. + +\option[-help-style, -h-style]Lists all styles and style categories, i.e. those permitted used in -v={styles-list}. + +\option[-help-style=, -h-style=]Gives description of style or style category. + +\option[-ver, --version]Print version number. + +\option[-lic, --license]License information. + +\end{description} diff --git a/tools/sub_tc_other.tex b/tools/sub_tc_other.tex new file mode 100755 index 0000000..dd6c05f --- /dev/null +++ b/tools/sub_tc_other.tex @@ -0,0 +1,28 @@ +%% Other TC instructions + +\begin{description} + +\option[break \parm{title}] +Break point which initiates a new subcount. The title is used to identify the following region in the summary output. + +\option[incbib \optiontext{or} includebibliography] +Sets bibliography inclusion, same as running \TeXcount{} with the option \code{-incbib}. + +\option[subst \parm{macro} \parm{text}] +This substitutes a macro with any text. The verbose output will show the substituted text: e.g. \code{\%TC:subst \bs{test} TEST} will cause a following \code{\bs{newcommand}\bs{test}\{TEST\}} to be changed into \code{\bs{newcommand} TEST\{TEST\}}, which \TeXcount{} will interpret differently. Use with care! + +\option[ignore] +Indicates start of a region to be ignored. End region with \code{\%TC:endignore}. + +\option[insert \parm{\TeX-code}] +Insert \TeX{} code for \TeXcount{} to process. + +\option[newcounter \parm{name} \opt{description}] +Define a new counter with the given name and description (optional). A corresponding parsing rule will also be added with the same name. + +\option[newtemplate \optiontext{and} template \opt{template-line}] +Specify a template for the summary output. The first line should just declare a new template using \code{\%TC:newtemplate}, while the subsequent lines use \code{\%TC:template} followed by text specifying the template. The line breaks in the template specification are not of importance: to specify a line break, use \code{\bs{n}}. + +\end{description} + + diff --git a/tools/texcount.pl b/tools/texcount.pl new file mode 100755 index 0000000..5a53b3b --- /dev/null +++ b/tools/texcount.pl @@ -0,0 +1,3798 @@ +#! /usr/bin/env perl +use strict; +use warnings; +use utf8; # Because the script itself is UTF-8 encoded +use Encode; +use Text::Wrap; +use Term::ANSIColor; + +BEGIN { + if ($^O=~/^MSWin/) { + require Win32::Console::ANSI; + Win32::Console::ANSI::->import(); + } +} + +##### Version information + +my $versionnumber="3.0"; +my $versiondate="2013 Jul 29"; + +###### Set global settings and variables + +### Global data about TeXcount +my %GLOBALDATA= + ('versionnumber' => $versionnumber + ,'versiondate' => $versiondate + ,'maintainer' => 'Einar Andreas Rodland' + ,'copyrightyears' => '2008-2013' + ,'website' => 'http://app.uio.no/ifi/texcount/' + ); + +### Options and states + +# Outer object (for error reports not added by a TeX object) +my $Main=getMain(); + +# Global options and settings +my $htmlstyle=0; # Flag to print HTML +my $texcodeoutput=0; # Flag to convert output to valid TeX text +my $encoding=undef; # Selected input encoding (default will be guess) +my @encodingGuessOrder=qw/ascii utf8 latin1/; # Encoding guessing order +my $outputEncoding; # Encoding used for output +my @AlphabetScripts=qw/Digit Is_alphabetic/; # Letters minus logograms: defined later +my @LogogramScripts=qw/Ideographic Katakana Hiragana Thai Lao Hangul/; # Scripts counted as whole words + +# Parsing rules options +my $includeTeX=0; # Flag to parse included files +my $includeBibliography=0; # Flag to include bibliography +my %substitutions; # Substitutions to make globally +my %IncludedPackages; # List of included packages + +# Counting options +my @sumweights; # Set count weights for computing sum +my $optionWordFreq=0; # Count words of this frequency, or don't count if 0 +my $optionWordClassFreq=0; # Count words per word class (language) if set +my $optionMacroStat=0; # Count macro, environment and package usage + +# Parsing details options +my $strictness=0; # Flag to check for undefined environments +my $defaultVerbosity='0'; # Specification of default verbose output style +my $defaultprintlevel=0; # Flag indicating default level of verbose output +my $printlevel=undef; # Flag indicating level of verbose output +my $showstates=0; # Flag to show internal state in verbose log +my $showcodes=1; # Flag to show overview of colour codes (2=force) +my $showsubcounts=0; # Write subcounts if #>this, or not (if 0) +my $separatorstyleregex='^word'; # Styles (regex) after which separator should be added +my $separator=''; # Separator to add after words/tokens + +# Final summary output options +my $showVersion=0; # Indicator that version info be included (1) or not (-1) +my $totalflag=0; # Flag to write only total summary +my $briefsum=0; # Flag to set brief summary +my $outputtemplate; # Output template +my $finalLineBreak=1; # Add line break at end + +# Global settings +my $optionFast=1; # Flag inticating fast method + +# Global variables and internal states (for internal use only) +my $blankline=0; # Number of blank lines printed +my $errorcount=0; # Number of errors in parsing +my %warnings=(); # Warnings +my %WordFreq; # Hash for counting words +my %MacroUsage; # Hash for counting macros, environments and packages + +# External sources +my $HTMLfile; # HTML file to use as HTML output template +my $CSSfile; # CSS file to use with HTML output +my $CSShref; # CSS reference to use with HTML output +my @htmlhead; # Lines to add to the HTML header +my $htmlopen; # text used to open the HTML file +my $htmlclose; # text used to close the HTML file + +# String data storage +my $STRINGDATA; + +# Other constants +my $_PARAM_='<param>'; # to identify parameter to _parse_unit +my $STRING_PARAMETER='{_}'; # used to log macro parameters +my $STRING_OPTIONAL_PARAM='[_]'; # used to log optional parameter +my $STRING_GOBBLED_OPTION='[]'; # used to log gobbled macro option +my $STRING_ERROR='<error>'; # used to log errors causing parsing to stop +my $REGEX_NUMBER=qr/^\d+$/; # regex to recognize a number + +###### Set CMD specific settings and variables + +## Preset command line options +# List of options (stings) separated by comma, +# e.g. ('-inc','-v') to parse included files and +# give verbose output by default. +my @StartupOptions=(); + +# CMD specific global variables +my @filelist; # List of files to parse +my $globalworkdir='./'; # Overrules workdir (default=present root) +my $workdir; # Current directory (default=taken from filename) +my $auxdir; # Directory for auxilary files, e.g. bbl (default=$workdir) +my $fileFromSTDIN=0; # Flag to set input from STDIN +my $_STDIN_='<STDIN>'; # File name to represent STDIN (must be '<...>'!) + +# CMD specific settings +$Text::Wrap::columns=76; # Page width for wrapped output + +###### Set state identifiers and methods + + +### Counter indices from 0 to $SIZE_CNT-1 +# 0: Number of files +# 1: Text words +# 2: Header words +# 3: Caption words +# 4: Number of headers +# 5: Number of floating environments +# 6: Number of inlined math +# 7: Number of displayed math +my $SIZE_CNT=8; +my $SIZE_CNT_DEFAULT=8; +my $CNT_FILE=0; +my $CNT_WORDS_TEXT=1; +my $CNT_WORDS_HEADER=2; +my $CNT_WORDS_OTHER=3; +my $CNT_COUNT_HEADER=4; +my $CNT_COUNT_FLOAT=5; +my $CNT_COUNT_INLINEMATH=6; +my $CNT_COUNT_DISPLAYMATH=7; + +# Labels used to describe the counts +my @countkey=('file','word','hword','oword','header','float','inmath','dsmath'); +my @countdesc=('Files','Words in text','Words in headers', + 'Words outside text (captions, etc.)','Number of headers','Number of floats/tables/figures', + 'Number of math inlines','Number of math displayed'); + +# Map keywords to counters +my %key2cnt; +add_keys_to_hash(\%key2cnt,$CNT_FILE,0,'file'); +add_keys_to_hash(\%key2cnt,$CNT_WORDS_TEXT,1,'text','word','w','wd'); +add_keys_to_hash(\%key2cnt,$CNT_WORDS_HEADER,2,'headerword','hword','hw','hwd'); +add_keys_to_hash(\%key2cnt,$CNT_WORDS_OTHER,3,'otherword','other','oword','ow','owd'); +add_keys_to_hash(\%key2cnt,$CNT_COUNT_HEADER,4,'header','heading','head'); +add_keys_to_hash(\%key2cnt,$CNT_COUNT_FLOAT,5,'float','table','figure'); +add_keys_to_hash(\%key2cnt,$CNT_COUNT_INLINEMATH,6,'inline','inlinemath','imath','eq'); +add_keys_to_hash(\%key2cnt,$CNT_COUNT_DISPLAYMATH,7,'displaymath','dsmath','dmath','ds'); + + +### Token types +# Set in $tex->{'type'} by call to _next_token +my $TOKEN_SPACE=-1; +my $TOKEN_COMMENT=0; +my $TOKEN_WORD=1; # word (or other form of text or text component) +my $TOKEN_SYMBOL=2; # symbol (not word, e.g. punctuation) +my $TOKEN_MACRO=3; # macro (\name) +my $TOKEN_BRACE=4; # curly braces: { } +my $TOKEN_BRACKET=5; # brackets: [ ] +my $TOKEN_MATH=6; +my $TOKEN_LINEBREAK=9; # line break in file +my $TOKEN_TC=666; # TeXcount instructions (%TC:instr) +my $TOKEN_END=999; # end of line or blank line + + +### Parsing states +# +## States for regions that should not be counted +# IGNORE = exclude from count +# FLOAT = float (exclude, but include captions) +# EXCLUDE_STRONG = strong exclude, ignore environments +# EXCLUDE_STRONGER = stronger exclude, do not parse macro parameters +# EXCLUDE_ALL = ignore everything except end marker: even { +# PREAMBLE = preamble (between \documentclass and \begin{document}) +## States for regions in which words should be counted +# TEXT = text +# TEXT_HEADER = header text +# TEXT_FLOAT = float text +## State change: not used in parsing, but to switch state then ignore contents +# TO_INLINEMATH = switch to inlined math +# TO_DISPLAYMATH = switch to displayed math +## Other states +# _NULL = default state to use if none other is defined +# _OPTION = state used to indicate that the next parameter is an option +# _EXCLUDE_ = cutoff, state <= this represents excluded text +## NB: Presently, it is assumed that additional states is added as 8,9,..., +## e.g. that states added through TC:newcounter correspond to the added counters. +# +my $STATE_IGNORE=-1; +my $STATE_MATH=-2; +my $STATE_FLOAT=-10; +my $STATE_EXCLUDE_STRONG=-20; +my $STATE_EXCLUDE_STRONGER=-30; +my $STATE_EXCLUDE_ALL=-40; +my $STATE_PREAMBLE=-99; +my $STATE_TEXT=1; +my $STATE_TEXT_HEADER=2; +my $STATE_TEXT_FLOAT=3; +my $STATE_TO_HEADER=4; +my $STATE_TO_FLOAT=5; +my $STATE_TO_INLINEMATH=6; +my $STATE_TO_DISPLAYMATH=7; +my $__STATE_EXCLUDE_=-10; +my $__STATE_NULL=1; +my $_STATE_OPTION=-1000; +my $_STATE_NOOPTION=-1001; +my $_STATE_AUTOOPTION=-1002; + +# Counter key mapped to STATE +my $PREFIX_PARAM_OPTION=' '; # Prefix for parameter options/modifiers +my %key2state; +add_keys_to_hash(\%key2state,$STATE_TEXT,1,'text','word','w','wd'); +add_keys_to_hash(\%key2state,$STATE_TEXT_HEADER,2,'headertext','headerword','hword','hw','hwd'); +add_keys_to_hash(\%key2state,$STATE_TEXT_FLOAT,3,'otherword','other','oword','ow','owd'); +add_keys_to_hash(\%key2state,$STATE_TO_HEADER,4,'header','heading','head'); +add_keys_to_hash(\%key2state,$STATE_TO_FLOAT,5,'float','table','figure'); +add_keys_to_hash(\%key2state,$STATE_TO_INLINEMATH,6,'inline','inlinemath','imath','eq'); +add_keys_to_hash(\%key2state,$STATE_TO_DISPLAYMATH,7,'displaymath','dsmath','dmath','ds'); +add_keys_to_hash(\%key2state,$STATE_IGNORE,0,'ignore','x'); +add_keys_to_hash(\%key2state,$STATE_MATH,'ismath'); +add_keys_to_hash(\%key2state,$STATE_FLOAT,-1,'isfloat'); +add_keys_to_hash(\%key2state,$STATE_EXCLUDE_STRONG,-2,'xx'); +add_keys_to_hash(\%key2state,$STATE_EXCLUDE_STRONGER,-3,'xxx'); +add_keys_to_hash(\%key2state,$STATE_EXCLUDE_ALL,-4,'xall'); +add_keys_to_hash(\%key2state,$_STATE_OPTION,'[',' option',' opt',' optional'); +add_keys_to_hash(\%key2state,$_STATE_NOOPTION,'nooption','nooptions','noopt','noopts'); +add_keys_to_hash(\%key2state,$_STATE_AUTOOPTION,'autooption','autooptions','autoopt','autoopts'); + +# When combining two states, use the first one; list must be complete! +my @STATE_FIRST_PRIORITY=( + $STATE_EXCLUDE_ALL, + $STATE_EXCLUDE_STRONGER, + $STATE_EXCLUDE_STRONG, + $STATE_FLOAT, + $STATE_MATH, + $STATE_IGNORE, + $STATE_PREAMBLE, + $STATE_TO_FLOAT, + $STATE_TO_HEADER, + $STATE_TO_INLINEMATH, + $STATE_TO_DISPLAYMATH); +my @STATE_MID_PRIORITY=(); +my @STATE_LAST_PRIORITY=( + $STATE_TEXT_FLOAT, + $STATE_TEXT_HEADER, + $STATE_TEXT); + +# Map state to corresponding word counter +my %state2cnt=( + $STATE_TEXT => $CNT_WORDS_TEXT, + $STATE_TEXT_HEADER => $CNT_WORDS_HEADER, + $STATE_TEXT_FLOAT => $CNT_WORDS_OTHER); + +# Transition state mapped to content state and counter +my %transition2state=( + $STATE_TO_HEADER => [$STATE_TEXT_HEADER,$CNT_COUNT_HEADER], + $STATE_TO_INLINEMATH => [$STATE_MATH ,$CNT_COUNT_INLINEMATH], + $STATE_TO_DISPLAYMATH => [$STATE_MATH ,$CNT_COUNT_DISPLAYMATH], + $STATE_TO_FLOAT => [$STATE_FLOAT ,$CNT_COUNT_FLOAT]); + +# Parsing state descriptions (used for macro rule help) +my %state2desc=( + $STATE_IGNORE => 'ignore: do not count', + $STATE_MATH => 'math/equation contents', + $STATE_FLOAT => 'float (figure, etc.): ignore all but special macros', + $STATE_EXCLUDE_STRONG => 'strong exclude: ignore environments', + $STATE_EXCLUDE_STRONGER => 'stronger exclude: ignore environments and macro paramters', + $STATE_EXCLUDE_ALL => 'exlude all: even {, only scan for end marker', + $STATE_PREAMBLE => 'preamble: from \documentclass to \begin{document}', + $STATE_TEXT => 'text: count words', + $STATE_TEXT_HEADER => 'header text: count words as header words', + $STATE_TEXT_FLOAT => 'float text: count words as float words (e.g. captions)', + $STATE_TO_HEADER => 'header: count header, then count words as header words', + $STATE_TO_FLOAT => 'float: count float, then count words as float/other words', + $STATE_TO_INLINEMATH => 'inline math: count as inline math/equation', + $STATE_TO_DISPLAYMATH => 'displayed math: count as displayed math/equation'); + +# Parsing state presentation style +my %state2style=( + $STATE_TEXT => 'word', + $STATE_TEXT_HEADER => 'hword', + $STATE_TEXT_FLOAT => 'oword', + ); + +# State: is a text state..."include state" is more correct +sub state_is_text { + my $st=shift @_; + return ($st>=$STATE_TEXT); +} + +# State: is a parsed/included region, text or preamble +sub state_is_parsed { + my $st=shift @_; + return ($st>=$STATE_TEXT || $st==$STATE_PREAMBLE); +} + +# State: get CNT corresponding to text state (or undef) +sub state_text_cnt { + my $st=shift @_; + return $state2cnt{$st}; +} + +# State: is an exclude state +sub state_is_exclude { + my $st=shift @_; + return ($st<=$__STATE_EXCLUDE_); +} + +# State: \begin and \end should be processed +sub state_inc_envir { + my $st=shift @_; + return ($st>$STATE_EXCLUDE_STRONG); +} + +# State as text (used with printstate) +# TODO: Should do a conversion based on STATE values. +sub state_to_text { + my $st=shift @_; + return $st; +} + +# Style to use with text state +sub state_to_style { + return $state2style{shift @_}; +} + +# Add new counter with the given key and description +sub add_new_counter { + my ($key,$desc,$like)=@_; + my $state=$SIZE_CNT; + my $cnt=$SIZE_CNT; + $key=lc($key); + if (!defined $like){$like=$CNT_WORDS_OTHER;} + $key2cnt{$key}=$cnt; + push @countkey,$key; + push @countdesc,$desc; + if (defined $sumweights[$like]) {$sumweights[$cnt]=$sumweights[$like];} + $key2state{$key}=$state; + $state2cnt{$state}=$cnt; + $state2style{$state}='altwd'; + push @STATE_MID_PRIORITY,$state; + $SIZE_CNT++; +} + +###### Set global definitions + + +### Break points +# Definition of macros that define break points that start a new subcount. +# The values given are used as labels. +my %BreakPointsOptions; +$BreakPointsOptions{'none'}={}; +$BreakPointsOptions{'part'}={%{$BreakPointsOptions{'none'}},'\part'=>'Part'}; +$BreakPointsOptions{'chapter'}={%{$BreakPointsOptions{'part'}},'\chapter'=>'Chapter'}; +$BreakPointsOptions{'section'}={%{$BreakPointsOptions{'chapter'}},'\section'=>'Section'}; +$BreakPointsOptions{'subsection'}={%{$BreakPointsOptions{'section'}},'\subsection'=>'Subsection'}; +$BreakPointsOptions{'default'}=$BreakPointsOptions{'subsection'}; +my %BreakPoints=%{$BreakPointsOptions{'none'}}; + +### Print styles +# Definition of different print styles: maps of class labels +# to ANSI codes. Class labels are as used by HTML styles. +my %STYLES; +my $STYLE_EMPTY=' '; +my $STYLE_BLOCK='-'; +my $NOSTYLE=' '; +$STYLES{'Errors'}={'error'=>'bold red'}; +$STYLES{'Words'}={'word'=>'blue','hword'=>'bold blue','oword'=>'blue','altwd'=>'blue'}; +$STYLES{'Macros'}={'cmd'=>'green','fileinc'=>'bold green'}; +$STYLES{'Options'}={'option'=>'yellow','optparm'=>'green'}; +$STYLES{'Ignored'}={'ignore'=>'cyan','math'=>'magenta'}; +$STYLES{'Excluded'}={'exclcmd'=>'yellow','exclenv'=>'yellow','exclmath'=>'yellow','mathcmd'=>'yellow'}; +$STYLES{'Groups'}={'document'=>'red','envir'=>'red','mathgroup'=>'magenta'}; +$STYLES{'Comments'}={'tc'=>'bold yellow','comment'=>'yellow'}; +$STYLES{'Sums'}={'cumsum'=>'yellow'}; +$STYLES{'States'}={'state'=>'cyan underline'}; +$STYLES{'<core>'}={%{$STYLES{'Errors'}},$STYLE_EMPTY=>$NOSTYLE,'<printlevel>'=>1}; +$STYLES{0}={%{$STYLES{'Errors'}},'<printlevel>'=>0}; +$STYLES{1}={%{$STYLES{'<core>'}},%{$STYLES{'Words'}},%{$STYLES{'Groups'}},%{$STYLES{'Sums'}}}; +$STYLES{2}={%{$STYLES{1}},%{$STYLES{'Macros'}},%{$STYLES{'Ignored'}},%{$STYLES{'Excluded'}}}; +$STYLES{3}={%{$STYLES{2}},%{$STYLES{'Options'}},%{$STYLES{'Comments'}},'<printlevel>'=>2}; +$STYLES{4}={%{$STYLES{3}},%{$STYLES{'States'}}}; +$STYLES{'All'}=$STYLES{4}; +my %STYLE=%{$STYLES{$defaultVerbosity}}; + +my @STYLE_LIST=('error','word','hword','oword','altwd', + 'ignore','document','cmd','exclcmd','option','optparm','envir','exclenv', + 'mathgroup','exclmath','math','mathcmd','comment','tc','fileinc','state','cumsum'); +my %STYLE_DESC=( + 'error' => 'ERROR: TeXcount error message', + 'word' => 'Text which is counted: counted as text words', + 'hword' => 'Header and title text: counted as header words', + 'oword' => 'Caption text and footnotes: counted as caption words', + 'altwd' => 'Words in user specified counters: counted in separate counters', + 'ignore' => 'Ignored text or code: excluded or ignored', + 'document' => '\documentclass: document start, beginning of preamble', + 'cmd' => '\macro: macro not counted, but parameters may be', + 'exclcmd' => '\macro: macro in excluded region', + 'option' => '[Macro options]: not counted', + 'optparm' => '[Optional parameter]: content parsed and styled as counted', + 'envir' => '\begin{name} \end{name}: environment', + 'exclenv' => '\begin{name} \end{name}: environment in excluded region', + 'mathgroup' => '$ $: counted as one equation', + 'exclmath' => '$ $: equation in excluded region', + 'math' => '2+2=4: maths (inside $...$ etc.)', + 'mathcmd' => '$\macro$: macros inside maths', + 'comment' => '% Comments: not counted', + 'tc' => '%TC:TeXcount instructions: not counted', + 'fileinc' => 'File to include: not counted but file may be counted later', + 'state' => '[state]: internal TeXcount state', + 'cumsum' => '[cumsum]: cumulative sum count'); + +###### Define what a word is and language options + + +# Patters matching a letter. Should be a single character or +# ()-enclosed regex for substitution into word pattern regex. +my @LetterMacros=qw/ae AE o O aa AA oe OE ss + alpha beta gamma delta epsilon zeta eta theta iota kappa lamda + mu nu xi pi rho sigma tau upsilon phi chi psi omega + Gamma Delta Theta Lambda Xi Pi Sigma Upsilon Phi Psi Omega + /; +my $specialchars='\\\\('.join('|',@LetterMacros).')(\{\}|\s*|\b)'; +my $modifiedchars='\\\\[\'\"\`\~\^\=](@|\{@\})'; +my %NamedLetterPattern; +$NamedLetterPattern{'restricted'}='@'; +$NamedLetterPattern{'default'}='('.join('|','@',$modifiedchars,$specialchars).')'; +$NamedLetterPattern{'relaxed'}=$NamedLetterPattern{'default'}; +my $LetterPattern=$NamedLetterPattern{'default'}; + +# List of regexp patterns that should be analysed as words. +# Use @ to represent a letter, will be substituted with $LetterPattern. +# Named patterns may replace or be appended to the original patterns. +# Apply_Options() results in a call to apply_language_options() which +# constructs $WordPattern based on $LetterPattern, @WordPatterns and +# alphabet/logogram settings. +my %NamedWordPattern; +$NamedWordPattern{'letters'}='@'; +$NamedWordPattern{'words'}='(@+|@+\{@+\}|\{@+\}@+)([\-\'\.]?(@+|\{@+\}))*'; +my @WordPatterns=($NamedWordPattern{'words'}); +my $WordPattern; # Regex matching a word (defined in apply_language_options()) + +### Macro option regexp list +# List of regexp patterns to be gobbled as macro option in and after +# a macro. +my %NamedMacroOptionPattern; +$NamedMacroOptionPattern{'default'}='\[[^\[\]\n]*\]'; +$NamedMacroOptionPattern{'relaxed'}='\[[^\[\]\n]*(\n[^\[\]\n]+)\n?\]'; +$NamedMacroOptionPattern{'restricted'}='\[(\w|[,\-\s\~\.\:\;\+\?\*\_\=])*\]'; +my $MacroOptionPattern=$NamedMacroOptionPattern{'default'}; + +### Alternative language encodings +my %NamedEncodingGuessOrder; +$NamedEncodingGuessOrder{'chinese'}=[qw/utf8 gb2312 big5/]; +$NamedEncodingGuessOrder{'japanese'}=[qw/utf8 euc-jp iso-2022-jp jis shiftjis/]; +$NamedEncodingGuessOrder{'korean'}=[qw/utf8 euc-kr iso-2022-kr/]; + + +###### Define character classes (alphabets) + + +### Character classes to use as Unicode properties + +# Character group representing digits 0-9 (more restrictive than Digits) +sub Is_digit { return <<END; +0030\t0039 +END +} + +# Character group representing letters (excluding logograms) +sub Is_alphabetic { return <<END; ++utf8::Alphabetic +-utf8::Ideographic +-utf8::Katakana +-utf8::Hiragana +-utf8::Thai +-utf8::Lao +-utf8::Hangul +END +} + +# Character group representing letters (excluding logograms) +sub Is_alphanumeric { return <<END; ++utf8::Alphabetic ++utf8::Digit +-utf8::Ideographic +-utf8::Katakana +-utf8::Hiragana +-utf8::Thai +-utf8::Lao +-utf8::Hangul +END +} + +# Character class for punctuation excluding special characters +sub Is_punctuation { return <<END; ++utf8::Punctuation +-0024\t0025 +-005c +-007b\007e +END +} + +# Character group representing CJK characters +sub Is_cjk { return <<END; ++utf8::Han ++utf8::Katakana ++utf8::Hiragana ++utf8::Hangul +END +} + +# Character group for CJK punctuation characters +sub Is_cjkpunctuation { return <<END; +3000\t303f +2018\t201f +ff01\tff0f +ff1a\tff1f +ff3b\tff3f +ff5b\tff65 +END +} + +###### Define core rules + +### Macros indicating package inclusion +# Will always be assumed to take one parameter (plus options). +# Gets added to TeXmacro. After that, values are not used, only membership. +my %TeXpackageinc=('\usepackage'=>1,'\RequirePackage'=>1); + +### Macros that are counted within the preamble +# The preamble is the text between \documentclass and \begin{document}. +# Text and macros in the preamble is ignored unless specified here. The +# value is the states (1=text, 2=header, etc.) they should be interpreted as. +# Note that only the first unit (token or {...} block) is counted. +# Gets added to TeXmacro. Is used within preambles only. +my %TeXpreamble; +add_keys_to_hash(\%TeXpreamble,['header'],'\title'); +add_keys_to_hash(\%TeXpreamble,['other'],'\thanks'); +add_keys_to_hash(\%TeXpreamble,['xxx','xxx'],'\newcommand','\renewcommand'); +add_keys_to_hash(\%TeXpreamble,['xxx','xxx','xxx'],'\newenvironment','\renewenvironment'); + +### In floats: include only specific macros +# Macros used to identify caption text within floats. +# Gets added to TeXmacro. Is used within floats only. +my %TeXfloatinc=('\caption'=>['otherword']); + +### How many tokens to gobble after macro +# Each macro is assumed to gobble up a given number of +# tokens (or {...} groups), as well as options [...] before, within +# and after. The %TeXmacro hash gives a link from a macro +# (or beginNAME for environment with no the backslash) +# to either an integer giving the number of tokens to ignore +# or to an array (specified as [rule,rule,...]) of length N where +# N is the number of parameters to be read with the macro. The +# array values tell how each is to be interpreted (see the parser state +# keywords for valid values). Thus specifying a number N is +# equivalent to specifying an array of N 'ignore' rules. +# +# For macros not specified here, the default value is 0: i.e. +# no tokens are excluded, but [...] options are. +my %TeXmacro=(%TeXpreamble,%TeXfloatinc,%TeXpackageinc); +add_keys_to_hash(\%TeXmacro,['text'], + '\textnormal','\textrm','\textit','\textbf','\textsf','\texttt','\textsc','\textsl','\textup','\textmd', + '\makebox','\mbox','\framebox','\fbox','\uppercase','\lowercase','\textsuperscript','\textsubscript', + '\citetext'); +add_keys_to_hash(\%TeXmacro,['[','text'], + '\item'); +add_keys_to_hash(\%TeXmacro,['[','ignore'], + '\linebreak','\nolinebreak','\pagebreak','\nopagebreak'); +add_keys_to_hash(\%TeXmacro,0, + '\maketitle','\indent','\noindent', + '\centering','\raggedright','\raggedleft','\clearpage','\cleardoublepage','\newline','\newpage', + '\smallskip','\medskip','\bigskip','\vfill','\hfill','\hrulefill','\dotfill', + '\normalsize','\small','\footnotesize','\scriptsize','\tiny','\large','\Large','\LARGE','\huge','\Huge', + '\normalfont','\em','\rm','\it','\bf','\sf','\tt','\sc','\sl', + '\rmfamily','\sffamily','\ttfamily','\upshape','\itshape','\slshape','\scshape','\mdseries','\bfseries', + '\selectfont', + '\tableofcontents','\listoftables','\listoffigures'); +add_keys_to_hash(\%TeXmacro,1, + '\begin','\end', + '\documentclass','\documentstyle','\hyphenation','\pagestyle','\thispagestyle', + '\author','\date', + '\bibliographystyle','\bibliography','\pagenumbering','\markright', + '\includeonly','\includegraphics','\special', + '\label','\ref','\pageref','\bibitem', + '\eqlabel','\eqref','\hspace','\vspace','\addvspace', + '\newsavebox','\usebox', + '\newlength','\newcounter','\stepcounter','\refstepcounter','\usecounter', + '\fontfamily','\fontseries', + '\alph','\arabic','\fnsymbol','\roman','\value', + '\typeout', '\typein','\cline'); +add_keys_to_hash(\%TeXmacro,2, + '\newfont','\newtheorem','\sbox','\savebox','\rule','\markboth', + '\setlength','\addtolength','\settodepth','\settoheight','\settowidth','\setcounter', + '\addtocontents','\addtocounter', + '\fontsize'); +add_keys_to_hash(\%TeXmacro,3,'\multicolumn','\addcontentsline'); +add_keys_to_hash(\%TeXmacro,6,'\DeclareFontShape'); +add_keys_to_hash(\%TeXmacro,['[','text','ignore'], + '\cite','\nocite','\citep','\citet','\citeauthor','\citeyear','\citeyearpar', + '\citealp','\citealt','\Citep','\Citet','\Citealp','\Citealt','\Citeauthor'); +add_keys_to_hash(\%TeXmacro,['ignore','text'],'\parbox','\raisebox'); +add_keys_to_hash(\%TeXmacro,['otherword'],'\marginpar','\footnote','\footnotetext'); +add_keys_to_hash(\%TeXmacro,['header'], + '\title','\part','\chapter','\section','\subsection','\subsubsection','\paragraph','\subparagraph'); +add_keys_to_hash(\%TeXmacro,['xxx','xxx','text'],'\multicolumn'); +add_keys_to_hash(\%TeXmacro,['xxx','xxx'],'\newcommand','\renewcommand'); +add_keys_to_hash(\%TeXmacro,['xxx','xxx','xxx'],'\newenvironment','\renewenvironment'); + +### Environments +# The %TeXenvir hash provides content parsing rules (parser states). +# Environments that are not defined will be counted as the surrounding text. +# +# Parameters taken by the \begin{environment} are defined in %TeXmacro. +# +# Note that some environments may only exist within math-mode, and +# therefore need not be defined here: in fact, they should not as it +# is not clear if they will be in inlined or displayed math. +my %TeXenvir; +add_keys_to_hash(\%TeXenvir,'ignore', + 'titlepage','tabbing','tabular','tabular*','thebibliography','lrbox'); +add_keys_to_hash(\%TeXenvir,'text', + 'document','letter','center','flushleft','flushright', + 'abstract','quote','quotation','verse','minipage', + 'description','enumerate','itemize','list', + 'theorem','thm','lemma','definition','corollary','example','proof','pf'); +add_keys_to_hash(\%TeXenvir,'inlinemath', + 'math'); +add_keys_to_hash(\%TeXenvir,'displaymath', + 'displaymath','equation','equation*','eqnarray','eqnarray*','align','align*',); +add_keys_to_hash(\%TeXenvir,'float', + 'float','picture','figure','figure*','table','table*'); +add_keys_to_hash(\%TeXenvir,'xall', + 'verbatim','tikzpicture'); + +# Environment parameters +my $PREFIX_ENVIR='begin'; # Prefix used for environment names +add_keys_to_hash(\%TeXmacro,1, + 'beginthebibliography','beginlrbox','beginminipage'); +add_keys_to_hash(\%TeXmacro,2, + 'beginlist'); +add_keys_to_hash(\%TeXmacro,['ignore'], + 'beginletter'); +add_keys_to_hash(\%TeXmacro,['xxx'], + 'begintabular'); +add_keys_to_hash(\%TeXmacro,['ignore','xxx'], + 'begintabular*'); +add_keys_to_hash(\%TeXmacro,['[','text'], + 'begintheorem','beginthm','beginlemma','begindefinition','begincorollary','beginexample','beginproof','beginpf'); +add_keys_to_hash(\%TeXmacro,['nooptions'], + 'beginverbatim'); + +### Macros that should be counted as one or more words +# Macros that represent text may be declared here. The value gives +# the number of words the macro represents. +my %TeXmacrocount=('\LaTeX'=>1,'\TeX'=>1,'beginabstract'=>['header','headerword']); + +### Macros for including tex files +# Allows \macro{file} or \macro file. If the value is 0, the filename will +# be used as is; if it is 1, the filetype .tex will be added if the +# filename is without filetype; if it is 2, the filetype .tex will be added. +my %TeXfileinclude=('\input'=>'input','\include'=>'texfile'); + +### Convert state keys to codes +convert_hash(\%TeXpreamble,\&keyarray_to_state); +convert_hash(\%TeXfloatinc,\&keyarray_to_state); +convert_hash(\%TeXmacro,\&keyarray_to_state); +convert_hash(\%TeXmacrocount,\&keyarray_to_cnt); +convert_hash(\%TeXenvir,\&key_to_state); + +###### Define package specific rules + +### Package rule definitions + +my %PackageTeXpreamble=(); # TeXpreamble definitions per package +my %PackageTeXpackageinc=(); # TeXpackageinc definitions per package +my %PackageTeXfloatinc=(); # TeXfloatinc definitions per package +my %PackageTeXmacro=(); # TeXmacro definitions per package +my %PackageTeXmacrocount=(); # TeXmacrocount definitions per package +my %PackageTeXenvir=(); # TeXenvir definitions per package +my %PackageTeXfileinclude=(); # TeXfileinclude definitions per package +my %PackageSubpackage=(); # Subpackages to include (listed in array [...]) + + +# Rules for bibliography inclusion +$PackageTeXmacrocount{'%incbib'}={'beginthebibliography'=>['header','hword']}; +$PackageTeXmacro{'%incbib'}={'\bibliography'=>1}; +$PackageTeXenvir{'%incbib'}={'thebibliography'=>'text'}; +$PackageTeXfileinclude{'%incbib'}={'\bibliography'=>'<bbl>'}; + +# Rules for package alltt +$PackageTeXenvir{'alltt'}={ + 'alltt'=>'xall'}; + +# Rules for package babel +# NB: Only core macros implemented, those expected found in regular documents +$PackageTeXenvir{'babel'}={ + 'otherlanguage'=>'text','otherlanguage*'=>'text'}; +$PackageTeXmacro{'babel'}={ + '\selectlanguage'=>1,'\foreignlanguage'=>['ignore','text'], + 'beginotherlanguage'=>1,'beginotherlanguage*'=>1}; + +# Rules for package comment +$PackageTeXenvir{'comment'}={ + 'comment'=>'xxx'}; + +# Rules for package color +$PackageTeXmacro{'color'}={ + '\textcolor'=>['ignore','text'],'\color'=>1,'\pagecolor'=>1,'\normalcolor'=>0, + '\colorbox'=>['ignore','text'],'\fcolorbox'=>['ignore','ignore','text'], + '\definecolor'=>3,\'DefineNamedColor'=>4}; + +# Rules for package endnotes +$PackageTeXmacro{'endnotes'}={'\endnote'=>['oword'],'\endnotetext'=>['oword'],'\addtoendnotetext'=>['oword']}; + +# Rules for package fancyhdr +$PackageTeXmacro{'fancyhdr'}={ + '\fancyhf'=>1,'\lhead'=>1,'\chead'=>1,'\rhead'=>1,'\lfoot'=>1,'\cfoot'=>1,'\rfoot'=>1}; + +# Rules for package geometry +$PackageTeXmacro{'geometry'}={ + '\geometry'=>1,'\newgeometry'=>1,'\restoregeometry'=>0,,'\savegeometry'=>1,'\loadgeometry'=>1}; + +# Rules for package graphicx +$PackageTeXmacro{'graphicx'}={ + '\DeclareGraphicsExtensions'=>1,'\graphicspath'=>1, + '\includegraphics'=>['[','ignore','ignore'], + '\includegraphics*'=>['[','ignore','[','ignore','[','ignore','ignore'], + '\rotatebox'=>1,'\scalebox'=>1,'\reflectbox'=>1,'\resizebox'=>1}; + +# Rules for package hyperref (urls themselves counted as one word) +# NB: \hyperref[label]{text} preferred over \hyperref{url}{category}{name}{text} +# NB: Macros for use in Form environments not implemented +$PackageTeXmacro{'hyperref'}={ + '\hyperref'=>['[','ignore','text'], + '\url'=>1,'\nolinkurl'=>1,'\href'=>['ignore','text'], + '\hyperlink'=>['ignore','text'],'\hypertarget'=>['ignore','text'], + '\hyperbaseurl'=>1,'\hyperimage'=>['ignore','text'],'\hyperdef'=>['ignore','ignore','text'], + '\phantomsection'=>0,'\autoref'=>1,'\autopageref'=>1, + '\hypersetup'=>1,'\urlstyle'=>1, + '\pdfbookmark'=>2,'\currentpdfbookmark'=>2,'\subpdfbookmark'=>2,'\belowpdfbookmark'=>2, + '\pdfstringref'=>2,'\texorpdfstring'=>['text','ignore'], + '\hypercalcbp'=>1,'\Acrobatmenu'=>2}; +$PackageTeXmacrocount{'hyperref'}={ + '\url'=>1,'\nolinkurl'=>1}; + +# Rules for package import +$PackageTeXfileinclude{'import'}={ + '\import'=>'dir file','\subimport'=>'subdir file', + '\inputfrom'=>'dir file','\subinputfrom'=>'subdir file', + '\includefrom'=>'dir file','\subincludefrom'=>'subdir file'}; + +# Rules for package inputenc +$PackageTeXmacro{'inputenc'}={ + '\inputencoding'=>1}; + +# Rules for package listings +$PackageTeXenvir{'listings'}={'lstlisting'=>'xall'}; +$PackageTeXmacro{'listings'}={'\lstset'=>['ignore'],'\lstinputlisting'=>['ignore']}; + +# Rules for package psfig +$PackageTeXmacro{'psfig'}={'\psfig'=>1}; + +# Rules for package sectsty +$PackageTeXmacro{'sectsty'}={ + '\allsectionsfont'=>1,'\partfont'=>1,'\chapterfont'=>1,'\sectionfont'=>1, + '\subsectionfont'=>1,'\subsubsectionfont'=>1,'\paragraphfont'=>1,'\subparagraphfont'=>1, + '\minisecfont'=>1,'\partnumberfont'=>1,'\parttitlefont'=>1,'\chapternumberfont'=>1, + '\chaptertitlefont'=>1,'\nohang'=>0}; + +# Rules for package setspace +$PackageTeXenvir{'setspace'}={ + 'singlespace'=>'text','singlespace*'=>'text','onehalfspace'=>'text','doublespace'=>'text', + 'spacing'=>'text'}; +$PackageTeXmacro{'setspace'}={ + 'beginspacing'=>1, + '\singlespacing'=>0,'\onehalfspacing'=>0,'\doublespacing'=>0, + '\setstretch'=>1,'\SetSinglespace'=>1}; + +# Rules for package subfiles +$PackageTeXfileinclude{'subfiles'}={ + '\subfile'=>'file'}; + +# Rules for package url +# NB: \url|...| variant not implemented, only \url{...} +# NB: \urldef{macro}{url} will not be counted +$PackageTeXmacro{'url'}={ + '\url'=>1,'\urldef'=>2,'\urlstyle'=>1,'\DeclareUrlCommand'=>['ignore','xxx']}; +$PackageTeXmacro{'setspace'}={ + '\url'=>1}; + +# Rules for package wrapfig +$PackageTeXenvir{'wrapfig'}={ + 'wrapfigure'=>'float','wraptable'=>'float'}; +$PackageTeXmacro{'wrapfig'}={ + 'beginwrapfigure'=>2,'beginwraptable'=>2}; + +# Rules for package xcolor (reimplements the color package) +# NB: only main macros (mostly from package color) included +$PackageTeXmacro{'xcolor'}={ + '\textcolor'=>['ignore','text'],'\color'=>1,'\pagecolor'=>1,'\normalcolor'=>0, + '\colorbox'=>['ignore','text'],'\fcolorbox'=>['ignore','ignore','text'], + '\definecolor'=>3,\'DefineNamedColor'=>4, + '\colorlet'=>2}; + + +###### Main script + + +################################################### + +MAIN(@ARGV); +exit; # Just to make sure it ends here... + +################################################### + + +######### +######### Main routines +######### + +# MAIN ROUTINE: Handle arguments, then parse files +sub MAIN { + my @args; + push @args,@StartupOptions; + push @args,@_; + Initialise(); + Check_Arguments(@args); + my @toplevelfiles=Parse_Arguments(@args); + Apply_Options(); + if (scalar @toplevelfiles>0 || $fileFromSTDIN) { + if ($showVersion && !$htmlstyle && !($briefsum && $totalflag)) { + print "\n=== LaTeX word count (TeXcount version $versionnumber) ===\n\n"; + } + conditional_print_style_list(); + my $totalcount=Parse_file_list(@toplevelfiles); + conditional_print_total($totalcount); + Report_Errors(); + if ($optionWordFreq || $optionWordClassFreq) {print_word_freq();} + if ($optionMacroStat) {print_macro_stat();} + } elsif ($showcodes>1) { + conditional_print_style_list(); + } else { + error($Main,'No files specified.'); + } + Close_Output(); +} + +# Initialise, overrule initial settings, etc. +sub Initialise { + _option_subcount(); + # Windows settings + if ($^O eq 'MSWin32') { + } elsif ($^O=~/^MSWin/) { + #DELETE: do not overrule colour setting + option_ansi_colours(0); + } +} + +# Check arguments, exit on exit condition +sub Check_Arguments { + my @args=@_; + my $arg=$args[0]; + if (!@args) { + print_version(); + print_short_help(); + exit; + } elsif ($arg=~/^(--?(h|\?|help)|\/(\?|h))$/) { + print_help(); + exit; + } elsif ($arg=~/^(--?(h|\?|help)|\/(\?|h))=(.*)$/) { + print_help_on_rule($4); + exit; + } elsif ($arg=~/^(--?(h|\?|help)|\/(\?|h))-?(opt|options?)$/) { + print_syntax(); + exit; + } elsif ($arg=~/^(--?(h|\?|help)|\/(\?|h))-?(opt|options?)=(.*)$/) { + print_syntax_subset($5); + exit; + } elsif ($arg=~/^(--?(h|\?|help)|\/(\?|h))-?(styles?)$/) { + print_help_on_styles(); + exit; + } elsif ($arg=~/^(--?(h|\?|help)|\/(\?|h))-?(styles?)=(\w+)$/) { + print_help_on_styles($5); + exit; + } elsif ($arg=~/^--?(ver|version)$/) { + print_version(); + exit; + } elsif ($arg=~/^--?(lic|license|licence)$/) { + print_license(); + exit; + } + return 1; +} + +# Parse arguments, set options (global) and return file list +sub Parse_Arguments { + my @args=@_; + my @files; + foreach my $arg (@args) { + if (parse_option($arg)) {next;} + if ($arg=~/^\-/) { + print "Invalid opton $arg \n\n"; + print_short_help(); + exit; + } + $arg=~s/\\/\//g; + push @files,$arg; + } + return @files; +} + +# Parse individual option parameters +sub parse_option { + my $arg=shift @_; + return parse_options_preset($arg) + || parse_options_parsing($arg) + || parse_options_counts($arg) + || parse_options_output($arg) + || parse_options_format($arg) + ; +} + +# Parse presetting options +sub parse_options_preset { + my $arg=shift @_; + if ($arg=~/^-(opt|option|options|optionfile)=(.*)$/) { + _parse_optionfile($2); + } + else {return 0;} + return 1; +} + +# Parse parsing options +sub parse_options_parsing { + my $arg=shift @_; + if ($arg eq '-') {$fileFromSTDIN=1;} + elsif ($arg eq '-merge') {$includeTeX=2;} + elsif ($arg eq '-inc') {$includeTeX=1;} + elsif ($arg eq '-noinc') {$includeTeX=0;} + elsif ($arg =~/^-(includepackage|incpackage|package|pack)=(.*)$/) {include_package($2);} + elsif ($arg eq '-incbib') {$includeBibliography=1;} + elsif ($arg eq '-nobib') {$includeBibliography=0;} + elsif ($arg eq '-dir') {$globalworkdir=undef;} + elsif ($arg=~/^-dir=(.*)$/) { + $globalworkdir=$1; + $globalworkdir=~s:([^\/\\])$:$1\/:; + } + elsif ($arg eq '-auxdir') {$auxdir=undef;} + elsif ($arg=~/^-auxdir=(.*)$/) { + $auxdir=$1; + $auxdir=~s:([^\/\\])$:$1\/:; + } + elsif ($arg =~/^-(enc|encode|encoding)=(.+)$/) {$encoding=$2;} + elsif ($arg =~/^-(utf8|unicode)$/) {$encoding='utf8';} + elsif ($arg =~/^-(alpha(bets?)?)=(.*)$/) {set_script_options(\@AlphabetScripts,$3);} + elsif ($arg =~/^-(logo(grams?)?)=(.*)$/) {set_script_options(\@LogogramScripts,$3);} + elsif ($arg =~/^-([-a-z]+)$/ && set_language_option($1)) {} + elsif ($arg eq '-relaxed') { + $MacroOptionPattern=$NamedMacroOptionPattern{'relaxed'}; + $LetterPattern=$NamedLetterPattern{'relaxed'}; + } + elsif ($arg eq '-restricted') { + $MacroOptionPattern=$NamedMacroOptionPattern{'restricted'}; + $LetterPattern=$NamedLetterPattern{'restricted'}; + } + else {return 0;} + return 1; +} + +# Parse count and summation options +sub parse_options_counts { + my $arg=shift @_; + if ($arg =~/^-sum(=(.+))?$/) {_option_sum($2);} + elsif ($arg =~/^-nosum/) {@sumweights=();} + elsif ($arg =~/^-(sub|subcounts?)(=(.+))?$/) {_option_subcount($3);} + elsif ($arg =~/^-(nosub|nosubcounts?)/) {$showsubcounts=0;} + elsif ($arg eq '-freq') {$optionWordFreq=1;} + elsif ($arg =~/^-freq=(\d+)$/) {$optionWordFreq=$1;} + elsif ($arg eq '-stat') {$optionWordClassFreq=1;} + elsif ($arg =~/^-macro-?(stat|freq)$/) {$optionMacroStat=1;} + else {return 0;} + return 1; +} + +# Apply sum option +sub _option_sum { + my $arg=shift @_; + if (!defined $arg) { + @sumweights=(0,1,1,1,0,0,1,1); + } elsif ($arg=~/^(\d+(\.\d*)?(,\d+(\.\d*)?){0,6})$/) { + @sumweights=(0,split(',',$1)); + } else { + print STDERR "Warning: Option value $arg not valid, ignoring option.\n"; + } +} + +# Apply subcount options +sub _option_subcount { + my $arg=shift @_; + $showsubcounts=2; + if (!defined $arg) { + %BreakPoints=%{$BreakPointsOptions{'default'}}; + } elsif (my $option=$BreakPointsOptions{$arg}) { + %BreakPoints=%{$option}; + } else { + print STDERR "Warning: Option value $arg not valid, using default instead.\n"; + %BreakPoints=%{$BreakPointsOptions{'default'}}; + } +} + +# Parse output and verbosity options +sub parse_options_output { + my $arg=shift @_; + if ($arg eq '-strict') {$strictness=1;} + elsif ($arg eq '-v') {set_verbosity_options('3');} + elsif ($arg =~/^-v([0-4=+\-].*)/) {set_verbosity_options($1);} + elsif ($arg =~/^-showstates?$/ ) {$showstates=1;} + elsif ($arg =~/^-(q|quiet)$/ ) {$printlevel=-1;} + elsif ($arg =~/^-(template)=(.*)$/ ) {_set_output_template($2);} + elsif ($arg eq '-split') {$optionFast=1;} + elsif ($arg eq '-nosplit') {$optionFast=0;} + elsif ($arg eq '-showver') {$showVersion=1;} + elsif ($arg eq '-nover') {$showVersion=-1;} + elsif ($arg =~/^-nosep(arator)?s?$/ ) {$separator='';} + elsif ($arg =~/^-sep(arator)?s?=(.*)$/ ) {$separator=$2;} + elsif ($arg =~/^-out=(.*)/ ) {close STDOUT; open STDOUT,'>',$1;} + else {return 0;} + return 1; +} + +# Set output template +sub _set_output_template { + my $template=shift @_; + $outputtemplate=$template; + if ($template=~/\{(SUM)[\?\}]/i && !@sumweights) { + @sumweights=(0,1,1,1,0,0,1,1); + } + if ($template=~/\{SUB\?/i && !$showsubcounts) { + _option_subcount(); + } +} + +# Parse output formating options +sub parse_options_format { + my $arg=shift @_; + if ($arg eq '-brief') {$briefsum=1;} + elsif ($arg eq '-total') {$totalflag=1;} + elsif ($arg eq '-0') {$briefsum=1;$totalflag=1;$printlevel=-1;$finalLineBreak=0;} + elsif ($arg eq '-1') {$briefsum=1;$totalflag=1;$printlevel=-1;} + elsif ($arg eq '-htmlcore' ) {option_ansi_colours(0);$htmlstyle=1;} + elsif ($arg eq '-html' ) {option_ansi_colours(0);$htmlstyle=2;} + elsif ($arg eq '-tex' ) {option_ansi_colours(0);$texcodeoutput=1;} + elsif ($arg =~/^\-(nocol|nc$)/) {option_ansi_colours(0);} + elsif ($arg =~/^\-(col)$/) {option_ansi_colours(1);} + elsif ($arg eq '-codes') {$showcodes=2;} + elsif ($arg eq '-nocodes') {$showcodes=0;} + elsif ($arg =~/^-htmlfile=(.+)$/) {$HTMLfile=$1;} + elsif ($arg =~/^-cssfile=(.+)$/) {$CSSfile=$1;} + elsif ($arg =~/^-css=file:(.+)$/) {$CSSfile=$1;} + elsif ($arg =~/^-css=(.+)$/) {$CSShref=$1;} + else {return 0;} + return 1; +} + +# Include options from option file +sub _parse_optionfile { + my $filename=shift @_; + open(FH,'<',$filename) + || die "Option file not found: $filename\n"; + my @options=<FH>; + close(FH); + s/^\s*(#.*|)//s for @options; + my $text=join('',@options); + $text=~s/(\n|\r|\r\n)\s*\\//g; + @options=split("\n",$text); + foreach my $arg (@options) { + __optionfile_tc($arg) + || parse_option($arg) + || die "Invalid option $arg in $filename.\n"; + } +} + +# Parse option file TC options +sub __optionfile_tc { + my $arg=shift @_; + $arg=~s/^\%\s*// || return 0; + if ($arg=~/^subst\s+(\\\w+)\s+(.*)$/i) { + $substitutions{$1}=$2; + } elsif ($arg=~/^(\w+)\s+([\\]*\w+)\s+([^\s\n]+)(\s+([0-9]+))?/i) { + tc_macro_param_option($Main,$1,$2,$3,$5) || die "Invalid TC option: $arg\n"; + } else { + print "Invalid TC option format: $arg\n"; + return 0; + } + return 1; +} + +# Parse file list and return total count +sub Parse_file_list { + my @files=@_; + my $listtotalcount=new_count('Total'); + foreach (@files) {s/\\/\//g; s/ /\\ /g;} + if (@files) { + @files=<@files>; # For the sake of Windows: expand wildcards! + for my $file (@files) { + my $filetotalcount=parse_file($file); + add_to_total($listtotalcount,$filetotalcount); + } + } + if ($fileFromSTDIN) { + my $filetotalcount=parse_file($_STDIN_); + add_to_total($listtotalcount,$filetotalcount); + } + return $listtotalcount; +} + +# Parse file and included files, and return total count +sub parse_file { + my $file=shift @_; + if (defined ($workdir=$globalworkdir)) {} + elsif ($file eq $_STDIN_) {$workdir='';} + else { + $workdir=$file; + $workdir =~ s/^((.*[\\\/])?)[^\\\/]+$/$1/; + } + if ($htmlstyle && ($printlevel>0 || !$totalflag)) {print "\n<div class='filegroup'>\n";} + my $filetotalcount=new_count('File(s) total: '.$file); + @filelist=(); + _add_file($filetotalcount,[$file,$workdir],'File: '.$file); + foreach my $f (@filelist) { + _add_file($filetotalcount,$f,'Included file: '.$f->[0]); + } + if (!$totalflag && get_count($filetotalcount,$CNT_FILE)>1) { + if ($htmlstyle) {formatprint("Sum of files: $file\n",'h2');} + print_count($filetotalcount,'sumcount'); + } + if ($htmlstyle && ($printlevel>0 || !$totalflag)) {print "</div>\n\n";} + return $filetotalcount; +} + +# Parse single file, included files will be appended to @filelist. +sub _add_file { + my ($filetotalcount,$pathfile,$title)=@_; + my @paths=@$pathfile; + my $file=shift @paths; + my $tex=TeXfile($file,$title); + if (!defined $tex) { + error($Main,'File not found or not readable: '.$file); + return; + } + $tex->{'PATH'}=\@paths; + parse($tex); + my $filecount=next_subcount($tex); + if (!$totalflag) {print_count($filecount);} + add_to_total($filetotalcount,$filecount); +} + +###### +###### Subroutines +###### + +###### CMD specific implementations + + +# Add file to list of files scheduled for parsing +sub include_file { + my ($tex,$state,$file,$refparam)=@_; + my %params=%$refparam; + my $type=$params{'<type>'}; + my @paths=@{$tex->{'PATH'}}; + my $filepath; + if ($type eq '<bbl>' && defined $auxdir) { + $auxdir=~s/([\\\/])*$/\//; + if (defined $globalworkdir) {@paths=[$auxdir];} + elsif ($auxdir=~/^(\w:)?[\\\/]/) {@paths=[$auxdir];} + else {@paths=[$workdir.$auxdir]} + } + foreach my $key (split(/\s+/,$type)) { + my $value=$params{$key}; + if ($key eq 'dir') {unshift @paths,$workdir.$value;} + elsif ($key eq 'subdir') {unshift @paths,$paths[0].$value;} + } + if ($file=~/^(\w:)?[\\\/]/) {$filepath=$file;} + else { + $filepath=_find_file_in_path($tex,$file,@paths) || BLOCK { + error($tex,'File '.$file.' not found in path ['.join(';',@paths).'].'); + return; + }; + } + if ($includeTeX==2) { + my $bincode=read_binary($filepath) || BLOCK { + error($tex,"File $filepath not readable."); + return; + }; + flush_next($tex); + line_return(0,$tex); + # DELETE: prepend_code($tex,$bincode,$file); + my $texstate=texcode_insert_text($tex,$bincode,$file,@paths); + parse_all($tex,$state); + texcode_restore_state($tex,$texstate); + } else { + push @filelist,[$filepath,@paths]; + } +} + +# Seach path list to find file +sub _find_file_in_path { + my $tex=shift @_; + my $file=shift @_; + foreach my $path (@_) { + if (!$path=~/[\\\/]$/) {$path.='/';} + my $filepath=$path.$file; + if (-e $filepath) {return $filepath;} + elsif ($filepath=~/\.tex$/i) {} + elsif (-e $filepath.'.tex') {return $filepath.'.tex';} + } + return undef; +} + +# Print count (total) if conditions are met +sub conditional_print_total { + my $sumcount=shift @_; + if ($totalflag || number_of_subcounts($sumcount)>1) { + if ($totalflag && $briefsum && @sumweights) { + print get_sum_count($sumcount),"\n"; + } else { + if ($htmlstyle) {formatprint('Total word count','h2');} + print_count($sumcount,'sumcount'); + } + } +} + +# Set or unset use of ANSI colours +sub option_ansi_colours { + my $flag=shift @_; + $ENV{'ANSI_COLORS_DISABLED'} = $flag?undef:1; +} + +# Print text using ANSI colours +sub ansiprint { + my ($text,$colour)=@_; + print Term::ANSIColor::colored($text,$colour); +} + +###### Option handling + + +# Apply options to set values +sub Apply_Options { + apply_encoding_options(); + if ($htmlstyle>1) {html_head();} + flush_errorbuffer($Main); + apply_language_options(); + if ($includeBibliography) {apply_include_bibliography();} + if ($showcodes>1 && !($STYLE{'<printlevel>'})) {%STYLE=%{$STYLES{'All'}};} + if ($showstates) {set_verbosity_options('+States');} + if (!@sumweights) {set_verbosity_options('-Sums');} + (defined $printlevel) + || (defined ($printlevel=$STYLE{'<printlevel>'})) + || ($printlevel=$defaultprintlevel); +} + +# Set or add styles included in the verbose output +sub set_verbosity_options { + my $verb=shift @_; + my $st; + if ($defaultprintlevel<1) {$defaultprintlevel=1;} + if ( $verb=~s/^([0-4])// || $verb=~s/^=([0-4])?// ) { + my $key=$1; + if (!defined $key) {%STYLE=(%{$STYLES{'<core>'}});} + elsif ($st=$STYLES{$key}) {%STYLE=(%$st);} + else { + error($Main,"Unknown verbosity option '$key'."); + %STYLE=(%{$STYLES{'<core>'}}); + } + } + while ($verb=~s/^([\+\-]?)(\w+)//) { + my $add=!($1 eq '-'); + my $key=$2; + if ($st=$STYLES{$key}) { + if ($add) {@STYLE{keys %$st}=values %$st;} + else {delete @STYLE{keys %$st};} + } elsif ($st=$STYLES{'All'}->{$key}) { + if ($add) {$STYLE{$key}=$st;} + elsif ($STYLE{$key}) {delete $STYLE{$key};} + } else { + error($Main,"Unknown verbosity option '$key'."); + } + } + if ($verb ne '') { + error($Main,"Unknown verbosity option format: $verb"); + } +} + +# Set or add scripts to array of scripts +sub set_script_options { + my ($scriptset,$str)=@_; + if ($str=~s/^\+//) {} else {splice(@$scriptset,0,scalar $scriptset);} + my @scripts=split(/[+,]/,$str); + foreach my $scr (@scripts) { + $scr=~tr/_/ /; + if ($scr eq 'Alphabetic') { + warning($Main,'Using alphabetic instead of Unicode class Alphabetic'); + $scr='alphabetic'; + } + if ($scr=~/^[a-z]/) {$scr='Is_'.$scr;} + if (_is_property_valid($scr)) {push @$scriptset,$scr;} + else {error($Main,"Unknown script $scr ignored.");} + } +} + +sub _is_property_valid { + my $script=shift @_; + if (! $script=~/^\w+$/) {return 0;} + eval {' '=~/\p{$script}/}; + if ($@) {return 0;} else {return 1;} +} + +# Set language option, return language if applied, undef if not +sub set_language_option { + my $language=shift @_; + if ($language=~/^(count\-?)all$/) { + @AlphabetScripts=qw/Digit Is_alphabetic/; + @LogogramScripts=qw/Ideographic Katakana Hiragana Thai Lao Hangul/; + } elsif ($language=~/^words(-?only)?$/) { + @LogogramScripts=(); + } elsif ($language=~/^(ch|chinese|zhongwen)(-?only)?$/) { + @LogogramScripts=qw/Han/; + if (defined $2) {$LetterPattern=undef;} + @encodingGuessOrder=@{$NamedEncodingGuessOrder{'chinese'}}; + return 'chinese'; + } elsif ($language=~/^(jp|japanese)(-?only)?$/) { + @LogogramScripts=qw/Han Hiragana Katakana/; + if (defined $2) {$LetterPattern=undef;} + @encodingGuessOrder=@{$NamedEncodingGuessOrder{'japanese'}}; + return 'japanese'; + } elsif ($language=~/^(kr|korean)(-?only)?$/) { + @LogogramScripts=qw/Han Hangul/; + if (defined $2) {$LetterPattern=undef;} + @encodingGuessOrder=@{$NamedEncodingGuessOrder{'korean'}}; + return 'korean'; + } elsif ($language=~/^(kr|korean)-?words?(-?only)?$/) { + if (defined $2) { + @AlphabetScripts=qw/Hangul/; + @LogogramScripts=qw/Han/; + } else { + @AlphabetScripts=qw/Digit Is_alphabetic Hangul/; + @LogogramScripts=qw/Han Katakana Hiragana Thai Lao/; + } + @encodingGuessOrder=@{$NamedEncodingGuessOrder{'korean'}}; + return 'korean-words'; + } elsif ($language=~/^(char|character|letter)s?(-?only)?$/) { + @WordPatterns=($NamedWordPattern{'letters'}); + if (defined $2) {@LogogramScripts=();} + $countdesc[1]='Letters in text'; + $countdesc[2]='Letters in headers'; + $countdesc[3]='Letters in captions'; + return 'letter'; + } else { + return undef; + } +} + +# Set encoding based on encoding options, guess encoding if not set +sub apply_encoding_options { + if (defined $encoding && $encoding eq 'guess') {$encoding=undef;} + if (!defined $encoding) { + } elsif (ref(find_encoding($encoding))) { + if (!$htmlstyle) {$outputEncoding=$encoding;} + } else { + error($Main,"Unknown encoding $encoding ignored."); + error_details($Main,'Valid encodings are: '.wrap('','',join(', ',Encode->encodings(':all')))); + $encoding=undef; + } + if (!defined $outputEncoding) {$outputEncoding='utf8';} + binmode STDIN; + binmode STDOUT,':encoding('.$outputEncoding.')'; +} + +# Apply language options +sub apply_language_options { + my @tmp; + if (defined $LetterPattern && @AlphabetScripts && scalar @AlphabetScripts>0) { + @tmp=@AlphabetScripts; + foreach (@tmp) {$_='\\p{'.$_.'}';} + my $letterchars='['.join('',@tmp).']'; + my $letter=$LetterPattern; + $letter=~s/@/$letterchars/g; + @WordPatterns=map { s/\@/$letter/g ; qr/$_/ } @WordPatterns; + } else { + @WordPatterns=(); + } + if (@LogogramScripts && scalar @LogogramScripts>0) { + @tmp=@LogogramScripts; + foreach (@tmp) {$_='\\p{'.$_.'}';} + push @WordPatterns,'['.join('',@tmp).']'; + } + if (scalar @WordPatterns==0) { + error($Main,'No script (alphabets or logograms) defined. Using fallback mode.'); + push @WordPatterns,'\\w+'; + } + $WordPattern=join '|',@WordPatterns; +} + +# Apply incbib rule +sub apply_include_bibliography { + include_package('%incbib'); +} + +# Process TC instruction, return 1 if applied, 0 if not (e.g. error) +sub tc_macro_param_option { + my ($tex,$instr,$macro,$param,$option)=@_; + if (!defined $param) { + error($tex,'%TC:'.$instr.' requires parameter rule specification.'); + return 0; + } + elsif ($instr eq 'macro') { + if (!($macro=~/^\\/)) { + warning($tex,'%TC:macro '.$macro.': should perhaps be \\'.$macro.'?'); + } + return _tc_macro_set_param($tex,\%TeXmacro,$instr,$macro,$param); + } + elsif ($instr eq 'exclude') { + warning($tex,'%TC:exclude is deprecated. Use %TC:macro instead.'); + return _tc_macro_set_param($tex,\%TeXmacro,$instr,$macro,$param); + } + elsif ($instr eq 'header') { + warning($tex,'%TC:header is deprecated. Instead use e.g. %TC:macro \\name [header].'); + $TeXmacrocount{$macro}=[$CNT_COUNT_HEADER]; + return _tc_macro_set_param($tex,\%TeXmacro,$instr,$macro,$param); + } + elsif ($instr eq 'preambleinclude') { + return _tc_macro_set_param($tex,\%TeXpreamble,$instr,$macro,$param); + } + elsif ($instr eq 'floatinclude') { + return _tc_macro_set_param($tex,\%TeXfloatinc,$instr,$macro,$param); + } + elsif ($instr=~/^(group|envir)$/) { + if (!defined $option) { + error($tex,'TC:'.$instr.' requires contents rule specification.'); + return 0; + } + defined ($option=key_to_state($option,$tex)) || return 0; + $TeXenvir{$macro}=$option; + return _tc_macro_set_param($tex,\%TeXmacro,$instr,$PREFIX_ENVIR.$macro,$param); + } + elsif ($instr=~/^(macrocount|macroword)$/) { + return _tc_macro_set_param($tex,\%TeXmacrocount,$instr,$macro,$param,\&key_to_cnt,$REGEX_NUMBER); + } + elsif ($instr eq 'fileinclude') { + # No parameter checking here, just pass on; errors reported when used instead. + $param=~s/^\[(.*)\]$/$1/; + $param=~s/,/ /g; + assert($param=~/^\w+(\s\w+)*$/,$tex,'Invalid %TC:fileinclude parameter: '.$param) + || return 0; + $TeXfileinclude{$macro}=$param; + } + elsif ($instr eq 'breakmacro') {$BreakPoints{$macro}=$param;} + else { + error($tex,'Unknown TC command: '.$instr); + return 0; + } + return 1; +} + +# Convert TC command parameters and add to hash +sub _tc_macro_set_param { + my ($tex,$hash,$instr,$macro,$param,$keymap,$validregex)=@_; + if (!defined $keymap) {$keymap=\&key_to_state; $validregex=$REGEX_NUMBER;} + $param=__interpret_tc_parameter($tex,$param,$keymap,$validregex); + if (!defined $param) {return 0;} + $hash->{$macro}=$param; + return 1; +} + +# Convert [...] into array of states, optionally return value if a valid pattern +sub __interpret_tc_parameter { + my ($tex,$param,$keymap,$validregex)=@_; + if ((defined $validregex) && ($param=~$validregex)) { + return $param; + } elsif ($param=~/^\[([0-9a-zA-Z,:]*)\]$/) { + my @params; + my $value; + foreach my $key (split(',',$1)) { + if ($key=~/^(\w*):(\w+)$/) { + defined ($value=$keymap->($PREFIX_PARAM_OPTION.$1,$tex)) || return undef; + push @params,$value; + $key=$2; + } + defined ($value=$keymap->($key,$tex)) || return undef; + push @params,$value; + } + return [@params]; + } + error($tex,'Invalid TC command parameter: '.$param); + return undef; +} + + +###### Macro rules handling + + +# Takes hash, value, list of keys and adds mappings to hash +sub add_keys_to_hash { + my $hash=shift @_; + my $value=shift @_; + foreach (@_) { + $hash->{$_}=$value; + } +} + +# Add source hash to target hash, optionally converting values +sub __add_to_hash { + my $target=shift @_; + my $source=shift @_; + my $convert=shift @_; + if (!defined $convert) {$convert=&__dummy;} + while (my ($key,$val)=each(%$source)) { + $target->{$key}=$convert->($val,@_); + } +} + +# Convert hash by applying function to all values +sub convert_hash { + my $hash=shift @_; + my $convert=shift @_; + while (my ($key,$val)=each(%$hash)) { + $hash->{$key}=$convert->($val,@_); + } +} + +# Convert key to parser state constant +sub key_to_state { + my ($key,$tex)=@_; + return key_convert(lc($key),$tex,\%key2state,'Unknown state key {key} replaced by IGNORE.',$STATE_IGNORE); +} + +# Convert key array to parser state constants +sub keyarray_to_state { + my ($array,$tex)=@_; + return keyarray_convert($array,$tex,\&key_to_state); +} + +# Convert key to counter constant +sub key_to_cnt { + my ($key,$tex)=@_; + return key_convert(lc($key),$tex,\%key2cnt,'Unknown counter key {key} ignored.'); +} + +# Convert key array to counter constants +sub keyarray_to_cnt { + my ($array,$tex)=@_; + return keyarray_convert($array,$tex,\&key_to_cnt); +} + +# Convert key to value using map +sub key_convert { + my ($key,$tex,$map,$error,$default)=@_; + my $value=$map->{$key}; + if (!defined $value) { + if (!defined $tex) {$tex=$Main;} + $error=~s/\{key\}/$key/g; + error($tex,$error); + return $default; + } + return $value; +} + +# Convert state keys to state nos for arrays +sub keyarray_convert { + my ($array,$tex,$conv)=@_; + if (ref($array) eq 'ARRAY') { + my @ar; + foreach my $key (@$array) { + if (defined (my $cnt=$conv->($key,$tex))) {push @ar,$cnt;} + } + $array=\@ar; + } + return $array; +} + +# Convert transition state to parsing state, and increment counter +sub transition_to_content_state { + my ($tex,$state)=@_; + my $tr=$transition2state{$state}; + if (defined $tr) { + if (my $cnt=$tr->[1]) {inc_count($tex,$tr->[1]);} + return $tr->[0]; + } + return $state; +} + +# Remove all rules +sub remove_all_rules { + %TeXpackageinc=(); + %TeXpreamble=(); + %TeXfloatinc=(); + %TeXmacro=(); + %TeXmacrocount=(); + %TeXenvir=(); + %TeXfileinclude=(); + %IncludedPackages=(); +} + +# Process package inclusion +sub include_package { + my ($incpackage,$tex)=@_; + if (defined $IncludedPackages{$incpackage}) {return;} + $IncludedPackages{$incpackage}=1; + # Add rules for package, preamble and float inclusion, and add to macro rules + _add_package(\%TeXpackageinc,\%PackageTeXpackageinc,$incpackage,\&__dummy,$tex); + _add_package(\%TeXpreamble,\%PackageTeXpreamble,$incpackage,\&keyarray_to_state,$tex); + _add_package(\%TeXfloatinc,\%PackageTeXfloatinc,$incpackage,\&keyarray_to_state,$tex); + _add_package(\%TeXmacro,\%PackageTeXpackageinc,$incpackage,\&keyarray_to_state,$tex); + _add_package(\%TeXmacro,\%PackageTeXpreamble,$incpackage,\&keyarray_to_state,$tex); + _add_package(\%TeXmacro,\%PackageTeXfloatinc,$incpackage,\&keyarray_to_state,$tex); + # Add regular rules + _add_package(\%TeXmacro,\%PackageTeXmacro,$incpackage,\&keyarray_to_state,$tex); + _add_package(\%TeXmacrocount,\%PackageTeXmacrocount,$incpackage,\&keyarray_to_cnt,$tex); + _add_package(\%TeXenvir,\%PackageTeXenvir,$incpackage,\&key_to_state,$tex); + _add_package(\%TeXfileinclude,\%PackageTeXfileinclude,$incpackage,\&__dummy,$tex); + if (my $subpackages=$PackageSubpackage{$incpackage}) { + foreach my $sub (@{$subpackages}) { + include_package($sub,$tex); + } + } +} + +# Add package rules if defined +sub _add_package { + my ($target,$source,$name,$convert,$tex)=@_; + my $sub; + if ($sub=$source->{$name}) { + __add_to_hash($target,$sub,$convert); + } +} + +# Dummy conversion function: returns arguments unchanged +sub __dummy { + return shift @_; +} + +###### TeX code handle + + + +# Return object capable of capturing errors for use when +# no TeXcode object is available. +sub getMain { + my %main=(); + $main{'errorcount'}=0; + $main{'errorbuffer'}=[]; + $main{'warnings'}={}; + return \%main; +} + + +## Make TeX handle for LaTeX code: the main TeXcount object +# The "TeX object" is a data containser: a hash containing +# - filepath: path of LaTeX file being parsed +# - filename: file name of LaTeX file +# - dirpath: path to directory containing LaTeX file +# - PATH: array of paths to search for included files +# - countsum: count object with total count (incl. subcounts) +# - subcount: count object for subcount (to be added to countsum) +# - subcounts: list of subcounts +# - errorcount: the number of errors reported +# plus following elements used for the processing the LaTeX code +# - line: the LaTeX paragraph being processed +# - texcode: what remains of LaTeX code to process (after line) +# - texlength: length of LaTeX code +# - next: the next token, i.e. the one being processed +# - type: the type of the next token +# - style: the present output style (for verbose output) +# - printstate: present parsing state (for verbose output only) +# - eof: set once the end of the input is reached +# which are passed to methods by passing the TeX object. It is used when parsing +# the LaTeX code, and discarded once the parsing is done. During parsing, counts +# are added to the subcount element; whenever a break point is encountered, the +# subcount is added to the countsum (next_subcount) and a new subcount object +# prepared. Note that this requires that the last subcount object be added to +# the countsum once the end of the document is reached. +sub TeXcode { + my ($bincode,$file,$title)=@_; + my $tex=_TeXcode_blank($file,$title); + _TeXcode_setcode($tex,$bincode); + more_texcode($tex); + return $tex; +} + +# Return a blank TeXcode object +sub _TeXcode_blank { + my ($file,$title)=@_; + if (defined $title) {} + elsif (defined $file) {$title='File: '.$file;} + else {$title='Word count';} + my %TeX=(); + $TeX{'errorcount'}=0; + $TeX{'warnings'}={}; + $TeX{'filepath'}=$file; + if (!defined $file) {} + elsif ($file=~/^<.+>$/) {$TeX{'filename'}=$file;} + elsif ($file=~/^(.*[\\\/])([^\\\/]+)$/) { + $TeX{'dirpath'}=$1; $TeX{'filename'}=$2; + } + else {$TeX{'dirpath'}=''; $TeX{'filename'}=$file;} + $TeX{'PATH'}=[]; + $TeX{'line'}=''; + $TeX{'next'}=undef; + $TeX{'type'}=undef; + $TeX{'style'}=undef; + $TeX{'printstate'}=undef; + $TeX{'eof'}=0; + my $countsum=new_count($title); + $TeX{'countsum'}=$countsum; + $countsum->{'TeXcode'}=\%TeX; + my $count=new_count('_top_'); + $TeX{'subcount'}=$count; + inc_count(\%TeX,$CNT_FILE); + return \%TeX; +} + +# Set the texcode element of the TeXcount object +sub _TeXcode_setcode { + my ($tex,$bincode)=@_; + $tex->{'texcode'}=_prepare_texcode($tex,$bincode); + $tex->{'texlength'}=length($tex->{'texcode'}); +} + +# Decode and return TeX/LaTeX code +sub _prepare_texcode { + my ($tex,$texcode)=@_; + $texcode=_decode_texcode($tex,$texcode); + foreach my $key (keys %substitutions) { + my $value=$substitutions{$key}; + $texcode=~s/(\w)\Q$key\E/$1 $value/g; + $texcode=~s/\Q$key\E/$value/g; + } + return $texcode; +} + +# Return text decoder +sub _decode_texcode { + my ($tex,$texcode)=@_; + my $decoder; + if (defined $encoding) { + $decoder=find_encoding($encoding); + eval {$texcode=$decoder->decode($texcode);}; + if ($@) { + error($tex,'Decoding file/text using the '.$decoder->name.' encoding failed.'); + } + } else { + ($texcode,$decoder)=_guess_encoding($texcode); + if (!ref($decoder)) { + error($tex,'Failed to identify encoding or incorrect encoding specified.'); + $tex->{'encoding'}='[FAILED]'; + return $texcode; + } + } + __set_encoding_name($tex,$decoder->name); + $texcode =~ s/^\x{feff}//; # Remove BOM (relevant for UTF only) + if ($texcode =~/\x{fffd}/ ) { + error($tex,'File/text was not valid '.$decoder->name.' encoded.'); + } + return $texcode; +} + +# Guess the right encoding to use +sub _guess_encoding { + my ($texcode)=@_; + foreach my $enc (@encodingGuessOrder) { + my $dec=find_encoding($enc); + if (ref($dec)) { + eval { + $texcode=$dec->decode($texcode,Encode::FB_CROAK) + }; + if (!$@) {return $texcode,$dec;} + } + } + return $texcode,undef; +} + +# Set name of current encoding +sub __set_encoding_name { + my ($tex,$enc)=@_; + my $cur=$tex->{'encoding'}; + if (!defined $enc) {$enc='[FAILED]';} # Shouldn't happen here though... + if (!defined $cur) {} + elsif ($enc eq 'ascii') {$enc=$cur;} + elsif ($cur eq 'ascii') {} + elsif ($cur ne $enc) { + error($tex,"Mismatching encodings: $cur versus $enc."); + } + $tex->{'encoding'}=$enc; +} + +# Apply substitution rule +sub apply_substitution_rule { + my ($tex,$from,$to)=@_; + $tex->{'line'}=~s/(\w)\Q$from\E\b\s*/$1 $to/g; + $tex->{'line'}=~s/\Q$from\E\b\s*/$to/g; + $tex->{'texcode'}=~s/(\w)\Q$from\E\b\s*/$1 $to/g; + $tex->{'texcode'}=~s/\Q$from\E\b\s*/$to/g; +} + +## Get more TeX code from texcode buffer if possible, return 1 if done +sub more_texcode { + my ($tex)=@_; + if ($tex->{'texcode'} eq '') {return 0;} + if ( $optionFast && $tex->{'texcode'} =~ s/^(.*?(\r{2,}|\n{2,}|(\r\n){2,}))//s ) { + $tex->{'line'}.=$1; # $1 ~ ${^MATCH} + return 1; + } + $tex->{'line'}.=$tex->{'texcode'}; + $tex->{'texcode'}=''; + return 1; +} + +## Prepend LaTeX code to TeXcode object +sub prepend_code { + my ($tex,$code,$filename)=@_; + my $prefix="\n% --- Start of included file $filename\n"; + my $suffix="\n% --- End of included file $filename\n"; + $code=_decode_texcode($tex,$code); + $tex->{'length'}+=length($code); + $tex->{'texcode'}=$prefix.$code.$suffix.$tex->{'line'}.$tex->{'texcode'}; + $tex->{'line'}=''; + more_texcode($tex); +} + +# Save and return TeXcode state, and replace with new text +sub texcode_insert_text { + my ($tex,$code,$filename,@path)=@_; + my %texstate=( + 'texcode'=>$tex->{'line'}.$tex->{'texcode'}, + 'eof'=>$tex->{'eof'}, + 'PATH'=>$tex->{'PATH'}); + my $prefix="\n% --- Start of included file $filename\n"; + my $suffix="\n% --- End of included file $filename\n"; + $code=_decode_texcode($tex,$code); + $tex->{'length'}+=length($code); + $tex->{'texcode'}=$prefix.$code.$suffix; + $tex->{'line'}=''; + more_texcode($tex); + return \%texstate; +} + +# Restore TeXcode state +sub texcode_restore_state { + my ($tex,$texstate)=@_; + while (my ($key,$value)=each %$texstate) { + $tex->{$key}=$value; + } +} + +## Returns size of TeX code in bytes +sub get_texsize { + my $tex=shift @_; + return $tex->{'texlength'} +} + + +###### TeX file reader + + +# Read LaTeX file into TeX object +sub TeXfile { + my ($filename,$title)=@_; + my ($header,$bincode); + if ($filename eq $_STDIN_) { + $header='File from STDIN'; + $bincode=_read_stdin(); + } else { + defined ($bincode=read_binary($filename)) || return undef; + $header='File: '.$filename; + } + if ($printlevel>0) { + formatprint($header."\n",'h2'); + $blankline=0; + } + my $tex=TeXcode($bincode,$filename,$title); + return $tex; +} + +# Read file to string without regard for encoding +sub read_binary { + my $filename=shift @_; + open(FH,$filename) || return undef; + binmode(FH); + my $bincode; + read(FH,$bincode,-s FH); + close(FH); + return $bincode; +} + +# Read file from STDIN +sub _read_stdin { + my @text=<STDIN>; + my $latexcode=join('',@text); + return $latexcode; +} + +###### Error handling + + +# Add warning to list of registered warnings (optionally to be reported at the end) +sub warning { + my ($tex,$text)=@_; + $warnings{$text}++; + # TODO: should only add warnings to subcount, not to TeXcode object + $tex->{'warnings'}->{$text}++; + if (my $count=$tex->{'subcount'}) {$count->{'warnings'}->{$text}++}; +} + +# Register error and print error message +sub error { + my ($tex,$text,$type)=@_; + if (defined $type) {$text=$type.': '.$text;} + $errorcount++; + $tex->{'errorcount'}++; + if (my $count=$tex->{'subcount'}) {$count->{'errorcount'}++}; + if (my $err=$tex->{'errorbuffer'}) {push @$err,$text;} + else {print_error($text);} +} + +# Print error details +sub error_details { + my ($tex,$text)=@_; + print STDERR $text,"\n"; +} + +# Make assertion: i.e. report error if not true and return truthfullness +sub assert { + my $assertion=shift @_; + if ($assertion) {return 1;} + error(@_); + return 0; +} + +# Print error message +sub print_error { + my $text=shift @_; + line_return(1); + if ($printlevel<0) { + print STDERR 'ERROR: ',$text,"\n"; + } elsif ($htmlstyle) { + print STDERR 'ERROR: ',$text,"\n"; + print_style($text,'error'); + } else { + print_style("!!! $text !!!",'error'); + } + line_return(1); +} + +# Print errors in buffer and delete errorbuffer +sub flush_errorbuffer { + my $source=shift @_; + my $err=$source->{'errorbuffer'} || return; + foreach (@$err) {print_error($_);} + $source->{'errorbuffer'}=undef; +} + +###### Parsing routines + + +# Parse LaTeX document +sub parse { + my ($tex)=@_; + if ($htmlstyle && $printlevel>0) {print "<div class='parse'><p>\n";} + parse_all($tex); + if ($htmlstyle && $printlevel>0) {print "</p></div>\n";} +} + +# Parse until end +sub parse_all { + my ($tex,$state)=@_; + if (!defined $state) {$state=$STATE_TEXT;} + while (!($tex->{'eof'})) { + _parse_unit($tex,$state); + } +} + +# Parse one block/unit with given state until end token +# If end token is set to $_PARAM_, unit is assumed to be a parameter +# and words parsed as individual letters (simple_token is set) +sub _parse_unit { + my ($tex,$state,$end)=@_; + my $simple_token=0; + ( assert(defined $state,$tex,'Undefined parser state!','BUG') + && assert(!ref($state),$tex,'Invalid parser state of type '.ref($state).'!','BUG') + ) || exit; + $state=transition_to_content_state($tex,$state); + _set_printstate($tex,$state,$end); + if (defined $end && $end eq $_PARAM_) { + $end=undef; + $simple_token=1; + } + my $next; + while (defined ($next=next_token($tex,$simple_token))) { + # Parse next token until token matches $end + set_style($tex,'ignore'); + if ($state==$STATE_MATH) {set_style($tex,'math');} + if ((defined $end) && ($end eq $next)) {return;} + # Determine how token should be interpreted + if ($state==$STATE_PREAMBLE && $next eq '\begin' && $tex->{'line'}=~/^\{\s*document\s*\}/) { + # \begin{document} + $state=$STATE_TEXT; + } + # Pick the first matching interpretation: + if ($state==$STATE_EXCLUDE_ALL) { + # ignore everything + } elsif ($tex->{'type'}==$TOKEN_SPACE) { + # space or other code that should be passed through without styling + flush_next($tex,' '); + } elsif ($tex->{'type'}==$TOKEN_TC) { + # parse TC instructions + _parse_tc($tex,$next); + } elsif ($tex->{'type'}==$TOKEN_WORD) { + # word + if (my $cnt=state_text_cnt($state)) { + _process_word($tex,$next,$state); + inc_count($tex,$cnt); + set_style($tex,state_to_style($state)); + } + } elsif ($next eq '{') { + # {...} group + set_style($tex,'ignore'); + _parse_unit($tex,$state,'}'); + set_style($tex,'ignore'); + } elsif ($next eq '}') { + error($tex,'Encountered } without corresponding {.'); + } elsif ($state==$STATE_EXCLUDE_STRONGER) { + # ignore remaining tokens + set_style($tex,'ignore'); + } elsif ($next eq '\documentclass') { + # defines document class and starts preamble + set_style($tex,'document'); + _parse_documentclass_params($tex); + while (!($tex->{'eof'})) { + _parse_unit($tex,$STATE_PREAMBLE); + } + } elsif ($tex->{'type'}==$TOKEN_MACRO) { + # macro call + _parse_macro($tex,$next,$state); + } elsif ($next eq '$') { + # math inline + _parse_math($tex,$state,$CNT_COUNT_INLINEMATH,'$'); + } elsif ($next eq '$$') { + # math display (unless already in inlined math) + if (!(defined $end && $end eq '$')) { + _parse_math($tex,$state,$CNT_COUNT_DISPLAYMATH,'$$'); + } + } elsif ($simple_token) { + #DELETE: simple tokens should be handled properly + #print STDERR '<',$next,'>',"\n"; + # handle as parameter that should not be counted + set_style($tex,'ignore'); + } + if (!defined $end) {return;} + } + defined $end && error($tex,'Reached end of file while waiting for '.$end.'.'); +} + +# Print state +sub _set_printstate { + my ($tex,$state,$end)=@_; + $tex->{'printstate'}=':'.state_to_text($state).':'.(defined $end?$end.':':''); + flush_next($tex); +} + +# Process word with a given state (>0, i.e. counted) +sub _process_word { + my ($tex,$word,$state)=@_; + $WordFreq{$word}++; +} + +# Parse unit when next token is a macro +sub _parse_macro { + my ($tex,$next,$state)=@_; + my $substat; + my @macro=($next); + if (my $label=$BreakPoints{$next}) { + if ($tex->{'line'}=~ /^[*]?(\s*\[.*?\])*\s*\{((.|\{.*\})*)\}/ ) { + $label=$label.': '.$2; + } + next_subcount($tex,$label); + } + if ($state==$STATE_MATH) {set_style($tex,'mathcmd');} + else {set_style($tex,state_is_text($state)?'cmd':'exclcmd');} + if ($next eq '\begin' && state_inc_envir($state)) { + _parse_envir($tex,$state); + push @macro,'<envir>'; + } elsif ($next eq '\end' && state_inc_envir($state)) { + error($tex,'Encountered \end without corresponding \begin'); + push @macro,$STRING_ERROR; + } elsif ($next eq '\verb') { + _parse_verb_region($tex,$state); + } elsif (state_is_parsed($state) && defined $TeXpackageinc{$next} ) { + _parse_include_package($tex); + push @macro,'<package>'; + } elsif (state_is_parsed($state) && defined (my $def=$TeXfileinclude{$next})) { + # include file (merge in or queue up for parsing) + set_style($tex,'cmd'); + if (defined ($substat=$TeXmacro{'@pre'.$next})) {__gobble_macro_parms($tex,$substat,$state);} + __count_macrocount($tex,$next,$state); + _parse_include_file($tex,$state,$def); + push @macro,'<filespec>'; + if (defined ($substat=$TeXmacro{'@post'.$next})) {__gobble_macro_parms($tex,$substat,$state);} + } elsif (($state==$STATE_FLOAT) && ($substat=$TeXfloatinc{$next})) { + # text included from float + set_style($tex,'cmd'); + push @macro,__gobble_macro_parms($tex,$substat,$__STATE_NULL); + } elsif ($state==$STATE_PREAMBLE && defined ($substat=$TeXpreamble{$next})) { + # parse preamble include macros + set_style($tex,'cmd'); + __count_macrocount($tex,$next,$STATE_TEXT); + push @macro,__gobble_macro_parms($tex,$substat,$__STATE_NULL); + } elsif (state_is_exclude($state)) { + # ignore + push @macro,__gobble_options($tex),'/*ignored*/'; + } elsif ($next eq '\(') { + # math inline + _parse_math($tex,$state,$CNT_COUNT_INLINEMATH,'\)'); + } elsif ($next eq '\[') { + # math display + _parse_math($tex,$state,$CNT_COUNT_DISPLAYMATH,'\]'); + } elsif ($next=~/^\\(def|edef|gdef|xdef)$/) { + # ignore \def... + $tex->{'line'} =~ s/^([^\{]*)\{/\{/; + flush_next($tex); + print_style($1,'ignore'); + _parse_unit($tex,$STATE_EXCLUDE_STRONG); + push @macro,'<macro>',$STRING_PARAMETER; + } elsif (defined ($substat=$TeXmacro{$next})) { + # macro: exclude options + __count_macrocount($tex,$next,$state); + push @macro,__gobble_macro_parms($tex,$substat,$state); + } elsif (defined __count_macrocount($tex,$next,$state)) { + # count macro as word (or a given number of words) + } elsif ($next =~ /^\\[^\w\_]$/) { + # handle \<symbol> as single symbol macro + push @macro,__gobble_options($tex),'/*symbol*/'; + } else { + push @macro,__gobble_options($tex),'/*defaultrule*/'; + } + $MacroUsage{join('',@macro)}++; +} + +# Parse TC instruction +sub _parse_tc { + my ($tex,$next)=@_; + set_style($tex,'tc'); + flush_next($tex); + assert($next=~s/^\%+TC:\s*(\w+)\s*//i + ,$tex,'TC command should have format %TC:instruction [[parameters]]') + || return; + my $instr=$1; + $instr=~tr/[A-Z]/[a-z]/; + if ($instr eq 'break') {next_subcount($tex,$next); + } elsif ($instr=~/^(incbib|includebibliography)$/) { + $includeBibliography=1; + apply_include_bibliography(); + } elsif ($instr eq 'ignore') {__gobble_tc_ignore($tex); + } elsif ($instr eq 'endignore') {error($tex,'%TC:endignore without corresponding %TC:ignore.'); + } elsif ($instr eq 'newtemplate') {$outputtemplate=''; + } elsif ($instr eq 'template') {$outputtemplate.=$next; + } elsif ($instr eq 'insert') { + $tex->{'line'}="\n".$next.$tex->{'line'}; + } elsif ($instr eq 'subst') { + if ($next=~/^(\\\S+)\s+(.*)$/) { + my $from=$1; + my $to=$2; + $substitutions{$from}=$to; + apply_substitution_rule($tex,$from,$to); + } else { + error($tex,'Invalid %TC:subst format.'); + } + } elsif ($instr eq 'newcounter') { + assert($next=~s/^(\w+)(=(\w+))?\s*//,$tex,'Should have format %TC:newcounter {key}[={like-key}] {description}') + || return; + my $key=$1; + my $like=$3; + if ($next eq '') {$next=$key;} + add_new_counter($key,$next,$like); + } elsif ($next=~/^([\\]*\S+)\s+([^\s]+)(\s+(-?\w+))?/) { + # %TC:instr macro param option + my $macro=$1; + my $param=$2; + my $option=$4; + tc_macro_param_option($tex,$instr,$macro,$param,$option); + } else { + error($tex,'Invalid TC command: '.$instr); + } +} + +# Parse math formulae +sub _parse_math { + my ($tex,$state,$cnt,$end)=@_; + my $localstyle; + if (state_is_text($state)) { + $localstyle='mathgroup'; + inc_count($tex,$cnt); + } else { + $localstyle='exclmath'; + } + set_style($tex,$localstyle); + _parse_unit($tex,$STATE_MATH,$end); + set_style($tex,$localstyle); +} + +# Parse \verb region +sub _parse_verb_region { + my ($tex,$state)=@_; + #flush_next($tex); + __gobble_macro_modifier($tex); + set_style($tex,'ignore'); + assert($tex->{'line'} =~ s/^([^\s])//,$tex,'Invalid \verb: delimiter required.') + || return; + my $dlm=$1; + print_style($dlm,'cmd'); + assert($tex->{'line'}=~s/^([^\Q$dlm\E]*)(\Q$dlm\E)// + ,$tex,'Invalid \verb: could not find ending delimiter ('.$dlm.').') + || return; + print_style($1,'ignore'); + print_style($2,'cmd'); +} + +# Parse environment +sub _parse_envir { + my ($tex,$state)=@_; + my $localstyle=state_is_text($state) ? 'envir' : 'exclenv'; + flush_next_gobble_space($tex,$localstyle,$state); + my ($envirname,$next); + if ($tex->{'line'} =~ s/^\{([^\{\}\s]+)\}[ \t\r\f]*//) { + # gobble environment name + $envirname=$1; + my @macro=('<envir:'.$envirname.'>'); + print_style('{'.$1.'}',$localstyle); + $next=$PREFIX_ENVIR.$envirname; + __count_macrocount($tex,$next,$state); + if (defined (my $def=$TeXmacro{$next})) { + push @macro,__gobble_macro_parms($tex,$def,$__STATE_NULL); + } else { + push @macro,__gobble_options($tex); + } + $MacroUsage{join('',@macro)}++; + } else { + $envirname='???'; $next='???'; + error($tex,'Encountered \begin without environment name provided.'); + } + # find new parsing state (or leave unchanged) + my $substat=$TeXenvir{$1}; + if (!defined $substat) { + $substat=$state; + if ($strictness>=1) { + warning($tex,'Using default rule for environment '.$envirname); + } + } else { + $substat=__new_state($substat,$state); + } + if (state_inc_envir($substat)) { + # Parse until \end arrives, and check that it matches + _parse_unit($tex,$substat,'\end'); + flush_next_gobble_space($tex,$localstyle,$state); + if ($tex->{'line'} =~ s/^\{([^\{\}\s]+)\}[ \t\r\f]*//) { + # gobble environment name + print_style('{'.$1.'}',$localstyle); + assert($envirname eq $1,$tex,"Environment \\begin{$envirname} ended with \\end{$1}."); + } else { + error($tex,"Environment ended while waiting for \end{$envirname}."); + } + } else { + # Keep parsing until appropriate \end arrives ignoring all else + while (!$tex->{'eof'}) { + _parse_unit($tex,$substat,'\end'); + if ($tex->{'line'} =~ s/^(\s*)(\{\Q$envirname\E\})[ \t\r\f]*//) { + # gobble \end parameter and end environment + flush_next($tex,$localstyle,$state); + print_style($1,$localstyle); + print_style($2,$localstyle); + return; + } + } + } +} + +# Parse and process file inclusion +sub _parse_include_file { + my ($tex,$state,$includetype)=@_; + my $include=state_is_parsed($state); + my $style=$include?'fileinc':'ignore'; + my $file; + my %params; + flush_next($tex); + $params{'<type>'}=$includetype; + if ($includetype eq '<bbl>') { + _parse_include_bbl($tex,$state,\%params); + return; + } + elsif ($includetype eq 'input') { + $tex->{'line'} =~ s/^(\s*\{([^\{\}\s]+)\})// + || $tex->{'line'} =~ s/^(\s*([^\{\}\%\\\s]+))// + || $tex->{'line'} =~ s/^(\s*\{(.+?)\})// + || BLOCK { + error($tex,'Failed to identify file name.'); + return; + }; + print_style($1,$style); + $file=$2; + } + else { + foreach my $param (split(/\s+|,/,$includetype)) { + if ($param=~/^<.*>$/) {next;} + $tex->{'line'} =~ s/(\s*\{)([^\{\}\%]*)(\})//|| BLOCK { + error($tex,"Failed to identify file parameter {$param}."); + return; + }; + print_style($1,'ignore'); + print_style($2,$style); + print_style($3,'ignore'); + if ($param eq 'file') {$file=$2;} + elsif ($param eq 'texfile') { + $file=$2; + if (!$file=~/\.tex$/i) {$file.='.tex';} + } + else {$params{$param}=$2;} + } + } + if (!defined $file) { + error($tex,'Undefined file name.'); + } + elsif ($includeTeX && $include) { + include_file($tex,$state,$file,\%params); + } +} + +# Parse and process bibliography file +sub _parse_include_bbl { + my ($tex,$state,$params)=@_; + if ($includeBibliography && state_is_text($state)) { + my $filename=$tex->{'filename'}; + $filename=~s/(\.tex)?$/\.bbl/i; + include_file($tex,$state,$filename,$params); + } +} + +# Parse and process package inclusion +sub _parse_include_package { + my ($tex)=@_; + set_style($tex,'document'); + __gobble_option($tex); + if ( $tex->{'line'}=~s/^\{(([\w\-]+)(\s*,\s*[\w\-]+)*)\}// ) { + print_style("{$1}",'document'); + foreach (split(/\s*,\s*/,$1)) { + $MacroUsage{"<package:$_>"}++; + include_package($_,$tex); + } + } else { + _parse_unit($tex,$STATE_IGNORE); + error($tex,'Could not recognise package list, ignoring it instead.'); + } +} + +# Parse \documentclass parameters and include rules +sub _parse_documentclass_params { + my ($tex)=@_; + my $options=__gobble_option($tex); + if ( $tex->{'line'}=~s/^(\{\s*([^\{\}\s]+)\s*\})// ) { + print_style("$1",'document'); + $MacroUsage{"<documentclass:$2>"}++; + include_package("class%$2",$tex); + } else { + _parse_unit($tex,$STATE_IGNORE); + error($tex,'Could not identify document class.'); + } +} + +# Count macrocount using given state +sub __count_macrocount { + my ($tex,$next,$state)=@_; + my $def=$TeXmacrocount{$next}; + my $cnt; + if (!defined $def) {return undef;} + elsif (ref($def) eq 'ARRAY') { + # TODO: Is this an appropriate style to use? + set_style($tex,state_to_style($state)); + flush_next($tex); + foreach $cnt (@{$def}) { + if ($cnt==$CNT_WORDS_TEXT) {$cnt=state_text_cnt($state);} + inc_count($tex,$cnt); + } + } + elsif ($cnt=state_text_cnt($state)) { + set_style($tex,state_to_style($state)); + flush_next($tex); + inc_count($tex,$cnt,$def); + } + return $def; +} + +# Gobble next option, return option or undef if none +sub __gobble_option { + my $tex=shift @_; + flush_next_gobble_space($tex); + if ($tex->{'line'}=~s/^($MacroOptionPattern)//) { + print_style($1,'option'); + return $1; + } + return undef; +} + +# Gobble all options, return the number of gobble options +sub __gobble_options { + my $n=0; + while (__gobble_option(@_)) {$n++} + return ($STRING_GOBBLED_OPTION)x$n; +} + +# Gobble macro modifyer (e.g. following *) +sub __gobble_macro_modifier { + my $tex=shift @_; + flush_next($tex); + if ($tex->{'line'} =~ s/^(\*)//) { + print_style($1,'option'); + return $1; + } + return; +} + +# Gobble macro parameters as specified in parm plus options +sub __gobble_macro_parms { + my ($tex,$parm,$oldstat)=@_; + my $n; + my @ret; + if (ref($parm) eq 'ARRAY') { + $n=scalar @{$parm}; + } else { + $n=$parm; + $parm=[($STATE_IGNORE)x$n]; + } + # TODO: Optionally gobble macro modifier? + if ($n>0) {push @ret,__gobble_macro_modifier($tex);} + my $auto_gobble_options=1; + for (my $j=0;$j<$n;$j++) { + my $p=$parm->[$j]; + if ($p==$_STATE_OPTION) { + # Parse macro option + $p=$parm->[++$j]; + if ($tex->{'line'}=~s/^(\s*\[)//) { + flush_next_gobble_space($tex); + print_style($1,'optparm'); + push @ret,$STRING_OPTIONAL_PARAM; + _parse_unit($tex,__new_state($p,$oldstat),']'); + set_style($tex,'optparm'); + } + } elsif ($p==$_STATE_NOOPTION) { + $auto_gobble_options=0; + } elsif ($p==$_STATE_AUTOOPTION) { + $auto_gobble_options=1; + } else { + # Parse macro parameter + if ($auto_gobble_options) {push @ret,__gobble_options($tex);} + push @ret,$STRING_PARAMETER; + _parse_unit($tex,__new_state($p,$oldstat),$_PARAM_); + } + } + #TODO: Drop default gobbling of option at end? + if ($auto_gobble_options) {push @ret,__gobble_options($tex);} + return @ret; +} + +# Parse through ignored LaTeX code +sub __gobble_tc_ignore { + my ($tex)=@_; + set_style($tex,'ignore'); + _parse_unit($tex,$STATE_EXCLUDE_ALL,'%TC:endignore'); + set_style($tex,'tc'); + flush_next($tex); +} + +# Return new parsing state given old and substate +sub __new_state { + my ($substat,$oldstat)=@_; + if (!defined $oldstat) {return $substat;} + foreach my $st (@STATE_FIRST_PRIORITY,@STATE_MID_PRIORITY,@STATE_LAST_PRIORITY) { + if ($oldstat==$st || $substat==$st) {return $st;} + } + error($Main,'Could not determine new state in __new_state!','BUG'); + return $oldstat; +} + + +###### Tokenisation routines + + +# Get next token skipping comments and flushing output buffer +sub next_token { + my ($tex,$simple_token)=@_; + if (!defined $simple_token) {$simple_token=0;} + my ($next,$type); + my $style=$tex->{'style'}; + if (defined $tex->{'next'}) {print_style($tex->{'next'},$tex->{'style'});} + $tex->{'style'}=undef; + while (defined ($next=_get_next_token($tex,$simple_token))) { + $type=$tex->{'type'}; + if ($type==$TOKEN_COMMENT) { + print_style($next,'comment'); + } elsif ($type==$TOKEN_LINEBREAK) { + if ($printlevel>0) {line_return(-1,$tex);} + } else { + return $next; + } + } + return $next; +} + +# Read, interpret and return next token +# If simple_token is set, words cannot form tokens +sub _get_next_token { + my ($tex,$simple_token)=@_; + my $next; + my $ch; + while (!$tex->{'eof'}) { + $ch=substr($tex->{'line'},0,1); + if ($ch eq '') { + if (!more_texcode($tex)) {$tex->{'eof'}=1;} + next; + } elsif ($ch=~/^[ \t\f]/) { + $tex->{'line'}=~s/^([ \t\f]+)//; + return __set_token($tex,$1,$TOKEN_SPACE); + } elsif ($ch eq "\n" || $ch eq "\r") { + $tex->{'line'}=~s/^(\r\n?|\n)//; + return __set_token($tex,$1,$TOKEN_LINEBREAK); + } elsif (!$simple_token && $tex->{'line'}=~s/^($WordPattern)//o) { + return __set_token($tex,$1,$TOKEN_WORD); + } elsif ($simple_token && $tex->{'line'}=~s/^(\w)//) { + # When parsing simple tokens (not words), only include single characters + # TODO: the handling of simple tokens should be improved + my $match=$1; + if ($match=~/^$WordPattern$/) {return __set_token($tex,$match,$TOKEN_WORD);} + else {return __set_token($tex,$match,$TOKEN_SYMBOL);} + } elsif ($ch eq '\\') { + if ($tex->{'line'}=~s/^(\\[{}%])//) {return __set_token($tex,$1,$TOKEN_SYMBOL);} + if ($tex->{'line'}=~s/^(\\([a-zA-Z@]+|[^a-zA-Z@[:^print:]]))//) {return __set_token($tex,$1,$TOKEN_MACRO);} + return __get_chtoken($tex,$ch,$TOKEN_END); + } elsif ($ch eq '$') { + $tex->{'line'}=~s/^(\$\$?)//; + return __set_token($tex,$1,$TOKEN_MATH); + } elsif ($ch eq '{' || $ch eq '}') { + return __get_chtoken($tex,$ch,$TOKEN_BRACE); + } elsif ($ch eq '[' || $ch eq ']') { + return __get_chtoken($tex,$ch,$TOKEN_BRACKET); + } elsif ($ch=~/^['"`:.,()[]!+-*=\/^_@<>~#&]$/) { + return __get_chtoken($tex,$ch,$TOKEN_SYMBOL); + } elsif ($ch eq '%') { + if ($tex->{'line'}=~s/^(\%+TC:\s*endignore\b[^\r\n]*)//i) { + __set_token($tex,$1,$TOKEN_TC); + return "%TC:endignore"; + } + if ($tex->{'line'}=~s/^(\%+[tT][cC]:[^\r\n]*)//) {return __set_token($tex,$1,$TOKEN_TC);} + if ($tex->{'line'}=~s/^(\%+[^\r\n]*)//) {return __set_token($tex,$1,$TOKEN_COMMENT);} + return __get_chtoken($tex,$ch,$TOKEN_COMMENT); + } else { + return __get_chtoken($tex,$ch,$TOKEN_END); + } + } + return undef; +} + +# Set next token and its type +sub __set_token { + my ($tex,$next,$type)=@_; + $tex->{'next'}=$next; + $tex->{'type'}=$type; + return $next; +} + +# Set character token and remove from line +sub __get_chtoken { + my ($tex,$ch,$type)=@_; + $tex->{'line'}=substr($tex->{'line'},1); + $tex->{'next'}=$ch; + $tex->{'type'}=$type; + return $ch; +} + + +###### Count handling routines + + + +## Make new count object +# The "count object" is a hash containing: +# - title: the title of the count (name of file, section, ...) +# - counts: a list of numbers (the counts: files, text words, ...) +# - subcounts: list of count objects (used by the TeX object) +# - warnings: list of warnings produced +# The elements are specified by the $CONT_* constants. +sub new_count { + my ($title)=@_; + my @cnt=(0) x $SIZE_CNT; + my %count=('counts'=>\@cnt,'title'=>$title,'subcounts'=>[],'errorcount'=>0,'warnings'=>{}); + return \%count; +} + +# Ensure the count array is of the right size +sub update_count_size { + my $count=shift @_; + my $counts=$count->{'counts'}; + while (scalar @{$counts}<$SIZE_CNT) {push @$counts,0;} +} + +# Increment TeX count for a given count type +sub inc_count { + my ($tex,$cnt,$value)=@_; + my $count=$tex->{'subcount'}; + if (!defined $value) {$value=1;} + ${$count->{'counts'}}[$cnt]+=$value; +} + +# Get count value for a given count type +sub get_count { + my ($count,$cnt)=@_; + my $counts=$count->{'counts'}; + if ($cnt<scalar @{$counts}) {return ${$counts}[$cnt];} + else {return 0;} +} + +# Compute sum count for a count object +sub get_sum_count { + my $count=shift @_; + my $sum=0; + for (my $i=scalar(@sumweights);$i-->1;) { + $sum+=get_count($count,$i)*$sumweights[$i]; + } + return $sum; +} + +# Returns the number of subcounts +sub number_of_subcounts { + my $count=shift @_; + if (my $subcounts=$count->{'subcounts'}) { + return scalar(@{$subcounts}); + } else { + return 0; + } +} + +# Returns the number of warnings (not distinct) +sub number_of_warnings { + my $count=shift @_; + my $n=0; + foreach my $m (values %{$count->{'warnings'}}) {$n+=$m;} + return $n; +} + +# Returns the number of distinct warnings +sub number_of_distinct_warnings { + my $count=shift @_; + return scalar keys %{$count->{'warnings'}}; +} + +# Is a null count? (counts 1-7 zero, title starts with _) +sub _count_is_null { + my $count=shift @_; + if (!$count->{'title'}=~/^_/) {return 0;} + for (my $i=1;$i<$SIZE_CNT;$i++) { + if (get_count($count,$i)>0) {return 0;} + } + if (scalar keys %{$count->{'warnings'}}) {return 0;} + if ($count->{'errorcount'}) {return 0;} + return 1; +} + +# Add one count to another +sub _add_to_count { + my ($a,$b)=@_; + update_count_size($a); + update_count_size($b); + for (my $i=0;$i<$SIZE_CNT;$i++) { + ${$a->{'counts'}}[$i]+=${$b->{'counts'}}[$i]; + } + while (my ($text,$n)=each %{$a->{'warnings'}}) { + $a->{'warnings'}->{$text}+=$n; + } + $a->{'errorcount'}+=$b->{'errorcount'}; +} + +# Add subcount to sum count and prepare new subcount +sub next_subcount { + my ($tex,$title)=@_; + add_to_total($tex->{'countsum'},$tex->{'subcount'}); + $tex->{'subcount'}=new_count($title); + return $tex->{'countsum'}; +} + +# Add count to total as subcount +sub add_to_total { + my ($total,$count)=@_; + _add_to_count($total,$count); + if (_count_is_null($count)) {return;} + push @{$total->{'subcounts'}},$count; + $count->{'parentcount'}=$total; +} + +###### Result output routines + + +# Close the output (STDOUT), e.g. adding HTML tail +sub Close_Output { + if ($htmlstyle>1) { + html_tail(); + } + close STDOUT; +} + +# Report if there were any errors occurring during parsing +sub Report_Errors { + if (defined $outputtemplate) {return;} + if ( !$briefsum && !$totalflag && $printlevel>=0 ) { + foreach (keys %warnings) {formatprint($_,'p','nb');print "\n";} + } + if ($errorcount==0) {return;} + if ($briefsum && $totalflag) {print ' ';} + if ($htmlstyle) { + print "<div class='error'><p>\n"; + print "There were $errorcount error(s) reported!\n"; + print "</p></div>\n"; + } elsif ($briefsum && $totalflag) { + print "(errors:$errorcount)"; + } else { + print "(errors:$errorcount)\n"; + } +} + +# Print word frequencies (as text only) +sub print_word_freq { + my ($word,$wd,$freq,%Freq,%Word,%Class); + my %regs; + foreach my $cg (@AlphabetScripts,@LogogramScripts) { + $regs{$cg}=qr/\p{$cg}/; + } + my $sum=0; + for $word (keys %WordFreq) { + $wd=lc $word; + $Freq{$wd}+=$WordFreq{$word}; + $sum+=$WordFreq{$word}; + $Word{$wd}=__lc_merge($word,$Word{$wd}); + } + if ($htmlstyle) {print "<table class='stat'>\n<thead>\n";} + __print_word_freq('Word','Freq','th'); + if ($htmlstyle) {print "</thead>\n";} + if ($optionWordClassFreq>0) { + for $word (keys %Freq) {$Class{__word_class($word,\%regs)}+=$Freq{$word};} + __print_sorted_freqs('langstat',\%Class); + } + if ($htmlstyle) {print "<tbody class='sumstat'>\n";} + __print_word_freq('All words',$sum,'td','sum'); + if ($htmlstyle) {print "</tbody>\n";} + if ($optionWordFreq>0) {__print_sorted_freqs('wordstat',\%Freq,\%Word,$optionWordFreq);} + if ($htmlstyle) {print "</table>\n";} +} + +# Merge to words which have the same lower case +sub __lc_merge { + my ($word,$w)=@_; + if (defined $w) { + for (my $i=length($word);$i-->0;) { + if (substr($word,$i,1) ne substr($w,$i,1)) { + substr($word,$i,1)=lc substr($word,$i,1); + } + } + } + return $word; +} + +# Return the word class based on script groups it contains +sub __word_class { + my ($wd,$regs)=@_; + my @classes; + $wd=~s/\\\w+({})?/\\{}/g; + foreach my $name (keys %{$regs}) { + if ($wd=~$regs->{$name}) {push @classes,$name;} + } + my $cl=join('+',@classes); + if ($cl) {} + elsif ($wd=~/\\/) {$cl='(macro)';} + else {$cl='(unidentified)';} + return $cl; +} + +# Print table body containing word frequencies +sub __print_sorted_freqs { + my ($class,$Freq,$Word,$min)=@_; + my $sum=0; + my ($word,$wd,$freq); + if (!defined $min) {$min=0;} + if ($htmlstyle) {print "<tbody class='$class'>\n";} + else {print "---\n";} + for $wd (sort {$Freq->{$b} <=> $Freq->{$a}} keys %{$Freq}) { + if (defined $Word) {$word=$Word->{$wd};} else {$word=$wd;} + $freq=$Freq->{$wd}; + if ($freq>=$min) { + $sum+=$freq; + __print_word_freq($word,$freq); + } + } + if ($min>0) {__print_word_freq('Sum of subset',$sum,'td','sum');} + if ($htmlstyle) {print "</tbody>\n";} +} + +# Print one word freq line +sub __print_word_freq { + my ($word,$freq,$tag,$class)=@_; + if (!defined $tag) {$tag='td';} + if (defined $class) {$class=" class='$class'";} else {$class='';} + if ($htmlstyle) { + print "<tr$class><$tag>$word</$tag><$tag>$freq</$tag></tr>\n"; + } else { + print $word,': ',$freq,"\n"; + } +} + +# Print macro usage statistics +sub print_macro_stat { + if ($htmlstyle) {print "<table class='stat'>\n<thead>\n";} + __print_word_freq('Macro/envir','Freq','th'); + if ($htmlstyle) {print "</thead>\n";} + if ($htmlstyle) {print "<tbody class='macrostat'>\n";} + foreach my $name (sort keys %MacroUsage) { + my $freq=$MacroUsage{$name}; + $name=text_to_printable($name); + $name=text_to_print($name); + if ($htmlstyle) { + print "<tr><td>$name</td><td>$freq</td></tr>\n"; + } else { + print $name,': ',$freq,"\n"; + } + } + if ($htmlstyle) {print "</tbody>\n";} + if ($htmlstyle) {print "</table>\n";} +} + + +###### Printing routines + + +# Print text using given style/colour +sub print_with_style { + my ($text,$style,$colour)=@_; + if ($style eq $NOSTYLE) { + print text_to_print($text); + } elsif ($htmlstyle) { + print "<span class='$style'>",text_to_print($text),'</span>'; + } else { + ansiprint(text_to_print($text),$colour); + if ($style=~/$separatorstyleregex/) {print $separator;} + } +} + +# Prepare text string for print: convert special characters +sub text_to_print { + my $text=join('',@_); + if ($htmlstyle) { + $text=~s/&/&/g; + $text=~s/</</g; + $text=~s/>/>/g; + $text=~s/[ \t]{2}/\ /g; + } elsif ($texcodeoutput) { + $text=~s/\\/\\textbackslash¤/g; + $text=~s/([%{}])/\\$1/g; + $text=~s/¤/{}/g; + } + return $text; +} + +# Return &#xxx; code for character ; +sub convert_to_charcode { + my $ch=shift @_; + return '&#'.ord($ch).';'; +} + +# Convert special characters to charcodes to make it printable ; +sub text_to_printable { + my $text=shift @_; + $text=~s/([^[:print:]])/convert_to_charcode($1)/gse; + return $text; +} + +# Print text, using appropriate tags for HTML +sub formatprint { + my ($text,$tag,$class)=@_; + my $break=($text=~s/(\r\n?|\n)$//); + if ($htmlstyle && defined $tag) { + print '<'.$tag; + if ($class) {print " class='$class'";} + print '>'.text_to_print($text)."</$tag>"; + } else { + print text_to_print($text); + } + if ($break) {print "\n";} +} + +# Add a line break to output +sub linebreak { + if ($htmlstyle) {print "<br>\n";} else {print "\n";} +} + +###### Routines for printing count summary + + +# Print count summary for a count object +sub print_count { + my ($count,$class)=@_; + if ($htmlstyle) {print "<div class='".($class||'count')."'>\n";} + if ($outputtemplate) { + _print_count_template($count,$outputtemplate); + } elsif ($briefsum && @sumweights) { + _print_sum_count($count); + } elsif ($briefsum) { + if ($htmlstyle) {print "<p class='count'>";} + _print_count_brief($count); + if ($htmlstyle) {print "</p>\n";} + } else { + _print_count_details($count); + } + if ($htmlstyle) {print "</div>\n";} +} + +# Return count,header,... list filling in header if missing +sub __count_and_header { + my $count=shift @_; + my $header=__count_header($count); + return $count,$header,@_; +} + +# Return count title or '' if missing +sub __count_header { + my $count=shift @_; + return $count->{'title'}||''; +} + +# Print total count (sum) for a given count object +sub _print_sum_count { + my ($count,$header)=__count_and_header(@_); + if ($htmlstyle) {print "<p class='count'>".text_to_print($header).": ";} + print get_sum_count($count); + if ($htmlstyle) {print "</p>\n";} + else {print ": ".text_to_print($header);} + print "\n"; +} + +# Print brief summary of count object +sub _print_count_brief { + my ($count,$header,$tag1,$tag2)=__count_and_header(@_); + my @cnt=@{$count->{'counts'}}; + if ($htmlstyle && $tag1) {print '<',$tag1,'>';} + print $cnt[$CNT_WORDS_TEXT],'+',$cnt[$CNT_WORDS_HEADER],'+',$cnt[$CNT_WORDS_OTHER]; + for (my $i=$SIZE_CNT_DEFAULT;$i<$SIZE_CNT;$i++) { + if ($cnt[$i]) {print '+',$cnt[$i],$countkey[$i];} + } + print ' (',$cnt[$CNT_COUNT_HEADER],'/',$cnt[$CNT_COUNT_FLOAT], + '/',$cnt[$CNT_COUNT_INLINEMATH],'/',$cnt[$CNT_COUNT_DISPLAYMATH],')'; + if ($htmlstyle && $tag2) { + print '</',$tag1,'><',$tag2,'>'; + $tag1=$tag2; + } else {print ' ';} + print text_to_print($header); + if ($htmlstyle && $tag1) {print '</',$tag1,'>';} + if ($finalLineBreak) {print "\n";} +} + +# Print detailed summary of count object +sub _print_count_details { + my ($count,$header)=__count_and_header(@_); + if ($htmlstyle) {print "<ul class='count'>\n";} + if ($header) {formatprint($header."\n",'li','header');} + if (my $tex=$count->{'TeXcode'}) { + if (!defined $encoding) {formatprint('Encoding: '.$tex->{'encoding'}."\n",'li');} + } + if (@sumweights) {formatprint('Sum count: '.get_sum_count($count)."\n",'li');} + for (my $i=1;$i<$SIZE_CNT;$i++) { + formatprint($countdesc[$i].': '.get_count($count,$i)."\n",'li'); + } + if (get_count($count,$CNT_FILE)>1) { + formatprint($countdesc[$CNT_FILE].': '.get_count($count,$CNT_FILE)."\n",'li'); + } + my $subcounts=$count->{'subcounts'}; + if ($showsubcounts && defined $subcounts && scalar(@{$subcounts})>=$showsubcounts) { + formatprint("Subcounts:\n",'li'); + if ($htmlstyle) {print "<span class='subcount'>\n";} + formatprint(" text+headers+captions (#headers/#floats/#inlines/#displayed)\n",'li','fielddesc'); + foreach my $subcount (@{$subcounts}) { + print ' '; + _print_count_brief($subcount,'li'); + } + if ($htmlstyle) {print "</span>\n";} + } + if ($htmlstyle) {print "</ul>\n";} else {print "\n";} +} + +# Print summary of count object using template +sub _print_count_template { + my ($count,$header,$template)=__count_and_header(@_); + $template=~s/\\n/\n/g; + if ($htmlstyle) {$template=~s/\n/<br>/g;} + my ($subtemplate,$posttemplate); + while ($template=~/^(.*)\{SUB\?((.*?)\|)?(.*?)(\|(.*?))?\?SUB\}(.*)$/is) { + __print_count_using_template($count,$1); # $1 ~ ${^PREMATCH} + if (number_of_subcounts($count)>1) { + if (defined $3) {print $3;} + __print_subcounts_using_template($count,$4); + if (defined $6) {print $6;} + } + $template=$7; # $7 ~ ${^POSTMATCH} + } + __print_count_using_template($count,$template); +} + +# Print counts using template +sub __print_count_using_template { + my ($count,$template)=@_; + while (my ($key,$cnt)=each %key2cnt) { + $template=__process_template($template,$key,get_count($count,$cnt)); + } + $template=~s/\{VER\}/\Q$versionnumber\E/gi; + $template=__process_template($template,'WARN|WARNING|WARNINGS',number_of_distinct_warnings($count)); + $template=__process_template($template,'NWARN|NWARNING|NWARNINGS',number_of_warnings($count)); + $template=__process_template($template,'ERR|ERROR|ERRORS',$count->{'errorcount'}); + $template=__process_template($template,'SUM',get_sum_count($count)); + $template=__process_template($template,'TITLE',$count->{'title'}||''); + $template=__process_template($template,'SUB',number_of_subcounts($count)); + print $template; +} + +# Print subcounts using template +sub __print_subcounts_using_template { + my ($count,$template)=@_; + my $subcounts=$count->{'subcounts'}; + if ($template && defined $subcounts && scalar(@{$subcounts})>=$showsubcounts) { + foreach my $subcount (@{$subcounts}) { + __print_count_using_template($subcount,$template); + } + } +} + +# Process template for specific label +sub __process_template { + my ($template,$label,$value)=@_; + if ($value) { + $template=~s/\{($label)\?(.*?)(\|(.*?))?\?(\1)\}/$2/gis; + } else { + $template=~s/\{($label)\?(.*?)\|(.*?)\?(\1)\}/$3/gis; + $template=~s/\{($label)\?(.*?)\?(\1)\}//gis; + } + if (!defined $value) {$value='';} + $template=~s/\{($label)\}/$value/gis; + return $template; +} + + +###### Routines for printing parsing details + + +# Print next token +sub flush_next { + my ($tex,$style)=@_; + my $ret=undef; + if (defined $style) {set_style($tex,$style);} + if (defined $tex->{'next'}) { + $ret=print_style($tex->{'next'},$tex->{'style'},$tex->{'printstate'}); + } + $tex->{'printstate'}=undef; + $tex->{'style'}=$STYLE_BLOCK; + return $ret; +} + +# Print next token and gobble following spaces (incl. line break and comments) +sub flush_next_gobble_space { + my ($tex,$style,$state)=@_; + my $ret=flush_next($tex,$style); + if (!defined $ret) {$ret=0;} + if (!defined $state) {$state=$STATE_IGNORE;} + my $prt=($printlevel>0); + while (1) { + if ($tex->{'line'}=~s/^([ \t\f]*)(\r\n|\r|\n)([ \t\f]*)//) { + if (!$prt) { + } elsif ($printlevel>1 || $ret) { + print $1; + line_return(-1,$tex); + my $space=$3; + if ($htmlstyle) {$space=~s/ /\ /g;} + print $space; + $ret=0; + } else { + line_return(0,$tex); + } + } elsif ($tex->{'line'}=~s/^([ \t\f]+)//) { + if ($prt) {print $1;} + } + if ($tex->{'line'}=~s/^(\%+[^\r\n]*)//) { + print_style($1,'comment'); + $ret=1; + } else {return;} + } +} + +# Set presentation style +sub set_style { + my ($tex,$style)=@_; + if (!(defined $tex->{'style'} && $tex->{'style'} eq $STYLE_BLOCK)) {$tex->{'style'}=$style;} +} + +# Print text using the given style, and log state if given +sub print_style { + my ($text,$style,$state)=@_; + (($printlevel>=0) && (defined $text) && (defined $style)) || return 0; + my $colour; + ($colour=$STYLE{$style}) || return; + if (defined $colour && $colour ne '-') { + print_with_style($text,$style,$colour); + if (defined $state) {print_style($state,'state');} + if ($style ne 'cumsum') {$blankline=-1;} + return 1; + } else { + return 0; + } +} + +# Conditional line return +sub line_return { + my ($blank,$tex)=@_; + if ($blank<0 && $printlevel<2) {$blank=1;} + if ($blank<0 || $blank>$blankline) { + if ((defined $tex) && @sumweights) { + my $num=get_sum_count($tex->{'subcount'}); + print_style(" [$num]",'cumsum'); + } + linebreak(); + $blankline++; + } +} + +###### Print help on style/colour codes + + +# Print output style codes if conditions are met +sub conditional_print_style_list { + if ($showcodes) {_print_style_list();} + return $showcodes; +} + +# Print help on output styles +sub _print_style_list { + if ($printlevel<=0) {return;} + if ($htmlstyle) {print '<div class="stylehelp"><p>';} + formatprint('Format/colour codes of verbose output:','h2'); + print "\n\n"; + foreach my $style (@STYLE_LIST) { + my $desc=$STYLE_DESC{$style}; + if ($desc=~/^(.*):\s+(.*)$/) { + _help_style_line($1,$style,$2); # $1~${^PREMATCH}, $2~${^POSTMATCH} + } else { + _help_style_line($desc,$style); + } + } + if ($htmlstyle) {print '</p></div>';} + print "\n\n"; +} + +# Print one line of help +sub _help_style_line { + my ($text,$style,$comment)=@_; + if (!defined $comment) {$comment='';} + if ($htmlstyle) {$comment=' .... '.text_to_print($comment);} + else {$comment=' .... '.$comment;} + if (print_style($text,$style)) { + print $comment; + linebreak(); + } +} + +###### Help routines + + +# Print TeXcount version +sub print_version { + wprintstringdata('Version'); +} + +# Print TeXcount reference text +sub print_reference { + wprintstringdata('Reference'); +} + +# Print TeXcount licence text +sub print_license { + wprintstringdata('License'); +} + +# Print short TeXcount help +sub print_short_help { + wprintstringdata('ShortHelp'); +} + +# Print TeXcount options list +sub print_syntax { + wprintstringdata('OptionsHead'); + wprintstringdata('Options','@ - :'); +} + +# Prinst TeXcount options containing substring +sub print_syntax_subset { + my $pattern=shift @_; + my $data=StringData('Options'); + if (!defined $data) { + error($Main,'No StringData Options.','BUG'); + return; + } + my @options; + foreach (@$data) { + if (/^\s*([^\s]+\s)*[^\s]*\Q$pattern\E/) {push @options,$_;} + } + if (scalar(@options)==0) {print "No options contained $pattern.\n";} + else { + print "Options containing \"$pattern\":\n\n"; + wprintlines('@ - :',@options); + } +} + +# Print complete TeXcount help +sub print_help { + print_help_title(); + print_syntax(); + print_help_text(); + print_reference(); +} + +# Print help title +sub print_help_title { + wprintstringdata('HelpTitle'); +} + +# Print help text +sub print_help_text { + wprintstringdata('HelpText'); + wprintstringdata('TCinstructions'); +} + +# Print help on specific macro or environment +sub print_help_on_rule { + my $arg=shift @_; + my $def; + my %rules=( + '\documentclass' => 'Initiates LaTeX document preamble.', + '\begin' => 'Treatmend depends on environment handling rules.', + '\def' => 'Excluded from count.', + '\verb' => 'Strong exclude for enclosed region.', + '$' => 'Opens or closes inlined equation', + '$$' => 'Opens or closes displayed equation.', + '\(' => 'Opens inlined equation.', + '\)' => 'Closes inlined equation initiated by \(.', + '\[' => 'Opens displayed equation.', + '\]' => 'Closes displayed equation initiated by \[.'); + if (!defined $arg || $arg=~/^\s*$/) { + print "\nSpecify macro or environment name after the -h= option.\n"; + return; + } + if ($arg=~/^[\w\-\%]+:/) { + remove_all_rules(); + %rules=(); + while ($arg=~s/^([\w\-\%]+)://) {include_package($1);} + } + if ($def=$rules{$arg}) { + print "\nSpecial rule (hard coded) for $arg\n"; + print $def."\n"; + } elsif ($arg=~/^\\/) { + my $hasrule=0; + print "\nRule(s) for macro $arg\n"; + if ($def=$TeXmacrocount{$arg}) { + $hasrule=1; + } + if ($def=$TeXfileinclude{$arg}) { + $hasrule=1; + print "Takes file name as parameter which is included in document.\n"; + } + if ($def=$TeXmacro{$arg}) { + $hasrule=1; + _print_rule_macro($arg,$def); + } + if ($def=$TeXfloatinc{$arg}) { + $hasrule=1; + print "\nIncluded inside floats:\n"; + _print_rule_macro($arg,$def); + } + if (!$hasrule) { + print "\nNo macro rule defined for $arg.\nParameters treated as surrounding text.\n"; + } + } else { + if ($def=$TeXenvir{$arg}) { + print "\nRule for environment $arg\n"; + _print_rule_macrocount($PREFIX_ENVIR.$arg); + _print_rule_envir($arg,$def); + } else { + print "\nNo default environment rule defined for $arg.\nContent handled as surrounding text.\n"; + } + } +} + +# Print macro handling rule +sub _print_rule_macro { + my ($arg,$def)=@_; + if (ref($def) eq 'ARRAY') { + my $optionflag=0; + print "Takes the following parameter(s):\n"; + foreach my $state (@{$def}) { + if ($state==$_STATE_OPTION) {$optionflag=1;} + elsif ($optionflag) { + $optionflag=0; + print " - Optional [] containing $state2desc{$state}\n"; + } else { + print " - $state2desc{$state}\n"; + } + } + } else { + print "Takes $def parameter(s): content ignored, i.e. not included in counts.\n"; + } +} + +# Print environment handling rule +sub _print_rule_envir { + my ($arg,$def)=@_; + print "Contents parsed as $state2desc{$def}\n"; + if ($def=$TeXmacro{$PREFIX_ENVIR.$arg}) { + _print_rule_macro($def); + } +} + +# Print macrocount rule, return rule +sub _print_rule_macrocount { + my $arg=shift @_; + my $def=$TeXmacrocount{$arg}; + if (!defined $def) {return undef;} + if (ref($def) eq 'ARRAY') { + my @wd=@$def; + foreach (@wd) {$_=$countdesc[$_];} + print 'Increments the following counters: ',join('; ',@wd),".\n"; + } else { + print "Counted as $def word(s).\n"; + } +} + +# Print style or style category summary +sub print_help_on_styles { + my $style=shift @_; + if (defined $style) { + if (my $def=$STYLES{$style}) { + _print_help_on_style_category($style); + print wrap('',' ',"$style = ".join(', ',grep(/^\w+$/,sort keys %$def))),".\n"; + } elsif ($def=$STYLE_DESC{$style}) { + print $style,' = ',$def,"\n"; + } else { + print "Unknown style or style category: $style.\n"; + } + } else { + print wrap('',' ','Styles: '.join(', ',@STYLE_LIST)),".\n\n"; + print wrap('',' ','Style categories: '.join(', ',sort grep(/^\w+$/,keys %STYLES))),".\n\n"; + print "Use -help-style={style} to get help on particular style or style category.\n"; + } +} + +# Print help on a particular style category (optional prefix and indentation) +sub _print_help_on_style_category { + my ($cat,$prefix,$indent)=@_; + if (!defined $prefix) {$prefix='';} + if (!defined $indent) {$indent=' ';} +} + +###### HTML routines + + +# Print HTML header +sub html_head { + if (defined $HTMLfile) { + if (open(FH,$HTMLfile)) { + my $text=join('',<FH>); + close FH; + if ( $text=~/^(.*)<!--+\s*texcount\s*--+>(.*)$/is + || $text=~/^(.*)\$\{texcount\}(.*)$/is + || $text=~/^(.*)(<\/body.*)$/is + ) { + $htmlopen=$1; + $htmlclose=$2; + } else { + error($Main,"Invalid HTML template format."); + } + } else { + error($Main,"HTML template file not found: $HTMLfile"); + } + } + if (defined $htmlopen) { + $htmlopen=~s/\$\{encoding\}/\Q$outputEncoding\E/g; + $htmlopen=~s/\$\{version\}/@{[_html_version()]}/g; + print $htmlopen; + $htmlopen=undef; + } else { + print "<html>\n<head>\n"; + print "<meta http-equiv='content-type' content='text/html; charset=$outputEncoding'>\n"; + _print_html_style(); + foreach (@htmlhead) {print $_,"\n";} + print "</head>\n\n<body>\n"; + print "\n<h1>LaTeX word count"; + if ($showVersion>0) {print ' (version ',_html_version(),')'} + print "</h1>\n"; + } +} + +# Print HTML tail +sub html_tail { + if (defined $htmlclose) { + print $htmlclose; + $htmlclose=undef; + } else { + print "</body>\n\n</html>\n"; + } +} + +# Return version number using HTML +sub _html_version { + my $htmlver=$versionnumber; + $htmlver=~s/\b(alpha|beta)\b/&$1;/g; + return $htmlver; +} + +# Print HTML STYLE element +sub _print_html_style { +if (defined $CSShref) { + print "<link rel='stylesheet' href='$CSShref' type='text/css'>\n"; + return; +} +if (defined $CSSfile) { + if (open(FH,$CSSfile)) { + print "<style>\n<!---\n",<FH>,"-->\n</style>\n"; + close(FH); + return; + } else { + error($Main,"CSS file not found: $CSSfile"); + } +} +print <<END +<style> +<!-- +body {width:auto;padding:5;margin:5;} +.error {font-weight:bold;color:#f00;font-style:italic; border: solid 1px red;} +.word,.hword,.oword,.altwd {border-left: 1px solid #CDF; border-bottom: 1px solid #CDF;} +.word {color: #009} +.hword {color: #009; font-weight: 700;} +.oword {color: #009; font-style: italic;} +.altwd {color: #00c; font-style: oblique;} +.cmd {color: #c00;} +.exclcmd {color: #f99;} +.option {color: #cc0;} +.optparm {color: #c90;} +.envir, .document {color: #900; font-weight:bold;} +.math {color: #6c0;} +.mathgroup {color: #090;} +.exclmath {color: #9f6;} +.mathcmd {color: #6c0;} +.ignore {color: #999;} +.exclenv {color:#c66;} +.tc {color: #999; font-weight:bold;} +.comment {color: #999; font-style: italic;} +.state {color: #990; font-size: 70%;} +.cumsum {color: #999; font-size: 80%;} +.fileinc {color: #696; font-weight:bold;} +.warning {color: #c00; font-weight: 700;} + +div.filegroup, div.parse, div.stylehelp, div.count, div.sumcount, div.error { + border: solid 1px #999; margin: 4pt 0pt; padding: 4pt; +} +div.stylehelp {font-size: 80%; background: #fffff0; margin-bottom: 16pt;} +div.filegroup {background: #dfd; margin-bottom: 16pt;} +div.count {background: #ffe;} +div.sumcount {background: #cec;} +div.error {background: #fcc;} +.parse {font-size: 80%; background: #f8fff8; border-bottom:none;} + +ul.count {list-style-type: none; margin: 4pt 0pt; padding: 0pt;} +.count li.header {font-weight: bold; font-style: italic;} +.subcount li.header {font-weight: normal; font-style: italic;} +.subcount li {margin-left: 16pt; font-size: 80%;} +.fielddesc {font-style: italic;} +.nb {color: #900;} + +table.stat {border:2px solid #339; margin:0 0 8pt 0; background:#666; border-collapse:collapse;} +table.stat tr {border:1px solid #CCC;} +table.stat th, table.stat td {padding:1pt 4pt;} +table.stat col {padding:4pt;} +table.stat thead {background: #CCF;} +table.stat tbody {border:1px solid #333;} +table.stat tbody.sumstat {background:#FFC;} +table.stat tbody.langstat {background:#FEE;} +table.stat tbody.wordstat {background:#EEF;} +table.stat tbody.macrostat {background:#EFE;} +table.stat .sum {font-weight:bold; font-style:italic;} +--> +</style> +END +} + +###### Read text data + + + +# Return the STRINGDATA hash (lazy instantiation) +sub STRINGDATA { + if (!defined $STRINGDATA) { + my @DATA=<DATA>; + foreach (@DATA) { + $_=~s/\$\{(\w+)\}/__apply_globaldata($1)/ge; + } + $STRINGDATA=splitlines(':{3,}\s*(\w+)?',\@DATA); + } + return $STRINGDATA; +} + +# Return value from STRINGDATA hash +sub StringData { + my $name=shift @_; + return STRINGDATA()->{$name}; +} + +# Insert value from GLOBALDATA +sub __apply_globaldata { + my $name=shift @_; + if (my $value=$GLOBALDATA{$name}) { + return $value; + } + return '[['.$name.']]'; +} + +# Print value from STRINGDATA using wprintlines +sub wprintstringdata { + my $name=shift @_; + my $data=StringData($name); + if (!defined $data) { + error($Main,"No StringData $name.",'BUG'); + return; + } + wprintlines(@_,@$data); +} + +# Divide array of lines by identifying headers +sub splitlines { + my ($pattern,$lines)=@_; + if (!defined $lines) {return;} + my $id=undef; + my %hash; + foreach my $line (@$lines) { + if ($line=~/^$pattern$/) { + $id=$1; + if (defined $id) { + $hash{$id}=[]; + if (defined $2) {push @{$hash{$id}},$2;} + } + } elsif (defined $id) { + chomp $line; + push @{$hash{$id}},$line; + } + } + return \%hash; +} + +# Print string with word wrapping and indentation using +# wprintlines. +sub wprint { + my $text=shift @_; + my @lines=split(/\n/,$text); + wprintlines(@lines); +} + +# Print with word wrapping and indentation. A line with +# @ - : +# sets two column tabs. A tab or multiple spaces is taken +# to indicate indentation. If the first column value is +# only a single '|', this is just an indication to skip +# one tab. +sub wprintlines { + my @lines=@_; + my $ind1=2; + my $ind2=6; + my $i; + foreach my $line (@lines) { + if ($line=~s/^@//) { + $ind2=1+index($line,':'); + $ind1=1+index($line,'-'); + if ($ind1<1) {$ind1=$ind2;} + next; + } + my $firstindent=0; + if ($line=~s/^(\t|\s{2,})(\S)/$2/) {$firstindent=$ind1;} + my $indent=$firstindent; + if ($line=~/^(.*\S)(\t|\s{2,})(.*)$/) { + $indent=$ind2; + if ($1 eq '|') {$line=' ';} + else {$line=$1.' ';} + $i=$indent-$firstindent-length($line); + if ($i>0) {$line.=' ' x $i;} + $line.=$3; # $3~${^POSTMATCH} + } + print wrap(' ' x $firstindent,' ' x $indent,$line)."\n"; + } +} + + +######################################## +##### TEXT DATA + +__DATA__ + +:::::::::::::::::::::::::::::::::::::::: +:::::::::: Version +TeXcount version ${versionnumber}, ${versiondate}. + +:::::::::: Reference +The TeXcount script is copyright of ${maintainer} (${copyrightyears}) and published under the LaTeX Project Public Licence. + +Go to the TeXcount web page + ${website} +for more information about the script, e.g. news, updates, help, usage tips, known issues and short-comings, or to access the script as a web application. Feedback such as problems or errors can be reported to einarro@ifi.uio.no. + +:::::::::: License +TeXcount version ${versionnumber} + +Copyright ${copyrightyears} ${maintainer} + +The TeXcount script is published under the LaTeX Project Public License (LPPL) + http://www.latex-project.org/lppl.txt +which grants you, the user, the right to use, modify and distribute the script. However, if the script is modified, you must change its name or use other technical means to avoid confusion. + +The script has LPPL status "maintained" with ${maintainer} being the current maintainer. + +:::::::::::::::::::::::::::::::::::::::: +:::::::::: ShortHelp +Syntax: texcount.pl [options] files + +Use option -help (or just -h) to get help; -help-options (-hopt) to get list of command line options, or -help-options=substring for help on all options containing substring. + +:::::::::::::::::::::::::::::::::::::::: +:::::::::: HelpTitle +*************************************************************** +* TeXcount version ${versionnumber}, ${versiondate} +* + +Count words in TeX and LaTeX files, ignoring macros, tables, formulae, etc. + +:::::::::::::::::::::::::::::::::::::::: +:::::::::: OptionsHead + +Syntax: texcount.pl [options] files + +Options: + +:::::::::: OptionsPrefix +@ - : +:::::::::: Options + -relaxed Uses relaxed rules for word and option handling: i.e. allows more general cases to be counted as either words or macros. + -restricted Restricts the rules for word and option handling. + -v Verbose (same as -v3). + -v0 Do not present parsing details. + -v1 Verbose: print parsed words, mark formulae. + -v2 More verbose: also print ignored text. + -v3 Even more verbose: include comments and options. + -v4 Same as -v3 -showstate. + -v=, v0=, ..., -v4= Set verbosity by adding/removing particular types of token (styles) to include in the verbose output. Use -help-style to get details of which tokens are included in each style, and of classes of tokens (style categories). + -showstate Show internal states (with verbose). + -brief Only prints a brief, one line summary of counts. + -q, -quiet Quiet mode, no error messages. Use is discouraged! + -strict Strict mode, warns against begin-end groups for which rule are not defined. + -sum, -sum= Make sum of all word and equation counts. May also use -sum=#[,#] with up to 7 numbers to indicate how each of the counts (text words, header words, caption words, #headers, #floats, #inlined formulae, #displayed formulae) are summed. The default sum (if only -sum is used) is the same as -sum=1,1,1,0,0,1,1. + -nosum Do not compute sum. + -sub, -sub= Generate subcounts. Option values are none, part, chapter, section or subsection. Default (-sub) is set to subsection, whereas unset is none. (Alternative option name is -subcount.) + -nosub Do not generate subcounts. + -col Use ANSI colours in text output. + -nc, -nocol No colours (colours require ANSI). + -nosep, -noseparator No separating character/string added after each word (default). + -sep=, -separator= Separating character or string to be added after each word. + -html Output in HTML format. + -htmlcore Only HTML body contents. + -htmlfile= HTML template file to use with the -html option. Use <!-- TeXcode --> to indicate where the output from TeXcount should be inserted. + -tex Encode TeX special characters for output into TeX code + -cssfile= CSS file to include instead of default styles. + -css= CSS href to include instead of default styles. Can use -css=file:{filename} instead of -cssfile={filename}. + -opt=, -optionfile= Read options/parameters from file. + - Read LaTeX code from STDIN. + -inc Parse included TeX files (as separate file). + -merge Merge included TeX files into code (in place). + -noinc Do not include included tex files (default). + -incbib Include bibliography in count, include bbl file if needed. + -nobib Do not include bibliography in count (default). + -incpackage= Include rules for the given package. + -total Do not give sums per file, only total sum. + -1 Same as -brief and -total. Ensures there is only one line of output. If used in conjunction with -sum, the output will only be the total number. (NB: Character is the number one, not the letter L.) + -template= Speficy an output template. Use {1},...,{7}, {SUM} and {TITLE} to include values, {1?...?1} etc. to conditionally include sections, {1?....|...?1} etc. to specify an alternative text if zero. To include subcounts, use {SUB?...?SUB} where ... is replaced with the template to use per subcount. Line shift may be specified using \\n. + -dir, -dir= Specify the working directory using -dir=path. Remember that the path must end with \\ or /. If only -dir is used, the directory of the provided file is used and paths (e.g. included files) are assumed to be absolute or relative to this. The default at startup is -dir=. which means that the directory from which TeXcount is run is the working directory with all paths absolute or relative to this. + -auxdir, -auxdir= Specify directory for auxilary files, e.g. the bibliograph (.bbl) file. If only -auxdir is used, the working directory (as determined by -dir or -dir=) is assumed. If -auxdir= is used with -dir=, it sets the path to the auxilary directiry. If -auxdir= is used with -dir, the working directory is determined from the location of the provided file, and the path to the auxilary directory is assumed to be absolute or relative to this. + -enc=, -encoding= Specify encoding (default is to guess the encoding). + -utf8, -unicode Selects Unicode (UTF-8) for input and output. This is automatic with -chinese, and is required to handle e.g. Korean text. Note that the TeX file must be save in UTF-8 format (not e.g. GB2312 or Big5), or the result will be unpredictable. + -alpha=, -alphabets= List of Unicode character groups (or digit, alphabetic) permitted as letters. Names are separated by ',' or '+'. If list starts with '+', the alphabets will be added to those already included. The default is Digit+alphabetic. + -logo=, -logograms= List of Unicode character groups interpreted as whole word characters, e.g. Han for Chinese characters. Names are separated by ',' or '+'. If list starts with '+', the alphabets will be added to those already included. By default, this is set to include Ideographic, Katakana, Hiragana, Thai and Lao. + -ch, -chinese, -zhongwen Turns on support for Chinese characters. TeXcount will then count each Chinese character as a word. + -jp, -japanese Turns on support for Japanese characters. TeXcount will count each Japanese character (kanji, hiragana, and katakana) as one word, i.e. not do any form of word segmentation. + -kr, -korean Turns on support for Korean. This will count hangul and han characters, i.e. with no word separation. NB: Experimental! + -kr-words, -korean-words Turns on support for Korean words, i.e. hangul words separated by characters. Han characters are still counted as characters. NB: Experimental! + -ch-only, ..., -korean-words-only As options -chinese, ..., -korean-words, but also excludes letter-based words or trims down the character set to the minimum. + -char, -character, -letters Counts letters/characters instead of words. Note that spaces and punctuation is not counted. + -char-only, ..., -letters-only Like -letters, but counts alphabetic letters only. + -countall, -count-all The default setting in which all characters are included as either alphabets og logograms. + -freq, -freq= Produce individual word frequency table. Optionally give minimal number of occurences to be listed. + -stat Produce statistics on language/script usage. + -macrostat, -macrofreq Print macro usage statistics. + -codes Display output style code overview and explanation. This is on by default. + -nocodes Do not display output style code overview. + -out= Write output to file, give filename as option value. + -h, -?, -help, /? Help text. + -h=, -?=, -help=, /?= Takes a macro or group name as option and returns a description of the rules for handling this if any are defined. If handling rule is package specific, use -incpackage=package name: -incpackage must come before -h= on the command line to take effect. + -help-options, -h-opt List all options. + -help-options=, -h-opt= List all options containing the provided string, e.g. -h-opt=dir or -h-opt=-v (the initial - in -v causes only options starting with v to be listed). + -help-style List the styles and style categories: i.e. those permitted used with -v={styles-list}. + -help-style= Give description of style or style category. + -ver, -version Print version number. + -lic, -license, -licence Licence information. + +:::::::::::::::::::::::::::::::::::::::: +:::::::::: HelpText +The script counts words as either words in the text, words in headers/titles or words in floats (figure/table captions). Macro options (i.e. \\macro[...]) are ignored; macro parameters (i.e. \\macro{...}) are counted or ignored depending on the macro, but by default counted. Begin-end groups are by default ignored and treated as 'floats', though some (e.g. center) are counted. + +Mathematical formulae are not counted as words, but are instead counted separately with separate counts for inlined formulae and displayed formulae. Similarly, the number of headers and the number of 'floats' are counted. Note that 'float' is used here to describe anything defined in a begin-end group unless explicitly recognized as text or mathematics. + +The verbose options (-v1, -v2, -v3, showstate) produces output indicating how the text has been interpreted. Check this to ensure that words in the text has been interpreted as such, whereas mathematical formulae and text/non-text in begin-end groups have been correctly interpreted. + +Summary, as well as the verbose output, may be produced as text (default) or as HTML code using the -html option. The HTML may then be sent to file which may be viewed with you favourite browser. + +Under UNIX, unless -nocol (or -nc) has been specified, the output will be colour coded using ANSI colour codes. Counted text is coloured blue with headers are in bold and in HTML output caption text is italicised. Use 'less -r' instead of just 'less' to view output: the '-r' option makes less treat text formating codes properly. Windows does not support ANSI colour codes, and so this is turned off by default. + +:::::::::: TCinstructions +Parsing instructions may be passed to TeXcount using comments in the LaTeX files on the format +@ - : + %TC:instruction arguments +and are used to control how TeXcount parses the document. The following instructions are used to set parsing rules which will apply to all subsequent parsing (including other files): + %TC:macro [macro] [param.states] + | macro handling rule, no. of and rules for parameters + %TC:macroword [macro] [number] + | macro counted as a given number of words + %TC:header [macro] [param.states] + | header macro rule, as macro but counts as one header + %TC:breakmacro [macro] [label] + | macro causing subcount break point + %TC:group [name] [param.states] [content-state] + | begin-end-group handling rule + %TC:floatinclude [macro] [param.states] + | as macro, but also counted inside floats + %TC:preambleinclude [macro] [param.states] + | as macro, but also counted inside the preamble + %TC:fileinclue [macro] [rule] + | file include, add .tex if rule=2, not if rule=0, if missing when rule=1 +The [param.states] is used to indicate the number of parameters used by the macro and the rules of handling each of these: the format is [#,#,...,#] with # representing one number for each parameter giving the parsing state to use for that parameter, alternatively just a single number (#) indicating how many parameters to ignore (parsing state 0). The option [content-state] is used to give the parsing state to use for the contents of a begin-end group. The main parsing states are 0 to ignore and 1 to count as text. + +Parsing instructions which may be used anywhere are: +@ - : + %TC:break [title] add subcount break point here + %TC:incbib include bibliography (same as running with -incbib) + %TC:ignore ignore region, end with %TC:endignore + %TC:insert [code] insert code for TeXcount to process as TeX code + %TC:newtemplate start a new template, ie delete the existing one + %TC:template [template] add another line to the template specification +See the documentation for more details. + +Command line options and most %TC commands (prefixed by % rather than %TC:) may be placed in an options file. This is particularly useful for defining your own output templates and macro handling rules. + +:::::::::::::::::::::::::::::::::::::::: + |