In this example, the use of a tokenizer function explains how the process of separating a string into tokens works. A programmer is usually interested in tokens because they serve as a raw input for terminal shells.
NCN Project, Purdue University
Cite this work
Researchers should cite this work as follows:
David A Saenz (2012), "Playing with tokens in C++," http://nanohub.org/resources/14159.
nanoHUB.org, a resource for nanoscience and nanotechnology, is supported by the National Science Foundation and other funding agencies. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.