Spiral

Spiral is a Python module that provides several different functions for splitting identifiers found in source code files. The name Spiral is a loose acronym based on "SPlitters for IdentifieRs: A Library".

Authors: Michael Hucka
Repository: https://github.com/casics/spiral
License: Unless otherwise noted, this content is licensed under the GPLv3 license.

🏁 Recent news and activities

April 2018: Version 1.1.0 fixes a bug that prevented importing Spiral, and another bug that cause setup.py to fail to install dependencies automatically. Additional enhancements include improved command-line help and internal code refactoring.

Table of Contents

• Introduction
• Please cite the paper
• Installation instructions
• Basic operation
• Performance of Ronin
• Other splitters in Spiral
• Limitations
• More information
• Getting help and support
• Contributing — info for developers
• Acknowledgments

☀ Introduction

Spiral is a Python 3 package that implements numerous identifier splitting algorithms. Identifier splitting (also known as identifier name tokenization) is the task of breaking apart program identifier strings such as getInt or readUTF8stream into component tokens: [get, int] and [read, utf8, stream]. The need for splitting identifiers arises in a variety of contexts, including natural language processing (NLP) methods applied to source code analysis and program comprehension. Spiral provides some basic naive splitting algorithms, such as a straightforward camel-case splitter, as well as more elaborate heuristic splitters, such as a new algorithm called Ronin.

♥️ Please cite the Spiral paper and the version you use

Article citations are critical for academic developers. If you use Spiral and you publish papers about your software, we ask that you please cite the Spiral paper:

Hucka, M. (2018). Spiral: splitters for identifiers in source code files. Journal of Open Source Software, 3(24), 653, https://doi.org/10.21105/joss.00653

Please also indicate the specific version of Spiral you use, to improve other people's ability to reproduce your results. You can use the Zenodo DOIs we provide for this purpose:

• Spiral release 1.0.1 ⇒ 10.5281/zenodo.1211835

✺ Installation instructions

For basic usage, the following is probably the simplest and most direct way to install Spiral on your computer:

sudo pip3 install git+https://github.com/casics/spiral.git

Alternatively, you can clone this GitHub repository and then run setup.py:

git clone https://github.com/casics/spiral.git
cd spiral
sudo python3 -m pip install .

The above should be all you need to run Spiral out of the box. If you plan on experimenting with alternative parameter values or alternative dictionaries, you will additionally need to install the following:

• Platypus-Opt, an optimization library (if you want to optimize parameters). Important: This Platypus is not the same as the other package called "Platypus" in PyPI. Make sure to get Platypus-Opt or install from the Platypus repo.

• Data modules from NLTK, particularly nltk_words and ntlk_wordnet from the nltk.corpus module and the nltk.stem module. These cannot be installed automatically by setup.py—please refer to the NLTK installation instructions for instructions on how to do it.

▶︎ Basic operation

Spiral is extremely easy to use. To use a Spiral splitter in a Python program, simply import a splitter and then call it.

from spiral.simple_splitters import pure_camelcase_split
print(pure_camelcase_split('TestString'))

Some splitters take optional parameters, and the more complex splitters have an init() function that you can optionally call to set additional parameters or load data. Currently, only Ronin and Samurai have initialization functions, and calling init() is optional—if you do not call it, Ronin and Samurai will call their init() functions automatically.

Here are some examples of using the Ronin splitter algorithm. The following input,

from spiral import ronin
for s in [ 'mStartCData', 'nonnegativedecimaltype', 'getUtf8Octets', 'GPSmodule', 'savefileas', 'nbrOfbugs']:
    print(ronin.split(s))

produces the following output:

['m', 'Start', 'C', 'Data']
['nonnegative', 'decimal', 'type']
['get', 'Utf8', 'Octets']
['GPS', 'module']
['save', 'file', 'as']
['nbr', 'Of', 'bugs']

Spiral also includes a command-line program named spiral in the bin subdirectory; it will split strings provided on the command line or in a file. (Note: Ronin and Samurai load internal data files when they start up. In normal use, called from an application program, the resulting startup delay will only happen once. In the command-line program, the data is reloaded at every invocation, causing a startup delay every time. The delay is not typical for normal Spiral usage.)

By the way, if your goal is to split identifiers obtained during source code mining and you need to filter out gibberish strings of characters before attempting to split them, you may find Nostril (the Nonsense String Evaluator) useful.

🎯 Performance of Ronin

Splitting identifiers is deceptively difficult. It is a research-grade problem for which no perfect solution exists. Even in cases where the input consists of identifiers that strictly follow conventions such as camel case, ambiguities can arise. For example, correctly splitting J2SEProjectTypeProfiler into J2SE, Project, Type, Profiler requires the reader to recognize J2SE as a unit. The task of splitting identifiers is made more difficult when there are no case transitions or other obvious boundaries in an identifier. And then there is the small problem that humans are often simply inconsistent!

Ronin is an advanced splitter that uses a variety of heuristic rules, English dictionaries, and tables of token frequencies obtained from mining source code repositories. Ronin includes a default table of term frequencies derived from an analysis of over 46,000 randomly selected software projects in GitHub that contained at least one Python source code file. The tokens were extracted using software from Spiral's parent project, CASICS (specifically, the extractor package), and the frequency table constructed using a procedure encoded in the small program create_frequency_file included with Spiral. Ronin also has a number of parameters that need to be tuned; this can be done using the optimization program optimize-ronin in the dev/optimization subdirectory. The default parameter values were derived by optimizing performance against two data sets available from other research groups:

1. The Loyola University of Delaware Identifier Splitting Oracle (Ludiso)
2. The INTT data set, extracted from the zip archive of INTT

Spiral includes copies of these data sets in the tests/data subdirectory. The parameters derived primarily by running against the INTT database of 18,772 identifiers and their splits. The following table summarizes the results:

Many of the "failures" against these sets of identifiers are actually not failures, but cases where Ronin gets a more correct answer or where there is a legitimate difference in interpretation. Here are some examples:

Names like fread could be considered appropriate to leave alone, but the f in fread actually stands for "file", and thus IMHO it makes sense to split it—splitting identifiers into meaningful tokens is the purpose of Ronin, after all. Conversely, tokens such as Utf8 should be left as units because they are meaningful. Differences involving some other terms such as color space are due to Ronin splitting terms that are typically considered separate words but are treated as one word in Ludiso, or vice versa. (The main entry in Wikipedia for "color space" is two words, while for "filename" it is one word.) This sometimes also arises because Ronin recognizes prefixes and sometimes does not split words because they would not be in normal written English, such as nonblocking instead of non blocking. Still other differences between Ronin's output and the expected splits given in Ludiso and INTT are due to inconsistencies in the Ludiso and INTT data sets. For example, sometimes a token within a larger identifier is split in one case but not in another. Finally, some other differences are cases where the Ludiso split seems to favor program-specific names. For example, QWheelEvent is split as [ QWheel, Event ] whereas Ronin will split it as [ Q, Wheel, Event ].

So, Ronin's performance may be better than the pure numbers imply. However, without checking every Ludiso case and manually reinterpreting the splits, we can't say for sure. All that aside, Ronin definitely gets many cases wrong.

⚙️ Other splitters in Spiral

Here is a list of all the splitters implemented in Spiral at this time:

The following table illustrates the differences between the simpler splitters.

Ronin and Samurai are more advanced than any of the simple splitters above. Ronin in particular does everything heuristic_splitter does, but handles far more difficult cases.

⚠️ Limitations

Ronin is the most advanced splitter in the Spiral package, but it is not perfect by any means. Certain limitations are known:

• Ronin is tuned for splitting program identifiers, which is not the same as splitting strings of concatenated words. With its default parameter values, it is not good at splitting strings like driveourtrucks which are not composed of tokens commonly encountered in source code contexts. (By the way, reducing the value of parameter low_freq_cutoff in ronin.init() to something like 10 makes Ronin more likely to split concatenated word strings like driveourtrucks or societynamebank. However, this comes at the expense of worsening scores on the INTT and Ludiso data sets.)

• Ronin was trained on source code repositories containing Python code. The default frequency table may not be optimal for splitting things that come from non-Python source files, although the test results on Ludiso and INTT show that it is pretty good on non-Python content.

• Spiral includes a set of predefined special terms in the file constants.py so that heuristic_split and the Ronin splitter can recognize certain character sequences that should be treated as units. Examples include utf8, ipv4, checkbox, J2SE and others. The use of specialized dictionaries like this is an approach taken in other splitters (including Simon Butler's INTT) and is the only known way to prevent incorrect splits of special terms, but it is also an obvious limitation: the list is surely incomplete (and can never be complete because new terms are invented all the time), and may even cause incorrect splits in some contexts.

• The current algorithm implemented in Ronin arose organically after a lot of trial-and-error experimentation, and as a consequence, is rather gnarly and difficult to explain. The implementation could be made more efficient and the algorithm clarified by a nice about of refactoring.

📚 More information

The name "Ronin" is a play on the use of the name "Samurai" by Enslen et al. (2009) for their identifier splitting algorithm. The core loop of Ronin is based on Samurai, but it would be inappropriate to call this implementation Samurai too. In an effort to imply the lineage of this modified algorithm, I chose "Ronin" (a name referring to a drifter samurai without a master, during the Japanese feudal period).

I implemented Samurai based on the description of the algorithm published in the following paper, then modified it repeatedly in an attempt to improve performance. Ronin is the result. A goal for Ronin was to produce a splitter that had good performance even without using a local frequency table.

The Ludiso data set is described in the following paper:

The INTT splitter and data set are described in the following paper:

⁇ Getting help and support

If you find an issue, please submit it in the GitHub issue tracker for this repository.

♬ Contributing — info for developers

A lot remains to be done on CASICS in many areas. We would be happy to receive your help and participation if you are interested. Please feel free to contact the developers either via GitHub or the mailing list [email protected].

Everyone is asked to read and respect the code of conduct when participating in this project.

❤️ Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant Number 1533792 (Principal Investigator: Michael Hucka). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

The installation of Spiral includes a dictionary containing words from the NTLK project, specifically the words and wordnet dictionaries. The word dictionary is public domain, but the wordnet dictionary is copyright 2006 by Princeton University and made available under license terms that permit free redistribution. For more information, please see Princeton University "About WordNet" (Princeton University, 2010).

The vector artwork of the segmented spiral illusion at the top of this page was obtained from www.123freevectors.com.