DynLex Dynamically Configurable Lexer Library
This is the main repository of the SilentByte DynLex Lexer Library.
DynLex is an easy-to-use library for PHP that provides the functionality to create and use dynamically configurable lexers accessed via a fluid interface.
Official documentations can be found here: http://docs.silentbyte.com/dynlex
Installation
To install the latest version, either checkout and include the source directly or use:
$ composer require silentbyte/sb-dynlexGeneral Usage
DynLex allows the definition of a set of lexer rules that determine how the input is scanned and what tokens can be created. The following code is a simple example that tokenizes words and numbers:
<?php
use SilentByte\DynLex\DynLexUtils;
use SilentByte\DynLex\DynLexBuilder;
$input = "Hello world 8273 this 919 28 is a 12 39 44 string"
. "consisting of 328 words 003 and numbers 283";
$lexer = (new DynLexBuilder())
->rule('[a-zA-z]+', 'word')
->rule('[0-9]+', 'number')
->skip('.')
->build();
$tokens = $lexer->collect($input);
DynLexUtils::dumpTokens($tokens);
// [Output]
// -------------------------------------
// tag off ln col value
// -------------------------------------
// word 0 1 1 Hello
// word 6 1 7 world
// number 12 1 13 8273
// word 17 1 18 this
// number 22 1 23 919
// ...
?>DynLex also allows the specification of lexer actions that will be executed each time the associated token is matched in the input stream. Extending the previous example, we can implement a program that counts the number of words and numbers within the input stream:
<?php
use SilentByte\DynLex\DynLexUtils;
use SilentByte\DynLex\DynLexBuilder;
$words = 0;
$numbers = 0;
$input = "hello world 8273 this 919 28 is a 12 39 44 string"
. "consisting of 328 words 003 and numbers 283";
$lexer = (new DynLexBuilder())
->rule('[a-z]+', 'word', function() use (&$words) { $words++; })
->rule('[0-9]+', 'number', function() use (&$numbers) { $numbers++; })
->skip('.')
->build();
$tokens = $lexer->collect($input);
DynLexUtils::dumpTokens($tokens);
echo "$words words found.\n";
echo "$numbers numbers found.\n";
// [Output]
// -------------------------------------
// tag off ln col value
// -------------------------------------
// word 0 1 1 hello
// word 6 1 7 world
// number 12 1 13 8273
// word 17 1 18 this
// number 22 1 23 919
// ...
// -------------------------------------
// 11 words found.
// 9 numbers found.
?>Using this concept, it is possible to easily create lexers for different kinds of applications. A more elaborate example that demonstrates how to use DynLex to create HTML syntax highlighters for programming languages can be found under examples/04-syntax-highlighting.php.
It is generally advised to check out the examples folder for further information and examples on how to use DynLex. Also have a look into the source code for more detailed documentation.
Contributing
See CONTRIBUTING.md.
FAQ
Under what license is DynLex released?
MIT license. Check out license.txt for details. More information regarding the MIT license can be found here: https://opensource.org/licenses/MIT
Why do rules sometimes not get matched correctly?
You have to ensure that rules that may conflict with each other are listed in the correct order from most specific to most general. For example, if you want to tokenize integers ([0-9]+) and floats ([0-9]+\.[0-9]+), the rule for floats must be listed before the rule for integers because the integer rule matches the first part of the float rule.