All Projects → renatahodovan → Grammarinator

renatahodovan / Grammarinator

Licence: other
ANTLR v4 grammar-based test generator

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Grammarinator

Fuzzinator
Fuzzinator Random Testing Framework
Stars: ✭ 164 (+1.23%)
Mutual labels:  hacktoberfest, fuzzing, test-automation
Beanmother
A library for setting up Java objects as test data.
Stars: ✭ 102 (-37.04%)
Mutual labels:  hacktoberfest, test-automation
Os Autoinst
OS-level test automation
Stars: ✭ 99 (-38.89%)
Mutual labels:  hacktoberfest, test-automation
Clusterfuzz Tools
Bugs are inevitable. Suffering is optional.
Stars: ✭ 111 (-31.48%)
Mutual labels:  fuzzing, fuzzer
Test Each
🤖 Repeat tests. Repeat tests. Repeat tests.
Stars: ✭ 89 (-45.06%)
Mutual labels:  fuzzing, test-automation
Crlf Injection Scanner
Command line tool for testing CRLF injection on a list of domains.
Stars: ✭ 91 (-43.83%)
Mutual labels:  fuzzing, fuzzer
Fisy Fuzz
This is the full file system fuzzing framework that I presented at the Hack in the Box 2020 Lockdown Edition conference in April.
Stars: ✭ 110 (-32.1%)
Mutual labels:  fuzzing, fuzzer
Example Go
Go Fuzzit Example
Stars: ✭ 39 (-75.93%)
Mutual labels:  fuzzing, fuzzer
Stryker4s
Mutation testing for Scala. Work in progress...
Stars: ✭ 118 (-27.16%)
Mutual labels:  hacktoberfest, test-automation
Aflplusplus
The fuzzer afl++ is afl with community patches, qemu 5.1 upgrade, collision-free coverage, enhanced laf-intel & redqueen, AFLfast++ power schedules, MOpt mutators, unicorn_mode, and a lot more!
Stars: ✭ 2,319 (+1331.48%)
Mutual labels:  fuzzing, fuzzer
Sienna Locomotive
A user-friendly fuzzing and crash triage tool for Windows
Stars: ✭ 130 (-19.75%)
Mutual labels:  fuzzing, fuzzer
Afl Patches
Patches to afl to fix bugs or add enhancements
Stars: ✭ 76 (-53.09%)
Mutual labels:  fuzzing, fuzzer
Elmyr
A utility to make Kotlin/Java tests random yet reproducible
Stars: ✭ 68 (-58.02%)
Mutual labels:  hacktoberfest, fuzzing
Coteafs Appium
📱 Wrapper Appium Framework in Java which supports Automation of Mobile and Tablet apps.
Stars: ✭ 93 (-42.59%)
Mutual labels:  hacktoberfest, test-automation
Mobileautomationframework
Single code base framework to test android and iOS app using appium (v6.1.0), maven, testng,java. Option to start appium server programmatically.
Stars: ✭ 66 (-59.26%)
Mutual labels:  hacktoberfest, test-automation
Ansvif
A Not So Very Intelligent Fuzzer: An advanced fuzzing framework designed to find vulnerabilities in C/C++ code.
Stars: ✭ 107 (-33.95%)
Mutual labels:  fuzzing, fuzzer
Janus
Janus: a state-of-the-art file system fuzzer on Linux
Stars: ✭ 139 (-14.2%)
Mutual labels:  fuzzing, fuzzer
Angora
Angora is a mutation-based fuzzer. The main goal of Angora is to increase branch coverage by solving path constraints without symbolic execution.
Stars: ✭ 669 (+312.96%)
Mutual labels:  fuzzing, fuzzer
Dirsearch
Web path scanner
Stars: ✭ 7,246 (+4372.84%)
Mutual labels:  fuzzing, fuzzer
Fuzzing Survey
The Art, Science, and Engineering of Fuzzing: A Survey
Stars: ✭ 116 (-28.4%)
Mutual labels:  fuzzing, fuzzer

============= Grammarinator

ANTLRv4 grammar-based test generator

.. image:: https://img.shields.io/pypi/v/grammarinator?logo=python&logoColor=white :target: https://pypi.org/project/grammarinator/ .. image:: https://img.shields.io/pypi/l/grammarinator?logo=open-source-initiative&logoColor=white :target: https://pypi.org/project/grammarinator/ .. image:: https://img.shields.io/github/workflow/status/renatahodovan/grammarinator/main/master?logo=github&logoColor=white :target: https://github.com/renatahodovan/grammarinator/actions .. image:: https://img.shields.io/coveralls/github/renatahodovan/grammarinator/master?logo=coveralls&logoColor=white :target: https://coveralls.io/github/renatahodovan/grammarinator

Grammarinator is a random test generator / fuzzer that creates test cases according to an input ANTLR_ v4 grammar. The motivation behind this grammar-based approach is to leverage the large variety of publicly available ANTLR v4 grammars_.

The trophy page_ of the found issues is available from the wiki.

.. _ANTLR: http://www.antlr.org .. _ANTLR v4 grammars: https://github.com/antlr/grammars-v4 .. _trophy page: https://github.com/renatahodovan/grammarinator/wiki

Requirements

  • Python_ >= 3.5
  • pip_ and setuptools Python packages (the latter is automatically installed by pip).
  • Java_ SE >= 7 JRE or JDK

.. _Python: https://www.python.org .. _pip: https://pip.pypa.io .. _Java: https://www.oracle.com/java/

Install

The quick way (to install the latest official release)::

pip3 install grammarinator

Or clone the project and run setuptools (to install the freshest development revision)::

python3 setup.py install

Usage

As a first step, Grammarinator takes an ANTLR v4 grammar_ and creates a test generator script in Python3. Such a generator can be subclassed later to customize it further if needed.

Basic command-line syntax of test generator creation::

grammarinator-process <grammar-file(s)> -o <output-directory> --no-actions

..

**Notes**

*Grammarinator* uses the `ANTLR v4 grammar`_ format as its input, which
makes existing grammars (lexer and parser rules) easily reusable. However,
because of the inherently different goals of a fuzzer and a parser, inlined
code (actions and conditions, header and member blocks) are most probably
not reusable, or even preventing proper execution. For first experiments
with existing grammar files, ``grammarinator-process`` supports the
command-line option ``--no-actions``, which skips all such code blocks
during fuzzer generation. Once inlined code is tuned for fuzzing, that
option may be omitted.

.. _ANTLR v4 grammar: https://github.com/antlr/grammars-v4

After having generated and optionally customized a fuzzer, it can be executed by the grammarinator-generate script (or by manually instantiating it in a custom-written driver, of course).

Basic command-line syntax of grammarinator-generate::

grammarinator-generate <generator> -r <start-rule> -d <max-depth> \
  -o <output-pattern> -n <number-of-tests> \
  -t <transformer1> -t <transformer2>

Beside generating test cases from scratch based on the ANTLR grammar, Grammarinator is also able to recombine existing inputs or mutate only a small portion of them. To use these additional generation approaches, a population of selected test cases has to be prepared. The preparation happens with the grammarinator-parse tool, which processes the input files with an ANTLR grammar (possibly with the same one as the generator grammar) and builds grammarinator tree representations from them (with .grt extension). Having a population of such .grt files, grammarinator-generate can make use of them with the --population cli option. If the --population option is set, then Grammarinator will choose a strategy (generation, mutation, or recombination) randomly at the creation of every new test case. If any of the strategies is unwanted, they can be disabled with the --no-generate, --no-mutate or --no-recombine options.

Basic command line syntax of grammarinator-parse::

grammarinator-parse <grammar-file(s)> -r
-i <input_file> -o

..

**Notes**

Real-life grammars often use recursive rules to express certain patterns.
However, when using such rule(s) for generation, we can easily end up in an
unexpectedly deep call stack. With the ``--max-depth`` or ``-d`` options,
this depth - and also the size of the generated test cases - can be
controlled.

Another specialty of the ANTLR grammars is that they support so-called
hidden tokens. These rules typically describe such elements of the target
language that can be placed basically anywhere without breaking the syntax.
The most common examples are comments or whitespaces. However, when using
these grammars - which don't define explicitly where whitespace may or may
not appear in rules - to generate test cases, we have to insert the missing
spaces manually. This can be done by applying a serializer (with the ``-s``
option) to the tree representation of the output tests. A simple serializer
- that inserts a space after every unparser rule - is provided by
*Grammarinator* (``grammarinator.runtime.simple_space_serializer``).

In some cases, we may want to postprocess the output tree itself (without
serializing it). For example, to enforce some logic that cannot be expressed
by a context-free grammar. For this purpose the transformer mechanism can be
used (with the ``-t`` option). Similarly to the serializers, it will take a
tree as input, but instead of creating a string representation, it is
expected to return the modified (transformed) tree object.

As a final thought, one must not forget that the original purpose of
grammars is the syntax-wise validation of various inputs. As a consequence,
these grammars encode syntactic expectations only and not semantic rules. If
we still want to add semantic knowledge into the generated test, then we can
inherit custom fuzzers from the generated ones and redefine methods
corresponding to lexer or parser rules in ways that encode the required
knowledge (e.g.: HTMLCustomGenerator_).

.. _HTMLCustomGenerator: examples/fuzzer/HTMLCustomGenerator.py

Working Example

The repository contains a minimal example_ to generate HTML files. To give it a try, run the processor first::

grammarinator-process examples/grammars/HTMLLexer.g4 examples/grammars/HTMLParser.g4 \
  -o examples/fuzzer/

Then, use the generator to produce test cases::

grammarinator-generate HTMLCustomGenerator.HTMLCustomGenerator -r htmlDocument -d 20 \
  -o examples/tests/test_%d.html -n 100 \
  -s HTMLGenerator.html_space_serializer \
  --sys-path examples/fuzzer/

.. _example: examples/

Compatibility

Grammarinator was tested on:

  • Linux (Ubuntu 16.04 / 18.04)
  • Mac OS X (Sierra 10.12 / High Sierra 10.13 / Mojave 10.14 / Catalina 10.15)
  • Windows (Server 2012 R2 / Server version 1809 / Windows 10)

Citations

Background on Grammarinator is published in:

  • Renata Hodovan, Akos Kiss, and Tibor Gyimothy. Grammarinator: A Grammar-Based Open Source Fuzzer. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating Test Case Design, Selection, and Evaluation (A-TEST 2018), pages 45-48, Lake Buena Vista, Florida, USA, November 2018. ACM. https://doi.org/10.1145/3278186.3278193

Copyright and Licensing

Licensed under the BSD 3-Clause License_.

.. _License: LICENSE.rst

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].