PythonSed

A full and working Python implementation of sed

Contents

General Information

Usage as a command line utility

Usage as a Python module

Sed dialect

Testing

Timing

To do list

General Information

Important note

pythonsed has been rewrited and completed by Frank Schäckermann. This new version has its own repository and is very close to be fully compatible with GNU sed 4.8. Any comment or pull request should be addressed to Frank.

Description

pythonsed is a full and working Python implementation of sed. Its reference is GNU sed 4.2 of which it implements almost all commands and features. It may be used as a command line utility or it can be used as a module to bring sed functionality to Python scripts.

A complete set of tests is available as well as a testing utility. These tests include scripts from various origins and cover all aspects of sed functionalities.

Compatibility

Compatibility status applies also to the testing utility test-suite.py.

License

PythonSed is released under the MIT license.

Install

To install, just clone or download the repository zip file and run the setup in download directory:

pip install .

This installs a command line utility named pythonsed and a package named PythonSed.

Usage as a command line utility

pythonsed is as console program receiving information from the command line. The format of the command line is:

pythonsed [options] -e<script expression> <input text file>

pythonsed [options] -f<script file> <input text file>

Note that pythonsed accepts only one script file or expression, and only one input file. options may be one or both of:

-n disable automatic printing

-ruse extended regular expressions

pythonsed may also use redirection to receive its input or send its output with the usual syntax:

cat myfile | pythonsed -f myscript1.sed | pythonsed -f myscript2.sed > myresultfile

It is also possible for pythonsed to receive its input from the keyboard by omitting any input file:

pythonsed -f myscript.sed

Usage as a Python module

An example covering all necessary symbols:

from PythonSed import Sed, SedException

sed = Sed()
try:
    sed.no_autoprint = True
    sed.regexp_extended = False
    sed.load_script('myscript.sed')
    sed.apply('myinput.txt')
except SedException as e:
    print(e.message)
except:
    raise

sed.apply() input parameter may be a string (which is interpreted as a filename) or file-like object (including streams). Note that sed.apply() returns the list of lines printed by the script. As a default, these lines are printed to stdout. sed.apply() has an output parameter which enables to inhibit printing the lines (output=None) or enables to redirect the output to some text file (output='somefile.txt') or to a file-like object (including streams). Note also that if myinput or myoutput are file-like objects, they must be closed by the caller.

The script may also be read from a string by using sed.load_string(my_script_string).

sed dialect

PythonSed implements all standard commands and regular expression features of sed. Its reference is GNU sed 4.2. It implements almost all its features except the most specific ones.

GNU sed manual page can serve as a reference for PythonSed given the differences described in the following.

Addresses

Regular expressions

Note that for any combination of quantifiers (*, +, ?, {}), consecutive quantifiers or a quantifier starting a regexp will launch an error. This is true in basic or extended regular expression modes.

Extended regular expressions

Using the -r switch enables to simplify regular expressions by removing the antislah character before the special characters +, ?, (, ), |, { and }. If these characters must appear as regular characters in a regexp, they must be slashed.

Commands

Compliant means compliant with GNU sed description.

The other commands specific to GNU sed are not implemented.

Testing

Description

The working of PythonSed is tested and compared to the behavior of GNU sed with a set of tests and a testing utility.

The tests are either coded in text files with .suite extension or may be stored in test directories as standard sed scripts.

The test suites are:

Testing utility

Tests are launched and checked with the test-suite.py Python script. This script uses either PythonSed package to run the sed scripts, or any sed executable. This enables to compare the working of PythonSed with the one of GNU sed.

The calling syntax is:

test-suite.py <testsuite> [number] [-b executable] [-x list of script references]

Text file test suites

When tests are stored in a text file (with .suite extension), they are made of four elements:

• the title of the test
• the script itself
• the input list of lines
• the expected result

The four elements of a test are separated with lines made of three identical characters, for instance:

---
Test substitution with global flag
---
s/an/AN/g
---
In Xanadu did Kubhla Khan
---
In XANadu did Kubhla KhAN
---

Note also that:

• the script section may be empty, enabling to test a script on various data without repeating the script.
• The input and output sections may be empty, enabling to test various scripts on the same data, without repeating the data.
• Flags are set with a comment on the first line. As usual, #n stops autoprint mode and extended regexp mode is set with #r or #nr.
• The expected result may be ??? when the test has no result and ends with an error.
• All text outside the test, i.e. before first delimiter or after last delimiter, is ignored and acts like a comment.

Directory test suites

When tests are stored in a directory, they are represented by three or four files with same name but different extensions:

• the script itself, with '.sed' extension
• the input of the script, with '.inp' extension
• the expected result of the script, with '.good' extension
• possibly a file, with '.flags' extension, containing the sed switches -n and/or -r.

Some other files may be used when using reading or writing commands in scripts. In that case, the expected written files must be named with extension '.wgoodN' where N is the number of the expected written file.

Timing

A python implementation of sed has to face legitimate questions about timing. Fortunately, results are not bad. Unfortunately, they seem correlated with version number. Timings are given in seconds.

Test conditions:

• Only script files are used (scripts from folders testsuiteN). This is to avoid measuring the time to extract scripts, inputs and results from .suite files.
• The given values are averaged from three consecutive test runs.

To do list

At one moment, one has to decide what will be in the release to come, and what can be delayed. Here are some features which would be nice to have but can be delayed to a future version.

• Better POSIX compliance:

• multiple scripts on the command line (-e, -f)

• multiple input file

• character classes

• Better error handling (display of the number of the line in error)

• Better error handling when testing (the error message could be tested)

• Use PythonSed as a basis for a sed debugger.

• ...