Description
Simple implementations of system ultilities in Dlang.
The primary purpose is to understand Dlang
and to learn system programming.
Learning notes are written in NOTES.md.
List of tools:
- free: display system memory by reading from
/proc/. Topics: Struct. Function overloading. - watch: Watch command output, may exit if output matches some regular expression. Topics: system command invocation, Getopt. Links: TODO and Examples.
- plotbar: Draw 2-d bar chat, a tool inspired by https://github.com/lebinh/goplot. Topics: Struct, Overloading, Testing. Links: TODO and Examples.
- jq: Simple
jsonreader, parses every json line from STDIN. Topics: JSON parser. Links: TODO and Examples. - hello: Simple Hello builtin command writtedn in Dlang, can be loaded into Bash environments. Topics: Bash, C.
- json.validator: Match json input against a set of rules.
Topics:
json,opApply, recursive. For examples, please see the unittest within the source file.
Getting started
To use any ultilities below, you need a dlang compiler and also
the dub package manager. The compiler can be installed by your
system package manager, for e.g,
$ pacman -S dmd # on ArchLinux
To intsall dub please follow this link https://github.com/dlang/dub#installation.
The latest ArchLinux database has dub in the official repository and
you can also install them with pacman -S dub.
Now from the top directory of project, you can start testing / running any tool as below
$ dub test -d 2 dusybox:free
$ dub run dusybox:free
When being compiled, the tool's binary is located under ./output/ directory.
For example, ./output/dzfree here dz is the common prefix for our tools
(dz sounds a bit similar to dusy, doesn't it?).
The testing and other build profile still generate binary files
under the top directory though.
The Makefile can help to run all tests and/or to build release versions
of all tools. You can override the list of tools with help of TOOLS=:
$ make releases TOOLS=watch
json validator
Source: lib/dusybox/json/validator.d.
A simple json validator inspired by the Python code
https://github.com/rycus86/webhook-proxy/blob/a8919cc82173b8e7a4cb0a2ba8a34a14996e159c/src/endpoints.py#L141.
For example, given a json data input = {"foo": [0,1,"2bar"]}
and a rule, {"foo": "[0-9]+"}, the validator returns true as every
item in the value input["foo"] matches against the regular expression
[0-9]+. However, the validator returns false when the rule is updated
to {"foo": "[0-9]{2}"}
free
Print information about system memory.
It's similar to the free command on your Linux system.
TODO
- Print Swap information (used/total)
- Support different Linux versions
- Print various information in a single command
- Print human-readable memory size
Examples
$ dub run dusybox:free
total user free shared buff/cache available
Mem (kB): 16337688 3181032 6025164 759692 7131492 12487640
Mem (mB): 15954 3106 5883 741 6964 12194
Mem (gB): 15 3 5 0 6 11
Mem (%): 100.00 19.47 36.88 4.65 43.65 76.43
watch
Execute a shell command and print its output to the standard output device
every one second. This is similar to the popular watch command.
TODO
- Document the external requirement (e.g,
libncursesw5-devonUbuntu-16.04) - Do not work inside
screen. Work-around:TERM=tmux dzwatch. See also https://github.com/D-Programming-Deimos/ncurses/issues/35. To fix this issue, you need to rebuild the application on the target machine. - Support
-ito work with case-insensitive regular expression - Redirect output from
stderr(This works out-of-the-box) - Print time information of the last interaction
- Print basic information about input command and iterator number.
- Wait 1 second after every execution. No more
--interval 1:) - Specify maxium number of executions with
-n <number> - Fix problem with overflow output, e.g, generated by
ps xauw. Wait for https://github.com/mpevnev/nice-curses/issues/2. - Exit if output matches some regular expression
- Exit if user presses
qorQ. Wait for https://github.com/mpevnev/nice-curses/issues/1. - Tool doesn't work with pipe commands, e.g,
ps x | grep foo: it reportscommand not founderror. As a work-around you can usebash -c "ps x | grep ff"instead.
Examples
$ dub run dusybox:watch -- free -m
$ dub run dusybox:watch -- ps x
$ dub run dusybox:watch -- -n 10 'ps xwa | grep "f[i]r"'
:: No 4/10, Cmd ["ps xwa | grep \"f[i]r\""]
15774 ? SNsl 3:01 /usr/lib/firefox/firefox
15776 ? ZN 0:00 [firefox] <defunct>
...
:: Reached maximum number of interation (2) at 2017-Sep-05 18:04:47.0811478.
Watch ElasticSearch cluster status and exit if cluster is green:
$ dzwatch -e '"status"\s+:\s+"green"' 'curl -s http://elk.example.net:9201/_cluster/health | json_pp'
plotbar
This tool is inspired by https://github.com/lebinh/goplot.
It visualizes your data as a simple bar chart.
goplot draws relative bars (compare bar height to the highest bar),
why this tool draws absolute bars (compare bar height to the sum of all bars).
The tool reads data from STDIN (the only source so far),
and fetches every entry in format
key value
It will generate error messages to STDERR in case some line doesn't
match the above format and/or their value is invalid.
TODO
- Detect if there is any input data from
STDIN - Option to draw relative bars
- Continuous mode (keep drawing new bar while reading from
stdin) - Support tab delimeter in
key valueline - Support negative data (2-direction bar chart)
- Display actual value after the bar
- Set the minium percent number to display (
-m min) - Display last n items (like
sort | tail) - Sort the output (if the input is sorted)
- Additive mode (Sum of duplicated items)
- Fix bug when parsing input data (previous
valueis reused.) - Move common part to a library file
- Avoid overflow (when input key is too long, and/or the bar is too high)
- Use
gnuplotinstead?
Examples
Find the biggest folder items, display ones consume great than 2% of total storage.
(The idea for this example comes from https://github.com/lebinh/goplot.)
Please note that you can't use \t character in this example: The input parser
doesn't understand tab.
$ dub run dusybox:plotbar -- -m 2 < <(2>/dev/null du -s /home/* | awk '{printf("%s %s\n", $2, $1)}')
/home/pi.fast : 9 % ========= (9466072)
/home/pi : 13 % ============= (14541032)
/home/btsync : 64 % ================================================================ (69425660)
/home/ebook : 8 % ======== (8600288)
/home/backup : 2 % == (2615004)
Display the ElasticSearch indices the have most documents.
Skip all indices that consumes less than 2% in the total number of documents.
$ curl -s 'elk.example.net:9201/_cat/indices?h=index,docs.count' | ./output/dzplotbar -m 2
aws-lambda-test-uat-test-20170824 : 9 % ========= (4986415)
api-gateway-execution-logs-test-uat-20170824 : 4 % ==== (2486179)
aws-lambda-test-uat-test-20170824 : 2 % == (1177304)
aws-lambda-test-dev-test-20170815 : 4 % ==== (2227446)
Display the biggest indexes (in stored size):
$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size&bytes=k' | ./output/dzplotbar -m 2
aws-lambda-test-uat-test-20170824 : 2 % == (2847921)
emr-20170904 : 2 % == (3364511)
aws-lambda-test-uat-test-20170824 : 4 % ==== (5544297)
aws-lambda-test-uat-test-20170821 : 2 % == (2853427)
Now find the biggest source (by discarding date suffixes):
$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size&bytes=k' \
| sed -re 's#-[0-9]{8}##g' \
| ./output/dzplotbar -m 5 2>/dev/null
aws-lambda-test-uat-test : 5 % ===== (3145751)
emr : 11 % =========== (6974423)
aws-lambda-test-uat-test2 : 11 % =========== (6622399)
cloudtrail-defaultloggroup : 11 % =========== (6726637)
Find the package that has most files on ArchLinux system
$ pacman -Ql | grep -vE '/$' | awk '{printf("%s 1\n", $1 );}' | ./output/dzplotbar -m 2
evince : 2 % == (3058)
efl-git : 2 % == (3563)
python2 : 3 % === (4646)
adwaita-icon-theme : 4 % ==== (5426)
mono : 2 % == (2443)
linux : 3 % === (3984)
linux-headers : 9 % ========= (12296)
python : 5 % ===== (6784)
ghc : 4 % ==== (5728)
claws-mail-themes : 3 % === (4689)
openssl : 2 % == (3252)
qt4 : 3 % === (3825)
perl : 2 % == (2393)
libxcb : 2 % == (2371)
ncurses : 3 % === (3678)
cmake : 2 % == (2267)
man-pages : 2 % == (3491)
gcc : 2 % == (2198)
Find the biggest packages on ArchLinux system
$ pacman -Qi | awk '
/Name/ {printf("%s", $NF);}
/Installed Size.+KiB/ {printf(" %s\n", $(NF-1))}
/Installed Size.+MiB/ {printf(" %s\n", $(NF-1) * 1024)}' \
| ./dzplotbar -m 2
mono : 4 % ==== (199782)
ghc-static : 17 % ================= (843428)
firefox : 3 % === (143370)
linux-firmware : 4 % ==== (206377)
chromium : 4 % ==== (212572)
python : 3 % === (131430)
ghc : 8 % ======== (425339)
gcc : 2 % == (119081)
jq
This is not https://github.com/stedolan/jq.
Instead, this tool reads line from STDIN and considers
each line as a JSON string. This is useful as I need to process
multiple JSON lines from nginx and/or ELK system.
If input line can be parsed, the result will be printed to stdout
(if the tool has not any argument), or each item from arguments
is looked up in the final JSON object. If the argument is
./output/dzjq .foo bar
then the .foo is used as a lookup key, while bar is printed literally.
If the program fails to query a key .foo, it prints foo instead.
TODO
- Detect if there is any input data from
STDIN - Handle delimeter
- Handle formatted string
- Handle object other than integer and/or string
- Nested key query
- Advanced query with array support
- Move common methods to a library file
- Option to flush
STDOUTbuffer on every processed input line - Add unit tests
- Literraly support
- Process lines from
STDINas invidual documents. See also https://github.com/stedolan/jq/issues/744.
Examples
Print key .a and .b, print 1 literally.
$ echo '{"a": 9, "b": {"c": 1}}' | dub run dusybox:jq -- .a 1 .b
9 1 {"c":1}
Print the original JSON string
$ echo '{"a": 9, "b": {"c": 1}}' | dub run dusybox:jq --
'{"a": 9, "b": {"c": 1}}'
Generate simple statistics from nginx access log file.
The format of log file is similar to
this one.
$ dub run dusybox:jq -- .host 1 < /home/pi/df/acces.log | ./output/dzplotbar -m 2
kibana.int.example.net : 25 % ========================= (269)
airflow.dev.example.net : 3 % === (33)
grafana.int.example.net : 70 % ====================================================================== (755)
airflow.staging.example.net : 3 % === (28)
How about the requests or statuses?
$ dub run dusybox:jq -- .request_uri 1 < /home/pi/df/acces.log | ./output/dzplotbar -m 2
/api/console/proxy?path=_aliases&method=GET : 4 % ==== (44)
/api/console/proxy?path=_mapping&method=GET : 4 % ==== (44)
/api/datasources/proxy/16 : 34 % ================================== (364)
/api/datasources/proxy/14 : 12 % ============ (132)
/api/datasources/proxy/13 : 5 % ===== (55)
/elasticsearch/_msearch : 4 % ==== (40)
/api/datasources/proxy/12 : 11 % =========== (122)
$ dub run dusybox:jq -- .status 1 < /home/pi/df/acces.log | ./output/dzplotbar -m 2
200 : 93 % ============================================================================================= (1013)
304 : 4 % ==== (43)
A Bash builtin command
We can write Bash built-in command in Dlang.
Thanks a lot evilrat on Dlang forum for the idea.
$ dub build dusybox:bash_builtin_hello
$ enable -f ./output/libdz_hello.so dz_hello
$ type -a dz_hello
dz_hello is a shell builtin
$ dz_hello
Hello, world. It's Hello builtin command written in Dlang.
$ help dz_hello
dz_hello: dz_hello
Hello, it's from Dlang.
A Hello builtin command written in Dlang.
$ enable -d dz_hello
$ dz_hello
-bash: dz_hello: command not found