latacora / Wernicke
Programming Languages
Projects that are alternatives of or similar to Wernicke
wernicke
A redaction tool for structured data. Run wernicke with JSON on stdin, get
redacted values out. Preserves structure and (to some extent) semantics. You
might want this because you have test data where the actual values are
sensitive. Because the changes are consistent within the data and the overall
data structure is preserved, there a better chance your data will stay suitable
for testing, even though it's been scrubbed.
Most people run wernicke on a shell, so you either have json_producing_thing | wernicke or wernicke < some_file.json > redacted.json. EDN is also supported.
See wernicke --help for additional information.
| Example input | Example output |
|---|---|
| IPs, MAC addresses, timestamps, various AWS identifiers, and a few other types of strings are redacted to strings of the same type: IPs to IPs, SGs to SGs, et cetera. If these strings have an alphanumeric id, that id will have the same length. | |
{
"long_val": "ABBBAAAABBBBAAABBBAABB",
"ip": "10.0.0.1",
"mac": "ff:ff:ff:ff:ff:ff",
"timestamp": "2017-01-01T12:34:56.000Z",
"ec2": "ip-10-0-0-1.ec2.internal",
"security_group": "sg-12345",
"vpc": "vpc-abcdef",
"aws_access_key": "AKIAXXXXXXXXXXXXXXXX",
"aws_role_cred": "AROAYYYYYYYYYYYYYYYY"
}
|
{
"long_val": "teyjdaeqEYGw18fRIt5vLo",
"ip": "254.65.252.245",
"mac": "aa:3e:91🆎3b:3a",
"timestamp": "2044-19-02T20:32:55.72Z",
"ec2": "ip-207-255-185-237.ec2.internal",
"security_group": "sg-887b8",
"vpc": "vpc-a9d96a",
"aws_access_key": "AKIAQ5E7IHRMOW7YABLS",
"aws_role_cred": "AROA6QA7SQTM6YWS4F0H"
}
|
| Redaction happens in arbitrarily nested structures. | |
{
"a": {
"b": [
"c",
"d",
{
"e": "10.0.0.1"
}
]
}
}
|
{
"a": {
"b": [
"c",
"d",
{
"e": "1.212.241.246"
}
]
}
}
|
| In addition to values in the tree, keys are also redacted, even nested ones. | |
{
"vpc-12345": {
"sg-abcdef": {
"instance_count": 5
}
}
}
|
{
"vpc-ec60f": {
"sg-086fd3": {
"instance_count": 5
}
}
}
|
| Redaction also happens in the middle of strings. | |
{
"x": "i-abc123 is in sg-12345"
}
|
{
"x": "i-26a1bf is in sg-77aff"
}
|
| The redacted values will change across runs (this is necessary to make redaction irreversible). | |
{
"ip": "10.0.0.1",
"mac": "ff:ff:ff:ff:ff:ff"
}
|
{
"ip": "246.220.253.214",
"mac": "dc:08:90:75:e3:91"
}
|
| Redacted values _are_ consistent within runs. If the input contains the same value multiple times it will get redacted identically. This allows you to still do correlation in the result. | |
{
"ip": "10.0.0.1",
"also_ip": "10.0.0.1"
}
|
{
"ip": "247.226.167.9",
"also_ip": "247.226.167.9"
}
|
(These examples were pretty-printed for viewing comfort, but wernicke does not do that for you. Try jq.)
Installation
Download from https://github.com/latacora/wernicke/releases
Configuration
We try to do something reasonable for most use cases. If you have a generally
useful redactions, please consider contributing them. However, sometimes
redaction behavior really does need to be configured. Pass an EDN literal on the
command line like so: wernicke --config '{:some-rules "detailed below"}'.
Right now this requires a pretty extensive understanding of how wernicke works--we want to make this more accessible, though! If there's a specific thing you want to accomplish, feel free to write a ticket.
Adding extra rules
For example, to redact all numbers, add the following structure to your EDN:
{:extra-rules
[{:name :numbers
:type :regex
:pattern "\\d*"}]}
The extra rules will be compiled before use, so e.g. you do not need to specify the parsed regex structure for this to work.
Disabling rules by name
Add the following structure to your EDN:
{:disabled-rules [:latacora.wernicke.patterns/arn-re]}
This still requires you to know what the rule names are. You can find these in
latacora.wernicke.core/default-config.
Development
To run the project directly from a source checkout:
$ clj -m latacora.wernicke.cli
To run the project's tests:
$ clj -A:test
To build a native image:
$ clj -A:native-image
(This requires GraalVM to be installed with SubstrateVM, and the GRAAL_HOME
environment variable to be set.)
Namesake
Named after Carl Wernicke, a German physician who did research on the brain. Wernicke's aphasia is a condition where patients demonstrate fluent speech with intact syntax but with nonsense words. This tool is kind of like that: the resulting structure is maintained but all the words are swapped out with (internally consistent) nonsense.
License
Copyright © Latacora, LLC
This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.