assafmo / Xioc
Licence: mit
Extract indicators of compromise from text, including "escaped" ones.
Stars: ✭ 148
Programming Languages
go
31211 projects - #10 most used programming language
Labels
Projects that are alternatives of or similar to Xioc
Grex
A command-line tool and library for generating regular expressions from user-provided test cases
Stars: ✭ 4,847 (+3175%)
Mutual labels: command-line-tool, regex, regexp
stringx
Drop-in replacements for base R string functions powered by stringi
Stars: ✭ 14 (-90.54%)
Mutual labels: regex, regexp, text-processing
perke
A keyphrase extractor for Persian
Stars: ✭ 60 (-59.46%)
Mutual labels: text-mining, data-mining, text-processing
Regex Automata
A low level regular expression library that uses deterministic finite automata.
Stars: ✭ 203 (+37.16%)
Mutual labels: regex, text-processing, regexp
Chr
🔤 Lightweight R package for manipulating [string] characters
Stars: ✭ 18 (-87.84%)
Mutual labels: extract, regex, text-processing
Sd
Intuitive find & replace CLI (sed alternative)
Stars: ✭ 2,755 (+1761.49%)
Mutual labels: command-line, regex, text-processing
teanaps
자연어 처리와 텍스트 분석을 위한 오픈소스 파이썬 라이브러리 입니다.
Stars: ✭ 91 (-38.51%)
Mutual labels: text-mining, data-mining, text-processing
Stringi
THE String Processing Package for R (with ICU)
Stars: ✭ 204 (+37.84%)
Mutual labels: regex, text-processing, regexp
Ripgrep
ripgrep recursively searches directories for a regex pattern while respecting your gitignore
Stars: ✭ 28,564 (+19200%)
Mutual labels: command-line-tool, command-line, regex
Artificial Adversary
🗣️ Tool to generate adversarial text examples and test machine learning models against them
Stars: ✭ 348 (+135.14%)
Mutual labels: data-mining, text-mining, text-processing
Cogcomp Nlpy
CogComp's light-weight Python NLP annotators
Stars: ✭ 115 (-22.3%)
Mutual labels: data-mining, text-mining, text-processing
Cascadia
Go cascadia package command line CSS selector
Stars: ✭ 67 (-54.73%)
Mutual labels: command-line-tool, command-line, extract
corpusexplorer2.0
Korpuslinguistik war noch nie so einfach...
Stars: ✭ 16 (-89.19%)
Mutual labels: text-mining, data-mining, text-processing
advanced-text-mining
TEANAPS 라이브러리를 활용한 자연어 처리와 텍스트 분석 방법론에 대해 다룹니다.
Stars: ✭ 15 (-89.86%)
Mutual labels: text-mining, data-mining, text-processing
Applied Text Mining In Python
Repo for Applied Text Mining in Python (coursera) by University of Michigan
Stars: ✭ 59 (-60.14%)
Mutual labels: text-mining, regex, text-processing
Command Line Text Processing
⚡ From finding text to search and replace, from sorting to beautifying text and more 🎨
Stars: ✭ 9,771 (+6502.03%)
Mutual labels: command-line, regex, text-processing
Textcluster
短文本聚类预处理模块 Short text cluster
Stars: ✭ 115 (-22.3%)
Mutual labels: text-mining, text-processing
Dynein
DynamoDB CLI written in Rust.
Stars: ✭ 126 (-14.86%)
Mutual labels: command-line-tool, command-line
Desktoppr
Simple command line tool to set the desktop picture on macOS
Stars: ✭ 127 (-14.19%)
Mutual labels: command-line-tool, command-line
xioc
Extract indicators of compromise from text, including "escaped" ones like hxxp://banana.com
, 1.1.1[.]1
and phish at malicious dot com
.
Installation
-
Download a precompiled binary from https://github.com/assafmo/xioc/releases
-
Or... Use
go get
:go get -u github.com/assafmo/xioc
-
Or... Use snap install (Ubuntu):
snap install xioc
-
Or use Ubuntu PPA:
curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add - sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list sudo apt update sudo apt install xioc
Features
- Extract IOCs (indicators of compromise) from an input text:
- IPv4
- IPv6
- Domain
- URL
- MD5
- SHA1
- SHA256
- Translate some kinds of "escaping"/"defanging" techniques:
-
(dot)
,[dot]
,(.)
,[.]
,{.}
to.
. -
(at)
,[at]
,(@)
,[@]
,{@}
to@
. -
hxxp
,hzzzp
,hxxxp
,hXXp
,h__p
,h**p
tohttp
.
-
- Command line interface
- Go library
Command line usage
$ xioc -h
Usage of xioc:
-o string
Extract only specified types.
Types must be comma seperated. E.g: xioc -o "ip4,domain,url,md5"
Available types:
- ip4
- ip6
- domain
- url
- email
- md5
- sha1
- sha256
-v Print version and exit
$ REPORT="https://unit42.paloaltonetworks.com/digital-quartermaster-scenario-demonstrated-in-attacks-against-the-mongolian-government/"
$ lynx -dump "$REPORT" | xioc
sha256 5beb50d95c1e720143ca0004f5172cb8881d75f6c9f434ceaff59f34fa1fe378
domain energy.gov.mn
email [email protected]
sha256 10090692ff40758a08bd66f806e0f2c831b4b9742bbf3d19c250e778de638f57
# ...
$ REPORT="https://unit42.paloaltonetworks.com/digital-quartermaster-scenario-demonstrated-in-attacks-against-the-mongolian-government/"
$ lynx -dump "$REPORT" | xioc -o email,sha256
sha256 5beb50d95c1e720143ca0004f5172cb8881d75f6c9f434ceaff59f34fa1fe378
email [email protected]
sha256 10090692ff40758a08bd66f806e0f2c831b4b9742bbf3d19c250e778de638f57
email [email protected]
# ...
Library usage
package main
import (
"fmt"
"github.com/assafmo/xioc/xioc"
)
func main() {
input := `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
banana.com
hxxp://i.robot.com/robots.txt
1.2.3.4
1.1.1[.]1
info at gmail dot com
hxxps://m.twitter[dot]com/`
fmt.Println(xioc.ExtractDomains(input)) // => [i.robot.com m.twitter.com gmail.com banana.com]
fmt.Println(xioc.ExtractSHA256s(input)) // => [e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855]
fmt.Println(xioc.ExtractMD5s(input)) // => []
fmt.Println(xioc.ExtractIPv4s(input)) // => [1.2.3.4 1.1.1.1]
fmt.Println(xioc.ExtractURLs(input)) // => [http://i.robot.com/robots.txt https://m.twitter.com/]
fmt.Println(xioc.ExtractEmails(input)) // => [[email protected]]
}
Sources
- Test email address: http://codefool.tumblr.com/post/15288874550/list-of-valid-and-invalid-email-addresses
- Domains can start with a number: https://serverfault.com/a/638270
- IPv6 Examples: http://www.gestioip.net/docu/ipv6_address_examples.html
- Fang and defang IOCs: https://github.com/ioc-fang/ioc_fanger
- Indicator of Compromise (De)Fanging Project: https://ioc-fang.hightower.space/
- InQuest/python-iocextract test data: https://github.com/InQuest/python-iocextract/tree/master/test_data
- Email address can be treated as case-insensitive: https://stackoverflow.com/a/9808332
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].