Blackweb

Blackweb is a project that collects and unifies public blocklists of domains (porn, downloads, drugs, malware, spyware, trackers, bots, social networks, warez, weapons, etc.) to make them compatible with Squid-Cache

Blackweb es un proyecto que recopila y unifica listas públicas de bloqueo de dominios (porno, descargas, drogas, malware, spyware, trackers, bots, redes sociales, warez, armas, etc) para hacerlas compatibles con Squid-Cache

DATA SHEET

GIT CLONE

git clone --depth=1 https://github.com/maravento/blackweb.git

HOW TO USE

blackweb.txt is already updated and optimized for Squid-Cache. Download it and unzip it in the path of your preference and activate Squid-Cache RULE / blackweb.txt ya viene actualizada y optimizada para Squid-Cache. Descárguela y descomprimala en la ruta de su preferencia y active la REGLA de Squid-Cache

Download

wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -

Checksum

wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/checksum.md5
md5sum blackweb.txt | awk '{print $1}' && cat checksum.md5 | awk '{print $1}'

Squid-Cache Rule

Edit: / Edite:

/etc/squid/squid.conf

And add the following lines: / Y agregue las siguientes líneas:

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
acl blackweb dstdomain "/path_to/blackweb.txt"
http_access deny blackweb

Squid-Cache Advanced Rules (recommended to use) / Reglas Avanzadas (recomendadas para usar)

Blackweb contains millions of domains, therefore it is recommended: / Blackweb contiene millones de dominios, por tanto se recomienda:

• Use allowdomains.txt to exclude domains (e.g.: accounts.youtube.com since Feb 2014, Google uses the subdomain accounts.youtube.com to authenticate its services) or false positives / Usar allowdomains.txt para excluir dominios (ejemplo: accounts.youtube.com desde Feb 2014, Google utiliza el subdominio accounts.youtube.com para autenticar sus servicios) o falsos positivos
• Use blockdomains.txt to add domains not included in blackweb.txt (e.g.: .youtube.com .googlevideo.com, .ytimg.com, etc) / Usar blockdomains.txt para agregar dominios no incluidos en blackweb.txt (ejemplo: .youtube.com .googlevideo.com, .ytimg.com, etc.)

acl allowdomains dstdomain "/path_to/allowdomains.txt"
acl blockdomains dstdomain "/path_to/blockdomains.txt"
acl blackweb dstdomain "/path_to/blackweb.txt"
http_access allow allowdomains
http_access deny blockdomains
http_access deny blackweb

IMPORTANT

Blackweb is designed exclusively for Squid-Cache and due to the large number of blocked domains it is not recommended to use it in other environments (DNSMasq, Pi-Hole, Hosts File, etc.), as it could slow it down or block it. Use it at your own risk / Blackweb está diseñado exclusivamente para Squid-Cache y debido a la gran cantidad de dominios bloqueados no se recomienda usarlo en otros entornos (DNSMasq, Pi-Hole, Hosts File, etc.), ya que podría ralentizarlo o bloquearlo. Úselo bajo su propio riesgo

For more information check Issue 10

UPDATE

⚠️ WARNING: BEFORE YOU CONTINUE

This section is only to explain how update and optimization process works. It is not necessary for user to run it. This process can take time and consume a lot of hardware and bandwidth resources, therefore it is recommended to use test equipment / Esta sección es únicamente para explicar cómo funciona el proceso de actualización y optimización. No es necesario que el usuario la ejecute. Este proceso puede tardar y consumir muchos recursos de hardware y ancho de banda, por tanto se recomienda usar equipos de pruebas

Blackweb Update

The update process of blackweb.txt consists of several steps and is executed in sequence by the script bwupdate.sh / El proceso de actualización de blackweb.txt consta de varios pasos y es ejecutado en secuencia por el script bwupdate.sh

wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/bwupdate/bwupdate.sh && chmod +x bwupdate.sh && ./bwupdate.sh

Dependencies

git subversion squid bash tar zip wget piconv curl python idn2 xargs awk notify-send

Bandwidth Check (optional)

To guarantee update execution, before starting, script check bandwidth (with Speedtest). If it is > 1 Mbit/s, update continues; else, it shows warning messages and it is recommended to interrupt update / Para garantizar la ejecución de la actualización, antes de comenzar, el script verifica el acho de banda (con Speedtest). Si es > 1 Mbit/s, la actualización continúa; de lo contrario, muestra mensajes de advertencia y se recomienda interrumpir la actualización

Capture Public Blocklists

Capture domains from downloaded public blocklists (see SOURCES) and unifies them in a single file / Captura los dominios de las listas de bloqueo públicas descargadas (ver FUENTES) y las unifica en un solo archivo

Domain Debugging

Remove overlapping domains ('.sub.example.com' is a subdomain of '.example.com'), does homologation to Squid-Cache format and excludes false positives (google, hotmail, yahoo, etc.) with a allowlist (allowurls.txt) / Elimina dominios superpuestos ('.sub.example.com' es un dominio de '.example.com'), hace la homologación al formato de Squid-Cache y excluye falsos positivos (google, hotmail, yahoo, etc.) con una lista de permitidos (allowurls.txt)

com
.com
.domain.com
domain.com
0.0.0.0 domain.com
127.0.0.1 domain.com
::1 domain.com
domain.com.co
foo.bar.subdomain.domain.com
.subdomain.domain.com.co
www.domain.com
www.foo.bar.subdomain.domain.com
domain.co.uk
xxx.foo.bar.subdomain.domain.co.uk

outfile:

.domain.com
.domain.com.co
.domain.co.uk

TLD Validation

Remove domains with invalid TLDs (with a list of Public and Private Suffix TLDs: ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, eTLD, etc., up to 4th level 4LDs) / Elimina dominios con TLD inválidos (con una lista de TLDs Public and Private Suffix: ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, eTLD, etc., hasta 4to nivel 4LDs)

.domain.exe
.domain.com

outfile:

.domain.com

Debugging Punycode-IDN

Remove hostnames larger than 63 characters (RFC 1035) and other characters inadmissible by IDN and convert domains with international characters (not ASCII) and used for homologous attacks to Punycode/IDNA format / Elimina hostnames mayores a 63 caracteres (RFC 1035) y otros caracteres inadmisibles por IDN y convierte dominios con caracteres internacionales (no ASCII) y usados para ataques homográficos al formato Punycode/IDNA

.президент.рф
.mañana.com
.bücher.com
.café.fr
.köln-düsseldorfer-rhein-main.de
.mūsųlaikas.lt
.sendesık.com

outfile:

.xn--d1abbgf6aiiy.xn--p1ai
.xn--maana-pta.com
.xn--bcher-kva.com
.xn--caf-dma.fr
.xn--kln-dsseldorfer-rhein-main-cvc6o.de
.xn--mslaikas-qzb5f.lt
.xn--sendesk-wfb.com

DNS Loockup

Most of the SOURCES contain millions of invalid and nonexistent domains (see internet live stats). Then, each domain is verified via DNS and invalid and nonexistent are excluded from Blackweb (sent to fault.txt). This process may take. By default it processes domains in parallel ≈ 6k to 12k x min, depending on the hardware and bandwidth / La mayoría de las FUENTES contienen millones de dominios inválidos e inexistentes (vea internet live stats). Entonces se verifica cada dominio vía DNS y los inválidos e inexistentes se excluyen de Blackweb (enviados a fault.txt). Este proceso puede tardar. Por defecto procesa en paralelo dominios ≈ 6k a 12k x min, en dependencia del hardware y ancho de banda

HIT google.com
FAULT testfaultdomain.com

TLD Block

Add Block TLDs to block any domain that contains one. Edit blocktlds.txt and add or remove the TLDs you want to block / Agrega Block TLDs para bloquear cualquier dominio que contenga alguno. Edite blocktlds.txt y agrege o elimine los TLDs que quiera bloquear

.adult
.porn
.xxx
.domain.adult
.domain.porn
.subdomain.domain.xxx
.domain.com

outfile:

.adult
.porn
.xxx
.domain.com

Run Squid-Cache with Blackweb

Run Squid-Cache with Blackweb and any error sends it to SquidError.txt on your desktop / Corre Squid-Cache con Blackweb y cualquier error lo envía a SquidError.txt en su escritorio

Check execution (/var/log/syslog):

Blackweb: Done 06/05/2019 15:47:14

Important about Blackweb Update

• The default path of blackweb is /etc/acl. You can change it for your preference / El path por default de blackweb es /etc/acl. Puede cambiarlo por el de su preferencia
• bwupdate.sh includes lists of domains related to remote support (Teamviewer, Anydesk, logmein, etc). They are commented by default (unless their domains are in the SOURCES). To block or exclude them you must activate the corresponding line in the script (# JOIN LIST), although is not recommended to avoid conflicts or false positives / bwupdate.sh incluye listas de dominios relacionados con soporte remoto (Teamviewer, Anydesk, logmein, etc). Están comentadas por defecto (excepto que sus dominios estén en las FUENTES). Para bloquearlas o excluirlas debe activar la línea correspondiente en el script (# JOIN LIST), aunque no se recomienda para evitar conflictos o falsos positivos

SOURCES

Blocklists

Active Blocklists

• ABPindo indonesianadblockrules
• Adaway
• adblock-leadgenerator-list
• adblockplus malwaredomains_full
• Anti-WebMiner
• anudeepND Blocklist (included: coinminer, adservers)
• betterwebleon dga-feed
• BlackJack8 iOSAdblockList (included: iOSAdblockList and Scam Websites, Crypto Miners and Fake new
• Capitole - Direction du Système d'Information (DSI)
• Carl Spam
• cedia.org.ec (included: domains, immortal_domains)
• chadmayfield (included: porn_all, porn top)
• Cibercrime-Tracker
• cobaltdisco Google-Chinese-Results-Blocklist
• crazy-max WindowsSpyBlocker
• Dawsey21 List
• Disconnect.me (included: simple_ad, simple_malvertising, simple_tracking)
• dshield.org (included: Low, Medium, High)
• ethanr dns
• fanboy-adblock
• firebog.net (included: AdguardDNS, Airelle-hrsk, Airelle-trc, BillStearns, Easylist, Easyprivacy, Kowabit, Prigent-Ads, Prigent-Malware, Prigent-Phishing, Shalla-mal, WaLLy3K)
• gfmaster adblock-korea
• Halt-and-Block-Mining
• hBlock
• hexxium
• hostsfile.mine.nu
• iploggerfilter
• Joelotz URL
• Joewein
• KADhosts
• malc0de
• Malwaredomainlist Hosts
• malspam emotet list
• Matomo-org
• MESD
• mitchellkrogza (included: Badd-Boyz-Hosts, Hacked Malware Web Sites, Nginx Ultimate Bad Bot Blocker, The Big List of Hacked Malware Web Sites, Ultimate Hosts Blacklist)
• MobileAdTrackers
• NanoFilters
• Netlab360 DGA Domains
• NoCoin adblock list
• notabug latvian-list
• nothingblock
• Oleksiig
• openphish
• Perflyst (included: android-tracking, SmartTV)
• Peter Lowe’s Ad and tracking server list
• Quedlin
• quidsup (included: notrack-blocklists, notrack-malware)
• Ransomware Abuse (included: CryptoWall, Locky, Domain Blocklist, Ransomware Abuse ,URL Blocklist ,TorrentLocker)
• Ransomware Database
• reddestdream
• securemecca.net and hostsfile.org
• Shallalist.de
• Someonewhocares
• StevenBlack (included: add.2o7Net, add.Risk, fakenews-gambling-porn-social, hosts, spam, uncheckyAds)
• Stopforumspam Toxic Domains
• tankmohit UnifiedHosts
• Taz SpamDomains
• txthinking
• URLhaus Malicious URL Blocklist
• vokins yhosts
• Web Annoyances Ultralist
• Winhelp2002
• YousList
• zerodot1 CoinBlockerLists (included: Host, host_browser, host_optional, list, list_browser, list_browser_UBO)
• Zeustracker

Inactive Blocklists (Added to: oldurls.txt)

• 280blocker
• BambenekConsulting
• CHEF-KOCH BarbBlock-filter-list
• hosts-file.net (included: ad_servers, emd, grm, hosts, psh)
• Malware Domains
• margevicius easylistlithuania
• Neohost
• Passwall SpamAssassin
• squidblacklist.org (included: dg-ads, dg-malicious.acl)
• UrlBlacklist

Allowlist (URL/TLD)

Active Allowlist

• iana
• ipv6-hosts (Partial)
• publicsuffix
• University Domains and Names Data List
• whoisxmlapi

Inactive Allowlist (Added to: allowurls.txt)

• O365IPAddresses (No longer support. See This post)

Debug Lists

• Allow URLs
• Block TLDs
• Block Social
• Block URLs
• Fault URLs
• Invalid TLDs
• Old URls
• Hosts
• Remote
• Ultrasurf Domains Blocklist

Worktools

• Debug Squid-Cache Errors
• Parse Domains (modified)
• CTFR
• idn2
• speedtest and bandwidth

BACKLINKS

• OSINT Framework. Domain Name/Domain Blacklists/Blackweb
• Wikipedia. Blacklist_(computing)
• Zeltser. Free Blocklists of Suspected Malicious IPs and URLs
• Segu-Info. Análisis de malware y sitios web en tiempo real
• covert.io. Getting Started with DGA Domain Detection Research
• Keystone Solutions. blocklists
• Secrepo. Samples of Security Related Data
• Soficas. CiberSeguridad - Protección Activa
• Xploitlab. Projects using WindowsSpyBlocker
• Awesome Open Source. Blackweb
• Lifars. Sites with blocklist of malicious IPs and URLs

CONTRIBUTIONS

We thank all those who have contributed to this project. Those interested can contribute, sending us links of new lists, to be included in this project / Agradecemos a todos aquellos que han contribuido a este proyecto. Los interesados pueden contribuir, enviándonos enlaces de nuevas listas, para ser incluidas en este proyecto

Special thanks to: Jhonatan Sneider

DONATE

BTC: 3M84UKpz8AwwPADiYGQjT9spPKCvbqm4Bc

LICENCES

maravento.com is licensed under a Creative Commons Reconocimiento-CompartirIgual 4.0 Internacional License.

© 2021 Maravento Studio

DISCLAIMER

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.