GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → dwisiswant0 → unew

dwisiswant0 / unew

Licence: MIT License
A tool for append URLs, skipping duplicates/paths & combine parameters.

Programming Languages

go
31211 projects - #10 most used programming language

Labels

url-parserwayback-machinealienvaultgo-tools

Projects that are alternatives of or similar to unew

httpparser
HTTP request, response and urls parser
Stars: ✭ 69 (-33.01%)
Mutual labels:  url-parser
WebCache
Chrome extension to view the cached version of the current webpage
Stars: ✭ 32 (-68.93%)
Mutual labels:  wayback-machine
libvcs
⚙️ Lite, typed, pythonic utilities for git, svn, mercurial, etc.
Stars: ✭ 43 (-58.25%)
Mutual labels:  url-parser
Ashok
Ashok is a OSINT Recon Tool , a.k.a 😍 Swiss Army knife .
Stars: ✭ 109 (+5.83%)
Mutual labels:  wayback-machine
uri
A type to represent, query, and manipulate a Uniform Resource Identifier.
Stars: ✭ 16 (-84.47%)
Mutual labels:  url-parser
urldedupe
Pass in a list of URLs with query strings, get back a unique list of URLs and query string combinations
Stars: ✭ 208 (+101.94%)
Mutual labels:  url-parser
emacs-go-tag
Edit field tags for golang struct fields
Stars: ✭ 49 (-52.43%)
Mutual labels:  go-tools
scrapy-wayback-machine
A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Stars: ✭ 92 (-10.68%)
Mutual labels:  wayback-machine
url
A C++ library that implements the URL WhatWG specification
Stars: ✭ 35 (-66.02%)
Mutual labels:  url-parser
vandal
Navigator for Web Archive
Stars: ✭ 146 (+41.75%)
Mutual labels:  wayback-machine
wayback
⏪ Tools to Work with the Various Internet Archive Wayback Machine APIs
Stars: ✭ 52 (-49.51%)
Mutual labels:  wayback-machine
Uri.js
Javascript URL mutation library
Stars: ✭ 6,119 (+5840.78%)
Mutual labels:  url-parser
galer
A fast tool to fetch URLs from HTML attributes by crawl-in.
Stars: ✭ 138 (+33.98%)
Mutual labels:  url-parser
Archivebox
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Stars: ✭ 12,383 (+11922.33%)
Mutual labels:  wayback-machine
Gops
A tool to list and diagnose Go processes currently running on your system
Stars: ✭ 5,404 (+5146.6%)
Mutual labels:  go-tools
Realize
Realize is the #1 Golang Task Runner which enhance your workflow by automating the most common tasks and using the best performing Golang live reloading.
Stars: ✭ 4,162 (+3940.78%)
Mutual labels:  go-tools

unew

u(rl)new — A tool for append URLs, skipping duplicates & combine parameters. Inspired by anew & qsreplace.

Usage

▶ cat urls.txt | unew
# or
▶ unew urls.txt
# or, save the results
▶ unew urls.txt output.txt

Flags

Usage of unew:

  -combine
        Combine parameters
  -r string
        Replace parameters value
  -skip-path value
        Skip specific paths (regExp pattern)

Install

with Go:

▶ go get -u github.com/dwisiswant0/unew

Workaround

If you have a urls.txt list as

https://twitter.com/dwisiswant0?href=evilzone.org
https://twitter.com/dwisiswant0
https://twitter.com/dwisiswant0?ref=github&utm_source=github
https://twitter.com/dwisiswant0/status/1305022512590278656
https://www.linkedin.com/in/dwisiswanto/
https://www.linkedin.com/in/dwisiswanto/?originalSubdomain=id
https://www.linkedin.com/in/dwisiswanto/?originalSubdomain=id&utm_medium=github

Regular

Sample workarounds:

▶ cat urls.txt | unew
https://twitter.com/dwisiswant0?href=evilzone.org
https://www.linkedin.com/in/dwisiswanto/

If the list contains multiple URLs with same path, it will save the first one and its parameters.

Combining parameters

But you can combine parameters if the same path exists by using -combine flag.

▶ cat urls.txt | unew -combine
https://twitter.com/dwisiswant0?href=evilzone.org&ref=github&utm_source=github
https://www.linkedin.com/in/dwisiswanto/?originalSubdomain=id&utm_medium=github

Query replacers

Use the -r flag if you want to change the value of all parameters.

▶ cat urls.txt | unew -combine -r "/etc/passwd"
https://twitter.com/dwisiswant0?href=%2Fetc%2Fpasswd&ref=%2Fetc%2Fpasswd&utm_source=%2Fetc%2Fpasswd
https://www.linkedin.com/in/dwisiswanto/?originalSubdomain=%2Fetc%2Fpasswd&utm_medium=%2Fetc%2Fpasswd

Skipping paths

In case if you want to pass specific/multiple URL paths, you can use -skip-path flag for it (can be set multiple times). But, you have to write it with regExp pattern.

▶ cat urls.txt | unew -skip-path "^/[\w]+/status/[0-9]+" -skip-path "/in/[\w]+"
https://twitter.com/dwisiswant0?href=evilzone.org
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].