All Projects → PuerkitoBio → Purell

PuerkitoBio / Purell

Licence: bsd-3-clause
tiny Go library to normalize URLs

Programming Languages

go
31211 projects - #10 most used programming language

Labels

Projects that are alternatives of or similar to Purell

Netswift
A type-safe, high-level networking solution for Swift apps
Stars: ✭ 21 (-94.63%)
Mutual labels:  url
Ffrouter
Powerful and easy-to-use URL routing library in iOS that supports URL Rewrite(强大、易用、支持 URL Rewrite的 iOS 路由库)
Stars: ✭ 263 (-32.74%)
Mutual labels:  url
Url Polyfill
Polyfill URL and URLSearchParams to match last ES7 specifications
Stars: ✭ 294 (-24.81%)
Mutual labels:  url
ngx-linkifyjs
Angular V8 wrapper for linkifyjs - library for finding links in plain text and converting them to HTML <a> tags via linkifyjs
Stars: ✭ 40 (-89.77%)
Mutual labels:  url
Parse Domain
Splits a hostname into subdomains, domain and (effective) top-level domains.
Stars: ✭ 261 (-33.25%)
Mutual labels:  url
Url Signer
Create and validate signed URLs with a limited lifetime
Stars: ✭ 273 (-30.18%)
Mutual labels:  url
cpplipa
C++ library package
Stars: ✭ 17 (-95.65%)
Mutual labels:  url
Uri Parser
RFC3986/RFC3987 compliant URI parser
Stars: ✭ 342 (-12.53%)
Mutual labels:  url
Frontexpress
An Express.js-Style router for the front-end
Stars: ✭ 263 (-32.74%)
Mutual labels:  url
Url Regex
Regular expression for matching URLs
Stars: ✭ 286 (-26.85%)
Mutual labels:  url
custom-permalinks
Set custom permalinks on a per-post basis in WordPress
Stars: ✭ 17 (-95.65%)
Mutual labels:  url
Uri
🌏 Functions for making sense out of URIs in PHP
Stars: ✭ 259 (-33.76%)
Mutual labels:  url
Native Url
Node's url module implemented using the built-in URL API.
Stars: ✭ 275 (-29.67%)
Mutual labels:  url
tall
Promise-based, No-dependency URL unshortner (expander) module for Node.js
Stars: ✭ 56 (-85.68%)
Mutual labels:  url
Mnmlurl
🔗 Minimal URL - Modern URL shortener with support for custom alias & can be hosted even in GitHub pages [DEPRECATED]
Stars: ✭ 311 (-20.46%)
Mutual labels:  url
requrl
Grab full URL from request.
Stars: ✭ 17 (-95.65%)
Mutual labels:  url
Django Multiurl
Have you ever wanted multiple views to match to the same URL? Now you can.
Stars: ✭ 268 (-31.46%)
Mutual labels:  url
Url
URL Standard
Stars: ✭ 343 (-12.28%)
Mutual labels:  url
E2guardian
E2guardian is a web content filter that can work in proxy, transparent or icap server modes
Stars: ✭ 340 (-13.04%)
Mutual labels:  url
Searchwithmybrowser
Open Cortana searches with your default browser.
Stars: ✭ 285 (-27.11%)
Mutual labels:  url

Purell

Purell is a tiny Go library to normalize URLs. It returns a pure URL. Pure-ell. Sanitizer and all. Yeah, I know...

Based on the wikipedia paper and the RFC 3986 document.

build status

Install

go get github.com/PuerkitoBio/purell

Changelog

  • v1.1.1 : Fix failing test due to Go1.12 changes (thanks to @ianlancetaylor).
  • 2016-11-14 (v1.1.0) : IDN: Conform to RFC 5895: Fold character width (thanks to @beeker1121).
  • 2016-07-27 (v1.0.0) : Normalize IDN to ASCII (thanks to @zenovich).
  • 2015-02-08 : Add fix for relative paths issue (PR #5) and add fix for unnecessary encoding of reserved characters (see issue #7).
  • v0.2.0 : Add benchmarks, Attempt IDN support.
  • v0.1.0 : Initial release.

Examples

From example_test.go (note that in your code, you would import "github.com/PuerkitoBio/purell", and would prefix references to its methods and constants with "purell."):

package purell

import (
  "fmt"
  "net/url"
)

func ExampleNormalizeURLString() {
  if normalized, err := NormalizeURLString("hTTp://someWEBsite.com:80/Amazing%3f/url/",
    FlagLowercaseScheme|FlagLowercaseHost|FlagUppercaseEscapes); err != nil {
    panic(err)
  } else {
    fmt.Print(normalized)
  }
  // Output: http://somewebsite.com:80/Amazing%3F/url/
}

func ExampleMustNormalizeURLString() {
  normalized := MustNormalizeURLString("hTTpS://someWEBsite.com:443/Amazing%fa/url/",
    FlagsUnsafeGreedy)
  fmt.Print(normalized)

  // Output: http://somewebsite.com/Amazing%FA/url
}

func ExampleNormalizeURL() {
  if u, err := url.Parse("Http://SomeUrl.com:8080/a/b/.././c///g?c=3&a=1&b=9&c=0#target"); err != nil {
    panic(err)
  } else {
    normalized := NormalizeURL(u, FlagsUsuallySafeGreedy|FlagRemoveDuplicateSlashes|FlagRemoveFragment)
    fmt.Print(normalized)
  }

  // Output: http://someurl.com:8080/a/c/g?c=3&a=1&b=9&c=0
}

API

As seen in the examples above, purell offers three methods, NormalizeURLString(string, NormalizationFlags) (string, error), MustNormalizeURLString(string, NormalizationFlags) (string) and NormalizeURL(*url.URL, NormalizationFlags) (string). They all normalize the provided URL based on the specified flags. Here are the available flags:

const (
	// Safe normalizations
	FlagLowercaseScheme           NormalizationFlags = 1 << iota // HTTP://host -> http://host, applied by default in Go1.1
	FlagLowercaseHost                                            // http://HOST -> http://host
	FlagUppercaseEscapes                                         // http://host/t%ef -> http://host/t%EF
	FlagDecodeUnnecessaryEscapes                                 // http://host/t%41 -> http://host/tA
	FlagEncodeNecessaryEscapes                                   // http://host/!"#$ -> http://host/%21%22#$
	FlagRemoveDefaultPort                                        // http://host:80 -> http://host
	FlagRemoveEmptyQuerySeparator                                // http://host/path? -> http://host/path

	// Usually safe normalizations
	FlagRemoveTrailingSlash // http://host/path/ -> http://host/path
	FlagAddTrailingSlash    // http://host/path -> http://host/path/ (should choose only one of these add/remove trailing slash flags)
	FlagRemoveDotSegments   // http://host/path/./a/b/../c -> http://host/path/a/c

	// Unsafe normalizations
	FlagRemoveDirectoryIndex   // http://host/path/index.html -> http://host/path/
	FlagRemoveFragment         // http://host/path#fragment -> http://host/path
	FlagForceHTTP              // https://host -> http://host
	FlagRemoveDuplicateSlashes // http://host/path//a///b -> http://host/path/a/b
	FlagRemoveWWW              // http://www.host/ -> http://host/
	FlagAddWWW                 // http://host/ -> http://www.host/ (should choose only one of these add/remove WWW flags)
	FlagSortQuery              // http://host/path?c=3&b=2&a=1&b=1 -> http://host/path?a=1&b=1&b=2&c=3

	// Normalizations not in the wikipedia article, required to cover tests cases
	// submitted by jehiah
	FlagDecodeDWORDHost           // http://1113982867 -> http://66.102.7.147
	FlagDecodeOctalHost           // http://0102.0146.07.0223 -> http://66.102.7.147
	FlagDecodeHexHost             // http://0x42660793 -> http://66.102.7.147
	FlagRemoveUnnecessaryHostDots // http://.host../path -> http://host/path
	FlagRemoveEmptyPortSeparator  // http://host:/path -> http://host/path

	// Convenience set of safe normalizations
	FlagsSafe NormalizationFlags = FlagLowercaseHost | FlagLowercaseScheme | FlagUppercaseEscapes | FlagDecodeUnnecessaryEscapes | FlagEncodeNecessaryEscapes | FlagRemoveDefaultPort | FlagRemoveEmptyQuerySeparator

	// For convenience sets, "greedy" uses the "remove trailing slash" and "remove www. prefix" flags,
	// while "non-greedy" uses the "add (or keep) the trailing slash" and "add www. prefix".

	// Convenience set of usually safe normalizations (includes FlagsSafe)
	FlagsUsuallySafeGreedy    NormalizationFlags = FlagsSafe | FlagRemoveTrailingSlash | FlagRemoveDotSegments
	FlagsUsuallySafeNonGreedy NormalizationFlags = FlagsSafe | FlagAddTrailingSlash | FlagRemoveDotSegments

	// Convenience set of unsafe normalizations (includes FlagsUsuallySafe)
	FlagsUnsafeGreedy    NormalizationFlags = FlagsUsuallySafeGreedy | FlagRemoveDirectoryIndex | FlagRemoveFragment | FlagForceHTTP | FlagRemoveDuplicateSlashes | FlagRemoveWWW | FlagSortQuery
	FlagsUnsafeNonGreedy NormalizationFlags = FlagsUsuallySafeNonGreedy | FlagRemoveDirectoryIndex | FlagRemoveFragment | FlagForceHTTP | FlagRemoveDuplicateSlashes | FlagAddWWW | FlagSortQuery

	// Convenience set of all available flags
	FlagsAllGreedy    = FlagsUnsafeGreedy | FlagDecodeDWORDHost | FlagDecodeOctalHost | FlagDecodeHexHost | FlagRemoveUnnecessaryHostDots | FlagRemoveEmptyPortSeparator
	FlagsAllNonGreedy = FlagsUnsafeNonGreedy | FlagDecodeDWORDHost | FlagDecodeOctalHost | FlagDecodeHexHost | FlagRemoveUnnecessaryHostDots | FlagRemoveEmptyPortSeparator
)

For convenience, the set of flags FlagsSafe, FlagsUsuallySafe[Greedy|NonGreedy], FlagsUnsafe[Greedy|NonGreedy] and FlagsAll[Greedy|NonGreedy] are provided for the similarly grouped normalizations on wikipedia's URL normalization page. You can add (using the bitwise OR | operator) or remove (using the bitwise AND NOT &^ operator) individual flags from the sets if required, to build your own custom set.

The full godoc reference is available on gopkgdoc.

Some things to note:

  • FlagDecodeUnnecessaryEscapes, FlagEncodeNecessaryEscapes, FlagUppercaseEscapes and FlagRemoveEmptyQuerySeparator are always implicitly set, because internally, the URL string is parsed as an URL object, which automatically decodes unnecessary escapes, uppercases and encodes necessary ones, and removes empty query separators (an unnecessary ? at the end of the url). So this operation cannot not be done. For this reason, FlagRemoveEmptyQuerySeparator (as well as the other three) has been included in the FlagsSafe convenience set, instead of FlagsUnsafe, where Wikipedia puts it.

  • The FlagDecodeUnnecessaryEscapes decodes the following escapes (from -> to): - %24 -> $ - %26 -> & - %2B-%3B -> +,-./0123456789:; - %3D -> = - %40-%5A -> @ABCDEFGHIJKLMNOPQRSTUVWXYZ - %5F -> _ - %61-%7A -> abcdefghijklmnopqrstuvwxyz - %7E -> ~

  • When the NormalizeURL function is used (passing an URL object), this source URL object is modified (that is, after the call, the URL object will be modified to reflect the normalization).

  • The replace IP with domain name normalization (http://208.77.188.166/ → http://www.example.com/) is obviously not possible for a library without making some network requests. This is not implemented in purell.

  • The remove unused query string parameters and remove default query parameters are also not implemented, since this is a very case-specific normalization, and it is quite trivial to do with an URL object.

Safe vs Usually Safe vs Unsafe

Purell allows you to control the level of risk you take while normalizing an URL. You can aggressively normalize, play it totally safe, or anything in between.

Consider the following URL:

HTTPS://www.RooT.com/toto/t%45%1f///a/./b/../c/?z=3&w=2&a=4&w=1#invalid

Normalizing with the FlagsSafe gives:

https://www.root.com/toto/tE%1F///a/./b/../c/?z=3&w=2&a=4&w=1#invalid

With the FlagsUsuallySafeGreedy:

https://www.root.com/toto/tE%1F///a/c?z=3&w=2&a=4&w=1#invalid

And with FlagsUnsafeGreedy:

http://root.com/toto/tE%1F/a/c?a=4&w=1&w=2&z=3

TODOs

  • Add a class/default instance to allow specifying custom directory index names? At the moment, removing directory index removes (^|/)((?:default|index)\.\w{1,4})$.

Thanks / Contributions

@rogpeppe @jehiah @opennota @pchristopher1275 @zenovich @beeker1121

License

The BSD 3-Clause license.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].