Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

Stars: ✭ 195 (-3.94%)

Mutual labels: crawler, spider

Js Reverse

JS逆向研究

Stars: ✭ 159 (-21.67%)

Mutual labels: crawler, spider

Proxy pool

Python爬虫代理IP池(proxy pool)

Stars: ✭ 13,964 (+6778.82%)

Mutual labels: crawler, spider

Yispider

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。内置一套爬虫定义规则（模版），可使用模版快速定义爬虫，也可当作框架手动开发爬虫。(兴趣使然的项目，用的不爽了就更新)

Stars: ✭ 158 (-22.17%)

Mutual labels: crawler, spider

Spoon

🥄 A package for building specific Proxy Pool for different Sites.

Stars: ✭ 173 (-14.78%)

Mutual labels: crawler, spider

Jlitespider

A lite distributed Java spider framework :-)

Stars: ✭ 151 (-25.62%)

Mutual labels: crawler, spider

Python3 Spider

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Stars: ✭ 2,129 (+948.77%)

Mutual labels: crawler, spider

Zhihu Crawler People

A simple distributed crawler for zhihu && data analysis

Stars: ✭ 182 (-10.34%)

Mutual labels: crawler, spider

Goribot

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Stars: ✭ 190 (-6.4%)

Mutual labels: crawler, spider

View All Similar Projects ➔

JSSoup

I'm a fan of Python library BeautifulSoup. It's feature-rich and very easy to use. But when I am working on a small react-native project, and I tried to find a HTML parser library like BeautifulSoup, I failed.
So I want to write a HTML parser library which can be so easy to use just like BeautifulSoup in Javascript.
JSSoup uses tautologistics/node-htmlparser as HTML dom parser, and creates a series of BeautifulSoup like API on top of it.
JSSoup supports both node and react-native.

Naming Style

JSSoup tries to use the same interfaces as BeautifulSoup so BeautifulSoup user can use JSSoup seamlessly. However, JSSoup uses Javascript's camelCase naming style instead of Python's underscore naming style. Such as find_all() in BeautifulSoup is replaced as findAll().

Install

$ npm install jssoup

How to use JSSoup

Import

//react-native
import JSSoup from 'jssoup'; 
// nodejs
var JSSoup = require('jssoup').default;

Make Soup

var soup = new JSSoup('<html><head>hello</head></html>');

The text element only contains whitespace will be ignored by default. To disable this feature, set second parameter of JSSoup to false. This parameter is "ignoreWhitespace" and will be passed into htmlparser.

var soup = new JSSoup('<html><head>hello</head></html>', false);

Name

var soup = new JSSoup('<html><head>hello</head></html>');
var tag = soup.find('head');
tag.name
// 'head'
tag.name = 'span'
console.log(tag)
//<span>hello</span>

Attributes

var soup = new JSSoup('<tag id="hi" class="banner">hello</tag>');
var tag = soup.nextElement;
tag.attrs
// {id: 'hi', class: 'banner'} 
tag.attrs.id = 'test';
console.log(tag)
// <tag id="test" class="banner">hello</tag>

Navigation

.previousElement, .nextElement

var data = `
<div>
  <a>1</a>
  <b>2</b>
  <c>3</c>
</div>
`
var soup = new JSSoup(data);
var div = soup.nextElement;
var b = div.nextElement.nextElement;
// b.string: '2'
var a = b.previousElement;
// a.string: '1'

.previousSibling, .nextSibling

var soup = new JSSoup(data);
var div = soup.nextElement;
var a = div.nextElement;
var b = a.nextSibling;
var c = b.nextSibling;
c.nextSibling == undefined;

.contents

div.contents
// [<a>1</a>, <b>2</b>, <c>3</c>]

.descendants

div.descendants
// [<a>1</a>, 1, <b>2</b>, 2, <c>3</c>, 3]

.parent

div.parent == soup

Edit

.extract()

b.extract();
div.contents
// [<a>1</a>, <c>3</c>]

.append()

b.extract();
div.append(b)
div.contents
// [<a>1</a>, <c>3</c>, <b>2</b>]

.insert(position, new Element)

d.prettify('', '')
// <d>4</d>
div.insert(1, d)
div.contents
// [<a>1</a>, <d>4</d>, <b>2</b>, <c>3</c>]

.replaceWith(new Element)

d.prettify('', '')
// <d>4</d>
b.replaceWith(d)
div.contents
// [<a>1</a>, <d>4</d>, <c>3</c>]

c.string.replaceWith('new')
div.contents
// [<a>1</a>, <d>4</d>, <c>new</c>]

Search

.findAll()

var data = `
<div>
  <div class="h1"></div>
  <a>hello</a>
</div>
`
var soup = new JSSoup(data);
soup.findAll('a')
// [<a>hello</a>]
soup.findAll('div', 'h1')
// [<div class="h1"></div>]

.find()

var data = `
<div>
  <p> hello </p>
  <p> world </p>
</div>
`
var soup = new JSSoup(data);
soup.find('p')
// <p> hello </p>

.findNextSibling()

var data = `
<div>
  <span> test </span>
  <div> div </div>
  <p> hello </p>
  <p> world </p>
</div>
`
var soup = new JSSoup(data);
var span = soup.find('span');
span.findNextSibling('p')
// <p> hello </p>

.findNextSiblings()

var data = `
<div>
  <span> test </span>
  <div> div </div>
  <p> hello </p>
  <p> world </p>
</div>
`
var soup = new JSSoup(data);
var span = soup.find('span');
span.findNextSiblings('p')
// <p> hello </p>
// <p> world </p>

.findPreviousSibling()

var data = `
<div>
  <p> hello </p>
  <p> world </p>
  <div> div </div>
  <span> test </span>
</div>
`
var soup = new JSSoup(data);
var span = soup.find('span');
span.findPreviousSibling('p')
// <p> world </p>

.findPreviousSiblings()

var data = `
<div>
  <p> hello </p>
  <p> world </p>
  <div> div </div>
  <span> test </span>
</div>
`
var soup = new JSSoup(data);
var span = soup.find('span');
span.findPreviousSiblings('p')
// <p> hello </p>
// <p> world </p>

Output

.prettify()

var soup = new JSSoup('<html><head>hello</head></html>');
soup.prettify()
// <html>
//  <head>
//   hello
//  </head>
// </html>

.getText(), .text

div.text
// '123'
div.getText('|')
// '1|2|3'

.string

b.string == '2';
var soup = new JSSoup('<html><head>hello</head></html>');
soup.string == 'hello';

Run Test

npm test

Status

There's a lot of work need to be done.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].

Stars: ✭ 203

Visit Git Page 🔗Visit User Page 🔗Visit Issues Page (3) 🔗