# nex-scraper

> Easily scrape your proxies from your favorite Websites

Latest version **1.0.6** (published 2020-12-14) · ISC license · 0 weekly downloads

## Install

```sh
npm install nex-scraper
pnpm add nex-scraper
yarn add nex-scraper
bun add nex-scraper
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.6 |
| Published | 2020-12-14 |
| First published | 2020-08-15 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 5 |
| Unpacked size | 28.7 KB |
| Known vulnerabilities | 0 (+1 in 1 direct dependencies) |
| Install scripts | no |
| Author | NeutronX |
| Maintainers | yaxeldragon |
| Keywords | proxies, http, https, socks4, socks5, socks, proxy, proxy-scrape, proxy scrape, scrape, scraper, proxy-scraper, proxy scraper, proxy list, proxy-list, free, free proxy, free proxy list, proxy checker, check, sockscheck, proxy-chcker, proxy_checker |

## Links

- npm: https://www.npmjs.com/package/nex-scraper
- Homepage: https://github.com/NeutronX-dev/nex-scraper
- npm.io page: https://npm.io/package/nex-scraper

## Dependencies (5)

- [ws](https://npm.io/package/ws.md) ^7.4.0
- [socks](https://npm.io/package/socks.md) ^1.1.10
- [request](https://npm.io/package/request.md) ^2.88.2
- [proxy-agent](https://npm.io/package/proxy-agent.md) ^4.0.0
- [socks-proxy-agent](https://npm.io/package/socks-proxy-agent.md) ^5.0.0

## Alternatives

- [@tsparticles/shape-image](https://npm.io/package/@tsparticles/shape-image.md) — 303.7K weekly downloads
- [@tsparticles/shape-line](https://npm.io/package/@tsparticles/shape-line.md) — 233.7K weekly downloads
- [stringify-attributes](https://npm.io/package/stringify-attributes.md) — 58.6K weekly downloads
- [mobile-drag-drop](https://npm.io/package/mobile-drag-drop.md) — 46.3K weekly downloads
- [@comunica/actor-rdf-parse-html](https://npm.io/package/@comunica/actor-rdf-parse-html.md) — 29.2K weekly downloads

## Recent versions

- 1.0.6 (latest) — 2020-12-14
- 1.0.4 — 2020-09-02
- 1.0.3 — 2020-08-26
- 1.0.2 — 2020-08-23
- 1.0.1 — 2020-08-21
- 1.0.0 — 2020-08-15

## README

# nex-scraper
Simple proxy scraper by **NeutronX** (`yaxeldragon`)
## Terms
When downloading this module/script you accept that
> The user/company should use it under his/her/their own responsability.

## How it Works
Once you input the **URLs** you want to look in, It will send a **`request`**, check if each proxy is an actual proxy using **`regex`**. Then It will give you access to **all proxies** in an array from an **`async`** funcion.
### Features
- [x] Custom **URL** Scrape.
- [x] Filter from an **HTML** document.
- [x] Check proxies with **`Regex`**.
- [x] Remove **all** Duplicates.
- [x] Easily access to all Proxies.

Area | Quick Description
------------ | -------------
[Get Started](#get-started) | How to start using **`nex-scraper`**
[How to Scrape](#how-to-scrape) | How to get the Proxies.
[Configuration](#configuration) | Customize **`nex-scraper`**.
[How to Check](#checking) | Check your proxies **`nex-scraper`**.
[Versions](#versions) | See what's new.

## Get Started
To install **`nex-scraper`** in your project you need to input the following in your console:
```
npm install nex-scraper
```
Once **`nex-scraper`** is installed you can import it in your project by using the following methods:
```js
const proxy = require('nex-scraper');
const proxies = new proxy([
    '(input URL here)',
    '(another URL here...)'
]);
// add more later on using
proxies.addURL([
    '(input URL here)',
    '(another URL here...)'
])
```
You can do as many **`objects`** as you want with the **`URLs`** you want.

## How to Scrape
your project by using the following methods:
```js
const proxy = require('nex-scraper');
const proxies = new proxy([
    'http://www.live-socks.net/2020/05/28-05-20-socks-5-servers.html',
    // ^^^^^^^ This Is HTML (It will get filtered)
    'https://api.proxyscrape.com/?request=getproxies&proxytype=socks5'
    // ^^^^^^^ This Is a normal API (RAW text)
]);
proxies.scrape().then(proxies => {
    // Here you can access the Proxies
});
```

# Configuration
Here you can decide weather you remove duplicates or not, if you want to it to filter something in specific, etc...
```js
// All configurations:
{
    removeDuplicates: true,     // bool
    filter: 'XXX.XXX.XXX:XXXX', // string
    forEach: () => {}           // function
}
```
> [How to enable/disable removing Duplicates.](#remove-duplicates)

> [Customise what to Filter.](#custom-filter)

> [forEach proxy.](#foreach-callback)
## Remove Duplicates
```js
const proxy = require('nex-scraper');
const proxies = new proxy([
    'URLs...'
]);
proxies.setConfig({
    removeDuplicates: true, // Default: true
});
```
This will **NOT** remove **ANY** duplicates, so you may get a same proxy more than one time.
## Custom Filter
```js
const proxy = require('nex-scraper');
const proxies = new proxy([
    'URLs...'
]);
proxies.setConfig({
    filter: "XXX.XXX.XXX.XXX:XXXX", // Default: true
});
```
**PD**: The default filter has a longer range than anything customizable.

This will **ONLY** return proxies that looks like **`XXX.XXX.XXX.XXX:XXXX`**. For example: **`XXX.XXX.XXX.XXX:XXXX`** to look like => **`123.456.789.123:4567`**.
### More examples:
```js
// -------------------------------------
proxies.setConfig({
    filter:  "XXX.XXX.XXX.XXX:XXXX",
    // Look: "123.456.789.123:4567"
});
// -------------------------------------
proxies.setConfig({
    filter:  "XX.XXX.XXX.XX:XXXX",
    // Look: "12.345.678.91:2345"
});
// -------------------------------------

```
## forEach Callback
```js
const proxy = require('nex-scraper');
const proxies = new proxy([
    'URLs...'
]);
proxies.setConfig({
    foreach: (proxy) => {
        console.log(`Gotten ${proxy}`);
    }
});
```
It will run the gotten callback for each proxy it gets.
# Checking
This area of this module will allow you to test the proxies you have just scraped, and supports **`SOCKS`**, and **`HTTPs`**.
## Preview: 
```js
const nex = require('nex-scraper');
const proxies = new nex([
    'URLs...'
]);
const checker = new nex.checker();
```

## `#constructor`: new \<checker_instance>(optional_URL)
Inside here goes a **WebSocket** URL to test on. And this is optional becuase it all ready includes an URL.

## `#setTimeout`: \<checker_object>.setTimeout(time)
Sets a limit on how much time each proxy spends checking.

## `#setHeaders`: \<checker_object>.setHeaders(headers)
Sets headers to the **`WebSockets`** connections.

## `#checkOne`: \<checker_object>.checkOne(type, proxy)
Checks one proxy with a **`type`** agent (http, or socks)
## `#checkMany`: \<checker_object>.checkMany(type, proxies_array)
Checks multiple proxy with a **`type`** agent (http, or socks)
## Versions
> [1.0.6](https://www.npmjs.com/package/nex-scraper/v/1.0.5) - Log:
> * Documentation Fix

> [1.0.5](https://www.npmjs.com/package/nex-scraper/v/1.0.5) - Log:
> * Checker Prototype

> [1.0.4](https://www.npmjs.com/package/nex-scraper/v/1.0.4) - Log:
> * Crash Fix

> [1.0.3](https://www.npmjs.com/package/nex-scraper/v/1.0.3) - Log:
> * Added [forEach Callback](#foreach-callback) Configuration.

> [1.0.2](https://www.npmjs.com/package/nex-scraper/v/1.0.2) - Log:
> * Added [version](#versions) Area.
> * Added [configuration](#configuration) Method.

> * [1.0.1](https://www.npmjs.com/package/nex-scraper/v/1.0.1) - Patched crashes when Scraping.

> * [1.0.0](https://www.npmjs.com/package/nex-scraper/v/1.0.0) - First Release.

# Happy Coding!

---
_Source: https://npm.io/package/nex-scraper · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._