0.11.14 • Published 5 years ago
proxybroker v0.11.14
node-proxybroker
Local installation
The proxybroker is implemented as typexs module. It's a framework for dynamic module invocation and integration. To use this implementation create a new directory and install the npm packages.
## Create the project directory and change to it
$ mkdir proxy-project && cd proxy-project
## Initialize npm project
$ npm init -y
## Mark project as typexs project,
## you need to set this key to enable the typexs functionality
$ node -p "JSON.stringify({...require('./package.json'), typexs: {name:'proxy-project'}}, null, 2)" > package.mod.json && \
mv package.mod.json package.json
## Install proxybroker
$ npm install proxybroker
## Create the configuration directory and file
$ mkdir config
$ touch config/typexs.yml
Add following content to the config/typexs.yml file:
# Configuration
proxy-broker:
startup: true
validator:
parallel: 50
judge:
selftest: true
remote_lookup: true
remote_ip: 127.0.0.1
ip: 0.0.0.0
request:
timeout: 5000
provider:
startup: true
parallel: 5
server:
proxyserver:
type: proxyserver
port: 3128
enable: true
timeout: 30000
# limit of search repeats if proxy failed
repeatLimit: 10
broker:
enable: true
timeouts:
incoming: 30000
forward: 2000
toProxy: true # default rotator will be used
schedules:
- name: fetch_proxies
offset: 4h
# startup: true
task:
name:
- proxy_fetch
validate: true
provider: __all__
- name: revalidate
offset: 30m
event:
name: validator_run_event
workers:
access:
- name: TaskQueueWorker
access: allow
Configuration
The configuration file must be under config/typexs.yml in the project directory.
TODO
proxy-broker
- startup
Usage
Proxyserver
Start the server with command:
typexs server
This command starts all instances of servers defined under the root key server in the configuration file.
Fetch proxies
Command:
typexs proxy-fetch [provider] [variant] -f json|csv -validate
- provider - the name of the defined proxy provider
- variant - is the possible different proxy
- -f - output format is default 'json', the other possible value is 'csv'
- --validate - run validation of grabbed data
- --store - store data also in backend
# Shows proxy variants
> typexs proxy-fetch
# Scan all proxy variants and return as json (can be piped to json file )
> typexs proxy-fetch __all__
> typexs proxy-fetch __all__ --validate --store
> typexs proxy-fetch __all__ > proxies.json
Validate only
Command:
typexs proxy-validate [string or filename] [-f json|csv] [--store]
Examples:
# Shows proxy variants
> typexs proxy-validate '127.0.0.1:3128'
> typexs proxy-validate '127.0.0.1:3128'
Server-mode
TODO
Docker
TODO
docker run -p 8080:8080 -p 8181:8181 -it proxybroker-pg
Testing
curl over proxy
$ curl -x http://localhost:3128 -L http://httpbin.org/headers
Notes
- If you running proxybroker locally or behind a NAT enviroment, you must expose the ports of the judge instance.