web-proxy v1.1.2

Web Proxy

Web Proxy (forward proxy) for intercepting and selectively caching HTTP requests.

Use Case

web-proxy has been designed to selectively cache outgoing HTTP requests for logging and re-iteration purposes, e.g. if you are running an inefficient web scrapping service or wish to re-run scrapping service using earlier fetched pages.

Demo

Illustration demonstrates cURL requests being made using web-proxy.

web-proxy is configured to:

• cache all HTTP GET requests that result in 200 response.
• to forward all resulting HTTP requests to further proxy (mitmproxy).

Command Line Usage

node ./bin/proxy --help

MySQL

Web Proxy can be used with a persistent data store. The only backend supported at the moment is MySQL.

To enable use of the MySQL backend, provide connections credentials at the time of starting the proxy.

Database schema can be obtained from ./database/proxy.sql. Note that table is using ROW_FORMAT=COMPRESSED. In order to benefit from the compression, ensure that the following MySQL variables are set:

innodb_file_format=BARRACUDA
innodb_file_per_table=ON

For more information, refer to http://stackoverflow.com/a/13636565/368691.

Proxy

Web Proxy can forward all outgoing HTTP requests to another proxy.

To enable forwarding, provide proxy credentials at the time of starting the proxy.

node ./bin/proxy --help

API

WebProxy can be used programmatically.

var WebProxy = require('../src/webproxy'),
    config = {},
    server;

/**
 * @param {Object} reference
 * @param {String} reference.method
 * @param {String} reference.url
 * @return {Null} Returning null will allow HTTP request to progress.
 * @return {Object} response
 * @return {Number} response.statusCode
 * @return {Object} response.headers
 * @return {String} response.body
 */
config.read = function (request) {
    //
};

/**
 * @param {Object} reference
 * @param {String} reference.method
 * @param {String} reference.url
 * @param {Object} response
 * @param {Number} response.statusCode
 * @param {Object} response.headers
 * @param {String} response.body
 */
config.write = function (request, response) {
    //
};

/**
 * @param {Object} config
 * @param {Function} config.read
 * @param {Function} config.write
 * @param {Object} config.logger
 */
server = WebProxy(config);

server.listen(9000);

Data Store

Data can be read/written using custom logic.

There are two existing data store interfaces:

Refer to the ./bin/proxy.js implementation to see a working example.