urllib v3.23.0
urllib
Request HTTP URLs in a complex world — basic and digest authentication, redirections, timeout and more.
Install
npm install urllib
Usage
TypeScript and ESM
import { request } from 'urllib';
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);
CommonJS
const { request } = require('urllib');
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);
API Doc
Method: async request(url[, options])
Arguments
- url String | Object - The URL to request, either a String or a Object that return by url.parse.
- options Object - Optional
- method String - Request method, defaults to
GET
. Could beGET
,POST
,DELETE
orPUT
. Alias 'type'. - data Object - Data to be sent. Will be stringify automatically.
- content String | Buffer - Manually set the content of payload. If set,
data
will be ignored. - stream stream.Readable - Stream to be pipe to the remote. If set,
data
andcontent
will be ignored. - writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and
callback
will be called withdata
setnull
after finished writing. - files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with
multipart/form-data
format, base onformstream
. Ifmethod
not set, will usePOST
method by default. - contentType String - Type of request data. Could be
json
(Notes: not useapplication/json
here). If it'sjson
, will auto setContent-Type: application/json
header. - dataType String - Type of response data. Could be
text
orjson
. If it'stext
, thecallback
eddata
would be a String. If it'sjson
, thedata
of callback would be a parsed JSON Object and will auto setAccept: application/json
header. Defaultcallback
eddata
would be aBuffer
. - fixJSONCtlChars Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is
false
. - headers Object - Request headers.
- timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Default is
5000
. You can usetimeout: 5000
to tell urllib use same timeout on two phase or set them seperately such astimeout: [3000, 5000]
, which will set connecting timeout to 3s and response 5s. - keepAliveTimeout
number | null
- Default is4000
, 4 seconds - The timeout after which a socket without active requests will time out. Monitors time between activity on a connected socket. This value may be overridden by keep-alive hints from the server. See MDN: HTTP - Headers - Keep-Alive directives for more details. - auth String -
username:password
used in HTTP Basic Authorization. - digestAuth String -
username:password
used in HTTP Digest Authorization. - followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to true.
- maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
- formatRedirectUrl Function - Format the redirect url by your self. Default is
url.resolve(from, to)
. - beforeRequest Function - Before request hook, you can change every thing here.
- streaming Boolean - let you get the
res
object when request connected, defaultfalse
. aliascustomResponse
- compressed Boolean - Accept
gzip, br
response content and auto decode it, default istrue
. - timing Boolean - Enable timing or not, default is
true
. - socketPath String | null - request a unix socket service, default is
null
. - highWaterMark Number - default is
67108864
, 64 KiB.
- method String - Request method, defaults to
Options: options.data
When making a request:
await request('https://example.com', {
method: 'GET',
data: {
'a': 'hello',
'b': 'world',
},
});
For GET
request, data
will be stringify to query string, e.g. http://example.com/?a=hello&b=world
.
For others like POST
, PATCH
or PUT
request,
in defaults, the data
will be stringify into application/x-www-form-urlencoded
format
if content-type
header is not set.
If content-type
is application/json
, the data
will be JSON.stringify
to JSON data format.
Options: options.content
options.content
is useful when you wish to construct the request body by yourself,
for example making a content-type: application/json
request.
Notes that if you want to send a JSON body, you should stringify it yourself:
await request('https://example.com', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
content: JSON.stringify({
a: 'hello',
b: 'world',
}),
});
It would make a HTTP request like:
POST / HTTP/1.1
host: example.com
content-type: application/json
{
"a": "hello",
"b": "world"
}
This exmaple can use options.data
with application/json
content type:
await request('https://example.com', {
method: 'POST',
headers: {
'content-type': 'application/json'
},
data: {
a: 'hello',
b: 'world',
}
});
Options: options.files
Upload a file with a hello
field.
await request('https://example.com/upload', {
method: 'POST',
files: __filename,
data: {
hello: 'hello urllib',
},
});
Upload multi files with a hello
field.
await request('https://example.com/upload', {
method: 'POST',
files: [
__filename,
fs.createReadStream(__filename),
Buffer.from('mock file content'),
],
data: {
hello: 'hello urllib with multi files',
},
});
Custom file field name with uploadfile
.
await request('https://example.com/upload', {
method: 'POST',
files: {
uploadfile: __filename,
},
});
Response Object
Response is normal object, it contains:
status
orstatusCode
: response status code.-1
meaning some network error likeENOTFOUND
-2
meaning ConnectionTimeoutError
headers
: response http headers, default is{}
size
: response sizeaborted
: response was aborted or notrt
: total request and response time in ms.timing
: timing object if timing enable.socket
: socket info
Run test with debug log
NODE_DEBUG=urllib:* npm test
Mocking Request
export from undici
import { strict as assert } from 'assert';
import { MockAgent, setGlobalDispatcher, request } from 'urllib';
const mockAgent = new MockAgent();
setGlobalDispatcher(mockAgent);
const mockPool = mockAgent.get('http://localhost:7001');
mockPool.intercept({
path: '/foo',
method: 'POST',
}).reply(400, {
message: 'mock 400 bad request',
});
const response = await request('http://localhost:7001/foo', {
method: 'POST',
dataType: 'json',
});
assert.equal(response.status, 400);
assert.deepEqual(response.data, { message: 'mock 400 bad request' });
Request through a http proxy
export from undici
import { ProxyAgent, request } from 'urllib';
const proxyAgent = new ProxyAgent('http://my.proxy.com:8080');
const response = await request('https://www.npmjs.com/package/urllib', {
dispatcher: proxyAgent,
});
console.log(response.status, response.headers);
Benchmarks
Connections 1
Tests | Samples | Result | Tolerance | Difference with slowest |
---|---|---|---|---|
http - no keepalive | 15 | 6.38 req/sec | ± 2.44 % | - |
http - keepalive | 10 | 6.77 req/sec | ± 2.35 % | + 6.13 % |
urllib2 - request | 45 | 40.13 req/sec | ± 2.88 % | + 528.66 % |
urllib3 - request | 10 | 58.51 req/sec | ± 2.52 % | + 816.64 % |
undici - pipeline | 5 | 59.12 req/sec | ± 2.47 % | + 826.18 % |
undici - fetch | 15 | 60.42 req/sec | ± 3.00 % | + 846.60 % |
undici - dispatch | 5 | 60.58 req/sec | ± 1.39 % | + 848.99 % |
undici - stream | 5 | 61.30 req/sec | ± 1.31 % | + 860.39 % |
undici - request | 5 | 61.74 req/sec | ± 2.03 % | + 867.20 % |
Connections 50
Tests | Samples | Result | Tolerance | Difference with slowest |
---|---|---|---|---|
urllib2 - request | 51 | 1465.40 req/sec | ± 14.40 % | - |
undici - fetch | 40 | 3121.10 req/sec | ± 2.82 % | + 112.99 % |
http - no keepalive | 45 | 3355.42 req/sec | ± 2.84 % | + 128.98 % |
http - keepalive | 51 | 5179.55 req/sec | ± 36.61 % | + 253.46 % |
urllib3 - request | 30 | 7045.86 req/sec | ± 2.93 % | + 380.82 % |
undici - pipeline | 50 | 8306.92 req/sec | ± 2.99 % | + 466.87 % |
undici - request | 51 | 9552.59 req/sec | ± 13.13 % | + 551.88 % |
undici - stream | 45 | 12523.45 req/sec | ± 2.97 % | + 754.61 % |
undici - dispatch | 51 | 12970.18 req/sec | ± 3.15 % | + 785.10 % |
Contributors
This project follows the git-contributor spec, auto updated at Mon Dec 04 2023 00:13:39 GMT+0800
.
License
19 days ago
28 days ago
1 month ago
1 month ago
2 months ago
3 months ago
3 months ago
4 months ago
4 months ago
6 months ago
7 months ago
6 months ago
6 months ago
10 months ago
7 months ago
10 months ago
8 months ago
7 months ago
7 months ago
7 months ago
10 months ago
11 months ago
1 year ago
1 year ago
11 months ago
11 months ago
1 year ago
10 months ago
1 year ago
10 months ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
2 years ago
2 years ago
1 year ago
2 years ago
2 years ago
2 years ago
1 year ago
1 year ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
2 years ago
3 years ago
3 years ago
3 years ago
3 years ago
3 years ago
4 years ago
4 years ago
4 years ago
4 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
11 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
12 years ago
13 years ago
13 years ago