hdfs247

NPM HDFS module

This is node.js interface to communicating with web HDFS interface.

How to use

Installing

Before we can use this module we need to install it first

$ npm install --save hdfs247

Using on Unsecure Cluster

var Hdfs = require('hdfs'),
  localpath = '/tmp/filetest.txt',
  remotepath = '/user/apps/filetest.txt';

var hdfs = new Hdfs({
  protocol: 'http',
  hostname: '192.168.1.225',
  port: 50070
});

hdfs.upload({
  'user.name': 'apps',
  overwrite: true,
  localpath: localpath,
  path: remotepath
}, function(error, response, body) {
  console.log(error);    // Error will be null if upload process is succeed.
  console.log(response); // Raw response from node.js request.
  console.log(body);     // Body of request result.
});

Using on Secure Cluster (Kerberized/SPNego)

var Hdfs = require('hdfs'),
  localpath = '/tmp/filetest.txt',
  remotepath = '/user/apps/filetest.txt';

var hdfs = new Hdfs({
  protocol: 'http',
  hostname: '192.168.1.225',
  port: 50070
});

hdfs.upload({
  'user.name': 'apps',
  overwrite: true,
  localpath: localpath,
  spnego_token: result.token,
  principal: spnego_principal,
  keytab: keytab,
  realm: realm,
  krb_fqdn: krb_fqdn,
  path: remotepath
}, function(error, response, body) {
  console.log(error);    // Error will be null if upload process is succeed.
  console.log(response); // Raw response from node.js request.
  console.log(body);     // Body of request result.
});

Functions

cekpath(param) ⇒ Object

Check the parameter, return path if there is path in parameter, and return / if no path in parameter.

prepvar(param) ⇒ Object

Prepare parameter, separate request option and request parameter, set default option, and explicitly set path.

Hdfs(data)

Hdfs class constructor

cekpath(param) ⇒ Object

Check the parameter, return path if there is path in parameter, and return / if no path in parameter.

Kind: global function
Returns: Object - Object with only path in it.

Param	Type	Description
param	Object	Parameter from user

prepvar(param) ⇒ Object

Prepare parameter, separate request option and request parameter, set default option, and explicitly set path.

Kind: global function
Returns: Object - Clean ready to use request parameter.

Param	Type	Description
param	Object	Mixed request parameter.

Hdfs(data)

Hdfs class constructor

Kind: global function

Param	Type	Description
data	Object	Object data seed.

• Hdfs(data)
  • .getfilestatus(param, callback)
  • .liststatus(param, callback)
  • .download(param, callback)
  • .upload(param, callback)
  • .open(param, callback)
  • .gethomedirectory(param, callback)
  • .getcontentsummary(param, callback)
  • .getfilechecksum(param, callback)
  • .concat(param, callback)
  • .append(param, callback)
  • .create(param, callback)
  • .rename(param, callback)
  • .mkdirs(param, callback)
  • .setpermission(param, callback)
  • .setowner(param, callback)
  • .setreplication(param, callback)
  • .settimes(param, callback)
  • .delete(param, callback)

hdfs.getfilestatus(param, callback)

Get file status from HDFS Possible properties of param is :

name	presence	type	default	description
path	optional	String	/	Path status that will be taken

Kind: instance method of Hdfs

Param	Type	Description
param	Object	Request parameter for get file status endpoint
callback	function	Callback function to return the result

hdfs.liststatus(param, callback)

List status of file and directory under specified directory

name	presence	type	default	description
path	optional	String	/	Path status that will be taken

Kind: instance method of Hdfs

Param	Type	Description
param	Object	Request parameter for get list of status file
callback	function	Callback function to return the result

hdfs.download(param, callback)

Download file from HDFS with without buffering.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Open_and_Read_a_File

Param properties :

name	presence	type	default	valid value	description
path	optional	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
offset	optional	Long	0	>=0	The starting byte position
length	optional	Long	null	>=0 or null	The number of bytes to be processed. null means the entire file
buffersize	optional	Int	Specified in hadoop configuration	>0	The size of the buffer used in transferring data.

Param	Type	Description
param	Object	Request parameter for download file from HDFS
callback	function	Callback function to return the result

hdfs.upload(param, callback)

Upload file to HDFS without buffering.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File

Param properties :

name	presence	type	default	valid value	description
path	optional	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
overwrite	optional	Boolean	false	true or false	If a file already exists, should it be overwritten?
blocksize	optional	Long	Specified in hadoop configuration	>0	The block size of a file.
replication	optional	Short	Specified in hadoop configuration	>0	The number of replications of a file.
permission	optional	octal	755	0 - 1777	The permission of a file/directory.
buffersize	optional	Int	Specified in hadoop configuration	>0	The size of the buffer used in transferring data.

Param	Type	Description
param	Object	Request parameter for upload file from HDFS
callback	function	Callback function to return the result

hdfs.open(param, callback)

Open file from HDFS, this function will buffer content file in memory

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Open_and_Read_a_File

Param properties :

name	presence	type	default	valid value	description
path	optional	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
offset	optional	Long	0	>=0	The starting byte position
length	optional	Long	null	>=0 or null	The number of bytes to be processed. null means the entire file
buffersize	optional	Int	Specified in hadoop configuration	>0	The size of the buffer used in transferring data.

Param	Type	Description
param	Object	Request parameter for open file from HDFS
callback	function	Callback function to return the result

hdfs.gethomedirectory(param, callback)

Get home directory of current accessing user.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Home_Directory

Param	Type	Description
param	Object	Request parameter for getting home directory.
callback	function	Callback function to return the result

hdfs.getcontentsummary(param, callback)

Get summary of content

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Content_Summary_of_a_Directory

name	presence	type	default	valid value	description
path	optional	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken

Param	Type	Description
param	Object	Request parameter for getting home directory.
callback	function	Callback function to return the result

hdfs.getfilechecksum(param, callback)

Get checksum of file

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_File_Checksum

name	presence	type	default	valid value	description
path	optional	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken

Param	Type	Description
param	Object	Request parameter for get checksum of file.
callback	function	Callback function to return the result

hdfs.concat(param, callback)

Concatenating file to another one. Warning : It's still experimental. We warn you.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Concat_Files

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
sources	Mandatory	String		A list of comma seperated absolute FileSystem paths without scheme and authority.	A list of source paths.

Param	Type	Description
param	Object	Request parameter for concatenating file.
callback	function	Callback function to return the result

hdfs.append(param, callback)

Append data to another file in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Append_to_a_File

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
buffersize	optional	Int	Specified in hadoop configuration	>0	The size of the buffer used in transferring data.

Param	Type	Description
param	Object	Request parameter for upload file from HDFS
callback	function	Callback function to return the result

hdfs.create(param, callback)

Create file in HDFS.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
overwrite	optional	Boolean	false	true or false	If a file already exists, should it be overwritten?
blocksize	optional	Long	Specified in hadoop configuration	>0	The block size of a file.
replication	optional	Short	Specified in hadoop configuration	>0	The number of replications of a file.
permission	optional	octal	755	0 - 1777	The permission of a file/directory.
buffersize	optional	Int	Specified in hadoop configuration	>0	The size of the buffer used in transferring data.

Param	Type	Description
param	Object	Request parameter for upload file from HDFS
callback	function	Callback function to return the result

hdfs.rename(param, callback)

Rename file or directory in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Rename_a_FileDirectory

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
destination	Mandatory	String	(an invalid path)	An absolute FileSystem path without scheme and authority.	The destination path.

Param	Type	Description
param	Object	Request parameter for rename file or directory.
callback	function	Callback function to return the result

hdfs.mkdirs(param, callback)

Make directory in Hdfs

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Make_a_Directory

Param properties :

name	presence	type	default	valid value	description
path	optional	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
permission	optional	octal	755	0 - 1777	The permission of a file/directory.

Param	Type	Description
param	Object	Request parameter for make directory in HDFS
callback	function	Callback function to return the result

hdfs.setpermission(param, callback)

Set permission of file in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Permission

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
permission	Mandatory	octal	755	0 - 1777	The permission of a file/directory.

Param	Type	Description
param	Object	Request parameter for set permission of file and directory in HDFS
callback	function	Callback function to return the result

hdfs.setowner(param, callback)

Set owner of file or directory

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Owner

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
owner	Optional	String	(means keeping it unchanged)	Any valid username.	The username who is the owner of a file/directory.
group	Optional	String	(means keeping it unchanged)	Any valid group name.	The name of a group.

Param	Type	Description
param	Object	Request parameter for set owner in HDFS
callback	function	Callback function to return the result

hdfs.setreplication(param, callback)

Set replication factor of file.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Replication_Factor

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
replication	Optional	Short	Specified in the configuration.	>0	The number of replications of a file.

Param	Type	Description
param	Object	Request parameter for set replication file.
callback	function	Callback function to return the result

hdfs.settimes(param, callback)

Set time and date of file in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Access_or_Modification_Time

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
modificationtime	Optional	Long	-1 (means keeping it unchanged)	-1 or a timestamp	The modification time of a file/directory.
accesstime	Optional	Long	-1 (means keeping it unchanged)	-1 or a timestamp	The access time of a file/directory.

Param	Type	Description
param	Object	Request parameter for setting time and date of file.
callback	function	Callback function to return the result

hdfs.delete(param, callback)

Delete file or directory in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Delete_a_FileDirectory

Param properties :

name	presence	type	default	valid value	description
path	Mandatory	String	/	An absolute FileSystem path without scheme and authority.	Path status that will be taken
recursive	Optional	Boolean	false	true or false	Should the operation act on the content in the subdirectories?

Param	Type	Description
param	Object	Request parameter for deleting file and or directory.
callback	function	Callback function to return the result