@wmfs/smithereens NPM

smithereens

Smash CSV files into more manageable files based on column values

Install

$ npm install smithereens --save

Usage

const smithereens = require('smithereens')

smithereens(
  [
    '/some/input/csv/files/people.csv'

    // people.csv:
    //
    // personNo,firstName,LastName,personType,action
    // 10,"Lisa","Simpson","c","u"
    // 20,"Homer","Simpson","a","u"
    // 30,"Bart","Simpson","c","d"
    // 40,"Marge","Simpson","a","d"
    // 50,"Maggie","Simpson","c","x"
    // 60,"Grampa","Simpson","x","u"
    // 70,"Milhouse","Van Houten","c","u"

  ],
  {

    outputDirRootPath: '/some/output/dir',

    parser: {
      quote: '"',
      delimiter: ',',
      newline: '\n',
      skipFirstLine: true,
      trimWhitespace: true
    },

    dirSplits: [
      {
        columnIndex: 3,
        valueToDirMap: {
          'c': 'children',
          'a': 'adults'
        }
      }
    ],

    fileSplits: {
      columnIndex: 4,
      valueToFileMap: {
        'u': {
          filename: 'changes',
          outputColumns: [
            {name: 'person_no', columnIndex: 0},
            {name: 'first_name', columnIndex: 1},
            {name: 'last_name', columnIndex: 2}
          ]
        },
        'd': {
          filename: 'deletes',
          outputColumns: [
            {name: 'person_no', columnIndex: 0}
          ]
        }
      }
    }
  },

  function (err, manifest) {

    // File output
    // -----------
    //   /some/output/dir
    //   ./adults
    //     changes.csv:
    //       person_no,first_name,last_name
    //       20,Homer,Simpson
    //     deletes.csv:
    //       person_no
    //       40
    //   ./children
    //     changes.csv:
    //       person_no,first_name,last_name
    //       10,Lisa,Simpson
    //       70,Milhouse,Van Houten
    //     deletes.csv:
    //       person_no
    //       30
    //     unknown.csv:
    //       50,Maggie,Simpson,c,x
    //   ./unknown
    //     changes.csv:
    //       person_no,first_name,last_name
    //       60,Grampa,Simpson

  }
)

smithereens(`sourceFilePaths`, `options`, `callback`)

Arg	Type	Description
`sourceFilePaths`	`string` \| `[string]`	A string or an array of strings identifying one or more files. Uses `glob` so `/some/dir/.csv` style patterns are supported, as is directory recursion via `/some/dir//.csv`
`options`	`object`	An object configuring how output should be produced. See Options for more information.
`callback`	`function`	To be of the form `function(err, manifest)`. Manifest contains a summary of the output files produced.

Options

Property	Type	Description
`outputDirRootPath`	`string`	An absolute directory path where to write output to. All missing directories will be created.
`parser`	`object`	An `parser` object for configuring how input CSV files should be parsed.
`dirSplits`	`[object]`	An array of of `dirSplit` objects
`fileSplits`	`object`	A `fileSplit` object

`parser` object

Configures how to parse incoming CSV lines. Uses csv-streamify under the bonnet.

Property	Type	Description
`skipFirstLine`	`boolean`	Should the first line of each file be ignored? Set to `true` if files include a header line, for example.
`delimiter`	`string`	Comma, semicolon, whatever - defaults to comma.
`newline`	`string`	Newline character (use \r\n for CRLF files).
`quote`	`string`	What's considered a quote.
`empty`	`string`	Empty fields are replaced by this value.

`dirSplit` object

Smithereens can break CSV files across a nested set of directories based on values defined in each line.

Property	Type	Description
`columnIndex`	`integer`	Each line of each CSV file will be parsed into an array of strings. This value identifies which value to split on.
`valueToDirMap`	`object`	A simple mapping of an expected string value (as identified by `columnIndex`) and the directory name that this line should be routed to.

`fileSplit` object

In a similar way, Smithereens can route lines to different files, based on the contents of a parsed CSV column.

Property	Type	Description
`columnIndex`	`integer`	Identifies which of the parsed string values from each CSV line should be used to determine a filename that a row should be routed to.
`valueToFileMap`	`object`	A key/value map where key is a string value that is expected via `columnIndex` and value is a `file` object.

`file` object

Defines which filename a CSV row should be routed to, along with some output-formatting configuration.

Property	Type	Description
`filename`	`string`	The filename which a row should be routed to. All output files will be in CSV format. Note that the `.csv` extension is added automatically, so don't include it here.
`outputColumns`	`[object]`	An array of `outputColumn` objects - each defining a column that should appear in the output file.

`outputColumn` object

Defines the values for each column in the output.

Property	Type	Description
`name`	string	The name of the column, used in the first line of the CSV output
`columnIndex`	integer	Identifies the column in the corresponding row of the parsed incoming CSV array to copy to the output. If `columnIndex` is given any `type` parameter is ignored.
`type`	string	Describes other output - `hash`, `uuid`, `constant`. `hash` generates a hash value from the contents of the corresponding input row, `uuid` outputs a unique id, `constant` output a fix value, given by the `value` property.
`value`	number or string	A fixed value to put output when the `type` property is `constant`.