dataf v0.3.0
DataFrame
- pandas-like data-frame library
- Column built on typed arrays
- tries to be memory efficient
- extensions to arrays
- great for tabular data
- reads CSV (only for now)
- work in progress
See JSDoc-generated API docs see gh-pages.
For more human-friendly docs keep reading.
Installation
$ npm install --save datafHuman-Friendly API
Preliminaries
Run the node REPL.
$ nodeImport the library (make sure it's installed).
const DF = require('dataf')Toy Datasets (shipped with the library)
DF.dataSets[ 'alcohol.csv', // alcohol consumption math students
'countries.csv', // geographical and economical data for all countries
'diabetes.csv',
'food.csv', // food choices
'got.csv', // game of thrones deaths
'happiness.csv', // world happiness 2017
'iris.csv',
'mushrooms.csv',
'pokemon.csv', // stats for all from all generations
'superheros.csv'
... ] All have been placed in the public domain.
Load the Iris DataSet
let iris = new DF('iris') // use `let`, you will be re-assigning a lotSelecting / Slicing Rows
Head / Tail
iris.head().print()
// .head(20) for the first 20 rows
// .tail() for last rows# u8 Id f32 SepalLe... f32 SepalWi... f32 PetalLe... f32 PetalWi... s Species
- ----- -------------- -------------- -------------- -------------- -------------
0 1 5.09 3.50 1.39 0.20 Iris-setos...
1 2 4.90 3.00 1.39 0.20 Iris-setos...
2 3 4.69 3.20 1.29 0.20 Iris-setos...
3 4 4.59 3.09 1.50 0.20 Iris-setos...
4 5 5.00 3.59 1.39 0.20 Iris-setos...
- ----- -------------- -------------- -------------- -------------- -------------
5B 20B 20B 20B 20B NaNNOTE the data types next to column names and memory indicators for every column.
Slicing
iris.slice(10, 20).print() // can be .slice(5) for .slice(5, end)# u8 Id f32 SepalLe... f32 SepalWi... f32 PetalLe... f32 PetalWi... s Species
- ----- -------------- -------------- -------------- -------------- -------------
0 11 5.40 3.70 1.50 0.20 Iris-setos...
1 12 4.80 3.40 1.60 0.20 Iris-setos...
- ----- -------------- -------------- -------------- -------------- -------------
2B 8B 8B 8B 8B NaNNOTE the library will try to compute the width of each column
Getting a Column (Column)
We know that there are 6 columns (try running iris.nCols). To get all column names run:
iris.colNames.print(100) // make sure it prints all Column s [Id, SepalLengthCm, SepalWidthCm, PetalLengthCm, PetalWidthCm, Species]If you want to extract a column (Column, see the Column API below) from a data frame try:
iris.Species.print(5) // last columnColumn s[Iris-setosa, Iris-setosa, Iris-setosa, Iris-setosa, Iris-setosa, ... 145 more]Here s stands for STRING. You may also see: f64, f32, i32, i16, i8, u32, u16 and u8.
NOTE some column names will have spaces or/and will clash with the API and you will have to use iris.col(2) OR iris.col('SepalWidthCm').
Columns can always be referred to by their index OR name.
Selecting columns (Data Frame)
Suppose you only want the first couple of columns:
iris.select(0, 1, -2).print(5) // the 1st, 2nd and the 2nd to lastThis show the first 5 rows of the new data frame with only: Id, SepalLength and PetalWidth.
# u8 Id f32 SepalLe... f32 PetalWi...
--- ----- -------------- --------------
0 1 5.09 0.20
1 2 4.90 0.20
2 3 4.69 0.20
3 4 4.59 0.20
4 5 5.00 0.20
... ... (145 more) ...
--- ----- -------------- --------------
150B 600B 600BIf you want to select a range of column: e.g. from the 1st to the 3rd try:
iris.sliceCols(0, 2).print(3) # u8 Id f32 SepalLe... f32 SepalWi...
--- ----- -------------- --------------
0 1 5.09 3.50
1 2 4.90 3.00
2 3 4.69 3.20
... ... (147 more) ...
--- ----- -------------- --------------
150B 600B 600BThis is equivalent to:
iris.sliceCols('Id', 'SepalWidthCm').print(3)Only Numeric Columns (remove string columns)
iris.numeric // all BUT the "Species" column (getter)Only String Columns (remove numeric columns)
iris.nominal // just the "Species" column (getter)Removing (Dropping) Columns
If you want to remove the 2nd and the second to last columns:
iris.drop(1, -2).print(3) # f32 SepalLe... f32 SepalWi... f32 PetalLe... s Species
--- -------------- -------------- -------------- -------------
0 5.09 3.50 1.39 Iris-setos...
1 4.90 3.00 1.39 Iris-setos...
2 4.69 3.20 1.29 Iris-setos...
... ... (147 more) ... ...
--- -------------- -------------- -------------- -------------
600B 600B 600B NaNNOTE those operations are not in-place meaning dropping produces a new data frame without specified columns.
Selecting Rows
With Specific Value
Signature: iris.where(val, colId, op). Where op is one of {"=" (default), ">", "<", ">=", "<="}.
iris.Species[0]
'Iris-setosa'
iris.where('Iris-setosa', -1) // -1 for last col
// ... DataFrame with subset of rows with just Iris-setosaMatching Predicate (Test)
Signature: iris.filter(rowTestFunc).
Signature: iris.filter(valTestFunc, colId).
iris.filter(species => species === 'Iris-setosa', -1)
// OR (expensive)
iris.filter(row => row[row.length - 1] === 'Iris-setosa')Accessing Values (preferred way)
iris.val(10, 'Species') // val(rowIdx, colId)
'Iris-setosa'Accessing Rows
One Row
Accessing a single row:
const row = iris.row(20) // 21st row
[ 21,
5.400000095367432,
3.4000000953674316,
1.7000000476837158,
0.20000000298023224,
'Iris-setosa' ]Iterating Over Values in a Single Row
const irow = iris.irow(10);
Array.from(irow)
[ 5.400000095367432,
3.700000047683716,
1.5,
0.20000000298023224,
'Iris-setosa' ]Iterating Over Many Rows
If you want to iterate over all the rows (this isn't very efficient) try:
const rowIt = iris.slice(0, 3).rowsIter // (getter)
for (const r of rowIt) {
console.log(r)
}
// you may also iterate over the dataframe (equivalent method)
for (const r of iris) {
console.log(r)
}
[ 1,
5.099999904632568,
3.5,
1.399999976158142,
0.20000000298023224,
'Iris-setosa' ]
[ 2,
4.900000095367432,
3,
1.399999976158142,
0.20000000298023224,
'Iris-setosa' ]
[ 3,
4.699999809265137,
3.200000047683716,
1.2999999523162842,
0.20000000298023224,
'Iris-setosa' ]Manipulation
In-Place Modification of Columns
Just assign:
// 2nd col
iris[1] = iris[1].map(s => s >= 5 ? 0 : 1)
// equivalent to:
iris.SepalLengthCm = iris.SepalLengthCm.map(s => s >= 5 ? 0 : 1)NOTE this might have to be dataset[' Col With Spaces'] = newCol.
Mapping Columns
Apply function to each element is selected column:
iris.map(-1, label => {
// there is an easier way to do this (see `DataFrame.labelEncode()`)
if (label === 'Iris-versi') {
return 0;
} else if (label === 'Iris-virgi') {
return 1;
} else {
return 2;
}
});NOTE use iris.map(null, f) to apply to all columns.
Mapping Shortcuts
null means it will be applied to all.
.trunc(colId | null).floor(colId | null).ceil(colId | null).round(colId | null).abs(colId | null).sqrt(colId | null).cbrt(colId | null).square(colId | null).cube(colId | null).add(colId | null, n).sub(colId | null, n).mul(colId | null, n).div(colId | null, n)
It's smart enough to know not to apply them to string columns if they don't
make sense (e.g. .abs()). String columns are ignored.
Rename Columns
iris.rename(0, 'First').rename(-2, 'Second to Last')
// or just
iris.rename(0, 'First', -2, 'Second to Last')Merging Data Frames
iris.concat(iris) // append all rows (axis 0)
iris.concat(iris, 1) // append all columns (axis 1)NOTE this library will manage duplicate column names.
iris.concat(iris, 1).colNames
[ 'Id',
'SepalLengthCm',
'SepalWidthCm',
'PetalLengthCm',
'PetalWidthCm',
'Species',
'Id2',
'SepalLengthCm2',
'SepalWidthCm2',
'PetalLengthCm2',
'PetalWidthCm2',
'Species2' ]Appending a Column
iris.appendCol(iris.Id, 'Id2') // .appendCol(col, colName)Shuffle, Reverse
iris.shuffle()
iris.reverse()Both are safe in that the won't modify in place.
Sort
Signature: iris.sort(colId, 'asc' (default) | 'des' ).
iris.sort('SepalWidthCm') // default is iris.sort(colId, 'asc')iris.sort('SepalWidthCm', 'des') // descending sort Transpose (inefficient)
Although it's inefficient it's sometimes useful to be able to swap the x and y axis. Suppose you want to do summation row-wise (not column wise):
// numeric will drop the 'Species' column
iris.numeric.transpose().sum() # u8 column f32 add
--- ---------- -------
0 0 11.19
1 1 11.50
2 2 12.39
3 3 13.39
4 4 15.19
... (145 more) ...
--- ---------- -------
150B 600BNOTE transposing will only work for homogeneous DataFrames (all nums or all strings).
Statistics & Math
Aggregate operations, each is DataFrame -> DataFrame:
MATH
.add().sub().mul().div()
STATS
.min().max().range().mean().var()variance.stdev()standard deviation.median().Q3().Q1().IQR()inter-quartile range.skewness().kurtosis().mad()mean absolute deviation
E.g.:
iris.IQR()# s column f32 IQR
- ------------- -------
0 Id 75.00
1 SepalLengt... 1.30
2 SepalWidth... 0.50
3 PetalLengt... 3.50
4 PetalWidth... 1.50
- ------------- -------
NaN 20BSample (get a random subset of rows)
Signatures:
Summary
iris.summary() // this will produce a summary data frame with info for every column# s column s dtype f32 min f32 max f32 range f32 mean f32 stdev
- ------------- ------- ------- ------- --------- -------- ---------
0 Id u8 1.00 150.00 149.00 75.50 43.30
1 SepalLengt... f32 4.30 7.90 3.59 5.84 0.82
2 SepalWidth... f32 2.00 4.40 2.40 3.05 0.43
3 PetalLengt... f32 1.00 6.90 5.90 3.75 1.75
4 PetalWidth... f32 0.10 2.50 2.40 1.19 0.76
5 Species s NaN NaN NaN NaN NaN
- ------------- ------- ------- ------- --------- -------- ---------
NaN NaN 24B 24B 24B 24B 24BAggregates
Counts (of unique values)
This is particularly useful for nominal / discrete attributes that take on a
small amount of values. E.g. Gender is one of {M, F} or Salary is one of {Low, Med, High}.
iris.counts(-1) // for the last column
// iris.ps(-1) // for normalized values# s Species u8 count
- ------------- --------
0 Iris-setos... 50
1 Iris-versi... 50
2 Iris-virgi... 50
- ------------- --------
NaN 3BCorrelations (A Matrix Operation)
For a correlation of each column with each other column (matrix):
iris.corr() // .corr(false) to *not* print the first column
// iris.cov() // covariance
// iris.dot() // dot product between each col
// iris.dist(1) // manhattan distance
// iris.dist(2) // euclidian distance
// iris.dist(2+) // minkowsky distance# s column f64 Id f64 SepalLe... f64 SepalWi... f64 PetalLe... f64 PetalWi...
- ------------- ------ -------------- -------------- -------------- --------------
0 Id 1.00 0.71 -0.39 0.88 0.89
1 SepalLengt... 0.71 1.00 -0.10 0.87 0.81
2 SepalWidth... -0.39 -0.10 1.00 -0.42 -0.35
3 PetalLengt... 0.88 0.87 -0.42 1.00 0.96
4 PetalWidth... 0.89 0.81 -0.35 0.96 1.00
- ------------- ------ -------------- -------------- -------------- --------------
NaN 40B 40B 40B 40B 40BPre-Processing
Remove NaN / Infinity / other
To remove all rows that have some value:
// from all cols i.e. remove all rows where any of the value is NaN
iris.removeAll(NaN)
// from 1th and 3rd cols and from col 'PetalLengthCm'
iris.removeAll(NaN, 0, 2, 'PetalLengthCm') Discretize (Bin)
iris.kBins('SepalLengthCm', 5); // 5 bins for this column
iris.kBins(null, 3); // 3 bins for all columns
iris.kBins(2, 3) // 3rd (2 idx) col, 3 bins
.col(2) // select ONLY 3rd column (index is 2), which is of type Column
.print(10) Column u8[2, 1, 2, 1, 2, 2, 2, 2, 1, 1, ... 40 more]NOTE this is smart enough only to target numeric attributes so string columns will be ignored (no need to run .numeric).
Feature (Column) Selection
Feature selection (i.e. select best columns, by default uses "var" -- variance):
Signature: iris.nBest(n, metric) where metric is one of:
"var""stdev""mean""mad""IQR""median""Q1""Q3""skewness""min""range""max"
OR a function from Column (one column) to a number (Column -> Num).
iris.drop('Id') // `Id` column is not very useful
.numeric // select all numeric cols
.nBest(2) // best 2 features using variance as score
.print(3) // show first 3 rows
// try: iris.drop('Id').numeric.nBest(2, 'mad').print(3) # f32 PetalLe... f32 SepalLe...
--- -------------- --------------
0 1.39 5.09
1 1.39 4.90
2 1.29 4.69
... (147 more) ...
--- -------------- --------------
600B 600BNormalization
However, using .nBest() in this way is very naive and you might want to normalize (scale to the same range) the values:
iris.drop('Id') // `Id` column is not very useful
.numeric // select all numeric cols
.normalize() // bring them to range [0, 1]
.nBest(2) // best 2 features using variance as score
.print(3) As you can see you might get different results:
# f32 PetalWi... f32 PetalLe...
--- -------------- --------------
0 0.04 0.06
1 0.04 0.06
2 0.04 0.05
... (147 more) ...
--- -------------- --------------
600B 600BLabel Encoding
It's a bit awkward to constantly have to drop the 'Species' column because it's a string column...
You can easily convert it to a numeric column:
From:
iris.select(-2, -1).print(48, 52) # f32 PetalWi... s Species
--- -------------- -------------
... (48 more) ...
48 0.20 Iris-setos...
49 0.20 Iris-setos...
50 1.39 Iris-versi...
51 1.50 Iris-versi...
... (98 more) ...
--- -------------- -------------
600B NaNTo:
iris.select(-2, -1).labelEncode().print(48, 52) # f32 PetalWi... u8 Species
--- -------------- ----------
... (48 more) ...
48 0.20 0
49 0.20 0
50 1.39 1
51 1.50 1
... (98 more) ...
--- -------------- ----------
600B 150BBy default all string columns will be label encoded (numeric columns will be ignored). You may specify the colIds e.g. df.labelEncode(0, -3, 'Target').
One-Hot Encoding
Signature: iris.oneHot(colId)
// expects the column to be unsigned int
iris.labelEncode('Species')
.oneHot('Species')
.print(48, 52) # u8 0 u8 1 u8 2
--- ---- --------- ----
... ... (48 more) ...
48 1 0 0
49 1 0 0
50 0 1 0
51 0 1 0
... ... (98 more) ...
--- ---- --------- ----
150B 150B 150BClipping (ensuring value is in range)
For demonstration let's make a 1-col data frame:
iris.select(1).print(3)# f32 SepalLe...
- --------------
0 5.09
1 4.90
2 4.69
- --------------
12BTo clip:
iris.select(1)
.clip(null, 4.88, 5) // null == all cols
.print(3) # f32 SepalLe...
--- --------------
0 5.00
1 4.90
2 4.88
... (147 more)
--- --------------
600BNotice that 5.09 got clipped to 5.00!
Outliers
To remove outliers (outside of Q1 to Q3) run:
iris.dropOutliers() // consider all cols
iris.dropOutliers(0, -2) // consider just 1st and second to last colsAdvanced Human-Friendly API
Data Types
If you want to get the data type for all columns try:
iris.dtypes
[ 'u8', 'f32', 'f32', 'f32', 'f32', 's' ] // read-onlyOr for a prettier output make a meta data frame with information about the previous data frame!
iris.dtype() // note difference between `iris.dtype()` (method) and `iris.dtypes` (getter)SIDENOTE .dtype() is an aggregate! This means it produces a data frame from applying a Column -> * operation to all columns.
# s column s dtype
- ------------- -------
0 Id u8
1 SepalLengt... f32
2 SepalWidth... f32
3 PetalLengt... f32
4 PetalWidth... f32
5 Species s
- ------------- -------
NaN NaNYou can force-cast columns:
iris.cast(2, 'u8') // passing `null` instead of `2` would run cast on all colsDown-Casting
You can also run iris.downcast() and let the library figure out the most efficient data type for each column so that data is not lost.
This is especially useful after truncating (floats are converted to integers).
Default:
# u8 Id f32 SepalLe... f32 SepalWi... f32 PetalLe... f32 PetalWi... s Species
- ----- -------------- -------------- -------------- -------------- -------------
0 1 5.09 3.50 1.39 0.20 Iris-setos...
1 2 4.90 3.00 1.39 0.20 Iris-setos...
2 3 4.69 3.20 1.29 0.20 Iris-setos...
- ----- -------------- -------------- -------------- -------------- -------------
3B 12B 12B 12B 12B NaNNow see how much memory can be saved:
iris.trunc().downcast().head(3)# u8 Id u8 SepalLe... u8 SepalWi... u8 PetalLe... u8 PetalWi... s Species
- ----- ------------- ------------- ------------- ------------- -------------
0 1 5 3 1 0 Iris-setos...
1 2 4 3 1 0 Iris-setos...
2 3 4 3 1 0 Iris-setos...
- ----- ------------- ------------- ------------- ------------- -------------
3B 3B 3B 3B 3B NaNMemory
Although this information is by default printed, you may produce a data frame with information about memory consumption of each column.
iris.memory()# s column u16 memory
- ------------- ----------
0 Id 150
1 SepalLengt... 600
2 SepalWidth... 600
3 PetalLengt... 600
4 PetalWidth... 600
- ------------- ----------
NaN 10BNOTE it's not calculated for string columns (notice that "Species" is missing).
To figure out how much your data frame is taking in total try:
iris.memory()
.col(-1)
.add()
2550 // bytesCopies
Deep Copy
If for some reason you need a deep-copy try (expensive):
iris.clone()Shallow Copy
Shallow copies are cheap:
iris.copy()Generalized Row Slicing
Sometimes you may want to get rows from 10th to 20th and e.g. 50th to 65th:
// [F, T],[F, T] // FROM - TO
iris.slice(9, 19, 49, 64)Generalized Column Slicing
The same applies to column slices:
iris.sliceCols(-3, -2, 0, 2)# f32 PetalLe... f32 PetalWi... u8 Id f32 SepalLe... f32 SepalWi...
- -------------- -------------- ----- -------------- --------------
0 1.39 0.20 1 5.09 3.50
1 1.39 0.20 2 4.90 3.00
2 1.29 0.20 3 4.69 3.20
- -------------- -------------- ----- -------------- --------------
12B 12B 3B 12B 12BExporting
HTML
iris.head(2).toHTML()<table>
<tr>
<th>Id</th>
<th>SepalLengthCm</th>
<th>SepalWidthCm</th>
<th>PetalLengthCm</th>
<th>PetalWidthCm</th>
<th>Species</th>
</tr>
<tr>
<td>1</td>
<td>5.099999904632568</td>
<td>3.5</td>
<td>1.399999976158142</td>
<td>0.20000000298023224</td>
<td>Iris-setosa</td>
</tr>
<tr>
<td>2</td>
<td>4.900000095367432</td>
<td>3</td>
<td>1.399999976158142</td>
<td>0.20000000298023224</td>
<td>Iris-setosa</td>
</tr>
</table>JSON
iris.head(2).toJSON(){
"Id": [1, 2],
"SepalLengthCm": [5.099999904632568, 4.900000095367432],
"SepalWidthCm": [3.5, 3],
"PetalLengthCm": [1.399999976158142, 1.399999976158142],
"PetalWidthCm": [0.20000000298023224, 0.20000000298023224],
"Species": ["Iris-setosa", "Iris-setosa"]
}CSV
iris.head(2).toCSV() Id,SepalLengthCm,SepalWidthCm,PetalLengthCm,PetalWidthCm,Species
1,5.099999904632568,3.5,1.399999976158142,0.20000000298023224,Iris-setosa
2,4.900000095367432,3,1.399999976158142,0.20000000298023224,Iris-setosaSettings
To set:
DF.opts.OPTION = VALUE;More Advanced Examples
Fix Column Names With Spaces (so that you can index df.Col)
const args = df.colNames
// replace spaces with '_'
.map(c => [c, c.replace(/\s+/, '_')])
// flatten
.reduce((pair1, pair2) => pair1.concat(pair2), []);
df = df.rename(...args) // spreadMatrix of Normalized Differences Between Means of Columns
This would normally take a lot of code:
iris.normalize()
.matrix(
(col1, col2) => Math.abs(col1.mean() - col2.mean()),
true, // show cols
true, // halves the computation time when f(c2, c1) == f(c1, c2)
0) // saves computation on the diagonal, when f(c, c) == idSave Memory
df = df.labelEncode() // string cols => unsigned int
.kBins(null, 5) // f64, f32, ... => unsigned int
.downcast() // optimize
// see memory
df.memory()
// see dtypes
df.dtype()
// megabytes
B = df.memory() // mem for each col
.add() // add up
.val(0, 1) // get total
MB = B / 1e6Column Human-Friendly API
TODO
Disclaimer
- I am not a statistician
- Unit tests for
DataFrameare not done yet - Alpha-stage
- I would not use it in production (yet)
- This isn't supposed to be an exact copy of pandas
- In some places it's not very efficient
- Date columns / mixed data types not supported. Every column must be either
numeric OR string. A single
DataFramemay have a combination of numeric and string columns. - I am a student.
License
MIT
Copyright 2019 Norbert Logiewa
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
4 years ago
4 years ago
4 years ago
4 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago