org.cylondata.cylon

Class Table

java.lang.Object

org.cylondata.cylon.DataRepresentation

org.cylondata.cylon.Table

public class Table
extends DataRepresentation

Table is the basic data manipulation endpoint of TwisterX API. This class doesn't hold any data, instead it acts as the mediator between the user's application and the native TwisterX layer. Data transformation, communication and persistence is handled entirely by the native layer.

Tables are immutable and transformations will create another table instance keeping the original table intact.

A unique ID will be assigned to each table at the time of creation. This can be considered as the identifier of a set of data. When transferring data to a different machine, or when persisting data to a disk, this ID will be always associated with the underlying set of data. Hence different APIs(Java, Python, REST based RPC) will be able to refer a set of data irrespective of where it actually got created.

Method Summary

Modifier and Type	Method and Description
void	clear() Clear the table and free memory associated with this table
Table	distributedJoin(Table rightTable, JoinConfig joinConfig) Apply the join algorithm across a distributed set of nodes/dataset
<I> Table	filter(int columnIndex, Filter<I> filterLogic) Filter out rows of a table based on a single column
static Table	fromColumns(List<Column> columns) Create a Table by combining a list of columns
static Table	fromCSV(CylonContext ctx, String path) This method will load a table by reading the data from a CSV file.
static Table	fromCSV(String path, List<org.apache.arrow.vector.types.Types.MinorType> dataTypes) This method will load a table by reading the data from a CSV file.
int	getColumnCount() Get the number of columns of the table
int	getRowCount() Get the number of rows of the table
List<Table>	hashPartition(List<Integer> hashColumns, int noOfPartitions) Partition a table based on the hash value of the specified columns
Table	join(Table rightTable, JoinConfig joinConfig) Join two tables based on the value of the columns
<I,O> Column<O>	mapColumn(int colIndex, Mapper<I,O> mapper) Maps the values of a column to another value
static Table	merge(CylonContext ctx, Table... tables) Merge a set of similar tables into a single table.
void	print() Prints the entire table to the console
void	print(int row1, int row2, int col1, int col2) Prints a section of the table to the console
List<Table>	roundRobinPartition(int noOfPartitions) Partition a table into n partitions of similar size
Table	select(Selector selector) This method can be used to filter out some rows from a table based on a user defined logic
Table	sort(int columnIndex) Sort the rows of a table based on the value of a column

Methods inherited from class org.cylondata.cylon.DataRepresentation

getId, unSupportedException

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

fromCSV

public static Table fromCSV(CylonContext ctx,
                            String path)

This method will load a table by reading the data from a CSV file.

Parameters:

path - path to the CSV file

Returns:

A Table instance that holds the data from the CSV file

fromColumns

public static Table fromColumns(List<Column> columns)

Create a Table by combining a list of columns

Parameters:

columns - List of columns

Returns:

A new table instance which holds the list of columns specified

fromCSV

public static Table fromCSV(String path,
                            List<org.apache.arrow.vector.types.Types.MinorType> dataTypes)

This method will load a table by reading the data from a CSV file. The behaviour will be similar to fromCSV(CylonContext, String), but additionally data types can be specified for each column.

Parameters:

path - path to the CSV file

dataTypes - List of data types, i^th index of the List should specify the data types of the i^th column.

Returns:

A Table instance that holds the data from the CSV file

getColumnCount

public int getColumnCount()

Get the number of columns of the table

Returns:

No of columns

getRowCount

public int getRowCount()

Get the number of rows of the table

Returns:

No of rows

join

public Table join(Table rightTable,
                  JoinConfig joinConfig)

Join two tables based on the value of the columns

Parameters:

rightTable - Table to be joined with this table

joinConfig - Configurations for the join operation

Returns:

Joined Table

distributedJoin

public Table distributedJoin(Table rightTable,
                             JoinConfig joinConfig)

Apply the join algorithm across a distributed set of nodes/dataset

Parameters:

rightTable - Table to be joined with this table

joinConfig - Configurations for the join operation

Returns:

Joined Table

mapColumn

public <I,O> Column<O> mapColumn(int colIndex,
                                 Mapper<I,O> mapper)

Maps the values of a column to another value

Type Parameters:

I - Input data type

O - Output data type

Parameters:

colIndex - Column index to be transformed(mapped)

mapper - Mapping logic

Returns:

An instance of Column which represents the mapped values

hashPartition

public List<Table> hashPartition(List<Integer> hashColumns,
                                 int noOfPartitions)

Partition a table based on the hash value of the specified columns

Parameters:

hashColumns - Indices of the columns to be hashed

noOfPartitions - No of partitions to generate

Returns:

List of partitioned tables

roundRobinPartition

public List<Table> roundRobinPartition(int noOfPartitions)

Partition a table into n partitions of similar size

Parameters:

noOfPartitions - no of partitions to generate

Returns:

List of partitioned tables

merge

public static Table merge(CylonContext ctx,
                          Table... tables)

Merge a set of similar tables into a single table.

Parameters:

tables - List of tables to be merged

Returns:

merged Table

sort

public Table sort(int columnIndex)

Sort the rows of a table based on the value of a column

Parameters:

columnIndex - index of the column to be usd for sorting

Returns:

Sorted Table instance

filter

public <I> Table filter(int columnIndex,
                        Filter<I> filterLogic)

Filter out rows of a table based on a single column

Type Parameters:

I - data type of the column

Parameters:

columnIndex - column to be used for filtering

filterLogic - filtering logic

Returns:

Table without the filtered out rows

select

public Table select(Selector selector)

This method can be used to filter out some rows from a table based on a user defined logic

Parameters:

selector - logic to select(filter) rows from the table

Returns:

resulting table, after filtering out rows

clear

public void clear()

Clear the table and free memory associated with this table

print

public void print()

Prints the entire table to the console

print

public void print(int row1,
                  int row2,
                  int col1,
                  int col2)

Prints a section of the table to the console

Parameters:

row1 - starting row index

row2 - ending row index

col1 - starting column index

col2 - ending column index