Develop a processor to bulk load massive amounts of data into any JDBC destination

Description

putJdbc is a NIFI Processor for bulk loading data into a database. It is tested for the extreme amount of data. With an approach that fits seamlessly into NIFI it has the ability to interact with the database to control ETL/ELT jobs as well as the special parametrization of bulk-load-connections. It takes a FlowFile, then executes a sequence of scripts and a load command to load and transform data. With a number of options, it is able to be configured using the best practice approaches for very large databases and high-speed ingestion.

The processor comes in two flavors:

  • putJdbc allows a parameterized insert statement to load into a database. The input can be any record-oriented data, that a RecordReader can parse.

  • putJdbcCsv is specially designed to load extreme amounts of data into Teradata using the Fastload-CSV feature of Teradata's JDBC driver. It takes a limited CSV Format and feeds IoT Platforms with up to 10 Million datapoints per second.

Assignee

Jagrut Sharma

Reporter

Juergen AtWork

Labels

None

Reviewer

None

Story Points

2

Time tracking

24h

Epic Link

Components

Sprint

None

Priority

Medium
Configure