Need a NiFi processor to identify the source data set for a feed

Description

We need a processor that would be put at the front of a flow that can be configured to load source data set metadata to be used down stream in the flow. Some requirements of the processor might be to:

  • Contain designer provided configuration settings that drives the feed input configuration stepper

  • Allow the template designer to limit the set of connectors/data sources supported by the feed

  • Retrieve and add data set metadata into the flow file at runtime

  • Allow branch on connector type to support alternate flow routes based the chosen type of input data set (file, JDBC, etc.)

The branching requirement above will probably be needed because, even though a template might designed to support a wide range connector types, certain alternate behavior could be necessary based on the type. For instance, if a file-based data set was chosen as the input, specific flow route may need to be followed in order to remove the source file as it is processed; something that would not be required when a JDBC data set was chosen.

This branching behavior could be implemented by a separate processor as well; which might well be needed anyway if the alternate routes will occur farther down the flow. The processor for this story might extend that branching processor to inherit that behavior. Alternatively, since NiFi already has a RouteOnAttribute processor, this routing feature can be dropped as long as the connector type was added as a flow file attribute by this processor, and there was no other info needed to make the routing decision.

Status

Assignee

Sean Felten

Reporter

Sean Felten

Labels

None

Reviewer

None

Story Points

5

Epic Link

Components

Sprint

Priority

High