Kylo reports the processing node coming AFTER the node actually failed as "FAILED"

Description

If a processing node routes the flow file to its "FAILURE" output queue, the node itself and not the next one fetching the flow file from the "FAILURE" queue should be marked as failed in Kylo UI.

This behaviour can be very confusing for the end user.

The failure handling logic in com.thinkbiganalytics.nifi.provenance.ProvenanceFeedLookup#setProcessorFlowType is incomplete, as it simply checks the INCOMING queue's name, and marks the CURRENT event as FAILURE if the incoming queue is a FAILURE queue. This is inconsistent, as the event for the (previous) node routing the flow file to the FAILURE queue is the one, where processing failed.

com.thinkbiganalytics.nifi.provenance.ProvenanceFeedLookup#setProcessorFlowType

Environment

None

Activity

Show:
Scott Reisdorf
April 24, 2017, 11:35 PM

This is because of a bug I reported to NiFi

https://issues.apache.org/jira/browse/NIFI-3453

Kylo gets NiFi Flow information from the Provenance Events that NiFi publishes after the processor has completed.
In order to walk back in the FlowFile to find out what processor previous to it the bug above needs to be fixed by NiFi.
Until then because the prior processor/event info is not available in the provenance its unable to mark the previous processor as being failed.

Peter Horvath
April 25, 2017, 1:34 AM

I'm wondering if it was possible to programatically traverse the flow definition itself to find the previous node?

Scott Reisdorf
April 25, 2017, 1:49 AM

Easier said than done. A few things could be done for simple flows, but not for more complicated flows
You can think of a flow in 2 ways.
1) The flow design (how it looks on the NiFi canvas when you are designing the flow)
We have a flow visitor that inspects and walks the flow as it is designed marking a processor and its related sources/destinations (connected processors)
2) The flow file at runtime (A specific instance of a running flow using the design in #1).
This is the flow file itself as it is running in NiFi.

To get a handle on the failed processor you need to look at #2, the running flow file.

You could traverse the flow definition (#1) and Kylo has all that information, however if a flow splits and joins and goes down a specific path based upon a running attribute you are not able to determine which path to take if you want to walk back in the definition to determine the previous processor that failed. For simple flows you could do this, but once a flow starts having splits, joins, funnels, merges it gets impossible to do this without knowing the running runtime flow previous connection in the provenance.

Peter Horvath
May 30, 2017, 8:07 PM

Would it be possible to describe this behaviour in Kylo documentation, e.g. in a Known issues section?

Assignee

Unassigned

Reporter

Peter Horvath

Labels

None

Reviewer

None

Story point estimate

None

Affects versions

Priority

Medium