Job failed step status not mapped to the correct processor (ExecuteScript)

Description

The failed flow from a ExecuteScript processor does not appear correctly in the Job status.
I'm processing and routing to failure in a Groovy script (executed by an ExecuteScript), and in most cases, the failed status in the Job Step status appears in the next processor step (post-failure/success action)

The only case when the Step status appears as Failed to the ExecuteScript processor is when the failure relationship is auto-terminated.

Attached is a feed example & template, script used and screenshot. The feed generates flowFiles and simply discards them at the end.

Not investigated too much, but the failed status appears to be picked-up mostly when the return is !=0, in eg:

Please advise about how to properly generate failed statuses.

Environment

NiFi 1.3
CDH 5.10

Activity

Show:
Scott Reisdorf
October 24, 2017, 3:12 AM

The issue is with NiFi and how it reports provenance. https://issues.apache.org/jira/browse/NIFI-3453

Kylo gets flow status from the Provenance Events that are emitted from the processors. Usually these events occur upon completion of a processor (although sometimes processors can generate start/SEND events).

When a processors finishes it will emit a provenance event indicating that the processor is done. This event is separate from the Relationship/status of the processor (i.e. failed/success). The only time the failed status can be obtained via the event itself is when it "auto terminates on failure". Because of this Kylo detects failure from the relationship of where the event came from and uses the logic where any connection that has the name "failure" it will treat as a failed job.

I opened that JIRA (NIFI-3453) a long time ago, but it has not been worked yet by NiFi. That is what is needed for Kylo to correctly navigate back to the actual processor that failed if it goes through a failure connection/relationship.

Claudiu Stanciu
October 24, 2017, 3:26 AM

Thank you for the follow-up. Can I at least somehow set the Job exit description, or it's picked up automagically?

Scott Reisdorf
October 24, 2017, 3:58 AM

Yes.
Add a property to your flowfile with the key of: "kylo.jobExitDescription".
When Kylo processes/finishes the job it will append the value of that property if it exists in the flow file to the job exit description

Claudiu Stanciu
October 24, 2017, 4:01 AM

Great, thank you!

Assignee

Scott Reisdorf

Reporter

Claudiu Stanciu

Labels

Reviewer

None

Story point estimate

None

Story Points

20

Components

Sprint

None

Affects versions

Priority

Medium