Kylo cluster doesn't work correctly

Description

Hi,

I am testing clustering feature in Kylo and I think it is unstable. Here is my setup:

  • 3 separate machines with one instance of Kylo on each of them.

  • ModeShape is configured to use a database as a shared persistent storage.

  • A load balancer is configured to balance the load between these 3 machines

With this setup, go to Kylo’s UI and create a feed, use timer as schedule, enable it immediately and go to look at Kylo’s Dashboard.
I tested this multiple times and most of the time I see the issue that some steps are not listed in feed’s steps.
Even worst, if the last step also missed then Kylo keeps reporting that feed is still running while it already finished in NiFi.

I dug deeper and I think this is because of synchronization issue. Feed is created and enabled immediately so it starts running right the way and NiFi start sending provenance events back to Kylo.
Since it is load balanced so some events are sent to hosts that ModeShape hasn’t finished propagating the changes to so Kylo could not find feed information and decides to ignore them.
When it happens to a feed it will happen to it all the time when the feed runs.
For reference here is the email thread https://groups.google.com/forum/#!topic/kylo-community/7eX9gVYAW7U

Thanks
-Binh

Environment

None

Assignee

Scott Reisdorf

Reporter

Binh Nguyen

Reviewer

None

Story point estimate

None

Components

Sprint

None

Fix versions

Priority

High
Configure