After upgrading my Kylo instance to 0.10.0 in my EMR environment the Validator step fails in the ingest template with the following error.
18/12/07 20:16:54 ERROR ApplicationMaster: User class threw exception: java.lang.IllegalStateException: Promise already completed.
java.lang.IllegalStateException: Promise already completed.
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
at scala.concurrent.Promise$class.success(Promise.scala:86)
at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$sparkContextInitialized(ApplicationMaster.scala:421)
at org.apache.spark.deploy.yarn.ApplicationMaster$.sparkContextInitialized(ApplicationMaster.scala:828)
at org.apache.spark.scheduler.cluster.YarnClusterScheduler.postStartHook(YarnClusterScheduler.scala:32)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:558)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:116)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2510)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at com.thinkbiganalytics.spark.datavalidator.Validator.main(Validator.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
This seems related to this commit:
https://github.com/Teradata/kylo/commit/a790d9a1a44cacb5473aba2beb67d384182fdc38#diff-079a844b77bc7573d6b48119ec355d2f
Related to KYLO-2239.