Spark Validator fails with Spark 2.3

Description

After upgrading my Kylo instance to 0.10.0 in my EMR environment the Validator step fails in the ingest template with the following error.

18/12/07 20:16:54 ERROR ApplicationMaster: User class threw exception: java.lang.IllegalStateException: Promise already completed.
java.lang.IllegalStateException: Promise already completed.
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
at scala.concurrent.Promise$class.success(Promise.scala:86)
at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$sparkContextInitialized(ApplicationMaster.scala:421)
at org.apache.spark.deploy.yarn.ApplicationMaster$.sparkContextInitialized(ApplicationMaster.scala:828)
at org.apache.spark.scheduler.cluster.YarnClusterScheduler.postStartHook(YarnClusterScheduler.scala:32)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:558)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:116)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2510)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at com.thinkbiganalytics.spark.datavalidator.Validator.main(Validator.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Environment

None

Status

Assignee

Jeremy Merrifield

Reporter

Jeremy Merrifield

Labels

None

Reviewer

None

Story point estimate

None

Components

Fix versions

Affects versions

0.10.0

Priority

Highest