When using AWS EMR, SparkLauncher does not set spark env
Description
It appears that when the data transformation template is used, ExecuteSparkJob in the reusable template fails to set the env using SparkLauncher (ExecuteSparkJob.java:518).
Reproducible steps using the Kylo Sandbox AMI and AWS EMR can be found attached and on the community forum.
This presents the error "No encryption/decryption key has been configured - please see configuration documentation" in the spark logs.
Environment
None
Activity
Show:
Ben S
April 30, 2019, 9:39 AM
I was able to get the ExecuteSparkJob working by building my own from the source code and adding the following:
Right after SparkLauncher's instantiation.
Gibran Akram
May 29, 2019, 12:01 PM
Did you alter the processor code for this workaround?
This blocks us from doing any transformation successfully.
Assignee
Unassigned
Reporter
Ben S
Labels
Reviewer
None
Story point estimate
None
Components
Affects versions
Priority
Configure