When using AWS EMR, SparkLauncher does not set spark env

Description

It appears that when the data transformation template is used, ExecuteSparkJob in the reusable template fails to set the env using SparkLauncher (ExecuteSparkJob.java:518).

Reproducible steps using the Kylo Sandbox AMI and AWS EMR can be found attached and on the community forum.

This presents the error "No encryption/decryption key has been configured - please see configuration documentation" in the spark logs.

Environment

None

Activity

Show:
Ben S
April 30, 2019, 9:39 AM

I was able to get the ExecuteSparkJob working by building my own from the source code and adding the following:

Right after SparkLauncher's instantiation.

Gibran Akram
May 29, 2019, 12:01 PM

Did you alter the processor code for this workaround?
This blocks us from doing any transformation successfully.

Assignee

Unassigned

Reporter

Ben S

Labels

Reviewer

None

Story point estimate

None

Components

Affects versions

Priority

Highest
Configure