Validator and Profiler spark context issues

Description

The Validator and Profiler were changed to share spark context.

  • The Validator class removed the

    call. This causes issues in some spark versions (i.e. spark 2.2 ) where the caller ExecuteSparkJob does a

    Without this sparkContext.stop() in the Validator the job will hang for the allotted timeout

2. The Profiler was refactored to share spark context.
Once the sparkContext is stopped in the Validator this doesnt work anymore

Environment

None

Assignee

Sean Felten

Reporter

Scott Reisdorf

Labels

None

Reviewer

None

Story point estimate

None

Time tracking

4h

Sprint

None

Fix versions

Affects versions

Priority

High
Configure