Allow number_of_partitions to be specified for Validator

Description

None

Activity

Show:
Jagrut Sharma
April 17, 2017, 8:53 PM

Validator now accepts a parameter that allows explicit setting of number of partitions the RDD will be split into. This can help with large files.

Example usage:
When using via NiFi, provide this to the Validate And Split Records processor configuration:

MainArgs: ${category},${feed},${feedts},${table_field_policy_json_file},--numPartitions,10

Done

Assignee

Jagrut Sharma

Reporter

Jagrut Sharma

Labels

None

Reviewer

None

Sprint

None

Fix versions

Priority

Medium