An extremely common data warehousing pattern is ingesting data into a dimension table and performing a 'Slowly Changing Dimension Type 2' Updates.
You will encounter this need when for example Customer, Product, Location(store) data data are ingested into a Big Data data warehouse.
You may see this as a result of ingesting claim data updates and a current view is required.
This operation is relatively easy to perform on an ACID based store but the logic is complex for Hive with its append only backing distributed file system.
This story will extend the Table Merge processor to accommodate the logic necessary to easily merge updates into an existing dimensional table stored in Hive on HDFS( or S3 or ADLS)?
The requested extension will need to know what the primary key parts are, what strategy (versioning or end dating) is being used, what the corresponding version/end date fields are.
FYI there are ~3 variants on the scd type 2 implementation, see Wikipedia, they have a decent write up.
Which variant(s) will you implement with this Jura?
Sent via my iPhone
On Feb 21, 2018, at 11:39 AM, Jeremy Merrifield (Jira) <jira@kylo-io.atlassian.net<jira@kylo-io.atlassian.net>> wrote:
https://avatar-cdn.atlassian.com/75d36576fc03908b47d1cd92016807d8?s=48&d=https%3A%2F%2Fsecure.gravatar.com%2Favatar%2F75d36576fc03908b47d1cd92016807d8%3Fd%3Dmm%26s%3D48%26noRedirect%3Dtrue Jeremy Merrifield<https://kylo-io.atlassian.net/secure/ViewProfile.jspa?name=jeremy.merrifield> updated an issue
Kylo<https://kylo-io.atlassian.net/browse/KYLO> / <mime-attachment.png><https://kylo-io.atlassian.net/browse/KYLO-415> KYLO-415<https://kylo-io.atlassian.net/browse/KYLO-415>
Support SCD Type 2 Merges<https://kylo-io.atlassian.net/browse/KYLO-415>
Change By: Jeremy Merrifield<https://kylo-io.atlassian.net/secure/ViewProfile.jspa?name=jeremy.merrifield>
Sprint: KYLO 0.9.1 Sprint 1 2
<mime-attachment.png> <https://kylo-io.atlassian.net/browse/KYLO-415#add-comment> Add Comment<https://kylo-io.atlassian.net/browse/KYLO-415#add-comment>
Get Jira notifications on your phone! Download the Jira Cloud app for Android<https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail> or iOS<https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8>