6.Load_the_DWH_Fact_Table.pdf -

Please enable JavaScript to view the full PDF

Load the DWH Fact Table Idea load the DWH fact table Level Basic Project Star Schema Integration Job Reading Order 6/7 Project Date @August 10, 2022 Purpose Requirements Job overview Components Connect ODS Connect DWH Truncate table Ledger Reject Get ODS FactLedger Get DWH DimAccount Lookup Account Insert records Ledger Reject Get DWH Entity Lookup Entity Insert records Ledger Reject Upsert records Ledger Get metadata Write metadata Purpose Once the data is loaded into the staging ODS area, the purpose of this job is to : Load the ODS facts into the DWH Capture lookup rejections into a dedicated table Insert the matching records into the DWH Keep a track record of the metadata Requirements We capture lookup rejections based on the defined requirements : Sometimes, some Ledger key information will not be recorded in the Account and Entity tables. Under such scenario, the client asks that the matching information is recorded in the DWH nevertheless. In other words, the client does not want to kill the entire ETL process if a couple of records are not matching the dimension tables. The non-matching data should be recorded in a separate table for a rejection analysis which will, most of the time, lead to a Ledger, Entity and/or Account reload. As usual, we update the audit table based on the defined requirements : Each data ETL load/reload batch should be traceable through a uniquely generated UID. All DML changes should be tracked in a metadata table. This metadata table should contain the number of rows deleted, inserted, updates, rejected, as well as the batch UID. Job overview Load the DWH Fact Table 1 Components Connect ODS Connect DWH Load the DWH Fact Table 2 Truncate table Ledger Reject Get ODS FactLedger Get DWH DimAccount Load the DWH Fact Table 3 Lookup Account Create a foreign key FK_ACCOUNT based on the dimension table surrogate key Catch lookup inner join reject Insert records Ledger Reject Load the DWH Fact Table 4 Get DWH Entity Lookup Entity Create a foreign key FK_ENTITY based on the dimension table surrogate key Catch lookup inner join reject Load the DWH Fact Table 5 Insert records Ledger Reject Upsert records Ledger Load the DWH Fact Table 6 Get metadata The number of fact lines rejected is defined by the number of ODS input lines minus the fact table DWH updates/inserts : ((Integer)globalMap.get("tDBInput_3_NB_LINE")) - ((Integer)globalMap.get("tDBOutput_1_NB_LINE_UPDATED")) - ((Integer)globalMap.get("tDBOutp Write metadata Load the DWH Fact Table 7