2.Project_Master_Execution.pdf -

Please enable JavaScript to view the full PDF

Project Master Execution administer and orchestrate the execution of all child jobs in the Idea project Level Basic Project Star Schema Integration Job Reading 2/7 Order Project Date @August 10, 2022 Purpose Project overview Requirements Job overview Context variables Components Connect ODS Connect DWH Generate UID Print input parameter load If(s) conditions Run(s) Commit Close Purpose The objective of this master job is to specify and orchestrate the execution of all child jobs in the project. The execution of the child jobs is modular, to satisfy the client reload requirements explained below. Project Master Execution 1 Project overview Before digging into the details of this master job, it is helpful to recall the project overall architecture : 1. The .csv source files are made available on a FTP server on a regular basis. 2. Those files are loaded, untransformed, on an ODS. 3. The data is then loaded in a DWH with few transformations and lookups. simplified project architecture Requirements The client will ask for file reloads under two scenarios : 1. The rejections may reveal that the Entity/Account tables are incomplete and should be re-loaded. 2. The client may realize that some records are missing from the Ledger information and should be loaded. Each data ETL load/reload batch should be traceable through a uniquely generated UID. Project Master Execution 2 Job overview Context variables To manage the reloads described earlier, it is not always necessary to reload the complete project. Instead only subsets of the project can be executed to satisfy the reload requirements : Project Master Execution 3 Components Connect ODS Connect DWH Project Master Execution 4 Generate UID Print input parameter load If(s) conditions context.ODSDimLoad Project Master Execution 5 context.ODSFactLoad context.DWHDimLoad context.DWHFactLoad Run(s) Project Master Execution 6 Commit Project Master Execution 7 Close Project Master Execution 8