HRQB 17 - Prep for functional tasks and pipelines #22
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Why these changes are being introduced: Debugging error will be easier if counts are available for the total number of records processed by a Task. This will reveal tasks that are not returning the expected number of records for downstream tasks. How this addresses that need: * Adds new properties on base Targets to produce a record count by reading the output Target data Side effects of this change: * None
Why these changes are being introduced: If one HRQBPipelineTask runs another pipeline task, it would be beneficial to see all pipelines in that lineage in the outputted file. How this addresses that need: * Adds parameter 'pipline_parent_name' to HRQBPipelineTask, which is considered when returning a pipeline name Side effects of this change: * Output target files include all calling pipelines in the filename Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
Why these changes are being introduced: When a pipeline is run via the luigi runner, it can support multiple parallel workers. It is unlikely we will need more than one concurrent worker, which means we do not need any threading or multiprocessing overhead. But this env var allows for that if needed. How this addresses that need: * Add new optional env var LUIGI_NUM_WORKERS * If not set, luigi.build() is run with 1 worker Side effects of this change: * By default, pipelines are run with a single worker Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
Why these changes are being introduced: Some tasks may require using data from Quickbase to inform upserts into other tables. This will require querying Quickbase tables using the QBClient. How this addresses that need: * Adds methods on QBClient to perform record queries * Methods are opinionated to use field labels vs ids * Methods return dataframes like other QBclient methods Side effects of this change: * QBClient is able to query records from tables Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
Why these changes are being introduced: Quickbase API may surface errors in two ways. First, there might be a non 2xx status code returned, which we will throw and immediate exception. Second, the request may have been partially successful, but the response indicates some failures for certain data upserted. For these we will log but continue. How this addresses that need: * Adds status code error handling in QBClient * Adds logging to QB tasks run method when errors in response Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/HRQB-17
ehanson8
approved these changes
May 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose and background context
While beginning on writing actual tasks and pipelines to load data into Quickbase, some additional helpers and functionality were required. This PR breaks that out into a smaller, standalone PR before implementing actual tasks.
To reiterate: this work lays the foundation for "real" tasks, which themselves should focus exclusively on getting data, transforming it, and loading it. Anytime we can offload that work to shared functionality, the better.
The primary are fairly well encapsulated in commits for review:
HRQBPipelineTaskcalls another, append all pipelines to the target output pathQBClientis able to query tables for records as data used in other tasksQBClienttasks are considered failed if API response shows errorsQBClientbase API URL is parameterized as env varHow can a reviewer manually see the effects of these changes?
This PR is a collection of small updates to support the writing of funtional tasks as a next step. As such, not readily testable in their own right.
Commits have, where applicable, new tests to demonstrate the new behavior.
Includes new or updated dependencies?
NOI
Changes expectations for external applications?
NO
What are the relevant tickets?
Developer
Code Reviewer(s)