feat: adding demographic durable function to allow for adding of 50,000 records #505
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…emograpghic records. the receiveCaasFile now calls the functions polls it every N ammount of milliseconds to stop the function from timming out on big files
application/CohortManager/src/Functions/CaasIntegration/RetrieveMeshFile/README.md
Outdated
Show resolved
Hide resolved
...tManager/src/Functions/CaasIntegration/receiveCaasFile/ProcessFileClasses/ProcessCaasFile.cs
Outdated
Show resolved
Hide resolved
...tManager/src/Functions/CaasIntegration/receiveCaasFile/ProcessFileClasses/ProcessCaasFile.cs
Outdated
Show resolved
Hide resolved
application/CohortManager/src/Functions/CaasIntegration/receiveCaasFile/receiveCaasFile.cs
Outdated
Show resolved
Hide resolved
application/CohortManager/src/Functions/Shared/Model/Enums/DemographicRunTImeStatus.cs
Outdated
Show resolved
Hide resolved
...Manager/src/Functions/screeningDataServices/DemographicDataService/DemographicDataService.cs
Outdated
Show resolved
Hide resolved
* feat: DTOS 00000 adding backup storage redundancy and setting it to local * feat: DTOS 00000 adding backup storage redundancy and setting it to local tfm fmt run * feat: DTOS 00000 adding backup storage redundancy and setting it to local pipeline ref updated
…24 (#493) * feat: DTOS 00000 enrolling ParticipantManagementDataService to int and preprod off pr * feat: added exception function to demographic and sorted the docker script * feat: increased the DB SKU to reflect current settings * fix: linting issue in preprod.tfvars * feat: change the PreProd SQL SKU to S3 * fix: comment the new function from docker image generation for now --------- Co-authored-by: Maciej Murawski <maciej.murawski@nordcloud.com>
* feat: add get primary care provider db lookup * feat: move chained rules into separate method * feat: dblookup to get bso code using primary care provider * feat: add extra bso dblookup * chore: rename pcp bso lookup to avoid clash with existing bso lookup * feat: move reason for removal transformations into separate class
* pagination * chore: added total pages, items and current page methods and to result object * chore: method name and formatting * feat: logic fix and added in comments * chore: update tests, DI - pagination service * chore: whitespace * fix: currentPage logic fix
* Add peering dendency checks * Increase database max size
…to make sure they are getting base 64 string to simulate input from queue. process caas file now correctly logs when the particiapnt form the databsae is null
…ecord is correct and builds
application/CohortManager/src/Functions/Shared/Model/WebhookRoot.cs
Outdated
Show resolved
Hide resolved
tests/UnitTests/DurableDemographicTests/DurableDemographicTests.cs
Outdated
Show resolved
Hide resolved
...c/Functions/DemographicServices/DemographicDataManagementFunction/DemographicDataFunction.cs
Outdated
Show resolved
Hide resolved
...tManager/src/Functions/CaasIntegration/receiveCaasFile/ProcessFileClasses/ProcessCaasFile.cs
Outdated
Show resolved
Hide resolved
...c/Functions/DemographicServices/DemographicDurableFunction/DemographicDurableFunction.csproj
Outdated
Show resolved
Hide resolved
tests/UnitTests/DurableDemographicTests/DurableDemographicTests.cs
Outdated
Show resolved
Hide resolved
…hekcing of compleation
…ent in GetInvalidFlag in paricipant model
...tManager/src/Functions/CaasIntegration/receiveCaasFile/ProcessFileClasses/ProcessCaasFile.cs
Outdated
Show resolved
Hide resolved
...tManager/src/Functions/CaasIntegration/receiveCaasFile/ProcessFileClasses/ProcessCaasFile.cs
Outdated
Show resolved
Hide resolved
| <ProjectReference Include="..\..\Shared\DataServices.Database\DataServices.Database.csproj" /> | ||
| <ProjectReference Include="..\..\Shared\DataServices.Client\DataServices.Client.csproj" /> | ||
| </ItemGroup> | ||
| <Target Name="CopyGrpcNativeAssetsToOutDir" AfterTargets="Build"> <ItemGroup> <NativeAssetToCopy Condition="$([MSBuild]::IsOSPlatform('OSX'))" Include="$(OutDir)runtimes/osx-arm64/native/*" /> </ItemGroup> <Copy SourceFiles="@(NativeAssetToCopy)" DestinationFolder="$(OutDir).azurefunctions/runtimes/osx-arm64/native" /> </Target> |
There was a problem hiding this comment.
Im not sure what this is and why its needed. Its specifically referencing OSX which is concerning.
There was a problem hiding this comment.
this is needed to be able to run the durable on Mac. for now Microsoft does not fully support Mac silicon for the durable functions. I have pushed the function to azure and it runs in linux in azure and works perfectly. taking this out will not allow us to run this locally on MAC OS. this gives us the packages and things we need that windows has by default
application/CohortManager/src/Functions/Shared/Common/CheckDemographic.cs
Outdated
Show resolved
Hide resolved
application/CohortManager/src/Functions/Shared/Data/Database/CreateDemographicData.cs
Outdated
Show resolved
Hide resolved
…old reocrds when updating
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
these changes are to allow for adding larger files, up to 50,000 records. this solution uses a durable function and batches records into the database for demographic data
Context
Type of changes
Checklist
Sensitive Information Declaration
To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.