Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Reef Check entities #1052

Merged
merged 17 commits into from
Dec 17, 2024
Merged

Add Reef Check entities #1052

merged 17 commits into from
Dec 17, 2024

Conversation

K-Markopoulos
Copy link
Collaborator

Integrate provided Reef Check Data into Aqualink.

  • Add Reef Check entities: ReefCheckSite, ReefCheckSurvey, ReefCheckOrganism, ReefCheckSubstrate
  • Add script to parse the xlsx files and create the entities.
  • Add basic api controllers for sites and surveys

image

@K-Markopoulos K-Markopoulos self-assigned this Nov 13, 2024
Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ This pull request was not sent to the PullRequest network because the pull request is a draft.

Comment on lines +290 to +300
@ApiProperty()
@VirtualColumn({
query: (survey) => `
SELECT satellite_temperature
FROM daily_data
WHERE DATE(daily_data.date) = DATE(${survey}.date)
AND daily_data.site_id = ${survey}.site_id
LIMIT 1
`,
})
satelliteTemperature: string;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is based on surveys service

.leftJoinAndMapOne(
'survey.latestDailyData',
'daily_data',
'data',
'data.site_id = survey.site_id AND DATE(data.date) = DATE(survey.diveDate)',
)

@ericboucher please verify whether this is the right way to compute the satellite temperature.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So ultimately we are pulling data from daily_data ? I think that's correct but it also means that we need to update the daily data table for all the site/survey date combos

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it would be quite inefficient to load all past data for all sites

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed that would require to add the daily_data entry for the date of the survey

@K-Markopoulos K-Markopoulos mentioned this pull request Nov 22, 2024
6 tasks
@K-Markopoulos K-Markopoulos marked this pull request as ready for review December 2, 2024 13:52
Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ This pull request was sent to the PullRequest network for review. Expert reviewers are now being matched to your request based on the code's requirements. Stay tuned!

What to expect from this code review:
  • Comments posted to any areas of potential concern or improvement.
  • Detailed feedback or actions needed to resolve issues that are found.
  • Turnaround times vary, but we aim to be swift.

@K-Markopoulos you can click here to see the review status or cancel the code review job.

Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PullRequest Breakdown

Reviewable lines of change

+ 1,453
- 4

99% TypeScript
1% JSON
<1% Markdown

Generated lines of change

+ 10
- 84

Type of change

Feature - These changes are adding a new feature or improvement to existing code.
1 Message
⚠️ Due to its size, this pull request will likely have a little longer turnaround time and will probably require multiple passes from our reviewers.

Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work here. I noticed just one issue noted inline for your consideration.

Image of Graham C Graham C


Reviewed with ❤️ by PullRequest

query: (survey) => `
SELECT satellite_temperature
FROM daily_data
WHERE DATE(daily_data.date) = DATE(${survey}.date)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential SQL injection vulnerability. I'd recommend applying validation on survey within this function to guard against this, even if this is not vulnerable today, but future refactors could inadvertently introduce this. Having this function check that the survey matches a regex is good security practice.

🔺 Vulnerability (Error)

Image of Graham C Graham C

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback on this. Based on the VirtualColumn description, the survey var is not user specified/input and it seems to be an alias to the entity name, therefore I think It's unlikely to cause an sql injection. As I understand it, an SQL injection is possible if someone renames the class to match the injection string, but even that would probably be invalid syntax.
That said, are you still concerned about this change? Should we sanitize the value before using it or simply hardcode the alias?

Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PullRequest reviewed the updates made to #1052 up until the latest commit (e632ccd). No further issues were found.

Reviewed by:

Image of Graham C Graham C

@@ -0,0 +1,700 @@
/* eslint-disable fp/no-mutating-methods */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a bit more comments in this file to make sure we will know how to adapt it and run it again with new data (and/or overlapping data).

@ericboucher
Copy link
Member

Great work, let's add comments and also expand the README to make a note of this addition

Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PullRequest reviewed the updates made to #1052 up until the latest commit (93ccfda). No further issues were found.

Reviewed by:

Image of Graham C Graham C

Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PullRequest reviewed the updates made to #1052 up until the latest commit (b66e360). No further issues were found.

Reviewed by:

Image of Graham C Graham C

@ericboucher ericboucher merged commit 2958800 into master Dec 17, 2024
3 checks passed
@ericboucher ericboucher deleted the reef-check-data branch December 17, 2024 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants