Merge pull request #102 from holukas/adding-xgboost

Adding xgboost

CHANGELOG.md

@@ -2,6 +2,55 @@
 
 ![DIIVE](images/logo_diive1_256px.png)
 
+## v0.75.0 | 26 Apr 2024
+
+### XGBoost gap-filling
+
+[XGBoost](https://xgboost.readthedocs.io/en/stable/index.html) can now be used to fill gaps in time series data.
+In `diive`, `XGBoost` is implemented in class `XGBoostTS`, which adds additional options for easily including e.g.
+lagged variants of feature variables, timestamp info (DOY, month, ...) and a continuous record number. It also allows
+direct feature reduction by including a purely random feature (consisting of completely random numbers) and calculating
+the 'permutation importance'. All features where the permutation importance is lower than for the random feature can
+then be removed from the dataset, i.e., the list of features, before building the final model.
+
+`XGBoostTS` and `RandomForestTS` both use the same base class `MlRegressorGapFillingBase`. This base class will also
+facilitate the implementation of other gap-filling algorithms in the future.
+
+Another fun (for me) addition is the new class `TimeSince`. It allows to calculate the time since the last occurrence of
+specific conditions. One example where this class can be useful is the calculation of 'time since last precipitation',
+expressed as number of records, which can be helpful in identifying dry conditions. More examples: 'time since freezing
+conditions' based on air temperature; 'time since management' based on management info, e.g. fertilization events.
+Please see the notebook for some illustrative examples.
+
+**Please note that `diive` is still under developement and bugs can be expected.**
+
+### New features
+
+- Added gap-filling class `XGBoostTS` for time series data,
+  using [XGBoost](https://xgboost.readthedocs.io/en/stable/index.html) (`diive.pkgs.gapfilling.xgboost_ts.XGBoostTS`)
+- Added new class `TimeSince`: counts number of records (inceremental number / counter) since the last time a time
+  series was inside a specified range, useful for e.g. counting the time since last precipitation, since last freezing
+  temperature, etc. (`diive.pkgs.createvar.timesince.TimeSince`)
+
+### Additions
+
+- Added base class for machine learning regressors, which is basically the code shared between the different
+  methods. At the moment used by `RandomForestTS` and `XGBoostTS`. (`diive.core.ml.common.MlRegressorGapFillingBase`)
+- Added option to change line color directly in `TimeSeries` plots (`diive.core.plotting.timeseries.TimeSeries.plot`)
+
+### Notebooks
+
+- Added new notebook for gap-filling using `XGBoostTS` with mininmal settings (`notebooks/GapFilling/XGBoostGapFillingMinimal.ipynb`)
+- Added new notebook for gap-filling using `XGBoostTS` with more extensive settings (`notebooks/GapFilling/XGBoostGapFillingExtensive.ipynb`)
+- Added new notebook for creating `TimeSince` variables (`notebooks/CalculateVariable/TimeSince.ipynb`)
+
+### Tests
+
+- Added test case for XGBoost gap-filling (`tests.test_gapfilling.TestGapFilling.test_gapfilling_xgboost`)
+- Updated test case for random forest gap-filling (`tests.test_gapfilling.TestGapFilling.test_gapfilling_randomforest`)
+- Harmonized test case for XGBoostTS with test case of RandomForestTS
+- Added test case for `TimeSince` variable creation (`tests.test_createvar.TestCreateVar.test_timesince`)
+
 ## v0.74.1 | 23 Apr 2024
 
 This update adds the first notebooks (and tests) for outlier detection methods. Only two tests are included so far and

README.md

@@ -22,8 +22,7 @@ Example notebooks can be found in the folder `notebooks`. More notebooks are add
 
 - Calculate z-aggregates in quantiles (classes) of x and
   y ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Analyses/CalculateZaggregatesInQuantileClassesOfXY.ipynb))
-- Daily
-  correlation ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Analyses/DailyCorrelation.ipynb))
+- Daily correlation ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Analyses/DailyCorrelation.ipynb))
 - Decoupling: Sorting bins
   method ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Analyses/DecouplingSortingBins.ipynb))
 - Find data gaps ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Analyses/GapFinder.ipynb))
@@ -40,6 +39,7 @@ Example notebooks can be found in the folder `notebooks`. More notebooks are add
 
 ### Create variable
 
+- Calculate time since last occurrence, e.g. since last precipitation ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/CalculateVariable/TimeSince.ipynb))
 - Calculate daytime flag, nighttime flag and potential radiation from latitude and
   longitude ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/CalculateVariable/Daytime_and_nighttime_flag.ipynb))
 - Day/night flag from sun angle
@@ -75,11 +75,9 @@ Example notebooks can be found in the folder `notebooks`. More notebooks are add
 
 ### Flux processing chain
 
-For info about the Swiss FluxNet flux levels,
-see [here](https://www.swissfluxnet.ethz.ch/index.php/data/ecosystem-fluxes/flux-processing-chain/).
+For info about the Swiss FluxNet flux levels, see [here](https://www.swissfluxnet.ethz.ch/index.php/data/ecosystem-fluxes/flux-processing-chain/).
 
-- Flux processing
-  chain ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/FluxProcessingChain/FluxProcessingChain.ipynb))
+- Flux processing chain ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/FluxProcessingChain/FluxProcessingChain.ipynb))
     - The notebook example shows the application of:
         - Level-2 quality flags
         - Level-3.1 storage correction
@@ -100,12 +98,10 @@ Format data to specific formats
 
 Fill gaps in time series with various methods
 
-- Linear
-  interpolation ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/LinearInterpolation.ipynb))
--
-RandomForestTS ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/RandomForestGapFilling.ipynb))
-- Quick random forest
-  gap-filling ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/QuickRandomForestGapFilling.ipynb))
+- XGBoostTS ([notebook example (minimal)](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/XGBoostGapFillingMinimal.ipynb), [notebook example (more extensive)](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/XGBoostGapFillingExtensive.ipynb))
+- RandomForestTS ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/RandomForestGapFilling.ipynb))
+- Linear interpolation ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/LinearInterpolation.ipynb))
+- Quick random forest gap-filling ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/GapFilling/QuickRandomForestGapFilling.ipynb))
 
 ### Outlier Detection