Skip to content

Commit

Permalink
Merge pull request #107 from holukas/diel-cycle-plot
Browse files Browse the repository at this point in the history
Diel cycle plot
  • Loading branch information
holukas authored May 14, 2024
2 parents e648180 + 1f0e841 commit 53d72fc
Show file tree
Hide file tree
Showing 14 changed files with 2,317 additions and 1,023 deletions.
37 changes: 35 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,37 @@

![DIIVE](images/logo_diive1_256px.png)

## v0.76.0 | 14 May 2024

### Diel cycle plot

The new class `DielCycle` allows to plot diel cycles per month or across all data for time series data. At the moment,
it plots the (monthly) diel cycles as means (+/- standard deviation). It makes use of the time info contained in the
datetime timestamp index of the data. All aggregates are calculated by grouping data by time and (optional) separately
for each month. The diel cycles have the same time resolution as the time component of the timestamp index, e.g. hourly.

![DIIVE](images/plotDielCycle_diive_v0.76.0.png)

### New features

- Added new class `DielCycle` for plotting diel cycles per month (`diive.core.plotting.dielcycle.DielCycle`)
- Added new function `diel_cycle` for calculating diel cycles per month. This function is also used by the plotting
class `DielCycle` (`diive.core.times.resampling.diel_cycle`)

### Additions

- Added color scheme that contains 12 colors, one for each month. Not perfect, but better than
before. (`diive.core.plotting.styles.LightTheme.colors_12_months`)

### Notebooks

- Added new notebook for plotting diel cycles (per month) (`notebooks/Plotting/DielCycle.ipynb`)
- Added new notebook for calculating diel cycles (per month) (`notebooks/Resampling/ResamplingDielCycle.ipynb`)

### Tests

- Added test case for new function `diel_cycle` (`tests.test_resampling.TestResampling.test_diel_cycle`)

## v0.75.0 | 26 Apr 2024

### XGBoost gap-filling
Expand Down Expand Up @@ -40,8 +71,10 @@ Please see the notebook for some illustrative examples.

### Notebooks

- Added new notebook for gap-filling using `XGBoostTS` with mininmal settings (`notebooks/GapFilling/XGBoostGapFillingMinimal.ipynb`)
- Added new notebook for gap-filling using `XGBoostTS` with more extensive settings (`notebooks/GapFilling/XGBoostGapFillingExtensive.ipynb`)
- Added new notebook for gap-filling using `XGBoostTS` with mininmal
settings (`notebooks/GapFilling/XGBoostGapFillingMinimal.ipynb`)
- Added new notebook for gap-filling using `XGBoostTS` with more extensive
settings (`notebooks/GapFilling/XGBoostGapFillingExtensive.ipynb`)
- Added new notebook for creating `TimeSince` variables (`notebooks/CalculateVariable/TimeSince.ipynb`)

### Tests
Expand Down
13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ by the [ETH Grassland Sciences group](https://gl.ethz.ch/) for [Swiss FluxNet](h
Recent updates: [CHANGELOG](https://github.com/holukas/diive/blob/main/CHANGELOG.md)
Recent releases: [Releases](https://github.com/holukas/diive/releases)

Example notebooks can be found in the folder `notebooks`. More notebooks are added constantly.
## Overview of example notebooks

- For many examples see notebooks here: [Notebook overview](https://github.com/holukas/diive/blob/main/notebooks/OVERVIEW.ipynb)
- More notebooks are added constantly...

## Current Features

Expand Down Expand Up @@ -105,14 +108,14 @@ Fill gaps in time series with various methods

### Outlier Detection

Single outlier tests create a flag where `0=OK` and `2=outlier`.

#### Multiple tests combined

- Step-wise outlier detection

#### Single tests

Single outlier tests create a flag where `0=OK` and `2=outlier`.

- Absolute limits ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/OutlierDetection/AbsoluteLimits.ipynb))
- Absolute limits, separately defined for daytime and nighttime data ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/OutlierDetection/AbsoluteLimitsDaytimeNighttime.ipynb))
- Incremental z-score: Identify outliers based on the z-score of increments
Expand All @@ -127,6 +130,7 @@ Single outlier tests create a flag where `0=OK` and `2=outlier`.

### Plotting

- Diel cycle per month ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Plotting/DielCycle.ipynb))
- Heatmap showing values (z) of time series as date (y) vs time (
x) ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Plotting/HeatmapDateTime.ipynb))
- Heatmap showing values (z) of time series as year (y) vs month (
Expand All @@ -143,6 +147,9 @@ Single outlier tests create a flag where `0=OK` and `2=outlier`.
- Stepwise MeteoScreening from
database ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/MeteoScreening/StepwiseMeteoScreeningFromDatabase.ipynb))

### Resampling
- Calculate diel cycle per month ([notebook example](https://github.com/holukas/diive/blob/main/notebooks/Resampling/ResamplingDielCycle.ipynb))

### Stats

- Time series
Expand Down
207 changes: 207 additions & 0 deletions diive/core/plotting/dielcycle.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
import calendar

import matplotlib.pyplot as plt
from pandas import Series, DataFrame

import diive.core.plotting.styles.LightTheme as theme
from diive.core.plotting.plotfuncs import set_fig
from diive.core.times.resampling import diel_cycle


class DielCycle:

def __init__(self, series: Series):
"""Plot diel cycles of time series.
Args:
series: Time series with datetime index.
The index must contain date and time info.
"""
self.series = series

self.var = self.series.name

self.fig = None
self.ax = None
self.showplot = False
self.title = None
self.txt_ylabel_units = None
self._diel_cycles_df = None

def get_data(self) -> DataFrame:
return self.diel_cycles_df

@property
def diel_cycles_df(self):
"""Return dataframe containing diel cycle aggregates."""
if not isinstance(self._diel_cycles_df, DataFrame):
raise Exception(f'No diel cycles dataframe available. Please run .plot() first.')
return self._diel_cycles_df

def plot(self,
ax: plt.Axes = None,
title: str = None,
color: str = None,
txt_ylabel_units: str = None,
mean: bool = True,
std: bool = True,
each_month: bool = False,
legend_n_col: int = 1,
**kwargs):

self.title = title
self.txt_ylabel_units = txt_ylabel_units

# Resample
self._diel_cycles_df = diel_cycle(series=self.series,
mean=mean,
std=std,
each_month=each_month)

months = set(self.diel_cycles_df.index.get_level_values(0).tolist())

self.fig, self.ax, self.showplot = set_fig(ax=ax)

counter_plotted = -1
n_months = len(months)
alpha = 0.05 if n_months > 10 else 0.1
auto_color = True if not color else False
for counter, month in enumerate(months):

means = self.diel_cycles_df.loc[month]['mean']
if means.isnull().all():
continue
else:
counter_plotted += 1

means_add_sd = self.diel_cycles_df.loc[month]['mean+sd']
means_sub_sd = self.diel_cycles_df.loc[month]['mean-sd']

if auto_color:
color = theme.colors_12_months()[counter_plotted]

# monthstr = calendar.month_name[month] if each_month else 'Mittelwert'
monthstr = calendar.month_abbr[month] if each_month else 'mean'
means.plot(ax=self.ax, label=f'{monthstr}', color=color, zorder=99, lw=2, **kwargs)

# label = "Mittelwert ± 1 Standardabweichung"
label = None
# label = "mean±1sd" if counter == 0 else ""
self.ax.fill_between(means.index.values,
means_add_sd.values,
means_sub_sd.values,
alpha=alpha, zorder=0, color=color, edgecolor='none',
label=label)

self._format(title=title, legend_n_col=legend_n_col)

if self.showplot:
self.fig.show()

def _format(self, title, legend_n_col):
# title = self.title if self.title else f"{self.series.name} in {self.series.index.freqstr} time resolution"
from diive.core.plotting.plotfuncs import default_format, format_spines, default_legend, add_zeroline_y
if title:
self.ax.set_title(title, color='black', fontsize=24)
# ax_xlabel_txt = "Uhrzeit"
ax_xlabel_txt = "Time (hours of day)"
default_format(ax=self.ax, ax_xlabel_txt=ax_xlabel_txt, ax_ylabel_txt=self.series.name,
txt_ylabel_units=self.txt_ylabel_units,
ticks_direction='in', ticks_length=8, ticks_width=2,
ax_labels_fontsize=20,
ticks_labels_fontsize=20)
format_spines(ax=self.ax, color='black', lw=1)
default_legend(ax=self.ax, ncol=legend_n_col)
add_zeroline_y(ax=self.ax, data=self.diel_cycles_df['mean'])
self.ax.set_xticks(['3:00', '6:00', '9:00', '12:00', '15:00', '18:00', '21:00'])
# self.ax.set_xticklabels(['3:00', '6:00', '9:00', '12:00', '15:00', '18:00', '21:00'])
self.ax.set_xticklabels([3, 6, 9, 12, 15, 18, 21])
self.ax.set_xlim(['0:00', '23:59:59'])
if self.showplot:
self.fig.tight_layout()


def example():
from diive.configs.exampledata import load_exampledata_parquet
df = load_exampledata_parquet()
series = df['NEE_CUT_REF_f'].copy()
dc = DielCycle(series=series)
title = r'$\mathrm{Mean\ CO_2\ flux\ (2013-2024)}$'
units = r'($\mathrm{µmol\ CO_2\ m^{-2}\ s^{-1}}$)'
dc.plot(ax=None, title=title, txt_ylabel_units=units,
each_month=True, legend_n_col=2)

# from diive.core.io.filereader import search_files, MultiDataFileReader
# filepaths = search_files(searchdirs=fr"F:\CURRENT\HON\2-FLUXRUN\out", pattern='*.csv')
# orig = MultiDataFileReader(filepaths=filepaths, filetype='EDDYPRO-FLUXNET-CSV-30MIN', output_middle_timestamp=True)
# origdf = orig.data_df
# origmeta = orig.metadata_df
# origfreq = origdf.index.freq # Original frequency
# from diive.core.io.files import save_parquet
# save_parquet(filename="merged", data=origdf, outpath=r"F:\CURRENT\HON\2-FLUXRUN\out")

# import numpy as np
# from diive.core.io.files import load_parquet
# df = load_parquet(filepath=r"L:\Sync\luhk_work\CURRENT\HON\2-FLUXRUN\out\merged.parquet")
#
# # CO2
# co2 = df['FC'].copy()
# keep = (df['USTAR'] > 0.07) & (df['FC_SSITC_TEST'] <= 1)
# co2[~keep] = np.nan
# keep = (co2 < 50) & (co2 > -50)
# co2[~keep] = np.nan
# from diive.pkgs.outlierdetection.zscore import zScoreDaytimeNighttime
# zs = zScoreDaytimeNighttime(series=co2,
# lat=47.418861,
# lon=8.491361,
# utc_offset=1,
# thres_zscore=3,
# showplot=False)
# zs.calc()
# flag = zs.get_flag()
# remove = flag == 2
# co2[remove] = np.nan
# keep = (co2.index.month >= 3) & (co2.index.month <= 5)
# co2[~keep] = np.nan
# title = r'$\mathrm{CO_2\ flux}$ (Mar-May 2024)'
# units = r'($\mathrm{µmol\ CO_2\ m^{-2}\ s^{-1}}$)'
#
# h2o = df['FH2O'].copy()
# keep = df['FH2O_SSITC_TEST'] <= 1
# h2o[~keep] = np.nan
# keep = (h2o < 50) & (h2o > -50)
# h2o[~keep] = np.nan
#
# from diive.pkgs.outlierdetection.zscore import zScoreDaytimeNighttime
# zs = zScoreDaytimeNighttime(series=h2o,
# lat=47.418861,
# lon=8.491361,
# utc_offset=1,
# thres_zscore=3,
# showplot=False)
# zs.calc()
# flag = zs.get_flag()
# remove = flag == 2
# h2o[remove] = np.nan
# keep = (h2o.index.month >= 3) & (h2o.index.month <= 5)
# h2o[~keep] = np.nan
#
# import matplotlib.gridspec as gridspec
# fig = plt.figure(facecolor='white', figsize=(14, 14), dpi=200)
# gs = gridspec.GridSpec(2, 1) # rows, cols
# gs.update(wspace=0.3, hspace=0.4, left=0.08, right=0.93, top=0.93, bottom=0.07)
# ax = fig.add_subplot(gs[0, 0])
# ax2 = fig.add_subplot(gs[1, 0])
# dc_co2 = DielCycle(series=co2)
# title = r'$\mathrm{CO_{2}\text{-}Austausch\ (März-Mai\ 2024)}$'
# units = r'($\mathrm{µmol\ CO_2\ m^{-2}\ s^{-1}}$)'
# dc_co2.plot(ax=ax, color='#388E3C', title=title, txt_ylabel_units=units)
# dc_h2o = DielCycle(series=h2o)
# title = r'$\mathrm{Wasserdampf\text{-}Austausch\ (März-Mai\ 2024)}$'
# units = r'($\mathrm{mmol\ H_2O\ m^{-2}\ s^{-1}}$)'
# dc_h2o.plot(ax=ax2, color='#1976D2', title=title, txt_ylabel_units=units)
# fig.show()


if __name__ == '__main__':
example()
14 changes: 14 additions & 0 deletions diive/core/plotting/plotfuncs.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@
from diive.core.times.times import current_datetime


# TODO generalize for other classes
def set_fig(ax: plt.Axes):
if ax:
# If ax is given, plot directly to ax, no fig needed
fig = None
# self.ax = self.ax
showplot = False
else:
# If no ax is given, create fig and ax and then show the plot
fig, ax = create_ax()
showplot = True
return fig, ax, showplot


def make_patch_spines_invisible(ax):
ax.set_frame_on(True)
ax.patch.set_visible(False)
Expand Down
17 changes: 17 additions & 0 deletions diive/core/plotting/styles/LightTheme.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,23 @@ def colors_12(shade):
return list_12colors


def colors_12_months():
"""Create list of 12 colors."""
list_12colors = [lightblue(400),
blue(900),
teal(500),
green(600),
yellow(800), # May
deeporange(500), # June
pink(600), # July
red(900), # August
deeppurple(400), # September
bluegray(400),
lime(600),
cyan(700)]
return list_12colors


def colors_24():
"""Create list of 24 colors that should be more or less distinguishable."""
list_24colors = [amber(300), deeppurple(400), indigo(300), blue(400), lightblue(300), cyan(300),
Expand Down
Loading

0 comments on commit 53d72fc

Please sign in to comment.