Input and output data files in the model

[datejada]

I can foresee three data files in the model that need to be considered as inputs and outputs for the model; see Figure below.

1. ESDL files (.esdl): These files will contain the available information that can be stored in this format. Generally speaking, it is capacities, efficiencies, investment costs, operational costs, and commission and decommissions dates.
2. Complementary information files (.csv?): Here we have two categories

• Extra information for assets: Information not available in the ESDL format, e.g., unit-commitment parameters.
• Configuration parameters: setting parameters such as the years to run, type of simulation (only dispatch or with investment), etc.

3. Profiles database (.sqllite?): This file/DB will contain a large amount of input data (e.g., availability profiles) and results (e.g., production profiles for each asset).

Maybe we need more types of files; I'm not sure. In addition, the pre-processing and post-processing can be either in Julia or Python; these steps should be able to link the information in all the data sources with the optimization model.

I hope these comments and general overview help in the discussion for next week's meeting to talk about the design of the model.

[Lokkij]

Nice overview! I have a few initial comments on the technologies/formats mentioned.

Complementary information files: splitting these between extra energy system information and simulation configuration is a good idea. As far as formats go, CSV or another table-based format can make sense for the energy system info, but the configuration I would keep in a configuration format such as TOML.

In my experience with EYE, extra information about the energy system outside of ESDL is always much more of a hassle to use and maintain - so I would recommend really keeping as much in ESDL as possible (though I realise this is not always possible).

Profiles database: at ACE, we use InfluxDB or PostgreSQL for most of our databases. SQLite is nice and simple, but it's not great when you want things like multiple users, concurrent accesses and connections to other tools such as Grafana or Apache Superset. In my experience, it's not much more complicated to maintain either when using the official Docker images.

I realise these subjects are very much something to be discussed still, but I figured I'd throw in my two cents here - do with them what you will! :)

[suvayu]

Complementary information files: ...

I would second the CSV + TOML recommendation

[suvayu]

@datejada, we should think about typical execution environments. What you have described is a typical data processing + computation pipeline. Depending on resource requirements, access restrictions, usability considerations, the preferred solution will differ.

[abelsiqueira]

Following our discussion at TNO 30/Aug/2023, we came up with things:

• Let's try to read the Graph data from ESDL and convert into a graph structure (https://github.com/JuliaGraphs/GraphIO.jl)
• Let's try to use PostgreSQL to read the time series.

This involves accessing ESDL information in Julia, and we came up we 4 different approaches, ordered by estimated time to implement:

1. Use PyCall.jl/PythonCall.jl to call the ESDL Python functions;
2. Create a separate Python script/package that reads the ESDL file and writes relevant files (e.g., GML format). In Julia, we read the files.
3. Use a C++ interface for Ecore to read the ESDL data directly.
4. Write a ESDL interface in Julia (Probably a Ecore reader and a ESDL reader using that).

Option 1 might be harder to maintain, given the usual issues handling Python executables and environments.
Option 2 is a good compromise with separation of concerns.
Option 3 might be too hard, more investigation is needed.
Option 4 would probably involve too many hours, and there is no budget for maintaining the created packages. If other groups adopt the ESDL format using JuMP/Julia, then it might be possible through team effort.

[abelsiqueira]

Did anyone manage to run the notebook (https://github.com/EnergyTransition/ESDL-PyEcore-Tutorial/blob/master/edrreader.py)?
Step 1.4 fails for me, apparently because the requested item from EDR is not available. Instead of whatever I should receive, I just receive a page.
Maybe I need to authenticate? If that is the case, where do I get access?

Reproduction:

import edrreader
import requests

url = edrreader.EDR_config['EDR_host'] + edrreader.EDR_config['EDR_path'] + '55ef73b3-fa4f-49e8-8ad6-2b1801c25dfe' + '?format=xml'

headers = {
        'Content-Type': "application/json",
        'Accept': "application/xml",
        'User-Agent': "ESDL Tutorials"
    }

r = requests.get(url, headers=headers)
r.text

[clizbe]

Here's the ESDL file for the Norse case study to use. (Should be .esdl but GitHub gets mad.)
Norse Mythology.txt

[clizbe]

This can be used for #62.

[datejada]

@Lokkij, I've included below the screenshot of the whole representation and the simplified versions of the Norse Case Study. For the whole representation, we have the ESDL file you developed before. We would like to have an ESDL file for the simplified version, keeping in mind that all the assets in the whole representation can have a profile associated with them. We also would like to have the JSON files for each ESDL representation.

Thanks!

@suvayu @sjvrijn @lsoucasse Any additional comment?

[lsoucasse]

Actually I already created the ESDL file for this simplified network a couple of weeks ago but without data in it. @Lokkij, I will send you the file by mail, I cannot do it in the chat.

[clizbe]

I think you can post it if you convert it to .txt - which shouldn't hurt anything.

[Lokkij]

Thanks Laurent! Let's use this file for now, as this is close to the representation we want in Julia so it should be easy to work with.

[clizbe]

@Lokkij @datejada @lsoucasse
Should we come up with a list of graph parameters that aren't in ESDL (such as unit commitment), so we can discuss with ACE whether they can be added?

[suvayu]

What do you mean by "graph parameters"? Is something missing? In #359 we can already get the edges to build the graph.

[clizbe]

Sorry I never replied. I meant parameters that are stored in the ESDL file. I now have a list of what we want so far and how it maps to ESDL. #8

[clizbe]

I think this discussion can be closed. I'll link it on the User Workflow discussion for future reference.