Open basic version of EMPIRE in Pyomo.
The EMPIRE model and all additional files in the git repository are licensed under the MIT license. In short, that means you can use and change the code of EMPIRE. Furthermore, you can change the license in your redistribution but must mention the original author. We appreciate if you inform us about changes and send a merge request via git. For further information please read the LICENSE file, which contains the license text, or go to https://opensource.org/licenses/MIT
The EMPIRE model is available in the Python-based, open-source optimization modelling language Pyomo. Running the model thus requires some coding skills in Python. To run the model, make sure Python, Pyomo and a third-party solver (gurobi, FICO xpress, or CPLEX) is installed and loaded to the respective computer or cluster. More information on how to install Python and Pyomo can be found here: http://www.pyomo.org/installation. Other python package dependencies can be found in the file environment.yml.
To download, you need to install Git and clone the repository. Note that the repository makes use of Git Large File Storage (LFS) which also needs to be installed for input data-files to be downloaded when cloning the repository. Once both Git and Git LFS has been successfully installed, the model is downloaded to the desired directory.
EMPIRE consists of a programming script used to run the model:
scripts/run.py: The main script used to run EMPIRE. This is the only script a user of EMPIRE needs to use and potentially modify.
Note: The main run-script (scipts/run.py), can run a small test instance of EMPIRE that usually finishes in 1-2 min. A normal test instance requires around 140 GB RAM and thus needs to be run on a high performance cluster (HPC).
The run script uses the empire package that consists of these core modules:
(1) empire.py: Contains the abstract formulation of EMPIRE in Pyomo. This script also contains code related to printing the results.
(2) scenario_random.py: Generates random operational scenarios as .tab-files through sampling.
(3) reader.py: Generates .tab-files input based on data provided in Excel workbooks.
(4) config.py: Defines two configuration objects used by Empire.
(5) model_runner.py: Methods used for setting up an Empire run.
In addition there are modules containing input and output clients, that can be used to read and alter input data, and read ouput/results data.
In the repository, the ‘Data handler’-folder contains the Excel workbooks that are used to store and modify input data. The workbooks are contained within folders representing instance-versions of EMPIRE, e.g. ‘europe_v50’. The ‘test’-folder contains input data for a small test-instance of EMPIRE. Within an instance-version in the ‘Data handler’-folder, there is a folder called ‘ScenarioData’ containing large data sets used to generate stochastic scenarios in EMPIRE. If EMPIRE is run with random scenario generation, representative time series are sampled once per scenario and season for each random input parameter.
The EMPIRE Model reads .tab-files, which provide all needed sets and input data. For editing and storing the data, excel-files are used. There are seven excel-files in total of which six contain indexed input data and one is to provide the indices/sets. The excel-files are sorted by the following categories: General data, generation data, country/node data, set/index data, transmission data, and storage data. These files contain multiple tables regarding for example investment costs and initial capacity.
For more details, please refer to the software documentation in the repository.
| Input name | Type | Default | Description |
|---|---|---|---|
| use_temporary_directory | True/False | False | If true, all instance-files related to solving EMPIRE is stored in the directory defined by temporary_directory (see below). This is useful when running a large instance of EMPIRE to avoid memory problems. |
| temporary_directory | String | './' | The path to which temporary files will be stored if use_temporary_directory = True; .lp-file is stored if write_in_lp_format = True; and .plk-file is stored if serialize_instance = True. |
| forecast_horizon_year | Integer | 2060 | The last strategic (investment) period used in the optimization run. |
| number_of_scenarios | Integer | 3 | The number of scenarios in every investment period. |
| length_of_regular_season | Integer | 168 | The number of hours to use in a regular season for optimization of system operation in every investment period. |
| discount_rate | Float | 0.05 | The discount rate. |
| wacc | Float | 0.05 | The weighted average cost of capital (WACC). |
| optimization_solver | String | "Xpress" | Specifies the solver. Options: “Xpress”, “Gurobi”, “CPLEX”. |
| use_scenario_generation | True/False | True | If true, new operational scenarios will be generated. NB! If false, .tab-files or sampling key must be manually added to the ‘ScenarioData’-folder in the version. |
| use_fixed_sample | True/False | False | If true, operational scenarios will be generated according to a fixed sampling key located in the ‘ScenarioData’-folder to ensure the same operational scenarios are generated. |
| load_change_module | True/False | False | |
| filter_make | True/False | False | |
| filter_use | True/False | False | |
| n_cluster | Integer | 10 | |
| moment_matching | True/False | ||
| n_tree_compare | Integer | 20 | |
| use_emission_cap | True/False | True | If true, emissions in every scenario are capped according to the specified cap in ‘General.xlsx’. If false, the CO2-price specified in ‘General.xlsx’ applies. |
| print_in_iamc_format | True/False | True | If true, selected results are printed on the standard IAMC-format in addition to the normal EMPIRE print. |
| write_in_lp_format | True/False | False | If true, the solver-file will be saved. Useful for debugging. |
| serialize_instance | True/False | False | If true, instance will be saved/pickled. Useful for printing alternative results. |
| north_sea | True/False | False | Whether the north sea is modelled or not. |
System Requirements: Running the base case of EMPIRE in Pyomo typically takes about 40 minutes and requires approximately 140 GB of RAM.
Initial Test: To ensure your setup is correctly configured, start with a test run using a smaller dataset. This test checks if your computer or cluster can successfully connect to the preferred solver.
Run the test using the following command:
C:\Users\name\path_to_folder> python scripts/run.py -d testNo-Optimization Test: To execute a test run without performing optimization, use:
C:\Users\name\path_to_folder> python scripts/run.py --test-run -d testThis can be useful for verifying basic setup and data handling without engaging the full optimization process.
Setup Requirements: Ensure that Pyomo and the preferred solver are installed.
Execution: Run the model using the run.py script. Specify the dataset located in the Data Handler folder. For example, to run the model with the europe_v51 dataset, use:
C:\Users\name\path_to_folder> python scripts/run.py -d europe_v51Example Script: For running multiple cases on an HPC, refer to the script scripts/copy_and_run_empire_on_hpc.sh. This script uses configurations from config/cluster.json and is designed for NTNU's HPC clusters: Solstorm and Idun.
Usage Linux or macOS: From the project directory, execute the following to run on the Solstorm cluster:
sh scripts/copy_and_run_empire_on_hpc.sh SolstormUsage Windows: Install "jq" on Windows:
curl -L -o /usr/bin/jq.exe https://github.com/stedolan/jq/releases/latest/download/jq-win64.exeFrom the project directory, execute the following command in Git Bash to run on the Solstorm cluster:
sh scripts/copy_and_run_empire_on_hpc.sh SolstormThis command copies the EMPIRE code to the Solstorm cluster and performs several runs managed by the SGE task manager. Ensure the "empire_env" conda environment is set up on the cluster with dependencies as listed in environment.yml.
The scripts/run_analysis.py script demonstrates how to modify input data at execution time using data managers.
We welcome any contribution the OpenEMPIRE, whether it is fixing a bug, adding a new feature, or improving documentation, your help is appreciated. For more information, see CONTRIBUTING.