Merge pull request #26 from scioip34/feature/data-load

Feature/data load

README.md

@@ -4,10 +4,10 @@ Agent based model framework to simulate collective foraging with visual private
 ## Introduction
 This repository hold the code base for the agent based model framework implemented in python/pygame to model and simualate agents collectively foraging in the environment.
 
-## Requirements
+### Requirements
 To run the simulations you will need python 3.8 or 3.9 and pip correspondingly. It is worth to set up a virtualenvironment using pipenv or venv for the project so that your global workspace is not polluted.
 
-## Test
+### Test
 To test the code:
   1. Clone the repo
   2. Activate your virtual environment (pipenv, venv) if you are using one
@@ -16,7 +16,9 @@ To test the code:
 
 ## Install Grafana and InfluxDB
 To monitor individual agents real time and save simulation data (i.e. write simulation data real time and save upon request at the end) we use InfluxDB and a grafana server for visualization. For this purpose you will need to install influx and grafana. If you don't do these steps you are still going to be able to run simulations, but you won't be able to save the resulting data or visualize the agent's parameters. This installation guide is only tested on Ubuntu. If you decide to use another op.system or you don't want to monitor and save simulation data, set `USE_IFDB_LOGGING` and `SAVE_CSV_FILES` parameters in the `.env` file to `0`.
-
+<details>
+  <summary>Click to expand for Grafana and InfluxDB installation details!</summary>
+
 ### Install Grafana
 1. run the following commands to add the grafana APT repository and install grafana
 ```bash
@@ -89,3 +91,108 @@ influx --execute "show users"
 ### Import Dashboard from repo
 1. Open your grafana app from the browser and on the left menu bar click on the "+" button and the on the "Import button"
 2. Upload the json file (that holds the blueprint of the grafana dashboard) from the repo under the path `abm/data/grafana_dashboard.json`
+
+</details>
+
+## Details of the package
+In this section the package is detailed for reproducibility and for ease of use. Among others you can read about the main restrictions and assumptions we used in our framework, how one can initialize the package with different parameters through `.env` files, and how the code is structured.
+
+### Code Elements
+The code is structured into a single installable python package called `abm`. Submodules of this package contain the main classes and methods that are used to implement the functionalities of our framework, such as `Agent`, `Resource` and `Simulation` classes among others. A dedicated `contrib` submodule provides parameters for running the simulations in python syntax. These parameters are either initialized from a `.env` file (described later) or they are not to be changed throughout simulations (such as passwords and database details) and therefore fixed in these scripts. Note that although we store passwords as text these are absolutely insensitive as they are only needed locally on a simulation computer to access the database in which we store simulation data (that is by nature not sensitive data).
+
+#### Submodules
+The package includes the following submodules:
+
+* `agent`: including the `Agent` class implementing a simple interactive agent that is able to move in the environment, update it's appearence in pygame, use visual social cues, find resource patches and exploit them. All necessary method that implements these behaviors are packed in the class and called in the `update` method that is used to update the agents status (position, orientation, exploited resources, decision parameters) in each timestep by pygame. This class inherits from the `pygame.Sprite` class and therefore can be used accordingly. A helper script of the submodule `supcalc.py` inlcudes some independent functions to calculate distances, norms, angles, etc.
+* `contrib`: including helper parameters of the package that can be later imported as `abm.contrib.<name_of_param_bundle>`. For further information about what the individual parameter bundles include within this submodule please read the comment in the beginning of these scripts.
+* `environment`: including classes and methods for environmental elements. As an example it includes the `Resource` class that implements a resource patch in the environment. Similar to the `Agent` class it inherits from pygame sprites and therefore the `update` method will call all relevant methods in each timestep.
+* `loader`: including all classes and methods to dynamically load data that was generated with the package. These methods are for example cvs and json readers and initializers that initialize input classes for Replay and DataAnalysis tools.
+* `metarunner`: including all classes and methods to run multiple simulations one after the other with programatically changed initialization parameters. The main classes are `Tunables` that define a criterion range with requested number of datapoints (e.g.: simulate with environment width from 300 to 600 via 4 datapoints). The `Constant` class that defines a fixed criterion throughout the simulations (e.g.: keep the simulation time `T` fixed at 1000). And `MetaProtocol` class that defines a batch of simulations with all combinations of the defined criteria according to the added `Tunable`s and `Constant`s. During running metaprotocols the corresponding initializations (as `.env` files) will be saved under the `data/metaprotocol/temp` folder. Only those `.env` files will be removed from here for which the simulations have been carried out, therefore the metaprotocol can be interrupted and finished later.
+* `monitoring`: including all methods to interface with InfluxDB, Grafana and to save the stored data from the database at the end of the simulation. The data will be saved into the `data/simualtion_data/<timestamp_of_simulation>` of the root abm folder. The data will consist of 2 relevant csv files (agent_data, resource_data) containing time series of the agent and resource patch status and a json file containing all parameters of the simulation for reproducibility.
+* `simulation`: including the main `Simulation` class that defines how the environment is visualized, what interactions the user can have with the pygame environment (e.g.: via cursor or buttons), and how the environment enforces some restrictions on agents, and how resources are regenerated. 
+
+### Functionality and Behavior
+Here you can read about how the framework works in large scale behavior and what restrictions and assumptions we used throughout the simulation.
+
+#### Behavior
+Upon starting the simulation an arena will pop up with given number of agents and resource patches. Details of these are controlled via `.env` variables and you can read more below. Agents will search for hidden resource patches and exploit/consume these with a given rate (resource unit/time) when found. The remaining resource units are shown on the patches as well as their quality (how much unit can be exploited in a time unit per agent). 
+
+Agents can behave according to 3 distinct behavioral states. These are: Exploration (individual uninformed state looking for resources with random movement and integration of individual and social cues in the meanwhile). Relocation (informed state in which the agent "decides" to join to another agent's patch). Exploitation (in which the agent consumes a resource patch and is recognized as a social visual cue for other agents). The mode of the agents are depicted with their colors, that is blue, purple and green respectively. Agents can collide with each other (red color) and in this case they avoid the collision by turning away from the other agents. Collision can be turned off during exploitation (ghost mode). Recognizing exploiting agents as social cues on the same patch can be turned off. 
+
+Each social cue (other exploiting agent) creates a visual projection on the focal agent's retina if in visual range (and the limited FOV allows). Relocation happens according to the overall excitation of the agent's retina. The focal agent steers right if the right hemifield is more excited and left if the left hemifield is more excited.
+
+During exploitation agents slow down and stop on patches.
+
+Agents decide on which mode to enter via a dedicated decision process. The decision process continously integrates private information (Did I find a new patch? How good the quality of the new patch is?) and social information (Do I see any other agents axploiting nearby? How many/how close according to visual projection field?). With the parameters of the decision process one con control how socially susceptible agents are and how much being in e.g. relocation inhibits exploitation and vica versa. Agents integrate infromation all the time and they can deliberately stop being in a behavioral mode to switch into another.
+
+#### Interaction
+During the simulation visualization can be turned off to speed up the run. In case it is turned on, the user is able to interact with the simulation as follows:
+
+* click (left) and move agents in space
+* rotate agents with mouse scroll
+* pause/unpause simulation with `space`
+* show social visual field with `return`
+* increase/decrease framrate with `f`/`s`
+* reset default framerate with `d`
+
+##### Environment variables as parameters
+To parametrize the simulation we use `.env` files. These include the main parameters line by line. This means, that a single `.env` file defines a simulation run fully. The env variables are as follows:
+
+<details>
+  <summary>Click to see all env variables!</summary>
+
+* `N`: number of agents
+* `N_RESOURCES`: number of resource patches
+* `T`: number of simulation timesteps
+* `INIT_FRAMERATE`: default framerate when visualization is on. Irrelevant for when visualization is turned off
+* `WITH_VISUALIZATION`: turns visualization on or off
+* `VISUAL_FIELD_RESOLUTION`: Resolution/size of agents' visual projection fields in pixels
+* `ENV_WIDTH`: width of the environment in pixels
+* `ENV_HEIGHT`: height of the environment in pixels
+* `RADIUS_AGENT`: radius of agents in pixels
+* `RADIUS_RESOURCE`: radius or resource patches in pixels
+* `MIN_RESOURCE_PER_PATCH`: minimum contained resource units of a resourca patch. real value will be random uniform between min and max values.
+* `MAX_RESOURCE_PER_PATCH`: maximum contained resource units of a resourca patch.
+* `REGENERATE_PATCHES`: turns on or off resource patch regeneration upon full depletion.
+* `AGENT_CONSUMPTION`: maximum resource consumption of agents (per time unit). Can be lower according to resource patch quality
+* `MIN_RESOURCE_QUALITY`: minimum quality of resourca patch. real quality will be random uniform between min and max quality.
+* `MAX_RESOURCE_QUALITY`: maximum quality of resource patches.
+* `TELEPORT_TO_MIDDLE`: pulling exploiting agents into the middle of the resource patch if turned on.
+* `GHOST_WHILE_EXPLOIT`: disabling collisions when the agents exploit when turned on.
+* `PATCHWISE_SOCIAL_EXCLUSION`: not taking into consideration agents on the same patch as social cues if turned on.
+* `AGENT_FOV`: Field of view of the agents. FOV is symmetric and defined with percent of pi. e.g if 0.6 then fov is (-0.6*pi, 0.6*pi). 1 is full 360 degree vision
+* `VISION_RANGE`: visual range in pixels
+* `VISUAL_EXCLUSION`: taking visual exclusion into account when calculating visual cues if turned on.
+* `SHOW_VISUAL_FIELDS`: always show visual fields of agents when turned on.
+* `SHOW_VISUAL_FIELDS_RETURN`: show visual fields of agents when return pressed if turned on
+* `SHOW_VISION_RANGE`: visualizing visual range and field of view of agents when turned on.
+* `USE_IFDB_LOGGING`: logs simulation data into a connected InfluxDB database when turned on (and InfluxDB is initialized)
+* `SAVE_CSV_FILES`: saves data from connected InfluxDB instance as csv files if turned on.
+
+ Parameters of the decision process as decsribed in rpopsal:
+* `DEC_TW`: time constant of w process
+* `DEC_EPSW`: social excitability
+* `DEC_GW`: social decay
+* `DEC_BW`: social process baseline
+* `DEC_WMAX`: social process limit
+* `DEC_TU`: time constant of u process
+* `DEC_EPSU`: individual excitability
+* `DEC_GU`: individual decay
+* `DEC_BU`: individual process baseline
+* `DEC_UMAX`: individual process limit
+* `DEC_SWU`: social to individual inhibition
+* `DEC_SUW`: individual to social inhibition
+* `DEC_TAU`: novelty time window of private information
+* `DEC_FN`: novelty multiplier
+* `DEC_FR`: quality multiplier
+
+Movement parameters:
+* `MOV_EXP_VEL_MIN`: minimum exploration velocity
+* `MOV_EXP_VEL_MAX`: maximum exploration velocity
+* `MOV_EXP_TH_MIN`: minimum exploration orientation change (per time unit)
+* `MOV_EXP_TH_MAX`: maximum exploration orientation change (per time unit)
+* `MOV_REL_DES_VEL`: relocation velocity
+* `MOV_REL_TH_MAX`: relocation maximal orientation change
+* `CONS_STOP_RATIO`: deceleration during exploitation
+
+</details>

abm/app.py

@@ -2,38 +2,38 @@
 
 # loading env variables from dotenv file
 from dotenv import dotenv_values
-
-envconf = dotenv_values(".env")
-
+import os
 
 def start():
-    sim = Simulation(N=int(envconf["N"]),
-                     T=int(envconf["T"]),
+    root_abm_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+    envconf = dotenv_values(os.path.join(root_abm_dir, ".env"))
+    sim = Simulation(N=int(float(envconf["N"])),
+                     T=int(float(envconf["T"])),
                      v_field_res=int(envconf["VISUAL_FIELD_RESOLUTION"]),
                      agent_fov=float(envconf['AGENT_FOV']),
-                     framerate=int(envconf["INIT_FRAMERATE"]),
-                     with_visualization=bool(int(envconf["WITH_VISUALIZATION"])),
-                     width=int(envconf["ENV_WIDTH"]),
-                     height=int(envconf["ENV_HEIGHT"]),
-                     show_vis_field=bool(int(envconf["SHOW_VISUAL_FIELDS"])),
-                     pooling_time=int(envconf["POOLING_TIME"]),
+                     framerate=int(float(envconf["INIT_FRAMERATE"])),
+                     with_visualization=bool(int(float(envconf["WITH_VISUALIZATION"]))),
+                     width=int(float(envconf["ENV_WIDTH"])),
+                     height=int(float(envconf["ENV_HEIGHT"])),
+                     show_vis_field=bool(int(float(envconf["SHOW_VISUAL_FIELDS"]))),
+                     pooling_time=int(float(envconf["POOLING_TIME"])),
                      pooling_prob=float(envconf["POOLING_PROBABILITY"]),
-                     agent_radius=int(envconf["RADIUS_AGENT"]),
-                     N_resc=int(envconf["N_RESOURCES"]),
-                     min_resc_perpatch=int(envconf["MIN_RESOURCE_PER_PATCH"]),
-                     max_resc_perpatch=int(envconf["MAX_RESOURCE_PER_PATCH"]),
+                     agent_radius=int(float(envconf["RADIUS_AGENT"])),
+                     N_resc=int(float(envconf["N_RESOURCES"])),
+                     min_resc_perpatch=int(float(envconf["MIN_RESOURCE_PER_PATCH"])),
+                     max_resc_perpatch=int(float(envconf["MAX_RESOURCE_PER_PATCH"])),
                      min_resc_quality=float(envconf["MIN_RESOURCE_QUALITY"]),
                      max_resc_quality=float(envconf["MAX_RESOURCE_QUALITY"]),
-                     patch_radius=int(envconf["RADIUS_RESOURCE"]),
-                     regenerate_patches=bool(int(envconf["REGENERATE_PATCHES"])),
-                     agent_consumption=int(envconf["AGENT_CONSUMPTION"]),
-                     ghost_mode=bool(int(envconf["GHOST_WHILE_EXPLOIT"])),
-                     patchwise_exclusion=bool(int(envconf["PATCHWISE_SOCIAL_EXCLUSION"])),
-                     teleport_exploit=bool(int(envconf["TELEPORT_TO_MIDDLE"])),
-                     vision_range=int(envconf["VISION_RANGE"]),
-                     visual_exclusion=bool(int(envconf["VISUAL_EXCLUSION"])),
-                     show_vision_range=bool(int(envconf["SHOW_VISION_RANGE"])),
-                     use_ifdb_logging=bool(int(envconf["USE_IFDB_LOGGING"])),
-                     save_csv_files=bool(int(envconf["SAVE_CSV_FILES"]))
+                     patch_radius=int(float(envconf["RADIUS_RESOURCE"])),
+                     regenerate_patches=bool(int(float(envconf["REGENERATE_PATCHES"]))),
+                     agent_consumption=int(float(envconf["AGENT_CONSUMPTION"])),
+                     ghost_mode=bool(int(float(envconf["GHOST_WHILE_EXPLOIT"]))),
+                     patchwise_exclusion=bool(int(float(envconf["PATCHWISE_SOCIAL_EXCLUSION"]))),
+                     teleport_exploit=bool(int(float(envconf["TELEPORT_TO_MIDDLE"]))),
+                     vision_range=int(float(envconf["VISION_RANGE"])),
+                     visual_exclusion=bool(int(float(envconf["VISUAL_EXCLUSION"]))),
+                     show_vision_range=bool(int(float(envconf["SHOW_VISION_RANGE"]))),
+                     use_ifdb_logging=bool(int(float(envconf["USE_IFDB_LOGGING"]))),
+                     save_csv_files=bool(int(float(envconf["SAVE_CSV_FILES"])))
                      )
     sim.start()

abm/contrib/decision_params.py

@@ -32,6 +32,6 @@
 S_uw = float(envconf.get("DEC_SUW", 0.01))  # strength from u to w
 
 ##### Calculating Private Information #####
-Tau = int(envconf.get("DEC_TAU", 10))
+Tau = int(float(envconf.get("DEC_TAU", 10)))
 F_N = float(envconf.get("DEC_FN", 2))  # novelty multiplier
 F_R = float(envconf.get("DEC_FR", 1))  # quality multiplier

abm/load_data_test.py

@@ -0,0 +1,3 @@
+from abm.loader.data_loader import LoadedSimulation
+
+dl = LoadedSimulation("/home/david/Desktop/ABM/abm/data/simulation_data/2022-01-14_13-50-59")