Skip to content

Commit

Permalink
Merge pull request #10 from mlgarrett/develop
Browse files Browse the repository at this point in the history 
Merge final changes
  • Loading branch information
@mlgarrett
mlgarrett authored Apr 23, 2021
2 parents 6b91cf3 + 6ea67f6 commit d6afe95
Show file tree
Hide file tree
Showing 26 changed files with 252 additions and 6,560 deletions.
28 changes: 6 additions & 22 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# GLACIES INDICIUM

Welcome to the code repository for Team Glacies Indicium at NAU.

You can visit our [Team Website](https://ceias.nau.edu/capstone/projects/CS/2021/GlaciesIndicium-F20/) to learn more about our project.

# HOW TO GET SET UP

1. Install Anaconda and the git tool on your machine.

- On Windows: Use the Start Menu to find and open the Anaconda/Miniconda command prompt/terminal.
Expand All @@ -21,6 +23,7 @@ You can visit our [Team Website](https://ceias.nau.edu/capstone/projects/CS/2021
6. Activate the virtual environment with `conda activate terracotta`. You should now be able to run terracotta commands, like `terracotta --help`.

# ATTRIBUTION

This project is based on the open-source project Terracotta, a Python-based GeoTIFF data exploration tool. The original Terracotta readme information can be found below.

---
Expand All @@ -44,7 +47,7 @@ and [Rasterio](https://github.com/mapbox/rasterio).
projections you want to have a look at in your browser?
`terracotta serve -r {name}.tif` and
`terracotta connect localhost:5000` get you there.
- We make minimal assumptions about your data, so *you stay in charge*.
- We make minimal assumptions about your data, so _you stay in charge_.
Keep using the tools you know and love to create and organize your
data, Terracotta serves it exactly as it is.
- Serverless deployment is a first-priority use case, so you don’t have
Expand Down Expand Up @@ -96,6 +99,7 @@ $ terracotta serve -d greenland.sqlite
#### Manually

You can use any HTTP-capable client, such as `curl`.

```bash
$ curl localhost:5000/datasets?tile=25XEL
{"page":0,"limit":100,"datasets":[{"date":"20170831","tile":"25XEL","band":"B02"},{"date":"20170831","tile":"25XEL","band":"B03"},{"date":"20170831","tile":"25XEL","band":"B04"}]}
Expand All @@ -106,6 +110,7 @@ Modern browsers (e.g. Chrome or Firefox) will render the JSON as a tree.
#### Interactively

Terracotta also includes a web client. You can start the client (assuming the server is running at http://localhost:5000) using

```bash
$ terracotta connect localhost:5000
* Serving Flask app "terracotta.client" (lazy loading)
Expand All @@ -119,24 +124,3 @@ $ terracotta connect localhost:5000
Then open the client page (http://127.0.0.1:5100/ in this case) in your browser.

![preview](docs/_figures/workflow-preview.png)

## Development

We gladly accept [bug reports](https://github.com/DHI-GRAS/terracotta/issues)
and [pull requests](https://github.com/DHI-GRAS/terracotta/pulls) via GitHub.
For your code to be useful, make sure that it is covered by tests and that
it satisfies our linting practices (via `mypy` and `flake8`).

To run the tests, just install the necessary dependencies via

```bash
$ pip install -e .[test]
```

Then, you can run

```bash
$ pytest
```

from the root of the repository.
14 changes: 10 additions & 4 deletions prep_scripts/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,16 @@ provided output folder.
For example, if your GeoTIFFs are in a folder in this directory called
`source_data/`, the command

`python3 raster-prep.py -d source_data/`
`python3 raster-prep.py -d source_data/ -r`

will output a set of processed, keyed GeoTIFFs into the `output/` folder within the Terracotta client.
will output a set of processed, keyed GeoTIFFs into the `mosaics/` folder within the Terracotta client and will remove the borders from the GeoTIFFs.

# State of the Script

The `raster-prep.py` script takes one argument, a directory containing
arbitrarily-many multiband GeoTIFFs.
- The `raster-prep.py` script takes two arguments, a directory containing
arbitrarily-many multiband GeoTIFFs and an optional argument to remove the borders from the GeoTIFF images.
- The `all_regions.json` file will automatically be filled with a valid JSON array containing every region in Antarctica in a hierarchical format.
- The `availible_regions.json` file will be dynamically created based on the regions found from the set of GeoTIFF images that is ingested in this script.

## Things the Script Currently Does

Expand All @@ -31,3 +33,7 @@ arbitrarily-many multiband GeoTIFFs.
- Reads the 4-coordinate boundaries of each file
- Extracts each band of each input file to its own separate file, with a
plain-English band key added to the file name
- Creates a hierarchical JSON file for every region in Antarctica
- Creates a hierarchical JSON file for each region in the set of ingested GeoTIFF images
- Removes the border from each GeoTIFF image
- Correlates each image's boundary coordinates to the regional boundary data
169 changes: 1 addition & 168 deletions prep_scripts/all_regions.json
View file
Original file line number Diff line number Diff line change
@@ -1,168 +1 @@
[
{
"name": "Northern Victoria Land",
"id": "a79b",
"subregions": [
{
"name": "McMurdo Dry Valleys",
"id": "b3e4",
"subregions": [
{
"name": "Asgard Range",
"id": "0419",
"subregions": [
{ "name": "Mt. Newall", "id": "c7ea" },
{ "name": "Round Mountain", "id": "7d31" }
]
},
{
"name": "Ferrar Glacier",
"id": "2756",
"subregions": [
{ "name": "Kukri Hills", "id": "7d71" },
{ "name": "Briggs Hills", "id": "70f3" },
{ "name": "Thomas Heights", "id": "94c0" }
]
},
{
"name": "Kukri Hills",
"id": "1559",
"subregions": [{ "name": "Mt. Coates", "id": "3a3b" }]
},
{
"name": "Quartermain Mountains",
"id": "5927",
"subregions": [
{ "name": "Beacon Valley", "id": "c453" },
{ "name": "Kennar Valley", "id": "21a6" },
{ "name": "Arena Valley", "id": "92fe" },
{ "name": "Pivot Peak", "id": "2544" }
]
},
{
"name": "Royal Society Range",
"id": "2440",
"subregions": [
{ "name": "Garwood Valley", "id": "61c9" },
{ "name": "Miers Valley", "id": "3de9" },
{ "name": "Heald Island", "id": "0c70" },
{ "name": "Blue Glacier", "id": "c834" },
{ "name": "Mt. Rucker", "id": "6c75" },
{ "name": "Trough Lake", "id": "bd48" },
{ "name": "Mt. Huggins", "id": "b078" },
{ "name": "The Spire", "id": "ae12" },
{ "name": "Table Mountain", "id": "1197" }
]
},
{
"name": "Taylor Valley",
"id": "5c17",
"subregions": [
{ "name": "Northwest Mountain", "id": "d05e" },
{ "name": "New Harbor", "id": "65e2" },
{ "name": "Fryxell Basin", "id": "1596" },
{ "name": "Kukri Hills", "id": "0065" },
{ "name": "Hoare Basin", "id": "8084" },
{ "name": "Mummy Pond", "id": "6c1c" },
{ "name": "Bonney Basin", "id": "d271" },
{ "name": "Pearse Valley", "id": "13d0" },
{ "name": "Simmons Lake", "id": "0da4" },
{ "name": "Friis Hills", "id": "747b" }
]
},
{
"name": "Wright Valley",
"id": "d289",
"subregions": [
{ "name": "The Labyrinth", "id": "5234" },
{ "name": "The Dais", "id": "c62e" },
{ "name": "South Fork", "id": "21c5" },
{ "name": "North Fork", "id": "a922" },
{ "name": "Koenig Valley", "id": "3dd5" },
{ "name": "Panorama Peak", "id": "2dff" },
{ "name": "Lake Vanda", "id": "2312" },
{ "name": "Meserve Glacier", "id": "0132" },
{ "name": "Lake Brownworth", "id": "2aaf" }
]
},
{ "name": "Bull Pass", "id": "e231", "subregions": [] },
{
"name": "Victoria Valley",
"id": "d92f",
"subregions": [
{ "name": "Lower Victoria", "id": "a5a7" },
{ "name": "Lake Vida", "id": "8634" },
{ "name": "Upper Victoria", "id": "1a36" }
]
},
{
"name": "Barwick Valley",
"id": "8818",
"subregions": [
{ "name": "Upper Barwick", "id": "4596" },
{ "name": "Lower Barwick", "id": "e09e" }
]
},
{ "name": "Balham Valley", "id": "70f6", "subregions": [] },
{
"name": "McKelvey Valley",
"id": "a379",
"subregions": [
{ "name": "Upper McKelvey", "id": "bbc1" },
{ "name": "Lower McKelvey", "id": "0c10" },
{ "name": "Halzen Mesa", "id": "954e" }
]
}
]
}
]
},
{
"name": "Central Transantarctic Mountains",
"id": "8b2f",
"subregions": [
{
"name": "Shackleton Glacier",
"id": "e7e1",
"subregions": [
{
"name": "Roberts Massif",
"id": "2eeb",
"subregions": [{ "name": "-Roberts Massif", "id": "d2b0" }]
},
{
"name": "Mount Black",
"id": "5b87",
"subregions": [
{ "name": "Bennett Platform", "id": "3d48" },
{ "name": "Kitching Ridge", "id": "b6a1" }
]
},
{ "name": "Vickers Nunatak", "id": "04c7", "subregions": [] },
{
"name": "Cumulus Hills",
"id": "138a",
"subregions": [
{ "name": "West Bluff", "id": "d0f3" },
{ "name": "Schroeder Hill", "id": "8389" },
{ "name": "Ellis Bluff", "id": "da3d" },
{ "name": "Collinson Ridge", "id": "38ad" },
{ "name": "Shenk Peak", "id": "a8a8" },
{ "name": "Cheu Valley", "id": "e816" },
{ "name": "Gatlin Glacier", "id": "0d8b" }
]
},
{ "name": "Mount Heekin", "id": "6bc1", "subregions": [] },
{ "name": "Pendant Ridge", "id": "e696", "subregions": [] },
{ "name": "Thanksgiving Point", "id": "5e50", "subregions": [] },
{ "name": "Mount Greenlee", "id": "7eb5", "subregions": [] },
{ "name": "Taylor Nunatak", "id": "c3d1", "subregions": [] },
{ "name": "Mount Cole", "id": "4138", "subregions": [] },
{ "name": "Mount Speed", "id": "6c5b", "subregions": [] },
{ "name": "Nilsen Peak", "id": "b827", "subregions": [] },
{ "name": "Mount Wendland", "id": "ad00", "subregions": [] }
]
}
]
}
]
[{"name": "Northern Victoria Land", "id": "1f7a", "subregions": [{"name": "McMurdo Dry Valleys", "id": "1438", "subregions": [{"name": "Asgard Range", "id": "cadc", "subregions": [{"name": "Mt. Newall", "id": "ca9c"}, {"name": "Round Mountain", "id": "83d1"}]}, {"name": "Ferrar Glacier", "id": "adfc", "subregions": [{"name": "Kukri Hills", "id": "3e74"}, {"name": "Briggs Hills", "id": "87b0"}, {"name": "Thomas Heights", "id": "2a47"}]}, {"name": "Kukri Hills", "id": "2866", "subregions": [{"name": "Mt. Coates", "id": "a19d"}]}, {"name": "Quartermain Mountains", "id": "aeeb", "subregions": [{"name": "Beacon Valley", "id": "d15d"}, {"name": "Kennar Valley", "id": "d712"}, {"name": "Arena Valley", "id": "c26a"}, {"name": "Pivot Peak", "id": "5387"}]}, {"name": "Royal Society Range", "id": "581d", "subregions": [{"name": "Garwood Valley", "id": "8b3a"}, {"name": "Miers Valley", "id": "da3b"}, {"name": "Heald Island", "id": "8b48"}, {"name": "Blue Glacier", "id": "e80d"}, {"name": "Mt. Rucker", "id": "7f7a"}, {"name": "Trough Lake", "id": "99f9"}, {"name": "Mt. Huggins", "id": "cefd"}, {"name": "The Spire", "id": "f5c1"}, {"name": "Table Mountain", "id": "cdee"}]}, {"name": "Taylor Valley", "id": "5d46", "subregions": [{"name": "Northwest Mountain", "id": "4b6a"}, {"name": "New Harbor", "id": "138d"}, {"name": "Fryxell Basin", "id": "abc2"}, {"name": "Kukri Hills", "id": "36a1"}, {"name": "Hoare Basin", "id": "900a"}, {"name": "Mummy Pond", "id": "3184"}, {"name": "Bonney Basin", "id": "b2e3"}, {"name": "Pearse Valley", "id": "10d3"}, {"name": "Simmons Lake", "id": "ded1"}, {"name": "Friis Hills", "id": "1aa2"}]}, {"name": "Wright Valley", "id": "3b73", "subregions": [{"name": "The Labyrinth", "id": "b7f9"}, {"name": "The Dais", "id": "561d"}, {"name": "South Fork", "id": "2009"}, {"name": "North Fork", "id": "cedd"}, {"name": "Koenig Valley", "id": "7ecc"}, {"name": "Panorama Peak", "id": "997e"}, {"name": "Lake Vanda", "id": "5fc0"}, {"name": "Meserve Glacier", "id": "1e99"}, {"name": "Lake Brownworth", "id": "83ea"}]}, {"name": "Bull Pass", "id": "4f62", "subregions": []}, {"name": "Victoria Valley", "id": "94a7", "subregions": [{"name": "Lower Victoria", "id": "95f6"}, {"name": "Lake Vida", "id": "302d"}, {"name": "Upper Victoria", "id": "1ccf"}]}, {"name": "Barwick Valley", "id": "17d2", "subregions": [{"name": "Upper Barwick", "id": "bfb2"}, {"name": "Lower Barwick", "id": "4b28"}]}, {"name": "Balham Valley", "id": "7105", "subregions": []}, {"name": "McKelvey Valley", "id": "8916", "subregions": [{"name": "Upper McKelvey", "id": "9ed1"}, {"name": "Lower McKelvey", "id": "c6a0"}, {"name": "Halzen Mesa", "id": "38d5"}]}]}]}, {"name": "Central Transantarctic Mountains", "id": "239e", "subregions": [{"name": "Shackleton Glacier", "id": "d8a8", "subregions": [{"name": "Roberts Massif", "id": "a9d8", "subregions": [{"name": "-Roberts Massif", "id": "e365"}]}, {"name": "Mount Black", "id": "7f54", "subregions": [{"name": "Bennett Platform", "id": "8e32"}, {"name": "Kitching Ridge", "id": "1817"}]}, {"name": "Vickers Nunatak", "id": "cbb6", "subregions": []}, {"name": "Cumulus Hills", "id": "9f15", "subregions": [{"name": "West Bluff", "id": "0cf8"}, {"name": "Schroeder Hill", "id": "57b5"}, {"name": "Ellis Bluff", "id": "987b"}, {"name": "Collinson Ridge", "id": "2393"}, {"name": "Shenk Peak", "id": "8299"}, {"name": "Cheu Valley", "id": "8b35"}, {"name": "Gatlin Glacier", "id": "1f20"}]}, {"name": "Mount Heekin", "id": "298a", "subregions": []}, {"name": "Pendant Ridge", "id": "c59b", "subregions": []}, {"name": "Thanksgiving Point", "id": "bc2f", "subregions": []}, {"name": "Mount Greenlee", "id": "fde4", "subregions": []}, {"name": "Taylor Nunatak", "id": "dea7", "subregions": []}, {"name": "Mount Cole", "id": "9154", "subregions": []}, {"name": "Mount Speed", "id": "324d", "subregions": []}, {"name": "Nilsen Peak", "id": "4d78", "subregions": []}, {"name": "Mount Wendland", "id": "c89a", "subregions": []}]}]}]
13 changes: 13 additions & 0 deletions prep_scripts/availible_regions.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Tier1,Tier2,Tier3,Tier4
Central Transantarctic Mountains,Shackleton Glacier,Nilsen Peak,Nilsen Peak
Central Transantarctic Mountains,Shackleton Glacier,Mount Speed,Mount Speed
Central Transantarctic Mountains,Shackleton Glacier,Mount Wendland,Mount Wendland
Central Transantarctic Mountains,Shackleton Glacier,Nilsen Peak,Nilsen Peak
Central Transantarctic Mountains,Shackleton Glacier,Mount Wendland,Mount Wendland
Central Transantarctic Mountains,Shackleton Glacier,Taylor Nunatak,Taylor Nunatak
Central Transantarctic Mountains,Shackleton Glacier,Mount Speed,Mount Speed
Central Transantarctic Mountains,Shackleton Glacier,Mount Wendland,Mount Wendland
Central Transantarctic Mountains,Shackleton Glacier,Mount Greenlee,Mount Greenlee
Central Transantarctic Mountains,Shackleton Glacier,Mount Speed,Mount Speed
Central Transantarctic Mountains,Shackleton Glacier,Mount Cole,Mount Cole
Central Transantarctic Mountains,Shackleton Glacier,Mount Greenlee,Mount Greenlee
2 changes: 1 addition & 1 deletion prep_scripts/bands.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import subprocess
import utilities

BAND_NAMES_FILE = '..\\terracotta\\client\\static\\data\\bandNames.json'
BAND_NAMES_FILE = Path('../terracotta/client/static/data/bandNames.json')


def extract_bands(data_folder_in, output_folder):
Expand Down
22 changes: 16 additions & 6 deletions prep_scripts/mosaics.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import subprocess


def build_mosaics(temp_folder, output_folder, region_centers):
def build_mosaics(temp_folder, output_folder, should_remove_border):
print("####### REGIONAL MOSAIC GENERATION ROUTINE #######")

pathlist = Path(temp_folder).glob('*.tif')
Expand All @@ -24,7 +24,8 @@ def build_mosaics(temp_folder, output_folder, region_centers):
imageGroupList += str(path) + " "
print("\nGroup: " + group)

run_mosaic_subprocess(imageGroupList, group, output_folder)
translate_string = create_translate_string(output_folder, group, should_remove_border)
run_mosaic_subprocess(imageGroupList, translate_string)


def create_file_groups(pathlist):
Expand All @@ -39,14 +40,23 @@ def create_file_groups(pathlist):
return groups


def run_mosaic_subprocess(group_list, group, output_folder):
vrtString = "gdalbuildvrt mosaic.vrt " + group_list
def create_translate_string(output_folder, group, should_remove_border):
translate_string = "gdal_translate -of GTiff "

subprocess.call(vrtString, shell=True)
if should_remove_border:
translate_string += "-a_nodata 0 "

translate_string = "gdal_translate -of GTiff -co \"TILED=YES\" mosaic.vrt " + \
translate_string += "-co \"TILED=YES\" mosaic.vrt " + \
str(output_folder) + "/" + group + ".tif"

return translate_string


def run_mosaic_subprocess(group_list, translate_string):
vrtString = "gdalbuildvrt mosaic.vrt " + group_list

subprocess.call(vrtString, shell=True)

subprocess.call(translate_string, shell=True)

os.remove("mosaic.vrt")
7 changes: 5 additions & 2 deletions prep_scripts/raster-prep.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,15 @@
REGIONS_FILE = 'all_regions.csv'
COORDINATES_FILE = 'L4_regions_sparse.csv'
REGIONS_JSON_FILE = 'all_regions.json'
DEFAULT_OUTPUT_FOLDER = '..\\terracotta\\client\\static\\mosaics\\'
DEFAULT_OUTPUT_FOLDER = Path('../terracotta/client/static/mosaics/')


def add_parser_arguments():
description = 'Process some GeoTIFFs according to a regional boundary key'
parser = argparse.ArgumentParser(description=description)
parser.add_argument('--data_folder', '-d', help='Location of some GeoTIFF files')
parser.add_argument('--remove_image_border', '-r', action='store_true',
help='Set nodata value to be 0 to remove black borders')

return parser

Expand All @@ -29,6 +31,7 @@ def main():
# get the path to the data folder and output folder from the argument
data_folder = Path(args.data_folder)
output_folder = Path(DEFAULT_OUTPUT_FOLDER)
should_remove_border = args.remove_image_border

region_centers = utilities.ingest_csv_file(COORDINATES_FILE)
print("found " + str(len(region_centers)) + " regions in key file")
Expand All @@ -42,7 +45,7 @@ def main():

input("press ENTER to continue...")

mosaics.build_mosaics(temp_folder, output_folder, region_centers)
mosaics.build_mosaics(temp_folder, output_folder, should_remove_border)

shutil.rmtree(temp_folder)

Expand Down
Loading

0 comments on commit d6afe95

Please sign in to comment.