diff --git a/docs/source/modelio.md b/docs/source/modelio.md index d5dfde2..5f30ca7 100644 --- a/docs/source/modelio.md +++ b/docs/source/modelio.md @@ -10,7 +10,7 @@ http://localhost:80/api/predict?fileurl=http://example.org/cutedogsandcats.jpg The API then returns the prediction in the specified format. For a thorough description of the API, have a look at its [documentation](https://modelhub.readthedocs.io/en/latest/modelhubapi.html).

#### As a Collaborator submitting a new Model -For single inputs, please create a configuration for your model according to the [example configuration](https://github.com/modelhub-ai/modelhub/blob/master/example_config_single_input.json). It is important that you keep the key `"single"` in the config, as the API uses this for accessing the dimension constraints when loading an image. Populate the rest of the configuration file as stated in the contribution guide and the [schema](https://github.com/modelhub-ai/modelhub/blob/master/config_schema.json). Validate your config file against our config schema with a JSON validator, e.g. [this one](https://www.jsonschemavalidator.net).
+For single inputs, please create a configuration for your model according to the [example configuration](https://github.com/modelhub-ai/modelhub/blob/master/examples/example_config_single_input.json). It is important that you keep the key `"single"` in the config, as the API uses this for accessing the dimension constraints when loading an image. Populate the rest of the configuration file as stated in the contribution guide and the [schema](https://github.com/modelhub-ai/modelhub/blob/master/config_schema.json). Validate your config file against our config schema with a JSON validator, e.g. [this one](https://www.jsonschemavalidator.net).
Take care to choose the right MIME type for your input, this format will be checked by the API when users call the predict function and load a file. We support a few extra MIME types in addition to the standard MIME types: @@ -45,7 +45,7 @@ If you need other types not supported in the standard MIME types and by our exte ### Input Configuration for Multiple Inputs #### As a User -When you use a model that needs more than a single input file for a prediction, you have to pass a JSON file with all the inputs needed for that model. You can have a look at an example [here](https://github.com/modelhub-ai/modelhub/blob/master/example_input_file_multiple_inputs.json).
+When you use a model that needs more than a single input file for a prediction, you have to pass a JSON file with all the inputs needed for that model. You can have a look at an example [here](https://github.com/modelhub-ai/modelhub/blob/master/examples/example_input_file_multiple_inputs.json).
The important points to keep in mind are: - There has to be a `format` key with `"application/json"` so that the API can handle the file - Each of the other keys describes one input and has to have a `format` (see the MIME types above) and a `fileurl` @@ -61,7 +61,7 @@ For a thorough description of the API, have a look at its [documentation](https: #### As a Collaborator submitting a new Model -For multiple inputs, please create a configuration for your model according to the [example configuration](https://github.com/modelhub-ai/modelhub/blob/master/example_config_multiple_inputs.json). The `format` key has to be present at the `input` level and must be equal to `application/json` as all input files will be passed in a json to the API. +For multiple inputs, please create a configuration for your model according to the [example configuration](https://github.com/modelhub-ai/modelhub/blob/master/examples/example_config_multiple_inputs.json). The `format` key has to be present at the `input` level and must be equal to `application/json` as all input files will be passed in a json to the API.
The other keys stand for one input file each and must contain a valid format (e.g. `application/dicom`) and dimensions. You can additionally add a description for the input.
diff --git a/docs/source/quickstart.md b/docs/source/quickstart.md index 3831cac..09d16b6 100644 --- a/docs/source/quickstart.md +++ b/docs/source/quickstart.md @@ -17,26 +17,25 @@ But since you are here, follow these steps to get modelhub running on your local Python 2.7 or Python 3.6 (or higher).

-3. **Download modelhub start script** +3. **Install the modelhub-ai package** - Download [_start.py_](https://raw.githubusercontent.com/modelhub-ai/modelhub/master/start.py) - (right click -> "save link as") from the [modelhub repository](https://github.com/modelhub-ai/modelhub) and place it into an empty folder. + Install the `modelhub-ai` package from PyPi using pip: `pip install modelhub-ai`.

4. **Run a model using start.py** - Open a terminal and navigate to the folder that contains _start.py_. For running models, write access + Open a terminal and navigate to a folder you want to work in. For running models, write access is required in the current folder. - Execute `python start.py squeezenet` in the terminal to run the squeezenet model from the modelhub collection. + Execute `modelhub-run squeezenet` in the terminal to run the squeezenet model from the modelhub collection. This will download all required model files (only if they do not exist yet) and start the model. Follow the instructions given on the terminal to access the web interface to explore the model. Replace `squeezenet` by any other model name in the collection to start a different model. To see a list of - all available models execute `python start.py -l`. + all available models execute `modelhub-list` or `modelhub -l`. You can also access a jupyter notebook that allows you to experiment with a model by starting a model with - the "-e" option, e.g. `python start.py squeezenet -e`. Follow the instructions on the terminal to open the notebook. + the "-e" option, e.g. `modelhub-run squeezenet -e`. Follow the instructions on the terminal to open the notebook. - See additional starting options by executing `python start.py -h`. + See additional starting options by executing `modelhub-run -h`.

diff --git a/framework/modelhubapi/pythonapi.py b/framework/modelhubapi/pythonapi.py index 37ace96..990448c 100644 --- a/framework/modelhubapi/pythonapi.py +++ b/framework/modelhubapi/pythonapi.py @@ -90,10 +90,11 @@ def predict(self, input_file_path, numpyToFile=True, url_root=""): Preforms the model's inference on the given input. Args: - input_file_path (str): Path to input file to run inference on. + input_file_path (str or dict): Path to input file to run inference on. Either a direct input file or a json containing paths to all input files needed for the model to predict. The appropriate structure for the json can be found in the documentation. + If used directly, you can also pass a dict with the keys. numpyToFile (bool): Only effective if prediction is a numpy array. Indicates if numpy outputs should be saved and a path to it is returned. If false, a json-serializable list representation of @@ -153,7 +154,9 @@ def _unpack_inputs(self, file_path): returns the file_path unchanged for single inputs It also converts the fileurl to a valid string (avoids html escaping) """ - if file_path.lower().endswith('.json'): + if isinstance(file_path, dict): + return self._check_input_compliance(file_path) + elif file_path.lower().endswith('.json'): input_dict = self._load_json(file_path) for key, value in input_dict.items(): if key == "format": diff --git a/framework/modelhubapi/restapi.py b/framework/modelhubapi/restapi.py index c24d29f..0b264db 100644 --- a/framework/modelhubapi/restapi.py +++ b/framework/modelhubapi/restapi.py @@ -223,13 +223,13 @@ def _delete_temp_files(self, folder): """ Removes all files in the given folder """ - for file in os.listdir(folder): - file_path = os.path.join(folder, file) - try: + try: + for file in os.listdir(folder): + file_path = os.path.join(folder, file) if os.path.isfile(file_path): os.unlink(file_path) - except Exception as e: - print(e) + except Exception as e: + print(e) def _jsonify(self, content): """ diff --git a/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/missing_key.json b/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/missing_key.json new file mode 100644 index 0000000..288917b --- /dev/null +++ b/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/missing_key.json @@ -0,0 +1,15 @@ +{ + "format": ["application/json"], + "t1c": { + "type": ["application/nii-gzip"], + "fileurl":"https://raw.githubusercontent.com/christophbrgr/modelhub-tests/master/testimage_nifti_91x109x91.nii.gz" + }, + "t2": { + "type": ["application/nii-gzip"], + "fileurl":"https://raw.githubusercontent.com/christophbrgr/modelhub-tests/master/testimage_nifti_91x109x91.nii.gz" + }, + "flair": { + "type": ["application/nii-gzip"], + "fileurl":"https://raw.githubusercontent.com/christophbrgr/modelhub-tests/master/testimage_nifti_91x109x91.nii.gz" + } +} diff --git a/framework/modelhubapi_tests/pythonapi_test.py b/framework/modelhubapi_tests/pythonapi_test.py index c6ffae8..5412a18 100644 --- a/framework/modelhubapi_tests/pythonapi_test.py +++ b/framework/modelhubapi_tests/pythonapi_test.py @@ -1,6 +1,7 @@ import unittest import os import numpy +import json import shutil from modelhubapi import ModelHubAPI from .apitestbase import TestAPIBase @@ -102,6 +103,12 @@ def tearDown(self): def test_predict_accepts_and_processes_valid_json(self): result = self.api.predict(self.this_dir + "/mockmodels/contrib_src_mi/sample_data/valid_input_list.json") self.assertEqual(result["output"][0]["prediction"][0], True) + + def test_predict_accepts_and_processes_valid_dict(self): + with open(self.this_dir + "/mockmodels/contrib_src_mi/sample_data/valid_input_list.json", "r") as f: + input_dict = json.load(f) + result = self.api.predict(input_dict) + self.assertEqual(result["output"][0]["prediction"][0], True) def test_predict_rejects_invalid_file(self): result = self.api.predict(self.this_dir + "/mockmodels/contrib_src_si/sample_data/testimage_ramp_4x2.png") @@ -141,7 +148,22 @@ def setUp(self): contrib_src_dir = os.path.join(self.this_dir, "mockmodels", "contrib_src_si") self.api = ModelHubAPI(model, contrib_src_dir) +class TestModelHubAPIUtitilyFunctions(unittest.TestCase): + def setUp(self): + model = ModelReturnsOneLabelList() + self.this_dir = os.path.dirname(os.path.realpath(__file__)) + # load config version 2 with only one output specified + contrib_src_dir = os.path.join(self.this_dir, "mockmodels", "contrib_src_mi") + self.api = ModelHubAPI(model, contrib_src_dir) + + + def tearDown(self): + pass + + def test_json_write_fails_for_invalid_path(self): + result = self.api._write_json({"some":"key"}, "this/does/not/exist/johndoe.json") + self.assertIn("error", result) class TestModelHubAPIModelReturnsOneLabelList(unittest.TestCase): diff --git a/framework/modelhubapi_tests/restapi_test.py b/framework/modelhubapi_tests/restapi_test.py new file mode 100644 index 0000000..db169f9 --- /dev/null +++ b/framework/modelhubapi_tests/restapi_test.py @@ -0,0 +1,235 @@ +import os +import io +from zipfile import ZipFile +import shutil +import json +from modelhubapi_tests.mockmodels.contrib_src_si.inference import Model +from modelhubapi_tests.mockmodels.contrib_src_mi.inference import ModelNeedsTwoInputs +from .apitestbase import TestRESTAPIBase +from modelhubapi import ModelHubAPI + + + +class TestModelHubRESTAPI_SI(TestRESTAPIBase): + + def setUp(self): + self.this_dir = os.path.dirname(os.path.realpath(__file__)) + self.contrib_src_dir = os.path.join(self.this_dir, "mockmodels", "contrib_src_si") + self.setup_self_temp_work_dir() + self.setup_self_temp_output_dir() + self.setup_self_test_client(Model(), self.contrib_src_dir) + + def tearDown(self): + shutil.rmtree(self.temp_work_dir, ignore_errors=True) + shutil.rmtree(self.temp_output_dir, ignore_errors=True) + pass + + def test_get_config_returns_correct_dict(self): + response = self.client.get("/api/get_config") + self.assertEqual(200, response.status_code) + config = json.loads(response.get_data()) + self.assert_config_contains_correct_dict(config) + + + def test_get_legal_returns_expected_keys(self): + response = self.client.get("/api/get_legal") + self.assertEqual(200, response.status_code) + legal = json.loads(response.get_data()) + self.assert_legal_contains_expected_keys(legal) + + + def test_get_legal_returns_expected_mock_values(self): + response = self.client.get("/api/get_legal") + self.assertEqual(200, response.status_code) + legal = json.loads(response.get_data()) + self.assert_legal_contains_expected_mock_values(legal) + + + def test_get_model_io_returns_expected_mock_values(self): + response = self.client.get("/api/get_model_io") + self.assertEqual(200, response.status_code) + model_io = json.loads(response.get_data()) + self.assert_model_io_contains_expected_mock_values(model_io) + + + def test_get_samples_returns_path_to_mock_samples(self): + response = self.client.get("/api/get_samples") + self.assertEqual(200, response.status_code) + samples = json.loads(response.get_data()) + samples.sort() + self.assertListEqual(["http://localhost/api/samples/testimage_ramp_4x2.jpg", + "http://localhost/api/samples/testimage_ramp_4x2.png"], + samples) + + + def test_samples_routes_correct(self): + response = self.client.get("/api/samples/testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + self.assertEqual("image/png", response.content_type) + + + def test_thumbnail_routes_correct(self): + response = self.client.get("/api/thumbnail/thumbnail.jpg") + self.assertEqual(200, response.status_code) + self.assertEqual("image/jpeg", response.content_type) + + + def test_get_model_files_returns_zip(self): + response = self.client.get("/api/get_model_files") + self.assertEqual(200, response.status_code) + self.assertEqual("application/zip", response.content_type) + + + def test_get_model_files_returned_zip_has_mock_content(self): + response = self.client.get("/api/get_model_files") + self.assertEqual(200, response.status_code) + test_zip_name = self.temp_work_dir + "/test_response.zip" + with open(test_zip_name, "wb") as test_file: + test_file.write(response.get_data()) + with ZipFile(test_zip_name, "r") as test_zip: + reference_content = ["model/", + "model/model.txt", + "model/config.json", + "model/thumbnail.jpg"] + reference_content.sort() + zip_content = test_zip.namelist() + zip_content.sort() + self.assertListEqual(reference_content, zip_content) + self.assertEqual(b"EMPTY MOCK MODEL FOR UNIT TESTING", + test_zip.read("model/model.txt")) + + + def test_predict_by_post_returns_expected_mock_prediction(self): + response = self._post_predict_request_on_sample_image("testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + result = json.loads(response.get_data()) + self.assert_predict_contains_expected_mock_prediction(result) + + + def test_predict_by_post_returns_expected_mock_meta_info(self): + response = self._post_predict_request_on_sample_image("testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + result = json.loads(response.get_data()) + self.assert_predict_contains_expected_mock_meta_info(result) + + + def test_predict_by_post_returns_error_on_unsupported_file_type(self): + response = self._post_predict_request_on_sample_image("testimage_ramp_4x2.jpg") + self.assertEqual(400, response.status_code) + result = json.loads(response.get_data()) + self.assertIn("error", result) + self.assertIn("Incorrect file type.", result["error"]) + + + def test_working_folder_empty_after_predict_by_post(self): + response = self._post_predict_request_on_sample_image("testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + self.assertEqual(len(os.listdir(self.temp_work_dir) ), 0) + + + # TODO this is not so nice yet, test should not require a download from the inet + # should probably use a mock server for this + def test_predict_by_url_returns_expected_mock_prediction(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-docker/master/framework/modelhublib_tests/testdata/testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + result = json.loads(response.get_data()) + self.assert_predict_contains_expected_mock_prediction(result) + + + # TODO this is not so nice yet, test should not require a download from the inet + # should probably use a mock server for this + def test_predict_by_url_returns_expected_mock_meta_info(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-docker/master/framework/modelhublib_tests/testdata/testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + result = json.loads(response.get_data()) + self.assert_predict_contains_expected_mock_meta_info(result) + + + # TODO this is not so nice yet, test should not require a download from the inet + # should probably use a mock server for this + def test_predict_by_url_returns_error_on_unsupported_file_type(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-docker/master/framework/modelhublib_tests/testdata/testimage_ramp_4x2.jpg") + self.assertEqual(400, response.status_code) + result = json.loads(response.get_data()) + self.assertIn("error", result) + self.assertIn("Incorrect file type.", result["error"]) + + + def test_working_folder_empty_after_predict_by_url(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-docker/master/framework/modelhublib_tests/testdata/testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + self.assertEqual(len(os.listdir(self.temp_work_dir) ), 0) + + + def test_predict_sample_returns_expected_mock_prediction(self): + response = self.client.get("/api/predict_sample?filename=testimage_ramp_4x2.png") + self.assertEqual(200, response.status_code) + result = json.loads(response.get_data()) + self.assert_predict_contains_expected_mock_prediction(result) + + + def test_predict_sample_on_invalid_file_returns_error(self): + response = self.client.get("/api/predict_sample?filename=NON_EXISTENT.png") + self.assertEqual(400, response.status_code) + +class TestModelHubRESTAPI_MI(TestRESTAPIBase): + + def setUp(self): + self.this_dir = os.path.dirname(os.path.realpath(__file__)) + self.contrib_src_dir = os.path.join(self.this_dir, "mockmodels", "contrib_src_mi") + self.setup_self_temp_work_dir() + self.setup_self_temp_output_dir() + self.setup_self_test_client(ModelNeedsTwoInputs(), self.contrib_src_dir) + self.client.api = ModelHubAPI(ModelNeedsTwoInputs(), self.contrib_src_dir) + self.client.api.get_config = self.monkeyconfig() + + def tearDown(self): + shutil.rmtree(self.temp_work_dir, ignore_errors=True) + shutil.rmtree(self.temp_output_dir, ignore_errors=True) + pass + + # this can change the config on the fly + def monkeyconfig(self): + return self.client.api._load_json(self.contrib_src_dir + '/model/config_4_nii.json') + + def test_get_config_returns_correct_dict(self): + response = self.client.get("/api/get_config") + self.assertEqual(200, response.status_code) + config = json.loads(response.get_data()) + self.assert_config_contains_correct_dict(config) + + # TODO this is not so nice yet, test should not require a download from the inet + # should probably use a mock server for this + def test_working_folder_empty_after_predict_by_url(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-engine/master/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/4_nii_gz_url.json") + self.assertEqual(200, response.status_code) + self.assertEqual(len(os.listdir(self.temp_work_dir) ), 0) + + # TODO this is not so nice yet, test should not require a download from the inet + # should probably use a mock server for this + def test_api_downloads_files_from_url(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-engine/master/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/4_nii_gz_url.json") + self.assertEqual(200, response.status_code) + + # TODO this is not so nice yet, test should not require a download from the inet + # should probably use a mock server for this + def test_api_manages_mix_of_url_and_local_paths(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-engine/master/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/4_nii_gz_mixed.json") + self.assertEqual(200, response.status_code) + + # TODO this is not so nice yet, test should not require a download from the inet + # should probably use a mock server for this + def test_api_rejects_wrong_url_in_json(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-engine/master/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/4_nii_gz_url_error.json") + self.assertEqual(400, response.status_code) + result = json.loads(response.get_data()) + self.assertIn("error", result) + + def test_api_fails_on_config_mismatch(self): + response = self.client.get("/api/predict?fileurl=https://raw.githubusercontent.com/modelhub-ai/modelhub-engine/master/framework/modelhubapi_tests/mockmodels/contrib_src_mi/sample_data/missing_key.json") + self.assertEqual(400, response.status_code) + result = json.loads(response.get_data()) + self.assertIn("error", result) + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/modelhub/LICENSE b/modelhub/LICENSE deleted file mode 100644 index 2674c8e..0000000 --- a/modelhub/LICENSE +++ /dev/null @@ -1,21 +0,0 @@ -MIT License - -Copyright (c) 2018 ModelHub - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/modelhub/README.md b/modelhub/README.md deleted file mode 100644 index 6f8beaf..0000000 --- a/modelhub/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# modelhub - -Crowdsourced through contributions by the research community, modelhub.ai is a repository of deep learning models for various data types. diff --git a/modelhub/modelhub/__init__.py b/modelhub/modelhub/__init__.py deleted file mode 100644 index 112b616..0000000 --- a/modelhub/modelhub/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -name = "modelhub" -test = "OK" diff --git a/modelhub/setup.py b/modelhub/setup.py deleted file mode 100644 index d92a618..0000000 --- a/modelhub/setup.py +++ /dev/null @@ -1,21 +0,0 @@ -import setuptools - -with open("README.md", "r") as fh: - long_description = fh.read() - -setuptools.setup( - name="modelhub-ai", - version="0.0.4", - author="modelhub", - author_email="info@modelhub.ai", - description="Crowdsourced through contributions by the research community, modelhub.ai is a repository of deep learning models for various data types.", - long_description=long_description, - long_description_content_type="text/markdown", - url="https://github.com/modelhub-ai", - packages=setuptools.find_packages(), - classifiers=[ - "Programming Language :: Python :: 2", - "License :: OSI Approved :: MIT License", - "Operating System :: OS Independent", - ], -)