Merge pull request #62 from awest1339/master

Fix PDF endpoints, add docs
mitre · Dec 6, 2017 · a91dfcf · a91dfcf
2 parents d60a268 + 44e0d92
commit a91dfcf
Show file tree

Hide file tree

Showing 35 changed files with 314 additions and 147 deletions.
diff --git a/docs/distributed_multiscanner.md b/docs/distributed_multiscanner.md
@@ -33,7 +33,7 @@ Distributed MultiScanner is intended to solve any combination of these problems
 ## Architecture ##
 This is the current architecture:
 
-![alt text](https://raw.githubusercontent.com/awest1339/multiscanner/celery/docs/distributed_ms_diagram.PNG)
+![Distributed MultiScanner Architecture](imgs/distributed_ms_diagram.PNG)
 
 When a sample is submitted (either via the web UI or the REST API), the sample is saved to the distributed file system (GlusterFS), a task is added to the distributed task queue (Celery), and an entry is added to the task management database (PostgreSQL). The worker nodes (Celery clients) all have the GlusterFS mounted, which gives them access to the samples for scanning. In our setup, we colocate the worker nodes with the GlusterFS nodes in order to reduce the network load of workers pulling samples from GlusterFS. When a new task is added to the Celery task queue, one of the worker nodes will pull the task and retrieve the corresponding sample from the GlusterFS via its SHA256 value. The worker node then performs the scanning work. Modules can be enabled / disabled via a configuration file. This configuration file is distributed to the workers by Ansible at setup time (details on this process later). When the worker finishes its scans, it will generate a JSON blob and index it into ElasticSearch for permanent storage. It will then update the task management database with a status of "Complete". The user will then be able view the report via the web interface or retrieve the raw JSON.
 

diff --git a/docs/imgs/Selection_001.png b/docs/imgs/Selection_001.png
diff --git a/docs/imgs/Selection_002.png b/docs/imgs/Selection_002.png
diff --git a/docs/imgs/Selection_003.png b/docs/imgs/Selection_003.png
diff --git a/docs/imgs/Selection_004.png b/docs/imgs/Selection_004.png
diff --git a/docs/imgs/Selection_005.png b/docs/imgs/Selection_005.png
diff --git a/docs/imgs/Selection_006.png b/docs/imgs/Selection_006.png
diff --git a/docs/imgs/Selection_007.png b/docs/imgs/Selection_007.png
diff --git a/docs/imgs/Selection_008.png b/docs/imgs/Selection_008.png
diff --git a/docs/imgs/Selection_009.png b/docs/imgs/Selection_009.png
diff --git a/docs/imgs/Selection_010.png b/docs/imgs/Selection_010.png
diff --git a/docs/imgs/Selection_011.png b/docs/imgs/Selection_011.png
diff --git a/docs/imgs/Selection_012.png b/docs/imgs/Selection_012.png
diff --git a/docs/imgs/Selection_013.png b/docs/imgs/Selection_013.png
diff --git a/docs/imgs/Selection_014.png b/docs/imgs/Selection_014.png
diff --git a/docs/imgs/Selection_015.png b/docs/imgs/Selection_015.png
diff --git a/docs/imgs/Selection_016.png b/docs/imgs/Selection_016.png
diff --git a/docs/imgs/Selection_017.png b/docs/imgs/Selection_017.png
diff --git a/docs/imgs/Selection_018.png b/docs/imgs/Selection_018.png
diff --git a/docs/imgs/Selection_019.png b/docs/imgs/Selection_019.png
diff --git a/docs/imgs/Selection_020.png b/docs/imgs/Selection_020.png
diff --git a/docs/imgs/Selection_021.png b/docs/imgs/Selection_021.png
diff --git a/docs/imgs/Selection_022.png b/docs/imgs/Selection_022.png
diff --git a/docs/imgs/Selection_023.png b/docs/imgs/Selection_023.png
diff --git a/docs/imgs/Selection_024.png b/docs/imgs/Selection_024.png
diff --git a/docs/distributed_ms_diagram.PNG → docs/imgs/distributed_ms_diagram.PNG b/docs/distributed_ms_diagram.PNG → docs/imgs/distributed_ms_diagram.PNG
diff --git a/docs/web.md b/docs/web.md
@@ -0,0 +1,108 @@
+# Web Interface #
+
+Submit Files for Analysis
+-------------------------
+
+![MultiScanner Web Interface](imgs/Selection_001.png)
+
+When you visit MultiScanner's web interface in a web browser, you'll be greeted by the file submission page. Drag files onto the large drop area in the middle of the page or click it or the "Select File(s)..." button to select one or more files to be uploaded and analyzed.
+
+Click on the "Advanced Options" button to change default options and set metadata fields to be added to the scan results.
+
+![Advanced Options](imgs/Selection_003.png)
+
+Metadata fields can be added or removed by editing web_config.ini. Metadata field values can be set for individual files by clicking the small black button below and to the right of that filename in the staging area.
+
+![File Options](imgs/Selection_004.png)
+
+Change from "Scan" to "Import" to import JSON analysis reports into MultiScanner.  This is intended only to be used with the JSON reports you can download from a report page in MultiScanner.
+
+![Import](imgs/Selection_005.png)
+
+By default, if you resubmit a sample that has already been submitted, MultiScanner will pull the latest report of that sample. If you want MultiScanner to re-scan the sample, set that option in Advanced Options.
+
+![Re-scan](imgs/Selection_006.png)
+
+If you have a directory of samples you wish to scan at once, we recommend zipping them and uploading the archive with the option to extract archives enabled. You can also specify a password, if the archive file is password- protected. Alternatively you can use the REST API for bulk uploads.
+
+![Archive Files](imgs/Selection_007.png)
+
+Click the "Scan it!" button to submit the sample to MultiScanner.
+
+![Scan It!](imgs/Selection_008.png)
+
+The progress bars that appear in the file staging area do not indicate the progress of the scan; a full bar merely indicates that the file has been uploaded to MultiScanner. Click on the file to go to its report page.
+
+![Submission Progress Bar](imgs/Selection_009.png)
+
+If the analysis has not completed yet, you'll see a "Pending" message.
+
+![Pending](imgs/Selection_010.png)
+
+Analyses and History Pages
+--------------------------
+
+Reports can be listed and searched in two different ways. The Analyses page lists the most recent report per sample.
+
+![Analyses Page](imgs/Selection_011.png)
+
+The History page lists every report of each sample. So if a file is scanned multiple times, it will only show up once on the Analyses page, but all of the reports will show up on the History page.
+
+![History Page](imgs/Selection_012.png)
+
+Both pages display the list of reports and allow you to search them. Click the blue button in the middle to refresh the list of reports.
+
+![Refresh Button](imgs/Selection_013.png)
+
+Click on a row in the list to go to that report, and click the red "X" button to delete that report from MultiScanner's Elasticsearch database.
+
+![Delete Button](imgs/Selection_014.png)
+
+Searching
+---------
+
+![Navbar Search](imgs/Selection_015.png)
+
+Reports can be searched from any page, with a few options. You can search Analyses to get the most recent scan per file, or search History to get all scans recorded for each file. Use the "Default" search type to have wildcards automatically appended to the beginning and end of your search term. Use the "Exact" search type to search automatically append quotes and search for the exact phrase. Finally, use the "Advanced" search type to search with the full power of Lucene query string syntax. Nothing will be automatically appended and you will need to escape any reserved characters yourself. When you click on one of the search results, the search term will be highlighted on the Report page and the report will be expanded and automatically scrolled to the first match.
+
+![Analyses/History Search](imgs/Selection_016.png)
+
+Report page
+-----------
+
+![Report Page](imgs/Selection_017.png)
+
+Each report page displays the results of a single analysis. Some rows in the report can be expanded or collapsed to reveal more data by clicking on the row header or the "Expand" button. Shift-clicking will also expand or collapse all of it's child rows.
+
+![Expand Button](imgs/Selection_024.png)
+
+The "Expand All" button will expand all rows at once. If they are all expanded, this will turn into a "Collapse All" button that will collapse them all again.
+
+![Expand All Button](imgs/Selection_018.png)
+
+As reports can contain a great deal of content, you can search the report to find the exact data you are looking for with the search field located under the report title. The search term, if found, will be highlighted, the matching fields will be expanded, and the page automatically scrolled to the first match.
+
+![In-Page Search](imgs/Selection_019.png)
+
+Reports can be tagged by entering text in the Tags input box and hitting the enter key. As you type, a dropdown will appear with suggestions from the tags already in the system. It will pull the list of tags from existing reports, but a pre-populated list of tags can also be provided in web_config.ini when the web interface is set up.
+
+![Tags](imgs/Selection_020.png)
+
+You can download the report in a number of different formats using the Download button on the right side. You can download a JSON-formatted version of the report containing all the same data shown on the page. You can also download a MAEC-formatted version of the reports from Cuckoo Sandbox. Finally, you can also download the original sample file as a password-protected ZIP file. The password will be "infected".
+
+![Download](imgs/Selection_021.png)
+
+Click on "Notes" to open a sidebar where analysts may enter notes or comments.
+
+![Notes](imgs/Selection_022.png)
+
+These notes and comments can be edited and deleted. Click the "<" button to collapse this sidebar.
+
+![Close Notes](imgs/Selection_023.png)
+
+Analytics
+---------
+
+![Analytics Page](imgs/Selection_002.png)
+
+The Analytics page displays various pieces of advanced analysis. For now, this is limited to ssdeep comparisons. The table lists samples, with those that have very similar ssdeep hashes grouped together. Other analytics will be added in the future. For more information, see [this page](../docs/analytics.md).
diff --git a/modules/Metadata/entropy.py b/modules/Metadata/entropy.py
@@ -0,0 +1,37 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+from __future__ import division, absolute_import, with_statement, print_function, unicode_literals
+from collections import Counter
+import math
+
+
+__author__ = "Austin West"
+__license__ = "MPL 2.0"
+
+TYPE = "Metadata"
+NAME = "entropy"
+DEFAULTCONF = {
+    'ENABLED': True
+}
+
+
+def check(conf=DEFAULTCONF):
+    return True
+
+
+def scan(filelist, conf=DEFAULTCONF):
+    '''Calculate entropy of a string'''
+    results = []
+    for fname in filelist:
+        with open(fname, 'rb') as f:
+            text = f.read()
+        chars, lns = Counter(text), float(len(text))
+        result = -sum(count/lns * math.log(count/lns, 2) for count in chars.values())
+        results.append((fname, result))
+
+    metadata = {}
+    metadata["Name"] = NAME
+    metadata["Type"] = TYPE
+    metadata["Include"] = False
+    return (results, metadata)
diff --git a/pdf_generator/__init__.py → utils/__init__.py b/pdf_generator/__init__.py → utils/__init__.py
diff --git a/utils/api.py b/utils/api.py
@@ -24,6 +24,7 @@
 PUT /api/v1/tasks/<task_id>/notes/<note_id> ---> Edit a note
 DELETE /api/v1/tasks/<task_id>/notes/<note_id> ---> Delete a note
 GET /api/v1/tasks/<task_id>/report?d={t|f}---> receive report in JSON, set d=t to download
+GET /api/v1/tasks/<task_id>/pdf ---> Receive PDF report
 POST /api/v1/tasks/<task_id>/tags ---> Add tags to task
 DELETE /api/v1/tasks/<task_id>/tags ---> Remove tags from task
 GET /api/v1/analytics/ssdeep_compare---> Run ssdeep.compare analytic
@@ -56,8 +57,6 @@
 from six import PY3
 import rarfile
 import zipfile
-from reportlab.platypus import TableStyle
-from reportlab.lib import colors, units
 import requests
 
 MS_WD = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -74,7 +73,7 @@
 import sql_driver as database
 import elasticsearch_storage
 import common
-from pdf_generator.generic_pdf import GenericPDF
+from utils.pdf_generator import create_pdf_document
 
 TASK_NOT_FOUND = {'Message': 'No task or report with that ID found!'}
 INVALID_REQUEST = {'Message': 'Invalid request parameters'}
@@ -846,6 +845,7 @@ def files_get_sha256_helper(sha256, raw=None):
             response.headers['Content-Disposition'] = 'inline; filename={}.zip'.format(sha256)
     return response
 
+
 @app.route('/api/v1/analytics/ssdeep_compare', methods=['GET'])
 def run_ssdeep_compare():
     '''
@@ -865,10 +865,11 @@ def run_ssdeep_compare():
             jsonify({'Message': 'Unable to complete request.'}),
             HTTP_BAD_REQUEST)
 
+
 @app.route('/api/v1/analytics/ssdeep_group', methods=['GET'])
 def run_ssdeep_group():
     '''
-    Runs sssdeep group analytic and returns list of groups as a list.
+    Runs ssdeep group analytic and returns list of groups as a list.
     '''
     try:
         ssdeep_analytic = SSDeepAnalytic()
@@ -880,138 +881,23 @@ def run_ssdeep_group():
             HTTP_BAD_REQUEST)
 
 
-@app.route('/api/v1/tasks/report/<task_id>/pdf', methods=['GET'])
+@app.route('/api/v1/tasks/<task_id>/pdf', methods=['GET'])
 def get_pdf_report(task_id):
+    '''
+    Generates a PDF version of a JSON report.
+    '''
     report_dict, success = get_report_dict(task_id)
 
     if not success:
         return jsonify(report_dict)
 
-    pdf = create_pdf_document(report_dict)
+    pdf = create_pdf_document(MS_WD, report_dict)
     response = make_response(pdf)
     response.headers['Content-Type'] = 'application/pdf'
     response.headers['Content-Disposition'] = 'attachment; filename=%s.pdf' % task_id
     return response
 
 
-def create_pdf_document(report):
-    '''
-    Method to create PDF report document from JSON.
-    '''
-    with open(os.path.join(MS_WD, 'pdf_generator/pdf_config.json')) as data_file:
-        pdf_components = json.load(data_file)
-
-    gen_pdf = GenericPDF(pdf_components)
-    gen_pdf.tlp_color = 'GREEN'
-
-    notice = gen_pdf.section('Notification', pdf_components['notification'], gen_pdf.style)
-
-    for n in notice:
-        gen_pdf.pdf_list.append(n)
-
-    summary = gen_pdf.section('Summary', '', gen_pdf.style)
-
-    for s in summary:
-        gen_pdf.pdf_list.append(s)
-
-    summary_data = [
-        ['Date Submitted', 'N\A'],
-        ['Artifact ID', 'N\A'],
-        ['Description', pdf_components['summary_description']],
-        ['Files Processed', '1'],
-        ['', report['Report']['filename']]
-    ]
-
-    gen_pdf.vertical_table(summary_data)
-
-    gen_pdf.line_break()
-
-    file_and_obs = gen_pdf.section('File Indicators and Observables', '', gen_pdf.style)
-
-    for f in file_and_obs:
-        gen_pdf.pdf_list.append(f)
-
-    # This list will store data for table under File Indicators and Observables
-    file_data = []
-
-    # This list will store data for Yara results. Currently, extracts description of rule. This is a horizontal table.
-    yara_data = [['Yara Rule', 'Yara Rule Description']]
-
-    # This list will store AV results. This is a horizontal table.
-    av_data = [['Antivirus', 'Scan Result']]
-
-    if 'Report' in report:
-        r = report['Report']
-        if 'filename' in r:
-            file_data.append(['File Name', r['filename']])
-        if 'Scan Time' in r:
-            file_data.append(['Scan Time', r['Scan Time']])
-        if 'libmagic' in r:
-            file_data.append(['Type', r['libmagic']])
-        if 'MD5' in r:
-            file_data.append(['MD5', r['MD5']])
-        if 'SHA1' in r:
-            file_data.append(['SHA1', r['SHA1']])
-        if 'SHA256' in r:
-            file_data.append(['SHA256', r['SHA256']])
-        if 'ssdeep' in r and 'ssdeep_hash' in r['ssdeep']:
-            file_data.append(['SSDEEP', r['ssdeep']['ssdeep_hash']])
-
-        if 'Yara' in r:
-            for v in r['Yara'].values():
-                if 'meta' in v and 'description' in v['meta']:
-                    yara_data.append([v['rule'], v['meta']['description']])
-                elif 'meta' in v and 'description' not in v['meta']:
-                    yara_data.append([v['rule'], "NO RULE DESCRIPTION"])
-
-        if 'AVG 2014' in r:
-            av_data.append(['AVG 2014', r['AVG 2014']])
-        if 'Microsoft Security Essentials' in r:
-            av_data.append(['Microsoft Security Essentials', r['Microsoft Security Essentials']])
-
-    gen_pdf.vertical_table(file_data)
-
-    gen_pdf.line_break()
-
-    av_table_style = TableStyle([
-        ('BACKGROUND', (0, 0), (-1, 0), colors.skyblue),
-    ])
-
-    gen_pdf.horizontal_table(av_data, av_table_style, (50 * units.mm, 140 * units.mm))
-
-    gen_pdf.line_break()
-
-    yara_table_style = TableStyle([
-        ('BACKGROUND', (0, 0), (-1, 0), colors.skyblue),
-    ])
-
-    gen_pdf.horizontal_table(yara_data, yara_table_style, (50 * units.mm, 140 * units.mm))
-
-    gen_pdf.line_break()
-
-    mitigation_recommendation = gen_pdf.section('Mitigation Recommendations', pdf_components['mitigation_recommendations'], gen_pdf.style)
-
-    for mr in mitigation_recommendation:
-        gen_pdf.pdf_list.append(mr)
-
-    mitigation_bullets = gen_pdf.bullet_list(pdf_components['mitigation_bullet_list'], 1)
-    gen_pdf.pdf_list.append(mitigation_bullets)
-
-    gen_pdf.line_break()
-
-    contact = gen_pdf.section('Contact Information', pdf_components['contact_information'], gen_pdf.style)
-
-    for c in contact:
-        gen_pdf.pdf_list.append(c)
-
-    faq = gen_pdf.section('Document FAQ', pdf_components['document_faq'], gen_pdf.style)
-
-    for f in faq:
-        gen_pdf.pdf_list.append(f)
-
-    return gen_pdf.build()
-
-
 if __name__ == '__main__':
 
     if not os.path.isdir(api_config['api']['upload_folder']):