diff --git a/docs/distributed_multiscanner.md b/docs/distributed_multiscanner.md index 2a84dec9..2b724945 100644 --- a/docs/distributed_multiscanner.md +++ b/docs/distributed_multiscanner.md @@ -33,7 +33,7 @@ Distributed MultiScanner is intended to solve any combination of these problems ## Architecture ## This is the current architecture: -![alt text](https://raw.githubusercontent.com/awest1339/multiscanner/celery/docs/distributed_ms_diagram.PNG) +![Distributed MultiScanner Architecture](imgs/distributed_ms_diagram.PNG) When a sample is submitted (either via the web UI or the REST API), the sample is saved to the distributed file system (GlusterFS), a task is added to the distributed task queue (Celery), and an entry is added to the task management database (PostgreSQL). The worker nodes (Celery clients) all have the GlusterFS mounted, which gives them access to the samples for scanning. In our setup, we colocate the worker nodes with the GlusterFS nodes in order to reduce the network load of workers pulling samples from GlusterFS. When a new task is added to the Celery task queue, one of the worker nodes will pull the task and retrieve the corresponding sample from the GlusterFS via its SHA256 value. The worker node then performs the scanning work. Modules can be enabled / disabled via a configuration file. This configuration file is distributed to the workers by Ansible at setup time (details on this process later). When the worker finishes its scans, it will generate a JSON blob and index it into ElasticSearch for permanent storage. It will then update the task management database with a status of "Complete". The user will then be able view the report via the web interface or retrieve the raw JSON. diff --git a/docs/imgs/Selection_001.png b/docs/imgs/Selection_001.png new file mode 100644 index 00000000..6d46951f Binary files /dev/null and b/docs/imgs/Selection_001.png differ diff --git a/docs/imgs/Selection_002.png b/docs/imgs/Selection_002.png new file mode 100644 index 00000000..a725e029 Binary files /dev/null and b/docs/imgs/Selection_002.png differ diff --git a/docs/imgs/Selection_003.png b/docs/imgs/Selection_003.png new file mode 100644 index 00000000..9609fcc4 Binary files /dev/null and b/docs/imgs/Selection_003.png differ diff --git a/docs/imgs/Selection_004.png b/docs/imgs/Selection_004.png new file mode 100644 index 00000000..03e3984d Binary files /dev/null and b/docs/imgs/Selection_004.png differ diff --git a/docs/imgs/Selection_005.png b/docs/imgs/Selection_005.png new file mode 100644 index 00000000..a72df539 Binary files /dev/null and b/docs/imgs/Selection_005.png differ diff --git a/docs/imgs/Selection_006.png b/docs/imgs/Selection_006.png new file mode 100644 index 00000000..6c210073 Binary files /dev/null and b/docs/imgs/Selection_006.png differ diff --git a/docs/imgs/Selection_007.png b/docs/imgs/Selection_007.png new file mode 100644 index 00000000..aaa346da Binary files /dev/null and b/docs/imgs/Selection_007.png differ diff --git a/docs/imgs/Selection_008.png b/docs/imgs/Selection_008.png new file mode 100644 index 00000000..5bfc60eb Binary files /dev/null and b/docs/imgs/Selection_008.png differ diff --git a/docs/imgs/Selection_009.png b/docs/imgs/Selection_009.png new file mode 100644 index 00000000..5bb6ffc1 Binary files /dev/null and b/docs/imgs/Selection_009.png differ diff --git a/docs/imgs/Selection_010.png b/docs/imgs/Selection_010.png new file mode 100644 index 00000000..05406dbd Binary files /dev/null and b/docs/imgs/Selection_010.png differ diff --git a/docs/imgs/Selection_011.png b/docs/imgs/Selection_011.png new file mode 100644 index 00000000..73241164 Binary files /dev/null and b/docs/imgs/Selection_011.png differ diff --git a/docs/imgs/Selection_012.png b/docs/imgs/Selection_012.png new file mode 100644 index 00000000..d5c3ea74 Binary files /dev/null and b/docs/imgs/Selection_012.png differ diff --git a/docs/imgs/Selection_013.png b/docs/imgs/Selection_013.png new file mode 100644 index 00000000..3f0134dd Binary files /dev/null and b/docs/imgs/Selection_013.png differ diff --git a/docs/imgs/Selection_014.png b/docs/imgs/Selection_014.png new file mode 100644 index 00000000..fea2e6a2 Binary files /dev/null and b/docs/imgs/Selection_014.png differ diff --git a/docs/imgs/Selection_015.png b/docs/imgs/Selection_015.png new file mode 100644 index 00000000..03e0276e Binary files /dev/null and b/docs/imgs/Selection_015.png differ diff --git a/docs/imgs/Selection_016.png b/docs/imgs/Selection_016.png new file mode 100644 index 00000000..7aaff1dc Binary files /dev/null and b/docs/imgs/Selection_016.png differ diff --git a/docs/imgs/Selection_017.png b/docs/imgs/Selection_017.png new file mode 100644 index 00000000..62f0a8a0 Binary files /dev/null and b/docs/imgs/Selection_017.png differ diff --git a/docs/imgs/Selection_018.png b/docs/imgs/Selection_018.png new file mode 100644 index 00000000..0596e71e Binary files /dev/null and b/docs/imgs/Selection_018.png differ diff --git a/docs/imgs/Selection_019.png b/docs/imgs/Selection_019.png new file mode 100644 index 00000000..98320fd6 Binary files /dev/null and b/docs/imgs/Selection_019.png differ diff --git a/docs/imgs/Selection_020.png b/docs/imgs/Selection_020.png new file mode 100644 index 00000000..0de26558 Binary files /dev/null and b/docs/imgs/Selection_020.png differ diff --git a/docs/imgs/Selection_021.png b/docs/imgs/Selection_021.png new file mode 100644 index 00000000..1380d0ae Binary files /dev/null and b/docs/imgs/Selection_021.png differ diff --git a/docs/imgs/Selection_022.png b/docs/imgs/Selection_022.png new file mode 100644 index 00000000..a281a02e Binary files /dev/null and b/docs/imgs/Selection_022.png differ diff --git a/docs/imgs/Selection_023.png b/docs/imgs/Selection_023.png new file mode 100644 index 00000000..5c728fbe Binary files /dev/null and b/docs/imgs/Selection_023.png differ diff --git a/docs/imgs/Selection_024.png b/docs/imgs/Selection_024.png new file mode 100644 index 00000000..31d5ad83 Binary files /dev/null and b/docs/imgs/Selection_024.png differ diff --git a/docs/distributed_ms_diagram.PNG b/docs/imgs/distributed_ms_diagram.PNG similarity index 100% rename from docs/distributed_ms_diagram.PNG rename to docs/imgs/distributed_ms_diagram.PNG diff --git a/docs/web.md b/docs/web.md new file mode 100644 index 00000000..71f3bfee --- /dev/null +++ b/docs/web.md @@ -0,0 +1,108 @@ +# Web Interface # + +Submit Files for Analysis +------------------------- + +![MultiScanner Web Interface](imgs/Selection_001.png) + +When you visit MultiScanner's web interface in a web browser, you'll be greeted by the file submission page. Drag files onto the large drop area in the middle of the page or click it or the "Select File(s)..." button to select one or more files to be uploaded and analyzed. + +Click on the "Advanced Options" button to change default options and set metadata fields to be added to the scan results. + +![Advanced Options](imgs/Selection_003.png) + +Metadata fields can be added or removed by editing web_config.ini. Metadata field values can be set for individual files by clicking the small black button below and to the right of that filename in the staging area. + +![File Options](imgs/Selection_004.png) + +Change from "Scan" to "Import" to import JSON analysis reports into MultiScanner. This is intended only to be used with the JSON reports you can download from a report page in MultiScanner. + +![Import](imgs/Selection_005.png) + +By default, if you resubmit a sample that has already been submitted, MultiScanner will pull the latest report of that sample. If you want MultiScanner to re-scan the sample, set that option in Advanced Options. + +![Re-scan](imgs/Selection_006.png) + +If you have a directory of samples you wish to scan at once, we recommend zipping them and uploading the archive with the option to extract archives enabled. You can also specify a password, if the archive file is password- protected. Alternatively you can use the REST API for bulk uploads. + +![Archive Files](imgs/Selection_007.png) + +Click the "Scan it!" button to submit the sample to MultiScanner. + +![Scan It!](imgs/Selection_008.png) + +The progress bars that appear in the file staging area do not indicate the progress of the scan; a full bar merely indicates that the file has been uploaded to MultiScanner. Click on the file to go to its report page. + +![Submission Progress Bar](imgs/Selection_009.png) + +If the analysis has not completed yet, you'll see a "Pending" message. + +![Pending](imgs/Selection_010.png) + +Analyses and History Pages +-------------------------- + +Reports can be listed and searched in two different ways. The Analyses page lists the most recent report per sample. + +![Analyses Page](imgs/Selection_011.png) + +The History page lists every report of each sample. So if a file is scanned multiple times, it will only show up once on the Analyses page, but all of the reports will show up on the History page. + +![History Page](imgs/Selection_012.png) + +Both pages display the list of reports and allow you to search them. Click the blue button in the middle to refresh the list of reports. + +![Refresh Button](imgs/Selection_013.png) + +Click on a row in the list to go to that report, and click the red "X" button to delete that report from MultiScanner's Elasticsearch database. + +![Delete Button](imgs/Selection_014.png) + +Searching +--------- + +![Navbar Search](imgs/Selection_015.png) + +Reports can be searched from any page, with a few options. You can search Analyses to get the most recent scan per file, or search History to get all scans recorded for each file. Use the "Default" search type to have wildcards automatically appended to the beginning and end of your search term. Use the "Exact" search type to search automatically append quotes and search for the exact phrase. Finally, use the "Advanced" search type to search with the full power of Lucene query string syntax. Nothing will be automatically appended and you will need to escape any reserved characters yourself. When you click on one of the search results, the search term will be highlighted on the Report page and the report will be expanded and automatically scrolled to the first match. + +![Analyses/History Search](imgs/Selection_016.png) + +Report page +----------- + +![Report Page](imgs/Selection_017.png) + +Each report page displays the results of a single analysis. Some rows in the report can be expanded or collapsed to reveal more data by clicking on the row header or the "Expand" button. Shift-clicking will also expand or collapse all of it's child rows. + +![Expand Button](imgs/Selection_024.png) + +The "Expand All" button will expand all rows at once. If they are all expanded, this will turn into a "Collapse All" button that will collapse them all again. + +![Expand All Button](imgs/Selection_018.png) + +As reports can contain a great deal of content, you can search the report to find the exact data you are looking for with the search field located under the report title. The search term, if found, will be highlighted, the matching fields will be expanded, and the page automatically scrolled to the first match. + +![In-Page Search](imgs/Selection_019.png) + +Reports can be tagged by entering text in the Tags input box and hitting the enter key. As you type, a dropdown will appear with suggestions from the tags already in the system. It will pull the list of tags from existing reports, but a pre-populated list of tags can also be provided in web_config.ini when the web interface is set up. + +![Tags](imgs/Selection_020.png) + +You can download the report in a number of different formats using the Download button on the right side. You can download a JSON-formatted version of the report containing all the same data shown on the page. You can also download a MAEC-formatted version of the reports from Cuckoo Sandbox. Finally, you can also download the original sample file as a password-protected ZIP file. The password will be "infected". + +![Download](imgs/Selection_021.png) + +Click on "Notes" to open a sidebar where analysts may enter notes or comments. + +![Notes](imgs/Selection_022.png) + +These notes and comments can be edited and deleted. Click the "<" button to collapse this sidebar. + +![Close Notes](imgs/Selection_023.png) + +Analytics +--------- + +![Analytics Page](imgs/Selection_002.png) + +The Analytics page displays various pieces of advanced analysis. For now, this is limited to ssdeep comparisons. The table lists samples, with those that have very similar ssdeep hashes grouped together. Other analytics will be added in the future. For more information, see [this page](../docs/analytics.md). diff --git a/modules/Metadata/entropy.py b/modules/Metadata/entropy.py new file mode 100644 index 00000000..5fe24d20 --- /dev/null +++ b/modules/Metadata/entropy.py @@ -0,0 +1,37 @@ +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. +from __future__ import division, absolute_import, with_statement, print_function, unicode_literals +from collections import Counter +import math + + +__author__ = "Austin West" +__license__ = "MPL 2.0" + +TYPE = "Metadata" +NAME = "entropy" +DEFAULTCONF = { + 'ENABLED': True +} + + +def check(conf=DEFAULTCONF): + return True + + +def scan(filelist, conf=DEFAULTCONF): + '''Calculate entropy of a string''' + results = [] + for fname in filelist: + with open(fname, 'rb') as f: + text = f.read() + chars, lns = Counter(text), float(len(text)) + result = -sum(count/lns * math.log(count/lns, 2) for count in chars.values()) + results.append((fname, result)) + + metadata = {} + metadata["Name"] = NAME + metadata["Type"] = TYPE + metadata["Include"] = False + return (results, metadata) diff --git a/pdf_generator/__init__.py b/utils/__init__.py similarity index 100% rename from pdf_generator/__init__.py rename to utils/__init__.py diff --git a/utils/api.py b/utils/api.py index 0d4313cc..5dd82a3e 100755 --- a/utils/api.py +++ b/utils/api.py @@ -24,6 +24,7 @@ PUT /api/v1/tasks//notes/ ---> Edit a note DELETE /api/v1/tasks//notes/ ---> Delete a note GET /api/v1/tasks//report?d={t|f}---> receive report in JSON, set d=t to download +GET /api/v1/tasks//pdf ---> Receive PDF report POST /api/v1/tasks//tags ---> Add tags to task DELETE /api/v1/tasks//tags ---> Remove tags from task GET /api/v1/analytics/ssdeep_compare---> Run ssdeep.compare analytic @@ -56,8 +57,6 @@ from six import PY3 import rarfile import zipfile -from reportlab.platypus import TableStyle -from reportlab.lib import colors, units import requests MS_WD = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) @@ -74,7 +73,7 @@ import sql_driver as database import elasticsearch_storage import common -from pdf_generator.generic_pdf import GenericPDF +from utils.pdf_generator import create_pdf_document TASK_NOT_FOUND = {'Message': 'No task or report with that ID found!'} INVALID_REQUEST = {'Message': 'Invalid request parameters'} @@ -846,6 +845,7 @@ def files_get_sha256_helper(sha256, raw=None): response.headers['Content-Disposition'] = 'inline; filename={}.zip'.format(sha256) return response + @app.route('/api/v1/analytics/ssdeep_compare', methods=['GET']) def run_ssdeep_compare(): ''' @@ -865,10 +865,11 @@ def run_ssdeep_compare(): jsonify({'Message': 'Unable to complete request.'}), HTTP_BAD_REQUEST) + @app.route('/api/v1/analytics/ssdeep_group', methods=['GET']) def run_ssdeep_group(): ''' - Runs sssdeep group analytic and returns list of groups as a list. + Runs ssdeep group analytic and returns list of groups as a list. ''' try: ssdeep_analytic = SSDeepAnalytic() @@ -880,138 +881,23 @@ def run_ssdeep_group(): HTTP_BAD_REQUEST) -@app.route('/api/v1/tasks/report//pdf', methods=['GET']) +@app.route('/api/v1/tasks//pdf', methods=['GET']) def get_pdf_report(task_id): + ''' + Generates a PDF version of a JSON report. + ''' report_dict, success = get_report_dict(task_id) if not success: return jsonify(report_dict) - pdf = create_pdf_document(report_dict) + pdf = create_pdf_document(MS_WD, report_dict) response = make_response(pdf) response.headers['Content-Type'] = 'application/pdf' response.headers['Content-Disposition'] = 'attachment; filename=%s.pdf' % task_id return response -def create_pdf_document(report): - ''' - Method to create PDF report document from JSON. - ''' - with open(os.path.join(MS_WD, 'pdf_generator/pdf_config.json')) as data_file: - pdf_components = json.load(data_file) - - gen_pdf = GenericPDF(pdf_components) - gen_pdf.tlp_color = 'GREEN' - - notice = gen_pdf.section('Notification', pdf_components['notification'], gen_pdf.style) - - for n in notice: - gen_pdf.pdf_list.append(n) - - summary = gen_pdf.section('Summary', '', gen_pdf.style) - - for s in summary: - gen_pdf.pdf_list.append(s) - - summary_data = [ - ['Date Submitted', 'N\A'], - ['Artifact ID', 'N\A'], - ['Description', pdf_components['summary_description']], - ['Files Processed', '1'], - ['', report['Report']['filename']] - ] - - gen_pdf.vertical_table(summary_data) - - gen_pdf.line_break() - - file_and_obs = gen_pdf.section('File Indicators and Observables', '', gen_pdf.style) - - for f in file_and_obs: - gen_pdf.pdf_list.append(f) - - # This list will store data for table under File Indicators and Observables - file_data = [] - - # This list will store data for Yara results. Currently, extracts description of rule. This is a horizontal table. - yara_data = [['Yara Rule', 'Yara Rule Description']] - - # This list will store AV results. This is a horizontal table. - av_data = [['Antivirus', 'Scan Result']] - - if 'Report' in report: - r = report['Report'] - if 'filename' in r: - file_data.append(['File Name', r['filename']]) - if 'Scan Time' in r: - file_data.append(['Scan Time', r['Scan Time']]) - if 'libmagic' in r: - file_data.append(['Type', r['libmagic']]) - if 'MD5' in r: - file_data.append(['MD5', r['MD5']]) - if 'SHA1' in r: - file_data.append(['SHA1', r['SHA1']]) - if 'SHA256' in r: - file_data.append(['SHA256', r['SHA256']]) - if 'ssdeep' in r and 'ssdeep_hash' in r['ssdeep']: - file_data.append(['SSDEEP', r['ssdeep']['ssdeep_hash']]) - - if 'Yara' in r: - for v in r['Yara'].values(): - if 'meta' in v and 'description' in v['meta']: - yara_data.append([v['rule'], v['meta']['description']]) - elif 'meta' in v and 'description' not in v['meta']: - yara_data.append([v['rule'], "NO RULE DESCRIPTION"]) - - if 'AVG 2014' in r: - av_data.append(['AVG 2014', r['AVG 2014']]) - if 'Microsoft Security Essentials' in r: - av_data.append(['Microsoft Security Essentials', r['Microsoft Security Essentials']]) - - gen_pdf.vertical_table(file_data) - - gen_pdf.line_break() - - av_table_style = TableStyle([ - ('BACKGROUND', (0, 0), (-1, 0), colors.skyblue), - ]) - - gen_pdf.horizontal_table(av_data, av_table_style, (50 * units.mm, 140 * units.mm)) - - gen_pdf.line_break() - - yara_table_style = TableStyle([ - ('BACKGROUND', (0, 0), (-1, 0), colors.skyblue), - ]) - - gen_pdf.horizontal_table(yara_data, yara_table_style, (50 * units.mm, 140 * units.mm)) - - gen_pdf.line_break() - - mitigation_recommendation = gen_pdf.section('Mitigation Recommendations', pdf_components['mitigation_recommendations'], gen_pdf.style) - - for mr in mitigation_recommendation: - gen_pdf.pdf_list.append(mr) - - mitigation_bullets = gen_pdf.bullet_list(pdf_components['mitigation_bullet_list'], 1) - gen_pdf.pdf_list.append(mitigation_bullets) - - gen_pdf.line_break() - - contact = gen_pdf.section('Contact Information', pdf_components['contact_information'], gen_pdf.style) - - for c in contact: - gen_pdf.pdf_list.append(c) - - faq = gen_pdf.section('Document FAQ', pdf_components['document_faq'], gen_pdf.style) - - for f in faq: - gen_pdf.pdf_list.append(f) - - return gen_pdf.build() - - if __name__ == '__main__': if not os.path.isdir(api_config['api']['upload_folder']): diff --git a/utils/pdf_generator/__init__.py b/utils/pdf_generator/__init__.py new file mode 100644 index 00000000..f36354f8 --- /dev/null +++ b/utils/pdf_generator/__init__.py @@ -0,0 +1,127 @@ +from __future__ import division, absolute_import, with_statement, print_function, unicode_literals + +import json +import os + +from reportlab.lib import colors, units +from reportlab.platypus import TableStyle + +from utils.pdf_generator import generic_pdf + + +def create_pdf_document(WD, report): + ''' + Method to create a PDF report based of a multiscanner JSON report. + + Args: + WD: Represents the working directory of the multiscanner. + report: A JSON object. + + ''' + with open(os.path.join(WD, 'utils/pdf_generator/pdf_config.json')) as data_file: + pdf_components = json.load(data_file) + + gen_pdf = generic_pdf.GenericPDF(pdf_components) + + notice = [] + + if pdf_components.get('notification', ''): + notice = gen_pdf.section('Notification', pdf_components.get('notification'), gen_pdf.style) + gen_pdf.pdf_list.extend(notice) + + summary = gen_pdf.section('Summary', '', gen_pdf.style) + gen_pdf.pdf_list.extend(summary) + + summary_data = [ + ['Date Submitted', report.get('Report', {}).get('Scan Time', 'N\A')], + ['Artifact ID', report.get('Report', {}).get('SHA256', 'N\A')], + ['Description', pdf_components.get('summary_description', 'N\A')], + ['Files Processed', '1'], + ['', report.get('Report', {}).get('filename', 'NO FILENAME AVAILABLE')] + ] + + gen_pdf.vertical_table(summary_data) + + gen_pdf.line_break() + + file_and_obs = gen_pdf.section('File Indicators and Observables', '', gen_pdf.style) + gen_pdf.pdf_list.extend(file_and_obs) + + # This list will store data for table under File Indicators and Observables + file_data = [] + + # This list will store data for Yara results. Currently, extracts description of rule. This is a horizontal table. + yara_data = [['Yara Rule', 'Yara Rule Description']] + + # This list will store AV results. This is a horizontal table. + av_data = [['Antivirus', 'Scan Result']] + + if 'Report' in report: + r = report.get('Report', {}) + if 'filename' in r: + file_data.append(['File Name', r.get('filename', '')]) + if 'Scan Time' in r: + file_data.append(['Scan Time', r.get('Scan Time', '')]) + if 'libmagic' in r: + file_data.append(['Type', r.get('libmagic', '')]) + if 'MD5' in r: + file_data.append(['MD5', r.get('MD5', '')]) + if 'SHA1' in r: + file_data.append(['SHA1', r.get('SHA1', '')]) + if 'SHA256' in r: + file_data.append(['SHA256', r.get('SHA256', '')]) + if 'ssdeep' in r: + file_data.append(['SSDEEP', r.get('ssdeep', {}).get('ssdeep_hash', '')]) + + if 'Yara' in r: + for v in r.get('Yara', {}).values(): + if 'meta' in v: + yara_data.append([v.get('rule', 'NO RULE NAME'), + v.get('meta', {}).get('description', 'NO RULE DESCRIPTION')]) + if 'AVG 2014' in r: + av_data.append(['AVG 2014', r.get('AVG 2014', '')]) + if 'Microsoft Security Essentials' in r: + av_data.append(['Microsoft Security Essentials', r.get('Microsoft Security Essentials', '')]) + + if file_data: + gen_pdf.vertical_table(file_data) + + gen_pdf.line_break() + + av_table_style = TableStyle([ + ('BACKGROUND', (0, 0), (-1, 0), colors.skyblue), + ]) + + gen_pdf.horizontal_table(av_data, av_table_style, (50 * units.mm, 140 * units.mm)) + + gen_pdf.line_break() + + yara_table_style = TableStyle([ + ('BACKGROUND', (0, 0), (-1, 0), colors.skyblue), + ]) + + gen_pdf.horizontal_table(yara_data, yara_table_style, (50 * units.mm, 140 * units.mm)) + + gen_pdf.line_break() + + mitigation_recommendation = gen_pdf.section('Mitigation Recommendations', pdf_components.get('mitigation_recommendations', ''), gen_pdf.style) + + for mr in mitigation_recommendation: + gen_pdf.pdf_list.append(mr) + + mitigation_bullets = gen_pdf.bullet_list(pdf_components.get('mitigation_bullet_list', ''), 1) + gen_pdf.pdf_list.append(mitigation_bullets) + + gen_pdf.line_break() + + contact = gen_pdf.section('Contact Information', pdf_components.get('contact_information', ''), gen_pdf.style) + + for c in contact: + gen_pdf.pdf_list.append(c) + + faq = gen_pdf.section('Document FAQ', pdf_components.get('document_faq', ''), gen_pdf.style) + + for f in faq: + gen_pdf.pdf_list.append(f) + + return gen_pdf.build() diff --git a/pdf_generator/generic_pdf.py b/utils/pdf_generator/generic_pdf.py similarity index 82% rename from pdf_generator/generic_pdf.py rename to utils/pdf_generator/generic_pdf.py index 19e94442..3b789344 100644 --- a/pdf_generator/generic_pdf.py +++ b/utils/pdf_generator/generic_pdf.py @@ -1,3 +1,5 @@ +from __future__ import division, absolute_import, with_statement, print_function, unicode_literals + import cgi import six @@ -8,7 +10,8 @@ from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle from reportlab.lib.units import inch, mm from reportlab.pdfgen import canvas -from reportlab.platypus import SimpleDocTemplate, Spacer, Image, Paragraph, ListFlowable, ListItem, TableStyle, Table +from reportlab.platypus import (SimpleDocTemplate, Spacer, Image, Paragraph, + ListFlowable, ListItem, TableStyle, Table) class NumberedCanvas(canvas.Canvas): @@ -29,9 +32,9 @@ def save(self): canvas.Canvas.save(self) def draw_page_number(self, page_count): - self.setFont("Helvetica-Bold", 7) + self.setFont('Helvetica-Bold', 7) self.drawRightString(203*mm, 12.7*mm, - "Page %d of %d" % (self._pageNumber, page_count)) + 'Page %d of %d' % (self._pageNumber, page_count)) class GenericPDF(object): @@ -49,14 +52,16 @@ def __init__(self, pdf_components): self.style.add(ParagraphStyle(name='bullet_list', parent=self.style['Normal'], fontSize=11)) - - self.buffer = six.StringIO() + if six.PY3: + self.buffer = six.BytesIO() + else: + self.buffer = six.StringIO() self.firstPage = True self.document = SimpleDocTemplate(self.buffer, pagesize=letter, rightMargin=12.7*mm, leftMargin=12.7*mm, topMargin=120, bottomMargin=80) - self.tlp_color = None + self.tlp_color = pdf_components.get('tlp_color', '') self.pdf_components = pdf_components self.pdf_list = [] @@ -68,21 +73,21 @@ def header_footer(self, canvas, doc): height_adjust = self.add_banner(canvas, doc) # Document Header - if self.pdf_components['hdr_image'] and self.firstPage: - header = Image(self.pdf_components['hdr_image'], height=25*mm, width=191*mm) + if self.pdf_components.get('hdr_image', None) and self.firstPage: + header = Image(self.pdf_components.get('hdr_image'), height=25*mm, width=191*mm) header.drawOn(canvas, doc.rightMargin, doc.height + doc.topMargin - 15*mm) self.firstPage = False elif self.firstPage: - header = Paragraph(self.pdf_components['hdr_html'], self.style['centered']) + header = Paragraph(self.pdf_components.get('hdr_html', ''), self.style['centered']) w, h = header.wrap(doc.width, doc.topMargin) header.drawOn(canvas, doc.leftMargin, doc.height + doc.topMargin - height_adjust * h) # Document Footer - if self.pdf_components['ftr_image']: - footer = Image(self.pdf_components['ftr_image'], 8.5 * inch, 1.8 * inch) + if self.pdf_components.get('ftr_image', None): + footer = Image(self.pdf_components.get('ftr_image'), 8.5 * inch, 1.8 * inch) footer.drawOn(canvas, 0, 0) else: - footer = Paragraph(self.pdf_components['ftr_html'], self.style['centered']) + footer = Paragraph(self.pdf_components.get('ftr_html', ''), self.style['centered']) w, h = footer.wrap(doc.width, doc.bottomMargin) footer.drawOn(canvas, doc.leftMargin, height_adjust * h) @@ -108,7 +113,7 @@ def add_banner(self, canvas, doc): textTransform='uppercase', alignment=TA_RIGHT)) - banner = Paragraph(self.span_text(self.bold_text('TLP:' + self.tlp_color), bgcolor="black"), self.style['banner_style']) + banner = Paragraph(self.span_text(self.bold_text('TLP:' + self.tlp_color), bgcolor='black'), self.style['banner_style']) w, h = banner.wrap(doc.width, doc.topMargin) banner.drawOn(canvas, doc.leftMargin, doc.height + doc.topMargin + (h + 12*mm)) w, h = banner.wrap(doc.width, doc.bottomMargin) @@ -158,8 +163,8 @@ def bullet_list(self, body, level): def vertical_table(self, data, table_style=None, col_widths=None): '''A table where the first column is bold. A label followed by values.''' - self.style["BodyText"].wordWrap = 'LTR' - self.style["BodyText"].spaceBefore = 2 + self.style['BodyText'].wordWrap = 'LTR' + self.style['BodyText'].spaceBefore = 2 if table_style: style = table_style @@ -175,8 +180,8 @@ def vertical_table(self, data, table_style=None, col_widths=None): else: cols = (35*mm, 140*mm) - data2 = [[Paragraph(self.bold_text(cell), self.style["BodyText"]) if idx == 0 - else Paragraph(cell, self.style["BodyText"]) + data2 = [[Paragraph(self.bold_text(cell), self.style['BodyText']) if idx == 0 + else Paragraph(cell, self.style['BodyText']) for idx, cell in enumerate(row)] for row in data] table = Table(data2, style=style, colWidths=cols) @@ -184,8 +189,8 @@ def vertical_table(self, data, table_style=None, col_widths=None): def horizontal_table(self, data, table_style=None, col_widths=None): '''A table where the first row is bold. The first row are labels, the rest values.''' - self.style["BodyText"].wordWrap = 'LTR' - self.style["BodyText"].spaceBefore = 2 + self.style['BodyText'].wordWrap = 'LTR' + self.style['BodyText'].spaceBefore = 2 if table_style: style = table_style @@ -201,8 +206,8 @@ def horizontal_table(self, data, table_style=None, col_widths=None): else: cols = (35*mm, 140*mm) - data2 = [[Paragraph(self.bold_text(cell), self.style["BodyText"]) if idx == 0 - else Paragraph(cell, self.style["BodyText"]) + data2 = [[Paragraph(self.bold_text(cell), self.style['BodyText']) if idx == 0 + else Paragraph(cell, self.style['BodyText']) for cell in row] for idx, row in enumerate(data)] table = Table(data2, style=style, colWidths=cols) diff --git a/pdf_generator/pdf_config.json b/utils/pdf_generator/pdf_config.json similarity index 92% rename from pdf_generator/pdf_config.json rename to utils/pdf_generator/pdf_config.json index 3fbfd1a7..31222d6e 100644 --- a/pdf_generator/pdf_config.json +++ b/utils/pdf_generator/pdf_config.json @@ -3,6 +3,7 @@ "hdr_image": "", "ftr_html": "", "ftr_image": "", + "tlp_color": "", "notification": "", "mitigation_recommendations": "", "mitigation_bullet_list": "", diff --git a/web/templates/layout.html b/web/templates/layout.html index fdbb900b..99170752 100644 --- a/web/templates/layout.html +++ b/web/templates/layout.html @@ -124,7 +124,7 @@

Report Page

Sample Download

Samples are downloaded as password-protected ZIP files. The password is infected.

Search

-

Reports can be searched from any page, with a few options. You can search Analyses to get the most recent scan per file, or search History to get all scans recorded for each file. Use the "Default "search type to have wildcards automatically appended to the beginning and end of your search term. Use the "Exact" search type to search automatically append quotes and search for the exact phrase. Finally, use the "Advanced" search type to search with the full power of Elasticsearch's query string syntax. Nothing will be automatically appended. When you click on one of the search results, the search term will be highlighted on the Report page.

+

Reports can be searched from any page, with a few options. You can search Analyses to get the most recent scan per file, or search History to get all scans recorded for each file. Use the "Default" search type to have wildcards automatically appended to the beginning and end of your search term. Use the "Exact" search type to search automatically append quotes and search for the exact phrase. Finally, use the "Advanced" search type to search with the full power of Lucene query string syntax. Nothing will be automatically appended and you will need to escape any reserved characters yourself. When you click on one of the search results, the search term will be highlighted on the Report page.


MultiScanner is copyright The MITRE Corporation, licensed under the Mozilla Public License, version 2.0.

diff --git a/web/templates/report.html b/web/templates/report.html index 624183c5..cabd54b2 100644 --- a/web/templates/report.html +++ b/web/templates/report.html @@ -495,6 +495,8 @@ window.location = "{{ api_loc }}/api/v1/tasks/{{ task_id }}/file?raw=false"; } else if (this.id == 'getMAEC') { window.location = "{{ api_loc }}/api/v1/tasks/{{ task_id }}/maec"; + } else if (this.id == 'getPDF') { + window.location = "{{ api_loc }}/api/v1/tasks/{{ task_id }}/pdf"; } }); }); @@ -534,6 +536,7 @@

Task {{ task_id }} Report

  • JSON
  • Sample
  • MAEC
    • +
  • PDF