ci: merge from main for next release (#5605)

* feat: Easy extraction of qualified volunteer list for nomcom chair (#5578)

* feat: Easy extraction of qualified volunteer list for nomcom chair

* fix: tune test setup to years where eligibility calculations can return nonempty

* chore: revert unintended change

* feat: default string when no affiliation is provided

* chore: pin django-oidc-provider (#5588)

* fix: add a link to the simplified volunteer view (#5583)

* chore: add bibxml-ids dir to container build. (#5590)

* chore: add bibxml-ids dir to container build. (#5590)

* feat: Process uploaded submissions asynchronously (#5580)

* fix: Use relative URL for submission status link

* refactor: Refactor/rename process_uploaded_submission async task

* feat: Add async task to process but not accept a submission

* feat: Replace upload_submission() with an async implementation (WIP)

* fix: Do not put Submission in "uploaded" state if an error occured

* refactor: Improve text/XML draft processing flow

* feat: Extract authors from text in async processing

* fix: Fix call signatures and abort submission on failed validation

* feat: Validate submission name format

* fix: Correctly validate emails from text submission

* fix: Clean up submission validation

* fix: Better display errors on upload_submission page

* feat: Reload submission status page when awaiting validation

* test: Fix call signatures; remove unused imports

* chore: Add type hint

* test: Update tests to match renamed task

* fix: Fix typo in error message

* test: Fix failing Api- and AsyncSubmissionTests

* Rename process_uploaded_submission to process_and_accept_...
* Remove outdated tests

Does not yet test new behavior.

* refactor: Break up submission_file() helper

* test: Refactor tests to run the async processing (wip)

* test: Drop test of bad PDF submission

The PDF submission field was removed, so no need to test it.

* test: Update more tests

* test: Bring back create_and_post_submission() and fix more tests

* fix: Drop to manual, don't cancel, on revision inconsistency

Fixes remaining failing SubmitTest tests

* style: Restyle upload_submission() with black

* test: Verify that async submission processing is invoked on upload

* test: Bring back old do_submission and fix tests

Properly separating the upload and async processing stages of submission
is a bigger refactoring than will fit right now. This better exercises
the submission pipeline.

* fix: Accept only XML for API submissions

* test: Test submission processing utilities

* feat: Improve status display for "validating" submissions

* chore: Remove obsolete code

* test: Update test to match amended text

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>

* chore(deps): update all Yarn dependencies (#5564)

* chore(deps): update all Yarn dependencies

* chore: fix yarn cache

---------

Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Co-authored-by: NGPixel <github@ngpixel.com>

* chore(deps): update all npm dependencies for dev/deploy-to-container (#5587)

Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>

* chore: Remove mysqlclient dependency (#5589)

* fix: close open things (#5593)

* fix: close open things

* fix: clean up test created files

* fix: remove one close too many

* chore: add git safe directory to docker init script

* chore: move git safe directory command to top

* chore: remove debugging file write from test (#5598)

---------

Co-authored-by: NGPixel <github@ngpixel.com>
Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>

dev/deploy-to-container/package.json

@@ -8,7 +8,7 @@
     "nanoid-dictionary": "5.0.0-beta.1",
     "slugify": "1.6.6",
     "tar": "^6.1.13",
-    "yargs": "^17.7.1"
+    "yargs": "^17.7.2"
   },
   "engines": {
     "node": ">=16"

dev/deploy-to-container/settings_local.py

@@ -23,9 +23,6 @@
 IDSUBMIT_IDNITS_BINARY = "/usr/local/bin/idnits"
 IDSUBMIT_REPOSITORY_PATH = "/test/id/"
 IDSUBMIT_STAGING_PATH = "/test/staging/"
-INTERNET_DRAFT_ARCHIVE_DIR = "/test/archive/"
-INTERNET_ALL_DRAFTS_ARCHIVE_DIR = "/test/archive/"
-RFC_PATH = "/test/rfc/"
 
 AGENDA_PATH = '/assets/www6s/proceedings/'
 MEETINGHOST_LOGO_PATH = AGENDA_PATH
@@ -65,6 +62,7 @@
 STATUS_CHANGE_PATH = '/assets/ietf-ftp/status-changes/'
 INTERNET_DRAFT_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
 INTERNET_ALL_DRAFTS_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
+BIBXML_BASE_PATH = '/assets/ietfdata/derived/bibxml'
 
 NOMCOM_PUBLIC_KEYS_DIR = 'data/nomcom_keys/public_keys/'
 SLIDE_STAGING_PATH = '/test/staging/'

dev/diff/settings_local.py

@@ -20,9 +20,6 @@
 IDSUBMIT_IDNITS_BINARY = "/usr/local/bin/idnits"
 IDSUBMIT_REPOSITORY_PATH = "test/id/"
 IDSUBMIT_STAGING_PATH = "test/staging/"
-INTERNET_DRAFT_ARCHIVE_DIR = "test/archive/"
-INTERNET_ALL_DRAFTS_ARCHIVE_DIR = "test/archive/"
-RFC_PATH = "test/rfc/"
 
 AGENDA_PATH = '/assets/www6s/proceedings/'
 MEETINGHOST_LOGO_PATH = AGENDA_PATH
@@ -62,6 +59,7 @@
 STATUS_CHANGE_PATH = '/assets/ietf-ftp/status-changes/'
 INTERNET_DRAFT_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
 INTERNET_ALL_DRAFTS_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
+BIBXML_BASE_PATH = '/assets/ietfdata/derived/bibxml'
 
 NOMCOM_PUBLIC_KEYS_DIR = 'data/nomcom_keys/public_keys/'
 SLIDE_STAGING_PATH = 'test/staging/'

dev/tests/settings_local.py

@@ -19,9 +19,6 @@
 IDSUBMIT_IDNITS_BINARY = "/usr/local/bin/idnits"
 IDSUBMIT_REPOSITORY_PATH = "test/id/"
 IDSUBMIT_STAGING_PATH = "test/staging/"
-INTERNET_DRAFT_ARCHIVE_DIR = "test/archive/"
-INTERNET_ALL_DRAFTS_ARCHIVE_DIR = "test/archive/"
-RFC_PATH = "test/rfc/"
 
 AGENDA_PATH = '/assets/www6s/proceedings/'
 MEETINGHOST_LOGO_PATH = AGENDA_PATH
@@ -61,6 +58,7 @@
 STATUS_CHANGE_PATH = '/assets/ietf-ftp/status-changes/'
 INTERNET_DRAFT_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
 INTERNET_ALL_DRAFTS_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
+BIBXML_BASE_PATH = '/assets/ietfdata/derived/bibxml'
 
 NOMCOM_PUBLIC_KEYS_DIR = 'data/nomcom_keys/public_keys/'
 SLIDE_STAGING_PATH = 'test/staging/'

docker/configs/settings_local.py

@@ -10,9 +10,6 @@
 IDSUBMIT_IDNITS_BINARY = "/usr/local/bin/idnits"
 IDSUBMIT_REPOSITORY_PATH = "test/id/"
 IDSUBMIT_STAGING_PATH = "test/staging/"
-INTERNET_DRAFT_ARCHIVE_DIR = "test/archive/"
-INTERNET_ALL_DRAFTS_ARCHIVE_DIR = "test/archive/"
-RFC_PATH = "test/rfc/"
 
 AGENDA_PATH = '/assets/www6s/proceedings/'
 MEETINGHOST_LOGO_PATH = AGENDA_PATH
@@ -52,6 +49,7 @@
 STATUS_CHANGE_PATH = '/assets/ietf-ftp/status-changes/'
 INTERNET_DRAFT_ARCHIVE_DIR = '/assets/archive/id'
 INTERNET_ALL_DRAFTS_ARCHIVE_DIR = '/assets/archive/id'
+BIBXML_BASE_PATH = '/assets/ietfdata/derived/bibxml'
 
 NOMCOM_PUBLIC_KEYS_DIR = 'data/nomcom_keys/public_keys/'
 SLIDE_STAGING_PATH = 'test/staging/'

docker/scripts/app-create-dirs.sh

@@ -21,6 +21,10 @@ for sub in \
     /assets/ietf-ftp/yang/ianamod \
     /assets/ietf-ftp/yang/invalmod \
     /assets/ietf-ftp/yang/rfcmod \
+    /assets/ietfdata \
+    /assets/ietfdata/derived \
+    /assets/ietfdata/derived/bibxml \
+    /assets/ietfdata/derived/bibxml/bibxml-ids \
     /assets/www6s \
     /assets/www6s/staging \
     /assets/www6s/wg-descriptions \

docker/scripts/app-init.sh

@@ -4,6 +4,9 @@ WORKSPACEDIR="/workspace"
 
 sudo service rsyslog start &>/dev/null
 
+# Add /workspace as a safe git directory
+git config --global --add safe.directory /workspace
+
 # Turn off git info in zsh prompt (causes slowdowns)
 git config oh-my-zsh.hide-info 1
 

ietf/doc/tests_bofreq.py

@@ -320,22 +320,26 @@ def test_submit(self):
             file = NamedTemporaryFile(delete=False,mode="w+",encoding='utf-8')
             file.write(f'# {username}')
             file.close()
-            for postdict in [
-                        {'bofreq_submission':'enter','bofreq_content':f'# {username}'},
-                        {'bofreq_submission':'upload','bofreq_file':open(file.name,'rb')},
-                     ]:
-                docevent_count = doc.docevent_set.count()
-                empty_outbox()
-                r = self.client.post(url, postdict)
-                self.assertEqual(r.status_code, 302)
-                doc = reload_db_objects(doc)
-                self.assertEqual('%02d'%(int(rev)+1) ,doc.rev)
-                self.assertEqual(f'# {username}', doc.text())
-                self.assertEqual(docevent_count+1, doc.docevent_set.count())
-                self.assertEqual(1, len(outbox))
-                rev = doc.rev
+            try:
+                with open(file.name, 'rb') as bofreq_fd:
+                    for postdict in [
+                                {'bofreq_submission':'enter','bofreq_content':f'# {username}'},
+                                {'bofreq_submission':'upload','bofreq_file':bofreq_fd},
+                            ]:
+                        docevent_count = doc.docevent_set.count()
+                        empty_outbox()
+                        r = self.client.post(url, postdict)
+                        self.assertEqual(r.status_code, 302)
+                        doc = reload_db_objects(doc)
+                        self.assertEqual('%02d'%(int(rev)+1) ,doc.rev)
+                        self.assertEqual(f'# {username}', doc.text())
+                        self.assertEqual(docevent_count+1, doc.docevent_set.count())
+                        self.assertEqual(1, len(outbox))
+                        rev = doc.rev
+            finally:
+                os.unlink(file.name)
+
             self.client.logout()
-            os.unlink(file.name)
 
     def test_start_new_bofreq(self):
         url = urlreverse('ietf.doc.views_bofreq.new_bof_request')
@@ -350,25 +354,28 @@ def test_start_new_bofreq(self):
         file = NamedTemporaryFile(delete=False,mode="w+",encoding='utf-8')
         file.write('some stuff')
         file.close()
-        for postdict in [
-                            dict(title='title one', bofreq_submission='enter', bofreq_content='some stuff'),
-                            dict(title='title two', bofreq_submission='upload', bofreq_file=open(file.name,'rb')),
-                        ]:
-            empty_outbox()
-            r = self.client.post(url, postdict)
-            self.assertEqual(r.status_code,302)
-            name = f"bofreq-{xslugify(nobody.last_name())[:64]}-{postdict['title']}".replace(' ','-')
-            bofreq = Document.objects.filter(name=name,type_id='bofreq').first()
-            self.assertIsNotNone(bofreq)
-            self.assertIsNotNone(DocAlias.objects.filter(name=name).first())
-            self.assertEqual(bofreq.title, postdict['title'])
-            self.assertEqual(bofreq.rev, '00')
-            self.assertEqual(bofreq.get_state_slug(), 'proposed')
-            self.assertEqual(list(bofreq_editors(bofreq)), [nobody])
-            self.assertEqual(bofreq.latest_event(NewRevisionDocEvent).rev, '00')
-            self.assertEqual(bofreq.text_or_error(), 'some stuff')
-            self.assertEqual(len(outbox),1)
-        os.unlink(file.name)
+        try:
+            with open(file.name,'rb') as bofreq_fd:
+                for postdict in [
+                                    dict(title='title one', bofreq_submission='enter', bofreq_content='some stuff'),
+                                    dict(title='title two', bofreq_submission='upload', bofreq_file=bofreq_fd),
+                                ]:
+                    empty_outbox()
+                    r = self.client.post(url, postdict)
+                    self.assertEqual(r.status_code,302)
+                    name = f"bofreq-{xslugify(nobody.last_name())[:64]}-{postdict['title']}".replace(' ','-')
+                    bofreq = Document.objects.filter(name=name,type_id='bofreq').first()
+                    self.assertIsNotNone(bofreq)
+                    self.assertIsNotNone(DocAlias.objects.filter(name=name).first())
+                    self.assertEqual(bofreq.title, postdict['title'])
+                    self.assertEqual(bofreq.rev, '00')
+                    self.assertEqual(bofreq.get_state_slug(), 'proposed')
+                    self.assertEqual(list(bofreq_editors(bofreq)), [nobody])
+                    self.assertEqual(bofreq.latest_event(NewRevisionDocEvent).rev, '00')
+                    self.assertEqual(bofreq.text_or_error(), 'some stuff')
+                    self.assertEqual(len(outbox),1)
+        finally:
+            os.unlink(file.name)
         existing_bofreq = BofreqFactory(requester_lastname=nobody.last_name())
         for postdict in [
                             dict(title='', bofreq_submission='enter', bofreq_content='some stuff'),

ietf/doc/tests_review.py

@@ -499,6 +499,8 @@ def make_test_mbox_tarball(self, review_req):
 
                 tar.add(os.path.relpath(tmp.name))
 
+                mbox.close()
+
         return mbox_path
 
     def test_search_mail_archive(self):

ietf/ipr/management/commands/process_email.py

@@ -23,8 +23,12 @@ def add_arguments(self, parser):
 
     def handle(self, *args, **options):
         email = options.get('email', None)
-        binary_input = io.open(email, 'rb') if email else sys.stdin.buffer
-        self.msg_bytes = binary_input.read()
+        if email:
+            binary_input = io.open(email, 'rb')
+            self.msg_bytes = binary_input.read()
+            binary_input.close()
+        else:
+            self.msg_bytes = sys.stdin.buffer.read()
         try:
             process_response_email(self.msg_bytes)
         except ValueError as e:
@@ -44,4 +48,4 @@ def make_failure_message(self, error, **extra):
                 'application', 'octet-stream',  # mime type
                 filename='original-message',
             )
-        return msg
+        return msg

ietf/meeting/tests_views.py

@@ -355,8 +355,7 @@ def test_meeting_agenda(self):
         # iCal, no session filtering
         ical_url = urlreverse("ietf.meeting.views.agenda_ical", kwargs=dict(num=meeting.number))
         r = self.client.get(ical_url)
-        with open('./ical-output.ics', 'w') as f:
-            f.write(r.content.decode())
+
         assert_ical_response_is_valid(self, r)
         self.assertContains(r, "BEGIN:VTIMEZONE")
         self.assertContains(r, "END:VTIMEZONE")
@@ -894,23 +893,27 @@ def build_session_setup(self):
 
     def test_session_draft_tarfile(self):
         session, filenames = self.build_session_setup()
-        url = urlreverse('ietf.meeting.views.session_draft_tarfile', kwargs={'num':session.meeting.number,'acronym':session.group.acronym})
-        response = self.client.get(url)
-        self.assertEqual(response.status_code, 200)
-        self.assertEqual(response.get('Content-Type'), 'application/octet-stream')
-        for filename in filenames:
-            os.unlink(filename)
+        try:
+            url = urlreverse('ietf.meeting.views.session_draft_tarfile', kwargs={'num':session.meeting.number,'acronym':session.group.acronym})
+            response = self.client.get(url)
+            self.assertEqual(response.status_code, 200)
+            self.assertEqual(response.get('Content-Type'), 'application/octet-stream')
+        finally:
+            for filename in filenames:
+                os.unlink(filename)
 
     @skipIf(skip_pdf_tests, skip_message)
     @skip_coverage
     def test_session_draft_pdf(self):
         session, filenames = self.build_session_setup()
-        url = urlreverse('ietf.meeting.views.session_draft_pdf', kwargs={'num':session.meeting.number,'acronym':session.group.acronym})
-        response = self.client.get(url)
-        self.assertEqual(response.status_code, 200)
-        self.assertEqual(response.get('Content-Type'), 'application/pdf')
-        for filename in filenames:
-            os.unlink(filename)
+        try:
+            url = urlreverse('ietf.meeting.views.session_draft_pdf', kwargs={'num':session.meeting.number,'acronym':session.group.acronym})
+            response = self.client.get(url)
+            self.assertEqual(response.status_code, 200)
+            self.assertEqual(response.get('Content-Type'), 'application/pdf')
+        finally:
+            for filename in filenames:
+                os.unlink(filename)
 
     def test_current_materials(self):
         url = urlreverse('ietf.meeting.views.current_materials')
@@ -6411,7 +6414,9 @@ def test_submit_and_approve_multiple_versions(self):
         path = os.path.join(submission.session.meeting.get_materials_path(),'slides')
         filename = os.path.join(path,session.sessionpresentation_set.first().document.name+'-01.txt')
         self.assertTrue(os.path.exists(filename))
-        contents = io.open(filename,'r').read()
+        fd = io.open(filename, 'r')
+        contents = fd.read()
+        fd.close()
         self.assertIn('third version', contents)
 
 
@@ -7946,12 +7951,13 @@ def test_add_proceedings_material_doc(self):
         """Upload proceedings materials document"""
         meeting = self._procmat_test_meeting()
         for mat_type in ProceedingsMaterialTypeName.objects.filter(used=True):
-            mat = self.upload_proceedings_material_test(
-                meeting,
-                mat_type,
-                {'file': self._proceedings_file(), 'external_url': ''},
-            )
-            self.assertEqual(mat.get_href(), f'{mat.document.name}:00')
+            with self._proceedings_file() as fd:
+                mat = self.upload_proceedings_material_test(
+                    meeting,
+                    mat_type,
+                    {'file': fd, 'external_url': ''},
+                )
+                self.assertEqual(mat.get_href(), f'{mat.document.name}:00')
 
     def test_add_proceedings_material_doc_invalid_ext(self):
         """Upload proceedings materials document with disallowed extension"""
@@ -8038,12 +8044,13 @@ def test_replace_proceedings_material(self):
             kwargs=dict(num=meeting.number, material_type=pm_doc.type.slug),
         )
         self.client.login(username='secretary', password='secretary+password')
-        r = self.client.post(pm_doc_url, {'file': self._proceedings_file(), 'external_url': ''})
-        self.assertRedirects(r, success_url)
-        self.assertEqual(meeting.proceedings_materials.count(), 2)
-        pm_doc = meeting.proceedings_materials.get(pk=pm_doc.pk)  # refresh from DB
-        self.assertEqual(pm_doc.document.rev, '01')
-        self.assertEqual(pm_doc.get_href(), f'{pm_doc.document.name}:01')
+        with self._proceedings_file() as fd:
+            r = self.client.post(pm_doc_url, {'file': fd, 'external_url': ''})
+            self.assertRedirects(r, success_url)
+            self.assertEqual(meeting.proceedings_materials.count(), 2)
+            pm_doc = meeting.proceedings_materials.get(pk=pm_doc.pk)  # refresh from DB
+            self.assertEqual(pm_doc.document.rev, '01')
+            self.assertEqual(pm_doc.get_href(), f'{pm_doc.document.name}:01')
 
         # Replace the uploaded document with a URL
         r = self.client.post(pm_doc_url, {'use_url': 'on', 'external_url': 'https://example.com/second'})
@@ -8066,12 +8073,13 @@ def test_replace_proceedings_material(self):
         self.assertEqual(pm_url.get_href(), 'https://example.com/third')
 
         # Now replace the URL doc with an uploaded file
-        r = self.client.post(pm_url_url, {'file': self._proceedings_file(), 'external_url': ''})
-        self.assertRedirects(r, success_url)
-        self.assertEqual(meeting.proceedings_materials.count(), 2)
-        pm_url = meeting.proceedings_materials.get(pk=pm_url.pk)  # refresh from DB
-        self.assertEqual(pm_url.document.rev, '02')
-        self.assertEqual(pm_url.get_href(), f'{pm_url.document.name}:02')
+        with self._proceedings_file() as fd:
+            r = self.client.post(pm_url_url, {'file': fd, 'external_url': ''})
+            self.assertRedirects(r, success_url)
+            self.assertEqual(meeting.proceedings_materials.count(), 2)
+            pm_url = meeting.proceedings_materials.get(pk=pm_url.pk)  # refresh from DB
+            self.assertEqual(pm_url.document.rev, '02')
+            self.assertEqual(pm_url.get_href(), f'{pm_url.document.name}:02')
 
     def test_remove_proceedings_material(self):
         """Proceedings material can be removed"""

ietf/nomcom/management/commands/feedback_email.py

@@ -42,8 +42,11 @@ def handle(self, *args, **options):
         except NomCom.DoesNotExist:
             raise CommandError("NomCom %s does not exist or it isn't active" % year)
 
-        binary_input = io.open(email, 'rb') if email else sys.stdin.buffer
-        self.msg = binary_input.read()
+        if email:
+            with io.open(email, 'rb') as binary_input:
+                self.msg = binary_input.read()
+        else:
+            self.msg = sys.stdin.buffer.read()
 
         try:
             feedback = create_feedback_email(self.nomcom, self.msg)