Skip to content

Commit

Permalink
[Documentation:Developer] update GSoC 2024 project ideas (#580)
Browse files Browse the repository at this point in the history 
Co-authored-by: Barb Cutler <Barb Cutler>
  • Loading branch information
@bmcutler
bmcutler authored Feb 5, 2024
1 parent 8bd6c22 commit 0f87cde
Showing 1 changed file with 214 additions and 81 deletions.
295 changes: 214 additions & 81 deletions _docs/developer/getting_started/project_ideas.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,127 +47,260 @@ developers.
new graders and some of our TA grading features are not adequately
tested by automated unit and end-to-end (Cypress) regression testing.

[Open Issues related to TA Grading](https://github.com/Submitty/Submitty/issues?q=is%3Aopen+is%3Aissue+label%3A%22TA+Grading+%2F+TA+UI%22)

_Expected Outcomes_: The goals of a project on TA grading would
include expantion of our automated testing of the TA Grading pages,
and to patch bugs uncovered by this improved testing. The project may be
expanded in scope to additionally propose and execute small or modest user
interface revisions that enhance the TA experience.
[Open Issues related to TA Grading](https://github.com/Submitty/Submitty/issues?q=is%3Aopen+is%3Aissue+label%3A%22TA+Grading+%2F+TA+UI%22)
[Open Issues related to Sample Data](https://github.com/Submitty/Submitty/issues?q=is%3Aopen+is%3Aissue+label%3A%22Sample+Data%22)

_Expected Outcomes_: The primary goals for this project include the
expansion of our automated testing of the TA Grading pages and to
patch bugs uncovered by this improved testing. The project may be
expanded in scope to additionally propose and execute small or
modest user interface revisions that enhance the TA experience,
especially for graders who are new to the interface and grading
process.

_Skills & Experience Required_: Some programming experience,
willingness to learn web and database development and
the Cypress end-to-end automated testing framework. Having served
as a teaching assistant or instructor with grading experience
design will be beneficial.
willingness to learn web and database development and the Cypress
end-to-end automated testing framework. Having served as a
teaching assistant with grading experience design will be
beneficial.

_Possible Mentors_: Barb Cutler, Nia Heermance
_Possible Mentors_: William Allen, Nia Heermance, Barb Cutler

_GSoC Project Size_: 90 or 175 hours

_Difficulty Level_: medium
_Difficulty Level_: introductory to medium


2. **Refactor and Performance Improvements for the Manual/TA Rubric Grading Interface**

[Overview of Rubric Grading Interface](/grader/rubric_grading/index)

Note: This project may be combined with the previous project idea.
The Manual/TA rubric grading interface is elaborate, highly-featured, and
customizable; however, the performance of these webpages is
problematic for large courses due to inefficient database queries and
delays to load data from the server that could/should be asynchronous.
The manual/TA rubric pages could benefit from a significant
technology refactor to use Vue/Vite, for example.

[Open Issues related to TA Grading](https://github.com/Submitty/Submitty/issues?q=is%3Aopen+is%3Aissue+label%3A%22TA+Grading+%2F+TA+UI%22)
_Note: This project may be combined with the previous project idea._

_Expected Outcomes_: A detailed software design for an organized, multi-stage
refactor of the manual/TA rubric pages and the execution/implementation of
a significant portion of the new design. The project will likely require the
extension and/or updating of our automated end-to-end (Cypress) testing and patching
bugs uncovered by this testing. The general interface for TA/Manual grading
should remain similar, but the project may include small user
interface revisions.
The Manual/TA rubric grading interface is elaborate,
highly-featured, and customizable; however, the performance of
these webpages is problematic for large courses due to inefficient
database queries and server communication delays to load data that
could/should be asynchronous. The manual/TA rubric pages could
benefit from a significant technology refactor to use Vue/Vite, for
example.

_Skills & Experience Required_: Web and database development experience.
Experience with end-to-end automated testing (Cypress) and and having served
as a teaching assistant or instructor with grading experience
design is beneficial but not required.
[Open Issues related to TA Grading](https://github.com/Submitty/Submitty/issues?q=is%3Aopen+is%3Aissue+label%3A%22TA+Grading+%2F+TA+UI%22)

_Possible Mentors_: Barb Cutler, Nia Heermance
_Expected Outcomes_: This project would first prepare a detailed
software design plan for an organized, multi-stage incremental
refactor of the manual/TA rubric pages and follow with the
execution/implementation of a significant portion of the new
design. The project could include the extension and/or updating of
our automated end-to-end (Cypress) testing and patching bugs
uncovered by this testing (as described in the previous project
idea). The project should include benchmarking along the way to
ensure that the refactor is improving the performance of the
Manual/TA Rubric Grading interface. The general interface for
TA/Manual grading should remain similar, but the project may
include small user interface revisions.

_Skills & Experience Required_: Web and database development
experience and general software design and implementation
experience. Experience with end-to-end automated testing (Cypress)
and and having served as a teaching assistant with grading
experience design is beneficial but not required.

_Possible Mentors_: Nia Heermance, William Allen, Barb Cutler

_GSoC Project Size_: 175 or 350 hours

_Difficulty Level_: medium or hard

_Difficulty Level_: medium to challenging



3. **Notebook Builder: UI To Streamline Instructor Configuration of Automated Grading**

Our system for automated testing and grading of student work is
very powerful, but the configuration process that instructors must
navigate is complex and time-consuming. While we provide a number
of examples, the number of choices for development of an
autograding configuration is overwhelming. The primary method for
creating an autograding configuration is to prepare a `config.json`
file (and any necessary additional files) and upload or store these
files on the server file system. We have a prototype Web GUI
interface we call the "Notebook Builder" but the current state of
the feature is undocumented and functionality is limited. We
would like to improve and expand this feature to facilitate
instructor creation of basic and moderate complexity autograding
configurations.

[Assignment Autograding Configuration Instructions](/instructor/autograding/specification)
[Notebook Assignment Configuration](/instructor/assignment_configuration/notebook)
[Tutorial Autograding Configuration Examples](https://github.com/Submitty/Tutorial/tree/main/examples)

This project will involve multiple modules of Submitty including
web UI development, integration, documentation, additional tutorial
examples, and extending output generation to instructor solutions
in compiled languages.

[Open Issues related to Autograding](https://github.com/Submitty/Submitty/labels/Autograding)
[Open Issues related to Notebook / Notebook Builder](https://github.com/Submitty/Submitty/issues?q=is%3Aopen+is%3Aissue+label%3A%22Notebook+%2F+Notebook+Builder%22)

_Expected Outcomes_: The primary focus of the project is the
revision and expansion of the Notebook Builder UI to increase the
number of autograding features that are supported. The UI should
be easy-to-use for instructors of
non-computer-science/non-programming courses and also instructors of courses with
introductory to moderate programming assignments. The size and
scope for a proposal in this area is flexible, depending on the
time commitment and prior skills of the applicant.

_Skills & Experience Required_: Some programming experience,
willingness to learn web and database development. Prior
experience with user interface design and an eye for quality user
design are beneficial. Having served as a teaching assistant or
instructor with experience in programming assignment design will be
beneficial but not required.

_Possible Mentors_: Barb Cutler, Chris Reed

3. **Streamline instructor configuration of automated grading**
_GSoC Project Size_: 90 or 175 or 350 hours

Currently, instructors must write a configuration as a `config.json`
(and any necessary additional files) and upload or store these
files on the local file system. We would like to provide an
alternate web GUI interface for creating basic or moderately
complex autograding configurations.
_Difficulty Level_: introductory or medium


[Assignment Configuration Instructions](/instructor/autograding/specification)
4. **Expansion of Examples and Documentation of Intermediate and Advanced Autograding Features**

We have preliminary support for automated creation of expected
output files (from and instructor solution -- currently limited to
Python) and randomized test case input. This project will involve multiple modules of
Submitty including
web UI development, integration, documentation, additional tutorial examples, and
extending output generation to
instructor solutions in compiled languages.
_Note: This project is related to previous project idea but is a distinct project._

[Open Issues related to Autograding](https://github.com/Submitty/Submitty/labels/Autograding)
Our system for automated testing and automated grading of student
work is very powerful and highly-customizable, but the
documentation for our moderate and advanced autograding features is
incomplete. While we provide a number of autograding examples,
some of the examples are out-of-date and do not represent our
current suggested best practices.

_Expected Outcomes_: The goal would be to
streamline the assignment configuration process for non-technical
instructors, relevant for use in
non-computer-science/non-programming courses.
[Assignment Autograding Configuration Instructions](/instructor/autograding/specification)
[Submitty Autograding Tutorial Examples](https://github.com/Submitty/Tutorial)
[Additional Autograding Examples](https://github.com/Submitty/Submitty/tree/master/more_autograding_examples)
[Prior GSoC Project: Drumil Patel](/developer/google_summer_of_code/2019_DrumilPatel)

_Skills & Experience Required_: Some programming experience, willingness to
learn web and database development. Having served as a teaching
assistant or instructor with experience in programming assignment
design will be beneficial.
We would like to reduce the learning curve for new instructors and
provide more tutorial examples of autograding for instructors
teaching courses of any level. Automated testing and automated
grading can be used in introductory programming courses in middle
and high schools, including AP Computer Science. It can also be
used by programming-intensive intermediate and upper level / senior
university-level systems coursework. Assignments that require can
be configured with custom Docker Images to provide access to
specific programming languages and libraries.

[Open Issues related to Docker Image Autograding](https://github.com/Submitty/Submitty/issues?q=label%3A%22Docker+Container+Autograding%22+)
[Docker Images for Autograding Common Programming Languages](https://github.com/Submitty/DockerImages/tree/main)
[Example Custom Docker Images University ](https://github.com/Submitty/DockerImagesRPI/tree/main/dockerfiles)
[Sample Java Assignments](/instructor/autograding/sample_assignments)

_Possible Mentors_: Barb Cutler, William Allen
_Expected Outcomes_: The project should begin with a review and
organization of existing sample and tutorial assignments and
current autograding functionality documentation. Out-of-date or
underdeveloped autograding configuration examples should be
expanded as necessary, and features that are missing documentation
and examples should be identified (e.g., generated random input and
output from instructor solution, customized docker containers,
autograding graphical output, autograding) and resolved by creating
new examples. Finally, we would like to create and support a
resource for the community of crowd-sourced complete programming
assignments/exercises with included autograding configuration.

_Skills & Experience Required_: Moderate to advanced programming
experience, willingness to learn web and database development.
Having served as a teaching assistant or instructor with experience
in programming assignment design will be beneficial.

_Possible Mentors_: Chris Reed, Barb Cutler

_GSoC Project Size_: 175 or 350 hours

_Difficulty Level_: medium or hard


4. **Instructional Materials and Documentation**
_Difficulty Level_: medium to challenging

We would like to reduce the learning curve for new instructors and
provide more starter material for instructors teaching introductory
programming courses in middle and high schools, including AP
Computer Science.

[Submitty Autograding Tutorial Examples](https://github.com/Submitty/Tutorial)
5. **AI/ML to Enhance and Steamline Manual / TA Grading**

[Additional Autograding Examples](https://github.com/Submitty/Submitty/tree/master/more_autograding_examples)
The use of a unified and retroactively editable rubric for
manual/TA grading can ensure consistency when grading large
courses, especially when more than one grader is working on a
single problem or assignment. However, it is usually still
necessary for the graders to inspect the student work one-at-a-time
and it can be difficult for the grader to remember the details of
all previously graded assignments and recognize patterns that
should be graded similarly. Furthermore, the process of manual
grading is time-consuming and thus detailed and thoughtful
constructive feedback to each individual is often not possible.

[Sample Java Assignments](/instructor/autograding/sample_assignments)
[Overview of Rubric Grading Interface](/grader/rubric_grading/index)

_Expected Outcomes_: Organization of existing sample and tutorial assignments and autograding
and current documentation. Review and curation of any publicly-available sample assignments
and creation of new sample assignments and autograding.
The goal of this project is to explore the potential to leverage
Artificial Intelligence and Machine Learning (AI/ML) to reduce the burden
of manual grading in large courses with either programming or
non-programming assignments. Automatically organizing student
submissions into groups that contain similar patterns and have
common strengths or flaws can ensure that student work is assessed
consistently and students receive appropriate and in-depth
feedback to aid their learning.

The Submitty project includes related technology for the static
analysis of student's code and tools to screen for plagiarism in
both student-submitted plain text assignments and software. Note
that the Submitty static analysis and plagiarism tools have been
tested and validated with datasets of sample student submissions;
however, for privacy and confidentiality reasons, these datasets are
not and cannot be part of the Submitty open-source materials.

[Autograding using Static Analysis](/instructor/autograding/static_analysis/index)
[Plagiarism Detection](/instructor/course_management/plagiarism)

_Expected Outcomes_: Detailed design plan for the integration of an
AI/ML framework into Submitty for the analysis and clustering the
assignment submissions by students based on common patterns in the
text and/or code. Implementation of a prototype AI/ML tool to
detect common patterns in student submissions and present this
information to the grader in an organized way to allow streamlined
bulk grading and feedback within the manual/TA grading interface.
As time permits (and based on the scope and time commitment)
evaluation of the effectiveness of this technique on real world
data and the potential for improving the efficiency of the
manual/TA grading process and the quality/accuracy and quantity of
useful constructive feedback to students.

_Skills & Experience Required_: Some programming experience, willingness to
learn web and database development. Having served as a teaching
assistant or instructor with experience in programming assignment
design will be beneficial.
_Skills & Experience Required_: Coursework and/or professional
experience in AI/ML and modern AI/ML technology. Moderate to
advanced programming experience, and willingness to learn web and
database development. Having served as a teaching assistant for a
large course with manual grading experience design will be
beneficial.

_Possible Mentors_: Barb Cutler, William Allen, Nia Heermance

_Possible Mentors_: Barb Cutler, William Allen
_GSoC Project Size_: 175 or 350 hours

_GSoC Project Size_: 175 hours
_Difficulty Level_: medium to challenging

_Difficulty Level_: easy or medium
6. **Other Topics**

The Submitty team welcomes GSoC project proposals on other topics
related to items in our GitHub issue tracker. A successful
application would select one or more issues of moderate scope
proportional to the applicant's time commitment and prior
experience. Be sure to join our [Zulip server](/contact) to meet
the Submitty mentors and other new developers and discuss your
interests and project plans.

_Skills & Experience Required_: Some prior programming experience,
willingness to learn web and database development, and additional
specific skills as appropriate.

_Possible Mentors_: Barb Cutler, Matthew Peveler, Shail Patel, William Allen, Chris Reed, Nia Heermance, Preston Carman

_GSoC Project Size_: 90 or 175 or 350 hours

_Difficulty Level_: introductory to medium to challenging


See also:

Expand Down

0 comments on commit 0f87cde

Please sign in to comment.