Does project proposal feedback result in better final projects?
tldr: Project proposal feedback didn't improve the quantity or quality of projects submitted.
Why did we do this?
On our June 2024 AI alignment course, participants completed an 5-week independent project. You can see the results of some of these projects on our website.
At the beginning of the project phase, they submitted a project proposal to us.
We wanted to know how useful our central team providing feedback on these proposals was. We thought feedback would likely be useful for some participants, but weren’t certain. Additionally, if it only helped a little, it might not be worth providing given how resource intensive it is for us to give.
We also realized that we wouldn’t have time to give high-quality feedback on all projects, and only had about enough capacity to review half the project proposals.
What did we do?
We ran a test on this version of the course: the central team provided feedback on a random sample of project proposals. All participants still got feedback from their cohorts and facilitator.
To get this random sample, we selected 50% of proposals submitted before 2024-08-29, by randomly generated Airtable ID. This meant we gave feedback on 41 projects, and didn’t give feedback on 40 projects. We excluded 1 project which we ended up giving feedback on at the participant’s request before assigning it to either sample set.
The feedback we gave included:
- inline comments on their planning document, which usually followed our template
- an overall comment, covering ways to narrow the project scope and ways to make their project more novel or useful
After submitting their projects, judges graded projects from 1 to 5 on three evaluation criteria. We calculated a total score using a weighted sum of evaluation criteria scores, weighting the communication and relevance scores at 1x, and research quality at 2x. Projects were also assigned a general bucket of low/moderate/high quality.
Results
There was no statistically significant difference between project proposals we gave feedback on and those that we didn’t, across:
- Percentage of final projects submitted
- Percentage of projects submitted that were deemed ‘high-quality’
- Any of the individual numeric scores, or the total score
In fact, many of the scores were slightly lower for projects we did give feedback on! (but not to a statistically significant degree)
Discussion
The results suggest that we should either not bother providing feedback, or need to substantially change how we are providing feedback.
We expected feedback to have at least a small positive effect, but it did not. We think it might not have had this effect because:
- The project phase sessions already focus a lot on peer and facilitator feedback, so participants receive good feedback without us
- The feedback primarily focused on their ideas, rather than their execution - and a lot of project success for beginners comes down to execution (e.g. making time available, having existing skills)
- Negative feedback might have been disheartening at the beginning of the project phase, causing people to put less energy into the projects
However, we have not (yet) validated any of these hypotheses!
Appendix: Detailed results
- % projects submitted (unreviewed): 85%
- % projects submitted (reviewed): 78%
- Average communication score (unreviewed): 3.2
- Average communication score (reviewed): 3.5
- Average relevance score (unreviewed): 3.1
- Average relevance score (reviewed): 3.1
- Average research quality score (unreviewed): 3.2
- Average research quality score (reviewed): 2.9
- Average total score (unreviewed): 12.7
- Average total score (reviewed): 12.5
- % high quality (unreviewed): 60%
- % high quality (reviewed): 49%