We study how to augment failing test input feedback with descriptive messages on the likely location and cause of errors for introductory functional programming assignments. We classify various problem-independent errors that students make related to recursive computational structure. Our algorithm for automatically identifying such errors involves comparing the student trace with an appropriate reference trace and leverages two key ideas. First, we compare traces at an abstract level that captures enough details to identify desired errors, but abstracts over varying programming styles. Second, we leverage correct student submissions to build a collection of reference solutions with distinct abstract traces, and then we identify the appropriate reference trace for comparison (that maximally covers some prefix of the incorrect trace) by mixing/matching parts from multiple reference traces. Our experiments over 1,500 incorrect attempts show that a small number of correct submissions suffices to provide accurate feedback for the vast majority of attempts.
We have implemented the idea in a tool called CamlBack which is available online and currently supports OCaml and Haskell. The website has been used in several university courses on functional programming.