Troubles and errors, rather than being a rare aberration in human communication, are highly frequent, with some authors estimating that edits, rephrasing, and amendments in response to some trouble signal occur every three turns [1]. Counteracting these errors are a set of robust repair mechanisms that have been widely documented in conversation analysis and cognitive science. While there is considerable conversation-analytic work on repair, most of it is restricted to speech. Furthermore, we are not aware of any prior work that analyses multimodal repair in instruction-based scenarios that are commonplace in human-robot interaction, that is, where a human instruction giver gives commands to a robotic instruction follower. Lacking a genuine human-robot interaction corpus collected from an instruction-based scenario, our underlying assumption is that much can be learned from a deep analysis of human-human corpora such as the PENTOREF corpus [2], and that the gained insights can inform the design of the next generation of multimodal dialogue systems. The aims of ongoing work are (1) to document how multimodal repair, that is repair involving more than one modality – here vision and speech – unfolds in instruction-based scenarios; (2) distil interactional regularities from the documented cases; and (3) derive a list of desiderata for future dialogue systems deployed on robotic instruction followers.
Topics of interest we would be interested to discuss on the workshop relate to our preliminary findings on how to best detect repair in human instructions given that negation words, although generally very strong indicators for the presence of a repair act, cannot exclusively be relied upon.
Secondly, repair utterances are likely not a crisp category that can be cleanly delineated, partly due to the multimodal nature of this type of dialogue and traditional accounts of repair having been built on speech only.

[1] Marcus Colman and Patrick Healey. 2011. The distribution of repair in dialogue.
In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 33.
[2] Sina Zarrieß, Julian Hough, Casey Kennington, Ramesh Manuvinakurike, David
DeVault, Raquel Fernández, and David Schlangen. 2016. PentoRef: A Corpus of
Spoken References in Task-oriented Dialogues. In: 10th edition of the Language
Resources and Evaluation Conference. Portorož (Slovenia).