Until the machines are fully autonomous and generate themselves, human design decisions affect Machine Learning outcomes every step of the way. This position paper outlines multiple stages at which design decisions affect machine learning outcomes, and how they interact. This includes: dataset curation and data pipelines, selection of optimization targets, and the designed dialogue with end-users with its implicit and explicit feedback mechanisms. We specifically also call out another user group that appears somewhat overlooked in the research literature – the data curators and editors often involved in selecting and annotating the data that machines learns from.