Response-time and accuracy costs as assessed in the context of the task-switching paradigm are usually thought to represent processes involved in the selection of abstract task sets. However, task sets are also applied to specific stimulus and response constellations, which in turn may become associated with task-set representations. To explore the consequence of such associations, we used a task-switching paradigm in which subjects had to select between two tasks (color or orientation discrimination) that were either associated with shared or unique stimulus/response locations on a touchscreen. When each task was associated with unique locations, error switch costs, stimulus–response congruency effects, as well as the characteristic task-switch × repetition-priming interaction were eliminated, and global selection costs were substantially reduced. These results demonstrate that to understand standard task-switching phenomena it is critical to consider links between lower level stimulus/response parameters and task sets.