When people frequently alternate between simple cognitive tasks, performance on stimuli which are assigned the same response in both tasks is typically faster and more accurate than on stimuli which require different responses for both tasks, thus indicating stimulus processing according to the stimulus–response (S–R) rules of the currently irrelevant task. It is currently under debate whether such response congruency effects are mediated by the activation of an abstract representation of the irrelevant task in working memory or by “direct” associations between specific stimuli and responses. We contrasted these views by manipulating concurrent memory load (Experiment 1) and the frequency of specific S–R associations (Experiment 2). While between-task response congruency effects were not affected by the amount of concurrent memory load, they were much stronger for stimuli that were processed frequently in the context of a competitor task. These findings are consistent with the idea that a large portion of the congruency effects stems from direct S–R associations and they do not support a sole mediation by task-set activation in working memory.