Backward inhibition (BI) reflects the suppression of a recently abandoned task set to allow for smooth transition to a new task even when the rules do not generate a response conflict. Competitor rule suppression (CRS) reflects the inhibition/suppression of irrelevant task rules when these rules generate a response conflict even if they have not recently been abandoned. We assessed whether BI and CRS are differentially affected by the difficulty in retrieving category-response mappings from memory. Retrieval demands were manipulated via the information provided by the task cues, which either indicated the relevant dimension (dimension cues; “color”) or the relevant dimension with its category-to-key mapping (mapping cues; “red green”, indicating that “red” and “green” go with the left/right responses, respectively). CRS was larger with dimension compared to mapping cues when cue-type varied between groups and was larger after trials involving dimension cues when cue-type varied on a trial-by-trial basis. In contrast, BI was not influenced by cue-type. These results suggest that task switching involve at least two distinct inhibitory processes, with CRS being related to the ease of retrieval of category-response mappings from memory.