Previous research has indicated that modality switching is considerably affected by modality compatibility. It has been shown that switch costs are higher for switching between relatively incompatible sensory-motor modality mappings (i.e., auditory-manual and visual-vocal) compared to switching between compatible mappings (i.e., auditory-vocal and visual-manual). So far, however, it has been unclear whether these findings are influenced by learning processes resulting from very small stimulus sets and a large number of stimulus repetitions. In the present study, we investigated the role of learning concept-to-category associations (Experiment 1) as well as influences of learning concept-to-modality mappings (Experiment 2) on sensory-motor modality switching in semantic categorizations. The results of both experiments revealed shorter overall reaction times due to learning. Additionally, learning of concept-to-category associations (Experiment 1) led to a significant reduction of modality switch costs. Interestingly, however, modality-compatibility effects were neither significantly influenced by learning of concept-to-category associations nor by learning of concept-to-modality mappings. Thus, the present study provides first evidence that learning on the semantic level influences modality switching but it does not significantly affect modality compatibility.