Presenting a target-like distractor in an RSVP task deteriorates the detection of a trailing target, because the visual system has difficulties in rejecting the erroneously accepted distractor. We investigated whether the rejection process is influenced by observers’ knowledge regarding possible distractors. Observers identified a letter (target) embedded in a stream of line patterns, rejecting a preceding distractor (digit). We informed the observers about either the category of distractors (“digit”) or the identity of the distractor (e.g., “5”). The distractors with certain distractor–target lags increased identification errors, indicating that the distractor rejection process temporarily interfered with the target identification. When the observers knew the distractor identity, the rejection process started later than when they knew only the distractor category. These results suggest that the rejection process may operate at either the category or the individual-item level; however, the setting of the rejection level is not under the observers’ control.