After observing a face with the eyes looking to the left or right (adaptation stimulus), the perception of the eye direction of the subsequent face (test stimulus) is biased in the opposite direction of the adapted eye direction; this is called the eye direction aftereffect (EDAE). In the present study, the adaptation stimuli were rotated 90° (clockwise or counterclockwise) or 180° relative to the viewer. The EDAE was measured using upright test stimuli. For the 90° rotation, prior observation of the ‘leftward’ and ‘rightward’ eye directions biased the perceived eye directions of the upright test stimuli to the right and left, respectively. These results suggest that the adaptation was induced utilizing an object-based (or face-based) reference frame. For the 180° rotation, however, the results suggest that the adaptation was induced in a viewer-centered reference frame. The involvement of an object-based reference frame suggests that the EDAE reflected the adaptation of a relatively higher-level mechanism at least when the rotation angle from the upright position did not exceed 90°.