To clarify how the brain understands the speaker's mind for verbal acts, we analyzed fMRI images obtained from 24 subjects when they judged linguistic meanings or emotional manners of spoken phrases. The target phrases had linguistically positive or negative meanings and were uttered warmheartedly or coldheartedly by a woman speaker. Significant interaction effects of meaning and manner were observed on the acoustic characteristics of utterances, such as FO range, and also on the perceptual behavior evaluated by response time and judgment correctness. When compared to the female subjects, the male subjects showed significantly stronger activation only in the right frontomedian cortex, which can be hypothesized to analyze and understand speaker's hidden but true intentions from speaking acts. These results suggest that emotion modulates linguistic processes not only in speech production but also in speech perception, and such modulations differ between the genders at least in perceptual processes.