Virtual Histology-Intravascular Ultrasound (VH-IVUS) is widely used for studying atherosclerosis plaque composition. However, one of the main limitations of the VH-IVUS relates to its dependence to the Electrocardiogram (ECG)-gated acquisition. To overcome this limitation, this paper proposes a robust image-based approach for characterization of the plaques using IVUS images. The proposed method consists of three main steps of (1) shadow detection: as an efficient preprocessing step to identify and remove acoustic shadow regions; (2) feature extraction: a combination of gray-scale based features and textural descriptors; and (3) classification: to classify each pixel into one of the three classes (calcium, necrotic core and fibro-fatty). In order to evaluate the efficiency of the proposed algorithm two in-vivo and ex-vivo data sets are considered. The kappa values of 0.639 on in-vivo and 0.628 on ex-vivo tests with VH-IVUS and the histology images labeled by the experts respectively indicate the effectiveness of the proposed algorithm.