The composite membranes comprising of sulfonated polymers as matrix and ionic liquids as ion-conducting medium in replacement of water are studied to investigate the effect of annealing of the sulfonated polymers. The polymeric membranes are prepared on recast Nafion containing the ionic liquid, 1-ethyl-3-methylimidazolium tetrafluoroborate ($EMIBF_4$). The composite membranes are characterized by thermogravitational analyses, ion conductivity and small-angle X-ray scattering. The composite membranes annealed at $190^{circ}C$ for 2 h after the fixed drying step showed better ionic conductivity, but no significant increase in thermal stability. The mean Bragg distance between the ionic clusters, which is reflected in the position of the ionomer peak (small-angle scattering maximum), is larger in the annealed composite membranes containing $EMIBF_4$ than the non-annealed ones. It might have been explained to be due to the different level of ion-clustering ability of the hydrophilic parts (i.e., sulfonic acid groups) in the non- and annealed polymer matrix. In addition, the ionic conductivity of the membranes shows higher for the annealed composite membranes containing $EMIBF_4$. It can be concluded that the annealing of the composite membranes containing ionic liquids due to an increase in ion-clustering ability is able to bring about the enhancement of ionic conductivity suitable for potential use in proton exchange membrane fuel cells (PEMFCs) at medium temperatures ($150-200^{circ}C$) in the absence of external humidification.