Discovering causal models hidden in the background of observational data has been a difficult issue. It is often necessary to deal with latent common causes and selection bias for constructing causal models in real data. Ancestral graph models are effective and useful for representing causal models with latent variables. The causal faithfulness condition, which is usually assumed for determining the models, is statistically known to often be weakly violated for finite data. One of the authors developed a constraint-based causal learning algorithm that is robust against the violations while assuming no latent variables. In this study, we applied and extended the thoughts of the algorithm to the inference of ancestral graphs. The practical validity and effectiveness of the algorithm are also confirmed by using some standard datasets in comparison with FCI and RFCI algorithms.