A novel method of hierarchical reinforcement learning, named OMQ, by integrating options into MAXQ is presented. In OMQ, the MAXQ is used as basic framework to design hierarchies experientially and learn online, and the option is used to construct hierarchies automatically. The performance of OMQ is demonstrated in taxi domain and compared with Option and MAXQ. The simulation results show that the OMQ is more practical than option and MAXQ in partial known environment