A new direction in speech recognition via statistical methods is to move from frame-based models, such as hidden Markov models, to segment-based models that provide a better framework for modeling the dynamics of the speech production mechanism. The stochastic segment model (SSM) is a joint model for a sequence of observations which provides explicit modeling of time correlation as well as a formalism for incorporating segmental features. The authors examine the modeling of time correlation within a segment. They consider three Gaussian model variations based on different assumptions about the form of statistical dependency, including a Gauss-Markov model, a dynamical system model, and a target state model, all of which can be formulated in terms of the dynamical system model. Evaluation of the different modeling assumptions is in terms of both phoneme classification performance and the predictive power of linear models.< >