The autoregressive moving average (ARMA) model

Stationarity

A stochastic process is covariance stationary or weakly stationary if for all $t$ and $s$

If a process is covariance stationary, the covariance between $y_t$ and $y_{t-s}$ depends only on $s$, the length of time separating the observations. It follows that for a covariance stationary process, ${\gamma }_s$ and ${\gamma }_{-s}$ would represent the same magnitude.

For a covariance stationary series, we can define the autocorrelation between $y_t$ and $y_{t-s}$

where ${\gamma }_0$ is the variance of $y_t$

Ergodicity

Imagine a battery of $I$ computers generating sequences $\{y_t^{(1)}{\}}_{t=-\mathrm{\infty }}^{\mathrm{\infty }}$, $\{y_t^{(2)}{\}}_{t=-\mathrm{\infty }}^{\mathrm{\infty }}$, $\dots$, $\{y_t^{(I)}{\}}_{t=-\mathrm{\infty }}^{\mathrm{\infty }}$ and consider selecting the observation associated with date $t$ from each sequence: $\{y_t^{(1)},y_t^{(2)},\dots ,y_t^{(I)}\}$ This would be described as a sample of $I$ realizations of the random variable $Y_t$. The expectation of the $t$th observation of a time series refers to the mean of the probability distribution

We might view this as the probability limit of the ensemble average

The above expectations of a time series in terms of ensemble averages may seem a bit contrived. Usually we have a single realization of size $T$ from the process $\{y_1^{(1)},y_2^{(1)},\dots ,y_T^{(1)}\}$ From these observations we would calculate the sample mean $\overline{y}$, which is a time average

A covariance stationary process is said to be ergodic for the mean if $\overline{y}$ converges in probability to $E(Y_t)$ as $T\to \mathrm{\infty }$.

Moving Average Processes

The First-Order Moving Average Process

Let $\{{\epsilon }_t\}$ be white noise and consider the process

where $\mu$ and $\theta$ could be any constants. This time series is called a first-order moving average process, denoted $MA(1)$.

Expectation The expectation of $y_t$ is

Variance The variance of $y_t$ is

Autocovariance The first autocovariance of $y_t$ is

Higher autocovariances are all zero. For all $j>1$

Autocorrelation The $j$th autocorrelation of a covariance stationary process is ${\rho }_j=\frac{{\gamma }_j}{{\gamma }_0}$

The $q$th-Order Moving Average Process

A $q$th-order moving average process, denoted $MA(q)$, is characterized by

Expectation The expectation of $y_t$ is

Variance The variance of $y_t$ is

Autocovariance The autocovariance of $y_t$ is

For all $j>q$

The Infinite-Order Moving Average Process

Consider the process when $q\to \mathrm{\infty }$

This could be described as an $MA(\mathrm{\infty })$ process.

The $MA(\mathrm{\infty })$ process is covariance stationary if it is square summable

It is often to work with a slightly stronger condition called absolutely summable

Expectation The mean of an $MA(\mathrm{\infty })$ process with absolutely summable is

Autocovariance The autocovariance of an $MA(\mathrm{\infty })$ process with absolutely summable is

Autoregressive Processes

The First-Order Autoregressive Process

A first-order autoregressive, denoted $AR(1)$, satisfies the following difference equation

We can derive the expectation and autocovariance of $AR(1)$ by the above corresponding $MA(\mathrm{\infty })$ process. We also can derive them by assuming $AR(1)$ process is covariance stationary.

Expectation Taking expectations both sides

Autocovariance

The $q$th-Order Autoregressive Process

The Autocorrelation Function

Autocorrelation function (ACF) and the partial autocorrelation function (PACF) are useful to determine the type of time series data.

For AR(1) model

• Method 1

So the autocorrelation function for AR(1)

• Method 2 If the process started at time zero

Take the expectation of $y_t$ and $y_{t+s}$

The Autocorrelation Function of an AR(2) Process

We assume that $a_0=0$, which implies that $E(y_t)=0$. Adding or subtracting any constant from a variable does not change its variance, covariance, correlation coefficient, etc.

Using Yule-Walker equations: multiply the second-order D.E by $y_{t-s}$ for s$=0,1,2,\cdots$, and take expectations

By definition, the autocovariances of a stationary series are such

We also know that coefficient on ${\epsilon }_t$ is unity so that $E{\epsilon }_t{y}_t={\sigma }^2$, and $E{\epsilon }_t{y}_{t-s}=0$, so

Now we can get the ACF

We know ${\rho }_0=1$, so

The Autocorrelation Function of an MA(1) Process

Consider the MA(1) process $y_t={\epsilon }_t+\beta {\epsilon }_{t-1}$

Applying the Yule-Walker equations

So the ACF of MA(1)

The Autocorrelation Function of an ARMA(1,1) Process Consider the ARMA(1,1) $y_t=a_1{y}_{t-1}+{\epsilon }_t+{\beta }_1{\epsilon }_{t-1}$

Solve the equations and get

And the AFC

The Partial Autoorrelation Function

In AR(1) process, $y_t$ and $y_{t-2}$ are correlated even though $y_{t-2}$ does not directly appear in the model. ${\rho }_2=corr(y_t,y_{t-1})\times corr(y_{t-1},y_{t-2})={\rho }_1^2$. All such “indirect” correlations are present in the ACF. In contrast, the partial autocorrelation between $y_t$ and $y_{t-s}$ climinates the effects of the intervening values $y_{t-1}$ through $y_{t-s+1}$.

Method to find the PACF:

• Form the series $\{y_t^{\ast }\}$, where $y_t^{\ast }\equiv y_t-\mu$
• Form the first-order autoregression equation:

where ${\varphi }_{11}$ is the partial autocorrelation between $y_t$ and $y_{t-1}$, ${\varphi }_{22}$ is the partial autocorrelation between $y_t$ and $y_{t-2}$. Repeating this process for all additional lags s yields the PACF.

Sample Autoorrelations

Given that a series is stationary, we can use the sample mean, variance and autocorrelations to estimate the parameters of the actual data-generating process.

The estimates of $\mu$, ${\sigma }^2$ and $\rho$:

Box and Pierce used the sample autocorrelations to form the statistic

If the data are generated from a stationary ARMA process, Q is asymptotically ${\chi }^2(s)$ distribution.

Ljung and Box test