Covariance Matrix Formula

Covariance Matrix

Covariance matrix is a type of matrix that is used to represent the covariance values between pairs of elements given in a random vector. The covariance matrix can also be referred to as the variance covariance matrix. This is because the variance of each element is represented along the main diagonal of the matrix.

A covariance matrix is always a square matrix. Furthermore, it is positive semi-definite, and symmetric. This matrix is very useful in stochastic modeling and principle component analysis. In this article, we will learn about the variance covariance matrix, its formula, examples, and various important properties associated with it.

What is Covariance Matrix?

Covariance matrix is a square matrix that displays the variance exhibited by elements of datasets and the covariance between a pair of datasets. Variance is a measure of dispersion and can be defined as the spread of data from the mean of the given dataset. Covariance is calculated between two variables and is used to measure how the two variables vary together.

Covariance Matrix Definition

Variance covariance matrix is defined as a square matrix where the diagonal elements represent the variance and the off-diagonal elements represent the covariance. The covariance between two variables can be positive, negative, and zero. A positive covariance indicates that the two variables have a positive relationship whereas negative covariance shows that they have a negative relationship. If two elements do not vary together then they will display a zero covariance.

Covariance Matrix Example

Suppose there are two data sets X = {3, 2} and Y = {7, 4}. The sample variance of dataset X = 0.5, and Y = 4.5. The covariance between X and Y is 1.5. The covariance matrix is expressed as follows:

A detailed description of how to find the variance covariance matrix will be covered in the upcoming sections.

Covariance Matrix Formula

To determine the covariance matrix, the formulas for variance and covariance are required. Depending upon the type of data available, the variance and covariance can be found for both sample data and population data. These formulas are given below.

Using these formulas, the general form of a variance covariance matrix is given as follows:

Covariance Matrix 2 × 2

A 2 × 2 matrix is one which has 2 rows and 2 columns. The formula for a 2 × 2 covariance matrix is given as follows:

Covariance Matrix 3 × 3

If there are 3 datasets, x, y, and z, then the formula to find the 3 × 3 covariance matrix is given below:

How To Calculate Covariance Matrix?

The number of variables determines the dimension of a variance-covariance matrix. For example, if there are two variables (or datasets) it indicates that the covariance matrix will be 2 dimensional. Suppose the math and science scores of 3 students are given as follows:

The steps to calculate the covariance matrix for the sample are given below:

• Step 1: Find the mean of one variable (X). This can be done by dividing the sum of all observations by the number of observations. Thus, (92 + 60 + 100) / 3 = 84
• Step 2: Subtract the mean from all observations; (92 – 84), (60 – 84), (100 – 84)
• Step 3: Take the sum of the squares of the differences obtained in the previous step. (92 – 84)² + (60 – 84)² + (100 – 84)².
• Step 4: Divide this value by 1 less than the total to get the sample variance of the first variable (X). var(X) = [(92 – 84)² + (60 – 84)² + (100 – 84)²] / (3 – 1) = 448
• Step 5: Repeat steps 1 to 4 to find the variances of all variables. Using these steps, var(Y) = 700.
• Step 6: Choose a pair of variables (X and Y).
• Step 7: Subtract the mean of the first variable (X) from all observations; (92 – 84), (60 – 84), (100 – 84).
• Step 8: Repeat step 7 for the second variable (Y); (80 – 60), (30 – 60), (70 – 60).
• Step 9: Multiply the corresponding observations. (92 – 84)(80 – 60), (60 – 84)(30 – 60), (100 – 84)(70 – 60).
• Step 10: Add these values and divide them by (n – 1) to get the covariance. cov(x, y) = cov(y, x) = [(92 – 84)(80 – 60) + (60 – 84)(30 – 60) + (100 – 84)(70 – 60)] / (3 – 1) = 520.
• Step 11: Repeat steps 6 to 10 for different pairs of variables.
• Step 12: Now using the general formula for covariance matrix arrange these values in matrix form. Thus, the variance covariance matrix for the example is given as

The same steps can be followed while calculating the covariance matrix for a population. The only difference is that the population variance and covariance formulas will be applied.

Properties of Covariance Matrix

Covariance matrix is a very important tool used by data scientists to understand and analyze multivariate data. Listed below are the various properties of this matrix that make it extremely useful.

• A covariance matrix is always a square matrix. This means that the number of rows of the matrix will be equal to the number of columns.
• The matrix is symmetric. Suppose M is the covariance matrix then M^T = M.
• It is positive semi-definite. Let u be a column vector, u^T is the transpose of that vector and M be the covariance matrix then u^TMu ≥ 0.
• All eigenvalues of the variance covariance matrix are real and non-negative.

Important Notes on Covariance Matrix

• The covariance matrix depicts the variance of datasets and covariance of a pair of datasets in matrix format.
• The diagonal elements represent the variance of a dataset and the off-diagonal terms give the covariance between a pair of datasets.
• The variance covariance matrix is always square, symmetric, and positive semi-definite.
• The general formula to represent a covariance matrix is

Examples on Covariance Matrix

Example 1: Find the population covariance matrix for the following table.

Solution: The formula for population variance is

μx = 56.5, n = 4

var(x) = [(68 – 56.5)² + (60 – 56.5)² + (58 – 56.5)² + (40 – 56.5)² ] / 4 = 104.75

μy = 30, n = 4

var(y) = [(29 – 30)² + (26 – 30)² + (30 – 30)² + (35 – 30)²] / 4 = 10. 5

cov(x, y) = -27

The variance covariance matrix is given as follows:

Example 2: Find the covariance matrix for the following sample data.

Solution: The sample variance formula is

Substituting the values of observations for each variable in this formula we get,

Example 3: How will you interpret the covariance matrix given below?

Solution: The variance covariance matrix can be interpreted as follows:

1) The diagonal elements 500, 340 and 800 indicate the variance in data sets X, Y and Z respectively. Y shows the lowest variance whereas Z displays the highest variance.

2) The covariance for X and Y is 320. As this is a positive number it means that when X increases (or decreases) Y also increases (or decreases)

3) The covariance for X and Z is -40. As it is a negative number it implies that when X increases Z decreases and vice – versa.

4) The covariance for Y and Z is 0. This means that there is no predictable relationship between the two data sets.

Example 1: The marks scored by 3 students in Physics and Biology are given below:

Calculate Covariance Matrix from the above data.

Solution:

Sample covariance matrix is given by

Here, μ_x = 84, n = 3

var(x) = [(92 – 84)² + (60 – 84)² + (100 – 84)²] / (3 – 1) = 448

Also, μ_y = 60, n = 3

var(y) = [(80 – 60)² + (30 – 60)² + (70 – 60)²] / (3 – 1) = 700

Now, cov(x, y) = cov(y, x) = [(92 – 84)(80 – 60) + (60 – 84)(30 – 60) + (100 – 84)(70 – 60)] / (3 – 1) = 520.

The population covariance matrix is given as:

Example 2. Prepare the population covariance matrix from the following table:

Solution:

Population variance is given by

Here, μ_x = 56.5, n = 4

var(x) = [(68 – 56.5)² + (60 – 56.5)² + (58 – 56.5)² + (40 – 56.5)² ] / 4 = 104.75

Also, μ_y = 30, n = 4

var(y) = [(29 – 30)² + (26 – 30)² + (30 – 30)² + (35 – 30)²] / 4 = 10. 5

cov(x, y) = -27

The population covariance matrix is given as:

Example 3. Interpret the following covariance matrix:

Solution:

1. The diagonal elements 60, 30, and 80 indicate the variance in data sets X, Y, and Z respectively. Y shows the lowest variance whereas Z displays the highest variance.
2. The covariance for X and Y is 32. As this is a positive number it means that when X increases (or decreases) Y also increases (or decreases)
3. The covariance for X and Z is -4. As it is a negative number it implies that when X increases Z decreases and vice-versa.
4. The covariance for Y and Z is 0. This means that there is no predictable relationship between the two data sets.

Example 4. Find the sample covariance matrix for the following data:

Solution:

FAQs on Covariance Matrix

What is Covariance Matrix in Math?

Covariance matrix is a square matrix that denotes the variance of variables (or datasets) as well as the covariance between a pair of variables. It is symmetric and positive semi definite.

What is the Formula for Covariance Matrix?

The general formula for the variance covariance matrix is given as follows:

What is the 2 × 2 Covariance Matrix Formula?

Suppose X and Y are two data sets then the 2 × 2 covariance matrix formula is given as

How to Find Covariance Matrix?

The steps to find the covariance matrix for a sample are as follows:

• Find the sample variance for all datasets using the formula

• Find the sample covariance between all pairs of datasets given by

• Substitute the values in the matrix

Using different formulas, the same steps can be applied to find the covariance matrix of population data.

What are the Properties of Covariance Matrix?

The main properties of a covariance matrix are listed as follows:

• It is a square matrix.
• It is symmetric.
• It is positive semi-definite.
• The eigenvalues are positive and real.

Is the Variance Covariance Matrix Symmetric?

Yes, variance covariance matrix is symmetric. Thus, if the transpose of a covariance matrix is taken, it will result in the original matrix. In other words, M^T = M, where M is the covariance matrix.

What are the Applications of Covariance Matrix?

Covariance matrix is widely used in the field of economics, financial engineering, and machine learning. The Cholesky decomposition makes use of the covariance matrix to perform a Monte Carlo simulation. This simulation is used to create various mathematical models.

What is the General Form of a 3 ⨯ 3 Covariance Matrix?

The general form of a 3 ⨯ 3 covariance matrix is given as follows:

What are the sectors where Covariance Matrix can be used?

Covariance Matrix is used in the field of Mathematics, Machine Learning, Finance and Economics. Covariance Matrix is used in Cholskey Decomposition to perfom Monte Carlo Simulation which is used to create Mathematical Models.

Related articles

Covariance Matrix Formula

✅ Covariance Formula ⭐️⭐️⭐️⭐️⭐

✅ Matrix Formula ⭐️⭐️⭐️⭐️⭐️

✅ Inverse Matrix Formula ⭐️⭐️⭐️⭐️⭐