The tables below show the rating results for each system at the conclusion of the 2019/2020 college football regular season. The RMSE numbers as calculated as previously described. Although the program generates a rating and ranking for > 200 college Div. 1 football teams, only the top 25 from the committee and for each system are shown in the table.

Table 3: RMSE vs. Playoff Selection Committee Rankings for Each Model (2019-20)

Elo	Glicko 2	BG	PageRank	HITS
7.53	5.52	68.85	11.06	10.95

Table 4: College Football 2019-2020 Season (Regular Season Only)

#	Committee	Elo	Score	Glicko 2	Score	Dev.	BG	Score	PageRank	Score	HITS	Score
1	LSU	Ohio State	1930.35	LSU	2177.13	133.3	LSU	170	Oklahoma	439.33	Clemson	160.53
2	Ohio State	LSU	1927.1	Ohio State	2175.98	128.11	Georgia	167	Kansas State	416.42	Ohio State	160.53
3	Clemson	Clemson	1884.14	Clemson	2098.62	135.25	Ohio State	163	Ohio State	247.38	LSU	160.53
4	Oklahoma	Memphis	1883.75	Oklahoma	2081.14	119.68	Penn State	160	LSU	218.44	Appalachian State	135.67
5	Georgia	Oklahoma	1874.69	Memphis	2065.32	123.61	Michigan	158	Baylor	193.15	Boise State	135.67
6	Oregon	Appalachian State	1812.39	Penn State	1998.85	127.77	Notre Dame	147	Temple	170.61	Memphis	133.89
7	Baylor	Boise State	1807.43	Oregon	1993.69	121.56	Florida	118	Memphis	169.15	Oklahoma	133.89
8	Wisconsin	Oregon	1792.9	Georgia	1991.81	125.32	Navy	115	Texas	151.21	Oregon	107.24
9	Florida	Notre Dame	1780.42	Boise State	1984.68	132.95	Auburn	111	Georgia	142.77	Georgia	105.45
10	Penn State	Georgia	1780.38	Baylor	1983.74	126.03	SMU	100	Oklahoma State	137.5	Utah	101.88
11	Utah	Navy	1768.75	Appalachian State	1982.46	131.68	Oregon	93	Wisconsin	115.14	Baylor	92.95
12	Auburn	Air Force	1762.78	Notre Dame	1968.72	126.07	USC	86	West Virginia	107.35	Minnesota	89.54
13	Alabama	Utah	1750.44	Utah	1961.37	121.67	Utah	81	BYU	104.86	Notre Dame	87.75
14	Michigan	Florida Atlantic	1744.23	Florida	1958.16	132.14	California	79	Oregon	96.24	Alabama	84.18
15	Notre Dame	Penn State	1743.85	Wisconsin	1953.66	122.91	Iowa	76	Georgia Southern	94.09	Air Force	84.18
16	Iowa	Florida	1741.85	Navy	1953.64	127.85	Washington State	75	Auburn	92.22	SMU	84.18
17	Memphis	Wisconsin	1737.6	Michigan	1948.62	124.13	Boise State	70	Cincinnati	91.83	Navy	84.18
18	Minnesota	Baylor	1731.16	SMU	1933.21	132.63	Clemson	67	Michigan	91.63	Penn State	82.39
19	Boise State	Michigan	1728.5	Auburn	1923.82	127.85	Oklahoma	65	Notre Dame	90.47	Florida	80.61
20	Appalachain State	Iowa	1723.57	Air Force	1919.14	127.61	Indiana	61	Penn State	88.92	UCF	73.62
21	Cincinnati	Auburn	1722.53	Minnesota	1912.2	133.86	Air Force	58	Appalachian State	85.11	San Diego State	70.04
22	USC	SMU	1706.03	Alabama	1905.39	135.89	Nebraska	57	Minnesota	84.58	Louisiana	69.89
23	Navy	Cincinnati	1705.69	Iowa	1900.24	126.42	Temple	49	South Carolina	79.94	Florida Atlantic	69.89
24	Virginia	USC	1694.15	Cincinnati	1892.5	121.71	Baylor	48	Iowa	78.57	Louisiana Tech	68.26
25	Oklahoma State	Alabama	1688.82	USC	1859.68	118.26	Stanford	47	Florida	77.77	Wisconsin	66.31

The 2020-2021 Season

The tables below show the rating results for each system at the conclusion of the 2020/2021 college football regular season. The RMSE numbers as calculated as previously described. Although the program generates a rating and ranking for > 200 college Div. 1 football teams, only the top 25 from the committee and for each system are shown in the table.

Note: In 2020/2021, with Covid-19 forcing many conferences to only play against teams within the conference, the graph-based rating systems suffered. For example, the BeatGraph approach purposefully discards some results in an attempt to create an acyclic graph. As a result, despite Alabama having a perfect score in the SEC, some results were discarded and they were ranked lower than others who simply played teams outside of their conference. Another challenge for the rating systems in the 2020/2021 season was that many teams did not play the same number of games, which is crucial when the regular season is only about a dozen games to begin with.

Table 5: RMSE vs. Playoff Selection Committee Rankings for Each Model (2020-21)

Elo	Glicko 2	BG	PageRank	HITS
11.69	11.02	43.71	19.52	9.84

Table 6: College Football 2020-2021 Season (Regular Season Only)

#	Committee	Elo	Score	Glicko 2	Score	Dev.	BG	Score	PageRank	Score	HITS	Score
1	Alabama	Coastal Carolina	1857.55	Cincinnati	2092.7	148.96	Coastal Carolina	54	Clemson	423.27	Coastal Carolina	194.88
2	Clemson	Cincinnati	1829.16	Coastal Carolina	2081.39	145.57	Oklahoma	50	Notre Dame	414.66	Alabama	194.88
3	Ohio State	Alabama	1825.76	Alabama	2060.45	153.97	TCU	47	Ball State	192.47	Cincinnati	159.45
4	Notre Dame	Clemson	1799.04	San Jose State	2059.23	170.39	BYU	44	Buffalo	175.54	Clemson	146.06
5	Texas A&M	San Jose State	1777.77	Clemson	2026.37	137.56	Texas	43	Miami (OH)	174.76	Notre Dame	146.06
6	Oklahoma	BYU	1760.24	Notre Dame	2001.18	141.82	Oklahoma State	41	Coastal Carolina	150.97	BYU	142.95
7	Florida	Notre Dame	1754.02	Ohio State	1996.41	170.66	Cincinnati	39	Oregon	121.35	Liberty	134.57
8	Cincinnati	Louisiana	1749.79	BYU	1986.14	146.24	Clemson	34	Iowa State	109.73	Louisiana	125.24
9	Georgia	Ohio State	1747.55	Louisiana	1934.81	148.57	Louisiana	33	Oklahoma	106.67	San Jose State	124.02
10	Iowa State	Texas A&M	1735.68	Texas A&M	1897.49	150.11	Tulsa	32	Stanford	95.72	Army	112.8
11	Indiana	Oklahoma	1733.08	Tulsa	1892.76	155.78	UCF	30	Oklahoma State	81.58	Texas A&M	107.52
12	Coastal Carolina	Ball State	1728.34	Ball State	1879.08	152.54	Appalachian State	30	Utah	80.89	Ohio State	106.3
13	North Carolina	Army	1705.9	Indiana	1873.64	163.66	Virginia Tech	30	Cincinnati	79.14	Oklahoma	104.41
14	Northwestern	North Carolina	1701.47	Miami	1869.83	141.43	Georgia State	28	Louisiana	77.58	Ball State	100.08
15	Iowa	Liberty	1684.24	North Carolina	1851.8	131.67	NC State	28	Oregon State	77.46	Marshall	99.14
16	BYU	Indiana	1681.46	Liberty	1844.53	156.95	Virginia	25	Alabama	76.3	Indiana	87.64
17	USC	NC State	1670.11	Oklahoma	1841.03	136.39	Boston College	23	North Carolina	74.66	North Carolina	85.75
18	Miami (FL)	Tulsa	1667.26	Georgia	1827.73	144.78	San Jose State	22	Kansas State	71.74	Miami	85.75
19	Louisiana	Miami	1662.99	Army	1826.71	133.56	Georgia Southern	21	Ohio State	69.64	Iowa	81.42
20	Texas	Georgia	1661.96	Buffalo	1823.82	179.19	Pittsburgh	20	USC	67.09	Northwestern	81.42
21	Oklahoma State	Iowa State	1661.53	NC State	1816.55	136.74	Stanford	20	Miami	66.92	NC State	76.42
22	San Jose State	Appalachian State	1655.66	Boise State	1799.52	164.36	Washington	18	Colorado	62.05	USC	76.14
23	NC State	Iowa	1651.3	Appalachian State	1793.31	132.65	Boise State	17	TCU	61.53	Nevada	72.09
24	Tulsa	Buffalo	1638.17	Iowa State	1790.24	127.52	Utah	16	California	60.02	Buffalo	69.92
25	Oregon	UAB	1622.23	USC	1780.53	180.49	Florida Atlantic	14	NC State	59.71	SMU	68.04

Return to Top

In this section, I describe the models I used to predict the winner of the NCAA College D1 football games. I used the R programming language and machine learning to build models based on classification algorithms described at cran.r-project.org . The algorithms classify (predict) games not yet played as either a home team victory or an away team victory. I use 4937 past games between season 2010 to 2020 for training and a different set of 670 past games over the same years for testing. For now, I build each model using data for the following variables:

• P5_DIFF: "0" If both or no teams are in a Power 5 conf. "1" if only the home team is in P5. "-1" if only the away team is in P5.
• H_W_M_A_W: (# of games the home team has won this season) - (# of games the away team has won this season)
• H_L_M_A_L: (# of games the home team has lost this season) - (# of games the away team has lost this season)
• H_APS_M_A_APS: (Avg. # of points the home team scored this season) - (Avg. # of points the away team scored this season)
• H_APLI_M_A_APLI: (Avg. # of points the home team conceded this season) - (Avg. # of points the away team conceded this season)
• H_P5W_M_A_P5W: (# of P5 teams the home team has beaten this season) - (# of P5 teams the away team has beaten this season)
• GLICKO_DIFF: (Home team's Glicko 2 rating score) - (Away team's Glicko 2 rating score)
• RR_DIFF: (Home team's recruit rank) - (Visiting team's recruit rank). Recruit ranking from 247sports.com.

Random Forest: Random Forest is a machine learning method that constructs many decision trees at training time as described here . I use the R programming language to build a model based on the ‘randomForest’ package described here . As shown in Figure 2 on the left side, for our test runs, the "out of bag" error converges to about 26% error rate (blue line) overall with a lower error rate when it predicts a home victory (~18%) and a higher error rate when it predicts an away victory (38%). Also shown in the figure on the right side is the variable importance plot. The mean decrease accuracy plot shows how much accuracy the model will lose if the variable is not included, higher values indicate the most important variables. The mean decrease in gini plot shows how much each variable contributes to the homogeneity of the trees in the model, higher values indicate the most important variables.

Support Vector Machines (SVM): A Support Vector Machine is a supervised machine learning algorithm. In my case, I use it for classification purposes where the SVMs attempt to find a hyperplane that best divides the dataset into two classes. More details on SVMs can be found here I use the R programming langauge to build a model based on the ‘e1071’ package described here .

Neural Net: An artificial Neural Network is a machine learning model inspired by the neural networks in the brain. In our case, the first hidden layer has four neurons and the second hidden layer has three neurons. The figure below shows an example of the Neural Network results from one of my runs. More details on a Neural Net can be found here I use the R programming language to build a model based on the ‘neural net’ package described here .

Decision Tree: The decision tree is a predictive model that we use to build a classification tree based on our variables. As shown in th figure below, each node represents an "if-else" statement that can be followed to determine whether the decision tree predicts a home or away win. More details on decision trees can be found here I use the R programming language to build a model based on the ‘partykit’ package described here .

Naive Bayes: A Naive Bayes classifier is a probablistic machine learning model that is using for classification tasks based on the Bayes theorem. More details on Naive Bayes can be found here . I use the R programming language to build a model based on the ‘naivebayes’ package described here . The figures below show a probability density plot for each variable used in the model.

k-Nearest Neighbor (k-NN): k-NN is a non-parametric classification method. Since we are solving a classification problem we are using k-NN classification and the output is either "away" or "home". An object is classified based on the class most common among its "k" nearest neighbors where "k" is a small positive integer. More details can be found here I use the R programming language to build a model based on the ‘class’ package described here .

Game Prediction Training and Test Results

In this section, I summarize the training and test results for each machine learning model we just described. As noted before, I used 4937 past games from seasons 2010 to 2020 for training and a different set of 670 past games in the same years for testing. The figures below show the confusion matrix and misclassification % for the training and test data. In the matrix the upper left and lower right quadrants represent where the prediction matched the result for each model while the opposite quadrants represent where the prediction did not.