clock menu more-arrow no yes

Filed under:

Evaluating the bullpen with Run Expectancy Statistics

New, 2 comments

RE24 is a win probability stat that can tell us how well relief pitchers performed

MLB: Houston Astros at Kansas City Royals Denny Medley-USA TODAY Sports

“Throw Strikes. Home plate don’t move.” —Satchel Paige

A week ago, I wrote about the run expectancy stat (“RE24” or “REW”) as a measure of team offense. What can RE24 tell us about team pitching? I’m not sure it can tell us much about starting pitchers that we can’t glean from traditional pitching stats like RA/9 or FIP. But the run expectancy stat may be a better approach to comparing bullpen performance.

Relief pitchers can be difficult to evaluate statistically. They may pitch only a portion of an innng. Runs allowed are notoriously bad indicators because the runners on base may be charged to the previous pitcher, and the outcome of runners put on base by a reliever may be determined by the next pitcher. Saves and holds are reliever stats that don’t consider the base-out situations when relievers entered the game. RE24 (“REW” when converted to wins) is based on each plate appearance faced by the pitcher, the base-out situation as the plate appearance began, and whether the pitcher improved or decreased the probability that run(s) will score in the inning. In short, RE24 measures run prevention at a granular level. RE24 is an outcome-oriented statistic which may or my not predict future performance.

Career REW for Baseball’s Best Relievers

Career REW for elite relievers demonstrates that RE24 / REW is a useful measure of reliever run prevention. The top five relievers in career REW are shown below. REW and WAR (Wins Above Replacement) cannot be compared directly because REW reflects wins above average rather than replacement level. Therefore, I estimated the reduction necessary to convert WAR to Wins Above Average (WAA). This seems to indicate that the elite relievers over perform their WAR or WAA measures based on run expectancy.


REW, WAA, Difference, Percent Increase Over WAA

Mariano Rivera 33.89/ 26.2/ 7.6/ 29%

Rich Gossage 20.96/ 13.0/ 7.9/ 61%

Billy Wagner 19.75/ 14.9 / 4.8/ 32%

Trevor Hoffman 18.68/ 15.0 / 3.7/ 24%

Lee Smith 17.79/ 13.2/ 4.511 / 34%

MLB: ALDS-Tampa Bay Rays at Houston Astros
Billy Wagner throws out first pitch in 2019 ALDS.
Thomas B. Shea-USA TODAY Sports

Career REW for relievers is shown here, the ranking of relievers based on career WAR is shown here. Note that Billy Wagner moves into the Top 5 relievers if REW is used (replacing Rollie Fingers). Indeed, Billy Wagner checks in as the third best reliever with REW, compared to a sixth best ranking in WAR. REW does not include the inning/score component of leverage, which could be a deficiency (depending on the purpose of the comparison). However, reliever WAR incorporates a leverage adjustment which favors closers. Yet all five of these elite closers improved run expectancy to a greater degree than indicated by their WAR.

2021 Team Rankings

The ranking of team bullpens by REW is shown at Fangraphs.com. It’s no secret that the Astros’ bullpen has blown too many games (68 melt downs, to be exact). The performance has been considerably better since the trade deadline. The REW below reflects games as of August 15, and therefore includes the performance of new reliever acquisitions over the past two weeks. For comparative purposes, I include WAA as calculated by Baseball Reference.com. The B-Ref WAA is based on RA/9 with several adjustments. The Astros rank No. 14 in bullpen REW (as of Aug. 15), as shown in the list below:

(REW / WAA)

  1. Padres 4.32 / 0.4
  2. Rays 3.91 / 0.4
  3. Giants 1.67 / 1.3
  4. Yankees 1.67 / 2
  5. Indians 1.51 / 3.1
  6. A’s 1.5 / 0.9
  7. Red Sox 1.12 / 1.8
  8. Blue Jays 0.93 / 0.6
  9. Mariners 0.35 / 1.8
  10. Dodgers -0.29 / -1.3
  11. Brewers -0.44 / 1.3
  12. White Sox -0.52 / -0.5
  13. Cardinals -0.76 / -4.3
  14. Astros -0.85 / -3.5
  15. Royals -1.11 / -1.1

I performed a statistical test (R-square) to identify potential variables that explain the REW results for bullpen run prevention. Those tests suggest that the following measures are ranked as the best explanatory variables for bullpen run prevention (R-square in parentheses) as measured by REW:

  1. ERA (.64); 2. Shutdowns Minus Meltdowns (SD-MD) (.61); 3. FIP (.60); 4. WHIP (.56) 5. K-BB% (.29); 6. SIERA (.29)

SD and MD are in the same win probability family of statistics as REW, and are usually extolled as superior measures to replace saves and blown saves. Individually, SD and MD are only minimally correlated with REW. But if meltdowns are subtracted from shutdowns (SD-MD), the resulting statistic is the second best explanation for variations in REW. This simple subtraction operation may be one of the better ways of comparing relief pitchers and team bullpens. (The Astros rank 21st in SD-MD, by the way.) Strike out rates and walk rates, by themselves, do not have high R-square values, but the K-BB% statistic on the Fangraphs advanced statistic table is a moderately good explainer of REW. It also turns out that FIP is significantly better at explaining REW run prevention than SIERA.

For what it’s worth, a team’s average bullpen fastball velocity is not correlated with the team’s REW.

RE24 Leaders Among Astros Relief Pitchers

Kendall Graveman is the prize relief pitcher acquistion for the Astros at the trade deadline. Graveman has the 5th highest RE24 among ML relievers in 2021. At the time of this writing Graveman’s RE24 is approximately the same as Craig Kimbrell (ranked 4th).

The Astros Top Five RE24 relievers are shown below. Christian Javier’s RE24 is atypical due to his considerable innings as a starter. Since I cannot separate out the starter innings for Javier’s RE24, a more comparable ranking is based on RE24 per inning pitched.

RE 24 Per Inning

1. Graveman 0.337

2. Pressly 0.207

3. Javier 0.173

4. Stanek 0.097

5. Taylor 0.064

MLB: AUG 13 Astros at Angels Photo by John Cordes/Icon Sportswire via Getty Images