Sabermetrics: Astros Bullpen Revisited

The improvement in Astros' wins and losses this year is closely tied to better bullpen results this season.  And the improved bullpen is the product of GM Jeff Luhnow's off-season objective of improving the team's relief pitching.  The free agent signings of Luke Gregerson and Pat Neshek are the most prominent changes in the bullpen.

Although improving the bullpen is an unassailable decision in hindsight, some critics at the time viewed it as a waste of money.  For example, this off-season Beyond the Boxscore article criticized Luhnow for "building a team backwards," asserting that the Astros are so far away from competing that money expended on the bullpen is folly.  As a retrospective, you may find that article entertaining.  I am particularly amused at the argument that Carlos Correa and other significant prospects have an ETA of 2017 or later.  However, I contend that these criticisms were the off the mark because they underestimated the extent that bullpen improvement, itself, could turn the Astros into a competitive team.

In December, I attempted to quantify how much changes in the Astros bullpen from 2014 to 2015 might improve the team's record.  I used a version of "with and without" method of analysis.   I estimated Gregerson's and Neshek performance, adding it into 2014 the bullpen results, subtracting an equivalent innings amount of negative performance by relievers who were released, and comparing the revised bullpen performance to the actual 2014 bullpen performance.  My December article is here.

My December 2014 analysis concluded that the range of improvement in the bullpen is 4.5 - 8.6 wins.  This projected improvement in W-L record is significantly higher than the typical estimates based only on the free agents' fWAR, because the typical approach ignores the removal of negative bullpen performers.  Acquiring back end relief pitchers has a ripple effect throughout the bullpen, as better relief pitchers replace the innings previously pitched by worse relievers.  This is sometimes called "chaining" and fWAR attempts to account for the effect through a leverage adjustment.  But that adjustment is generalized, and doesn't specifically consider the degree of negative reliever performance--below replacement level--in the Astros' 2014 bullpen.

I will now revisit my previous projection and compare it to the actual 2015 bullpen performance.  Obviously the season isn't over, and the current performance can change.  But we are around the two-thirds mark in the season, and the current bullpen results have had a major impact on the Astros' first place rank in the AL West.   I'll give you a hint: the Astros' bullpen improvement is better than I anticipated in  December 2014.

In my December article, the largest increase in bullpen wins is based upon utilizing RE24 wins.  The RE24 metric, part of the win probability family of statistics, is sometimes suggested as a useful measure of relief pitcher performance. Here is my description of RE24 from that article:

RE24 is a win probability statistic based on the 24 base-out states.  An example of a base-out state is "bases loaded, one out." The number of base-out combinations is 24. The run expectancy for each of the base-out states is the average probability of runs scoring in that situation. RE24 measures whether the pitcher produced a change in run expectancy which is more or less than the average run expectancy for the particular base-out situation.  REW converts RE24 into wins.  The proponents of RE24 contend that RE24 is particularly useful for representing relief pitcher performance.  A relief pitcher may enter the game to face any one of the 24 base out situations, and RE24 measures his effectiveness based on the situation.

Amazingly, only three relievers in the 2014 bullpen had a positive RE24.

My previous article projected a REW improvement of 7.5 wins above average.  Because WAR is based upon replacement level, I estimated that the WAR value is approximately a 9 win improvement---which is quite large compared to the win projections we normally associate with bullpen changes.

So far this season, the Astros' bullpen shows a REW improvement of 11.2 wins above average compared to last year.  This surpasses my previous projection, even though the season is not complete.  If the current trend continued, the REW improvement could exceed 14 wins above average by the end of the season.  Stating the improvement on a replacement level basis would result in even a higher win improvement total.

This comparison suggests that the Astros' win-loss record would be less than .500 without the improvement to the   bullpen.  My previous article also projected an improvement in SIERA.  

I projected a bullpen SIERA of 3.16, which would have been a 0.42 improvement over 2014.  However, the Astros' current SIERA is 3.01, which is an improvement of 0.57. Given that the Astros' bullpen SIERA is lower than its FIP, this suggests that fWAR (which is based on FIP) may understate the Astros' bullpen improvement.

The rankings based on the 30 major league bullpens also drive home the point.   In 2014, the Astros were dead last--30th ranked--in RE24.  My December article pointed out that the projected RE24 improvement would have placed the Astros' bullpen in 10th place.  The Astros' current RE24 for the bullpen is ranked 3d, behind only the Royals and Cardinals.   Also, keep in mind that RE24 is affected by fielding quality--so, the improvement in the bullpen relievers is also part of a strategy that included improvement in the Astros' fielding.

According to SIERA, the Astros had the 7th worst bullpen in 2014.  My previous article suggested that the Astros' bullpen would climb to an 8th ranking in SIERA.  However, the Astros' bullpen currently is ranked 2d in SIERA, just .002 behind the 1st ranked Yankees.

The method used in my December article is static in nature.  There are a number of factors which favorably affected the bullpen, but which were not projected.  For example, we didn't even know whether Will Harris would make the Astros' 25 man roster in December 2014.  We knew that he looked to be a promising waiver pick up.  But nobody would have predicted the strong run prevention impact on the bullpen.  Another illustration is Josh Fields' improvement over 2014. Fields has a sub-3 ERA,compared to a 4+ ERA in 2014.   My inclination at the time was to believe Fields would benefit from regression in 2015--but my December projection didn't attempt a detailed projection of relief pitcher regression.  As mentioned above, solid improvement in the Astros' defense also benefited the reliever corps, but that was not included in the December projection.

Do you like the improvement in the Astros' bullpen results?