One of the perpetual debates in baseball is: "Are your five best pitchers always your starters?" And generally, the answer seems to be yes. But there are exceptions, pitchers with a limited repertoire, exceptionally young pitchers or pitchers with durability limits tend to perform better as relievers. Another debate is "when do you use your best reliever?" Often times, you'll want to use him in high leverage situations, generally late and close situations. Sadly, poorly defined statistics like the save have confused the issue and excellent pitchers that might otherwise prevent lead loss or keep a one run defecit from becoming too large to overcome are reserved only for situations that can get the pitcher a save. This has lead to the movement from using firemen to using closers. Personally, I'll take the fireman role.
Regardless, some pitchers have shown that they excel in the clutch (that's not to say that don't excel at other times, of course), and it's these pitchers that qualify for the fireman role. Pretty much any great starter would qualify, of course.
In addition to situation agnostic pitching stats, relievers are measured by their contribution to the odds of their team winning. The basic concept is called Win Probability Added (WPA) and it's a measure of how much the pitcher increases an average team's probability of winning according the location of runners (the base states), the number of outs, the inning and the score. A player with a positive WPA leaves his team in a better position to win the game while one with a negative WPA leaves them in a worse position. Batting WPA is similar, except it is added or deducted per plate appearance or stolen base attempt. No one has managed to add in fielding WPA yet. WPA data goes back to 1974.
There's also derived stats called shutdowns and meltdowns that are essentially better versions of saves and blown saves. A pitcher gets a shutdown if he increases his team's chances of winning by more than 6% and a meltdown if he lowers them by more than 6%. I used these stats in addition to context independent stats such as ERA, FIP and WHIP.
It's important to note that WPA and associated stats are much more descriptive than predictive.
Selection Mariano Rivera
It's hard to pick anyone else. There are definitely greater pitchers than Rivera, but it's hard to say any of them would actually be better over one inning. Pedro Martinez, maybe. Such a pitcher would probably end up as a starter, anyway, though. Rivera's cutter is legendary and it's effective against both handed batters. He uses some other pitches, including a fastball, but his placement and movement on the cutter make it maybe the most feared pitch in baseball history. Strong control, good stuff (though not really swing and miss, only a 8.26 K/9 rate) and the ability to induce weak contact have made Rivera the greatest relief pitcher of all time. And truly great enough to make this roster.
His career pitching line is 2.21/2.75/49/62 (ERA/FIP/ERA-/FIP-). Teams have scored less than half as many earned runs against Rivera compared to an average pitch across his career. He's also got good ground ball tendencies, creating 53.2% ground balls and only 30% fly balls and with only 6.1% home runs per fly ball, he's rarely homered on. Truly a hard pitcher to turn the game around on.
Used almost exclusively in high leverage situations, he's really given his team a boost. His total WPA is 54.70 (according to Fangraphs), in the same neighborhood as batters like George Brett and Eddie Murray. For pitchers, he's second only to Roger Clemens and just ahead of Greg Maddux and Randy Johnson.
From a shutdown/meltdown standpoint, Rivera is 547/113. That is to say, he's left his team with a signficantly better chance to win almost five times as often as he's significantly hurt them. Not surprisingly, he's the all time Shutdown leader (ahead of Trevor Hoffman and Lee Smith), but nowhere near the meltdown leader. Simply put, Marianno Rivera is the greatest high leverage pitcher since 1974 and probably of all time.
And everybody knows it.
If you want a few more strike outs from your fireman/closer or think you're going face a unusually large number of left handed batters, Wagner makes a good choice. His lifetime K/9 is 11.92 (33.2%) and he's one of the few relievers that have a claim to match Rivera on ability, if not as much on longevity. His FIP- (normalized fielding independent pitching, lower is better) is only one higher than Rivera's. His Shutdown Meltdown ratio is also a great 389/100 (3.89) (higher than likely Hall of Famer Hoffman) which, while higher than nearly everyone else, is still less than Rivera's 4.84.
Context independent pitching line: 2.31/2.73/54/63.
WPA added: 28.78
When it comes to stuff, it's hard to find a better pitcher than Nolan Ryan. Ryan is the hardest pitcher to hit in the history of baseball, with a batting average against of only .200 and a strike out percentage of 25.3%. More than one of every four batters Ryan faced as a starter struck out against him. Relievers, who rarely face the same batter twice in a game, usually get a big boost in K%, so we'd expect Ryan to do so, too. Ryan's glaring weakness, though, was his high walk rate (12.4%), granting 4.67 free bases per 9. Exceptionally durable (and the all time leader in both strikeouts and walks), Ryan is a Hall of Fame starting pitcher, but not an all time top 10 in terms of results. But I imagine that as a reliever, particularly used more like a closer than a fireman, coming in with the bases empty in the 7th or later to protect a lead, he'd be amazing. No batter getting the opportunity to adjust and knowing that even a walk or two doesn't help if no one can make contact to drive them in.
Pitching line (as a starter): 3.19/2.97/90/84