Two spots in the Baseball at the 2004 Summer Olympics were available to baseball teams of the Pan American Baseball Confederation. To decide which teams would qualify, a tournament was held in 2003 in Panama, from October 30 to November 10. The tournament was won by Cuba, with Canada also qualifying for the Olympics.

Notably, the United States team did not qualify, after losing to Mexico in the quarterfinals. This elicited shock in the American media^[1] as the USA was the previous gold medal winner, and was the first country to play baseball and is the country where baseball has the largest following. The major reason for defeat was the scheduling of the tournament, which meant the Americans could not even use minor league players.

Although the tournament was originally scheduled to include 13 teams, four withdrew before their first game: Venezuela, the Dominican Republic, Aruba and the Bahamas. As the Bahamas were a last-minute withdrawal, the tournament structure was left unbalanced, with one pool larger than the other.