Early career

Baker signed for Aston Villa as a 13-year-old in 2004, and rose through the ranks at the club, making his Academy debut in 2007. He was a pivotal figure in the centre of defence alongside Ciaran Clark in the 2007–08 season, as Villa clinched the national title, defeating Manchester City. Baker finished the season with one goal from 23 appearances for the Academy team, and two goals from seven appearances for the Reserves, who also tasted success by clinching the Premier Reserve League South.

The defender progressed to the first team in the summer of 2008 by making the squad for the tour of Switzerland, of which Baker played in both matches against FC Wil and FC Zürich. In July 2008, Baker was named on the bench for the second leg of the UEFA Intertoto Cup tie against Odense Boldklub. A week later he was again named on the bench, this time for the trip to Icelandic side Fimleikafélag Hafnarfjarðar in the second qualifying round of the UEFA Cup.

On 23 October 2009, Baker signed for Lincoln City on a month-long loan deal, alongside fellow defender Eric Lichaj. Lincoln City manager and former Aston Villa striker Chris Sutton handed Baker his league debut at Sincil Bank on 24 October 2009, in a 0–0 draw with Torquay United. Baker impressed at Lincoln, and his loan deal was eventually extended to the end of the season.^[4]

Baker was banned from attending Aston Villa's League Cup final against Manchester United at Wembley Stadium on 28 February 2010 after being caught trying to sell his allocation of tickets via the social networking website Facebook. Baker had attempted to sell five tickets at a price of £200 each, but later said that he regretted his actions.^[5][6]

2010–11 season

Baker was brought in to the Aston Villa squad for the derby match against Birmingham City on 16 January 2011. He was an unused substitute as Villa drew 1–1.^[7] Baker made his starting debut against Wigan Athletic on 25 January at DW Stadium. On 29 January, his second ever start, Baker gained an assist for the third goal but was then sent off for a dangerous tackle in an FA Cup fourth round tie against Blackburn Rovers. On 26 February, Baker made his second league start for Villa against Blackburn Rovers, but went off injured in the first half after a collision with Blackburn goalkeeper Paul Robinson. Villa announced on 2 June 2011 that Baker had signed a new three-year contract with the club.^[8]

2011–12 season

On 22 November he joined Millwall on a one-month loan deal.^[9] He was assigned the squad number 27 and made his debut four days later, helping Millwall keep a clean sheet against Crystal Palace.^[10] He made six league appearances before his loan spell ended in December with a 1–0 win against Portsmouth on 26 December being his last game.^[11] He returned to Villa and made his first appearance of the season coming on as a late substitute for Richard Dunne against Manchester City on 12 February.^[12]

2012–13 season

During the 2012–13 season, Baker established himself as a centre back with great aerial ability. Due to an injury to Ron Vlaar, Baker was partnered with fellow academy graduate Ciaran Clark and impressed during a difficult spell where Villa conceded many goals. On occasion he also played at left back in the absence of Joe Bennett. On 9 March, manager Paul Lambert praised his maturity for producing another outstanding performance after slicing the ball into his own net to give Reading a first-half lead.^[13] In May 2013 Baker signed a new three-year deal with Aston Villa which would keep him there until 2016.

2013–14 season

Baker made his first appearance against Arsenal but was substituted in the 17th minute due to injury. He missed the next games against Chelsea, Liverpool and Newcastle before making his return to the squad as an unused substitute against Norwich. He made his full return starting at Villa Park in a dramatic 3–2 win over Manchester City, before being dropped to the bench against Hull. After that, Baker and Vlaar were Paul Lambert's preferred defensive partnership, however, due to Vlaar's constant injury problems, Baker was often partnered with fellow academy graduate Ciaran Clark.

2014–15 season

Although Baker was a regular throughout the 2013–14 season, he was an unused substitute in Villa's opening fixture, a 1–0 victory against Stoke. He made his first appearance of the season as an 82nd minute substitution for Aly Cissokho. After injury ruled captain Ron Vlaar out, Baker made his first start against Liverpool, putting in a great performance alongside Philippe Senderos, keeping a clean sheet in a 1–0 victory. He was set to start against Arsenal, however, like many of his teammates suffered from an illness and was left out of the squad. Baker went on to start against Chelsea, Manchester City and Everton. He wasn't included in the squad against Queens Park Rangers, before starting against Spurs. On 8 November 2014, during the 0–0 draw against West Ham United, Baker picked up a knee injury. After a long lay off, Baker was an unused substitute against Leicester before starting against Liverpool on 17 January 2015.

2015–16 season

On 31 July 2015, Baker ended transfer speculation when he signed a new four-year contract with the club.^[14]

On 1 September 2015, Baker joined Bristol City on a season-long loan.^[15]

2016–17 season

Following Aston Villa's relegation from the Premier League, Baker returned to Aston Villa. Under Roberto Di Matteo, Baker often found himself behind the club's new arrivals Tommy Elphick and James Chester, being used as a substitute. Following an injury to Elphick, and the replacement of Di Matteo for Steve Bruce, Baker formed an impressive partnership with Chester. On 19 November, Baker scored his first goal for Aston Villa against Brighton and Hove Albion, heading in an Albert Adomah delivery. The match ended 1–1.