The album was recorded in 1966 and 1967 during the time Gaye and Terrell's first collaboration album United was released. After recording You're All I Need, Tammi Terrell collapsed onstage while performing with Gaye at the Hampden–Sydney College homecoming in Virginia. She was later diagnosed with a brain tumor, and could no longer record nor perform live. According to Gaye, the final Gaye/Terrell album, Easy, would be completed by having Valerie Simpson fill in for Tammi Terrell on most of the album's songs, and having Gaye overdub archived Terrell solo tracks for two tracks. Simpson vehemently denies this and says they had to record Tammi's vocals in pieces in order to get the project done, but in the Unsung documentary about Terrell, she admits to singing guide vocals intended for the unavailable Terrell.
Nickolas Ashford and Valerie Simpson produced the singles "Ain't Nothing Like the Real Thing", "Keep On Loving Me Honey," and "You're All I Need to Get By."[2] Harvey Fuqua and Johnny Bristol produced the rhythm tracks and Terrell's vocal for "You Ain't Livin' 'Til You're Lovin'," with Ashford and Simpson completing the production and supervising Gaye's vocal.[2] Fuqua and Bristol also produced "I'll Never Stop Loving You Baby" while Motown CEO Berry Gordy, Jr.'s brother Robert Gordy produced "I Can't Help But Love You."[2] Harvey Fuqua and Johnny Bristol produced "Come On and See Me," "Baby Dont'cha Worry," "Give In, You Just Can't Win," "When Love Comes Knocking At Your Heart," "That's How It Is (Since You've Been Gone)," and "Memory Chest" as Tammi Terrell solo tracks in 1965 and 1966, and had Gaye overdub his vocals to them to create duet versions.[2]
The hit title track was revived 25 years later as a duet between rapper Method Man and hip-hop soul singer Mary J. Blige. Their rendition, titled "I'll Be There for You/You're All I Need", won Method Man and Blige a Grammy Award for Best Rap Vocal Performance by a Duo or Group in 1995.
