He has since gone back on his previous expectations insisting that she is actually the "loveliest person" he has ever met. The 21-year-old described how the music industry in the states is responsible for turning many successful artists into 'douche bags'.
The 'Lego House' man met Swift while helping pen the track 'Everything Has Changed' for her fourth album 'Red'. The track leaked online earlier this week, but has since been removed from YouTube.
He said: "Working with Taylor Swift was cool, writing with her was cool. She's like a really sound person," reports ContactMusic.
"With American acts in England, once you get to a certain point you know everyone is humble and cool, but in America when you get to a certain level you kind of get a douche bag pass. Certain artists are allowed to be douches."
Sheeran went on to explain how it was his 'expectation' that he would find Swift irritating because of the success she has had since hitting the big-time at the age of sixteen.
Swift shows love for Sheeran's album, +
He added: "Taylor Swift is one of these people who is the biggest selling female artist in the world right now. Her last record sold just as much as Adele's I think, which is pretty mental to think.
"I expected her to be one of these artists who is allowed to have a douche bag pass but she was the loveliest person I've met."
Meanwhile the Brit-winner spoke about how he sees songwriting as a good contingency in case his career as a performer goes "tits up".
Upcoming One Direction single, 'Little Things', is a song that was written by Sheeran four years ago, and he went on to tell SB.TV that songwriting for other artists is something he would like to do more of.
"Careers can end instantly but when you're a songwriter you can have a career for the whole of your life.
"If everything goes tits up for me tomorrow then the cut I've got on the One Direction album, the Taylor Swift album, the Wretch, the Devlins, I think it's quite an important thing to get my name out there as a songwriter."
Watch Ed Sheeran and Taylor Swift perform 'Lego House' together below.
Photos: Ed Sheeran's headline slot at the iTunes festival 2012
Issue Four of the Gigwise Print magazine is on pre-order now! Order here.