Are all comments with links spam?

Is link building by commenting against Google Webmaster Guidelines? What if it's a topically relevant site and the comment is meaningful?