There are several very good reasons why you should care if your website works well on a phone.
The first reason is a very practical one: more than half of all website views are now done on a phone. The phone has become our collective web browser of choice. Even if you don't use a phone that much, many of your customers probably do. If your website is difficult to use on a phone, people will just move on.
The second reason has to do with search engine ranking. Google recently changed its search algorithms to reward mobile friendly webistes, and conversely penalize mobile unfriendly sites. This is just a recognition of how dominant the mobile platform has become, that sites which are not mobile friendly are simply viewed as less relevant by Google. Everybody knows that your website is not much good if it can't be found on the search engines. If it is not mobile friendly, the changes of it being found goes down significantly.