I was forced into specialty and I hate it

I've been working at my store now for about a year and a half. I would say a little less than a year ago I came in to work one day and my boss announced I was taking over gift and greeting cards. I was not asked if I wanted to do this, just told. I've always loved to read, and my knowledge of books is exceedingly vast. I'm familiar with a variety of subjects/genres, backlist, and upcoming titles. My boss constantly talks about it, customers comment on it, and several of my co workers have said things too. But because I'm more or less the lead specialty person (only working part time mind you) basically every shift by boss is working I'm told to work on specialty over everything else. Our back room can be overflowing with books, toys, etc but I'm only supposed to work on gift. I hate it. This is not why I took this job and I dread going into work when my boss is working because I know I won't be able to so much as touch a book. I don't mind doing it every now and then and greeting cards aren't really an issue, but I'm honestly so sick of it and I don't know what to do. Is it even something I can ask to be taken off of, or am I just stuck?