Can the plan work?

Keep buying, never selling, restrict supply, price keeps rising... Are there any risks or downsides to this plan? With more BTC being removed from circulation, can price only rise? Can a small amount of outstanding coins still move the price down? What if there was only one BTC left that was not stockpiled by the big holders? Any buying makes the price go up, but does the same go with any selling?