Liberal party membership

Could some please explain why the biggest party in Canada only had 400,000 members? With a population of 41,000,000 I would imagine much bigger party participation.