Conservative leader alleging non-citizens may have voted admits there's no hard evidence