What plugins did you try to remove? And how did you go about it?
Where do you see the lag? Is it in regular typing, switching windows? Just bootup?
I googled lazyredraw which pointed me to this: set lazyredraw · Issue #78 · tpope/vim-sensible · GitHub
especially the comment of kecebongsoft is interesting.