I found that Policy objects are not hard to write and test against. Unlike authentication, where I think you would be very ill-advised to try it yourself, authorization has fewer gotchas, and it’s easier to look at your tests and make sure you’re checking the right things. You can also build compound policies by decorating policies with other policies if you design your policies in the right way.
Haven’t looked at Pundit, but I would advise trying to build some very simple objects on your own, find the pain points, and then see if Pundit fixes those.