When rate limiting jobs, I use a simple redis rate limit lua script that I posted here: https://mhenrixon.com/articles/client-side-rate-limiter , and I also use a similar scheduler to spread the jobs out. This is how I do it
Sometimes you have to get ahead of things. Rate limits suck but are there for a reason, waiting until the API you are calling raises a rate limit error is bad, really bad. This is how you get ahead and prevent the external API from raising rate limit errors.