That description is consistent with having reverse dns lookup enabled for logging, first an initial delay while the ip number is resolved, then fast while it's the cached value that's used.
You might want to check that that isn't the cause.
Reductionist and proud of it.
Being ignorant is not so much a shame, as being unwilling to learn. Benjamin Franklin
Chase after the truth like all hell and you'll free yourself, even though you never touch its coat tails. Clarence Darrow
A person who won't read has no advantage over one who can't read. Mark Twain