kubernetes – K8s – IdentitySever4 – Nginx – Error returned from introspection endpoint: Gateway Time-out

We need help on a critical issue we face in the K8s environments. The problem is not reproducible on the windows or docker-compose instances. The final user perceives it as a system performance issue. We discovered that after calling various APIs (each request is sent to the IndentityServer service for authorization), we get “Error returned from introspection endpoint: Gateway Time-out” in the logs of the service that sends the requests and nothing is logged in the IndentityServer logs. The IdentityServer service seems to have insufficient resources and is struggling with the number of requests.
We added more resources and enabled HPA but there has been no improvement. After every x (random number, between 200 and 600) requests, some API calls stay in pending status for a while and then fail (log out – log in solves the issue immediately and it reapers after another set of API calls). If several users trigger requests, each user manages to reproduce the issue but never at the same moment (for one user the requests are pending, but for another one, the requests are resolved for a while).