nginx – PHP-FPM slows down after a while – Opcache File Cache

My set up is a large drupal website with 50,000 requests per day. My main issue is that Php-fpm slows down after 10-15 minutes of a restart (1500+ code files). Cache is only full 40% (174mb of 512mb). Initial server response time is over 700ms but goes down to 170ms for first few minutes of a restart as per Google site speed testing.

I found this answer with similar issue and have added Opcache file cache with some improvement.
forums.cpanel.net/threads/php-fpm-restart-necessary-for-site-speed.650549/#post-2654105

Current Opcache Settings

 [opcache]
    opcache.enable=1
    opcache.memory_consumption=512
    opcache.interned_strings_buffer=64
    opcache.max_accelerated_files=50000
    opcache.max_wasted_percentage=15
    opcache.validate_timestamps=0
    opcache.revalidate_freq=0
    opcache.mmap_base=0x20000000
    opcache.file_cache="var/www/cache"
    opcache.file_cache_fallback=1

Any other ideas on how to optimise the above to achieve consistent php-fpm performance. (Set up is Php-fpm 7.2, Nginx, Drupal, Mariadb on AWS server with 64G memory)

Tried adding File cache. It has shaved off 100ms but probably a lot more can be done.