reverse proxy – nginx: restrict proxied IP access?

We are using cloudflare proxy, so real IP is hidden. But if you would know real IP, you can still access site using it. Is there a way to limit access by that IP?

I mean:

by domain: https://some-domain.com -> OK
by proxy IP: https://1.2.3.4 -> OK
by real IP: https://5.6.7.8 -> to restrict.

Or it is pointless to try to restrict it?

Sample of nginx.conf server blocks:

    server {
        listen 80 default_server;
        server_name some-domain.com;
        return 301 https://$host$request_uri;
    }
    charset utf-8;
    server {
        server_name some-domain.com;
        listen 443 ssl http2;
        ...
        ...
    }