Can’t see Magento 2 front nor admin page on excluded IP when in maintenance mode. Using Varnish

So as I said, I am trying to see my website while others see it in maintenance.
BTW I have Varnish configured.

What I did:

magento maintenance:enable --ip=

What hapened:

At first I was able to see only frontend from excluded IP, but admin was regular “Service temporary unavailable” page.
Than I red about maintenance.flag in var and how it is what causes maintenance mode. So I deleted it to deactivate maintenance and again created it with touch. After that frontend was also unavailable from my excluded IP.

System:

nginx
magento 2.3.3
mariadb 10.*
php 7.2
Varnish [latest as of today]

I followed official magento 2 installation guide along with this for setup of varnish and letsencrypt.

I gathered from other questions that it is problem with varnish and it can be fixed with forwarding headers, but not sure how to do this on Nginx server.