camellia-redis-proxy support proxy_protocol to get real client ip in 4-layer-proxy

[caojiajun]

Module: camellia-redis-proxy
Content: camellia-redis-proxy support proxy_protocol to get real client ip in 4-layer-proxy

[caojiajun]

@InputOutputZ https://github.com/netease-im/camellia/blob/master/docs/redis-proxy/other/proxy_protocol.md

could you pls help me to testing this feature?
include such complex testcase:

• use haproxy as a 4-layer-proxy of camellia-redis-proxy
• enable proxy_protcol in haproxy and camellia-redis-proxy
• enable tls in camellia-redis-proxy and redis-cli

[InputOutputZ]

@InputOutputZ https://github.com/netease-im/camellia/blob/master/docs/redis-proxy/other/proxy_protocol.md

could you pls help me to testing this feature? include such complex testcase:

• use haproxy as a 4-layer-proxy of camellia-redis-proxy
• enable proxy_protcol in haproxy and camellia-redis-proxy
• enable tls in camellia-redis-proxy and redis-cli

Sure, I will test it now.

[caojiajun]

@InputOutputZ https://github.com/netease-im/camellia/blob/master/docs/redis-proxy/other/proxy_protocol.md
could you pls help me to testing this feature? include such complex testcase:

• use haproxy as a 4-layer-proxy of camellia-redis-proxy
• enable proxy_protcol in haproxy and camellia-redis-proxy
• enable tls in camellia-redis-proxy and redis-cli

Sure, I will test it now.

thank you very much!

you can compare enable/disable proxy_protocol

• if disable it, client info/client list will return haproxy ip as the client ip
• if enable it, client info/client list will return the real ip as the client ip

[caojiajun]

pls use the latest code, I fix a bug just now. @InputOutputZ

[InputOutputZ]

I managed to test only following cases, because I can't disable TLS.

• enable tls in camellia-redis-proxy and redis-cli
• use haproxy as a 4-layer-proxy of camellia-redis-proxy

HAProxy.cfg

frontend ft_redis
       mode tcp
       bind :7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
       bind [::]:7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
       use_backend camellia_redis_bk

backend camellia_redis_bk
       server redis *:7619 ssl verify required crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem

Per tests works just fine, expected to returns upstream server IP address when I connect through HAProxy.

When I connect directly to Camellia Redis Proxy it returns client IP address.

• enable tls in camellia-redis-proxy and redis-cli
• enable proxy_protcol in haproxy and camellia-redis-proxy

HAProxy.cfg, I tested send-proxy-v2 and send-proxy and both worked just fine.

frontend ft_redis
       mode tcp
       bind :7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
       bind [::]:7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
       use_backend camellia_redis_bk

backend camellia_redis_bk
       server redis *:7619 ssl verify required crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem send-proxy-v2

Per tests works just fine, expected to return client ip address when connecting to Camellia Redis Proxy via HAProxy.

But when I connect directly to Camellia Redis Proxy while proxy-protocol-enable is true, I get this error in terminal:-

Could not connect to Redis at api.zakaria.website:7619: SSL_connect failed: Undefined error: 0

I think it can be considered as expected behaviour since Camellia Redis Proxy is configured to use the proxy protocol and I'm making connection to without send proxy protocol?

I hope my tests helped to confirm the new feature is working perfectly and let me know if there is anything else I should test it.

With thanks to all your work.

[caojiajun]

@InputOutputZ thank you very much for your tests work.

all test results is expected!

And I try to compatible enable/disable proxy_protocol for haproxy and direct_connect, but I need more discuss

[caojiajun]

@InputOutputZ I tried several methods to compatible, finally I decide to keep the way it is.

Thank you very much for you tests work again!

[caojiajun]

@InputOutputZ in latest code, you can specify the proxy_protocol_ports, e.g. tls-port enable and no-tls-port disable.
as the default, both tls-port/no-tls-port enabled proxy_protocol, if proxy_protocol_enable setting to true

you can see config in doc: https://github.com/netease-im/camellia/blob/master/docs/redis-proxy/other/proxy_protocol.md

[InputOutputZ]

@InputOutputZ in latest code, you can specify the proxy_protocol_ports, e.g. tls-port enable and no-tls-port disable. as the default, both tls-port/no-tls-port enabled proxy_protocol, if proxy_protocol_enable setting to true

you can see config in doc: https://github.com/netease-im/camellia/blob/master/docs/redis-proxy/other/proxy_protocol.md

I think it's better solution. Thanks for this.

[InputOutputZ]

@caojiajun I thought about client list, and it returns all connected clients including unauthenticated and I wonder if you can add extra field, beside cmd and user and the rest of the fields, to indicate authentication status e.g :-

id=729 addr=10.8.0.2:65381 laddr=77.68.117.221:7619 name= db=0 age=14 idle=0 sub=0 psub=0 multi=-1 cmd=client user=default auth=false

Given the user is not authenticated but connected to redis-cli and:-

id=729 addr=10.8.0.2:65381 laddr=77.68.117.221:7619 name= db=0 age=14 idle=0 sub=0 psub=0 multi=-1 cmd=client user=default auth=true

Given the user has already authenticated via AUTH command after connecting to redis-cli?

With thanks.

[InputOutputZ]

Btw, refer to Client List documentation, it said user field is the authenticated username of the client while now I have not authenticated just the user and from other session when I run client list I get the unauthenticated session having user=default field, when it shouldn't.

[caojiajun]

I test direct connect to redis, unauthenticated client connection return user=default

but I found if a client connection unauthenticated, the cmd field is NULL

[InputOutputZ]

I test direct connect to redis, unauthenticated client connection return user=default

but I found if a client connection unauthenticated, the cmd field is NULL

Aha, but in mine I get cmd=command, so I guess if cmd was equal to NULL/command then it means the user is unauthenticated?

I run Redis 7.0.12.

[InputOutputZ]

Update regardless to the authentication status, cmd field changes to recent commands ran by the connected user.

From authenticated.

From unauthenticated.

[caojiajun]

Update regardless to the authentication status, cmd field changes to recent commands ran by the connected user.

From authenticated.

From unauthenticated.

yep, cmd update always, ignore the reply is error or success.

[caojiajun]

Update regardless to the authentication status, cmd field changes to recent commands ran by the connected user.

From authenticated.

From unauthenticated.

because redis-cli will send command cmd to redis firstly, it's the feature of redis-cli

[caojiajun]

after authenticated, you can manual send command cmd

[caojiajun]

the best solution is add auth filed in reply as you say, but redis official do not has this feature. So I hesitated whether to do so in proxy

[InputOutputZ]

the best solution is add auth filed in reply as you say, but redis official do not has this feature. So I hesitated whether to do so in proxy

Up to you but I recommend it to take CRP to the next level, because it's helpful in different situations e.g. to keep tracking of who is connected and authenticated to Redis Cluster for security purposes and I think it's needed since redis-cli it doesn't offer extra layer of protection such as 2FA , and to avoid false positives, it's helpful to have authentication status field available, and scanning the web cybersecurity services are a huge source of false positives.

[caojiajun]

I will add auth field in reply of proxy. You convinced me.

[caojiajun]

pls try the latest code @InputOutputZ

[InputOutputZ]

pls try the latest code @InputOutputZ

Tested and works like a charm. Thanks for this :)

[caojiajun]

@InputOutputZ could you pls leave a star, and leave your company in this issue #10 (For open source users only)
thank you very much!

[InputOutputZ]

@InputOutputZ could you pls leave a star, and leave your company in this issue #10 (For open source users only)
thank you very much!

Sorry @caojiajun but I don't have a company yet I just left a star. Thanks.

[caojiajun]

#171

@InputOutputZ