Avaya CM TCP session reuse resulting in 403 Forbidden

1. Brekeke Product Name and Version:
Brekeke SIP Server 3.8.6.4/501

2. Java version:
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 10.0.1

3. OS type and the version:
CentOS Linux release 7.5.1804 (Core) kernel 3.10.0-862.14.4.el7.x86_64

4. UA (phone), gateway or other hardware/software involved:
Avaya Communication Manager

5. Your problem:
Avaya CM responds with 403 Forbidden Unknown Far-End when Brekeke re-uses an existing TCP session for the SIP INVITE.

Additional background information: We have 2 Brekeke SIP servers set up in a cluster. This issue is only observed happening from the cluster IP address, and only when reusing an existing TCP session, started from the Avaya.

The sequence is always:
1) Avaya opens TCP session to Brekeke SIP servers cluster IP, TCP port 5060.
2) Avaya sends SIP OPTIONS request.
3) Brekeke sends back 603 DECLINE.
4) The TCP session is left active.
5) The Brekeke receives a call request for the Avaya and sends a SIP INVITE over this active TCP session.
6) The Avaya responds with 403 Forbidden (Unknown Far-End).
7) The TCP session is left active. If time passes, the Avaya will send a TCP ACK to keep the session active.
8) The Brekeke responds to the Avaya keep alive ACK and then 5 seconds later sends FIN to end the session.

The 403 forbidden Unknown Far-End rejection from the Avaya is only observed AFTER there is an INVITE from the primary peer's IP, and then an additional INVITE is sent from the cluster IP on said pre-existing TCP session to the Avaya. For example:
Primary IP INVITE to Avaya on TCP 5060
Cluster IP INVITE to Avaya on ephemeral port (source port 5060, because it was already an existing session TO the Brekeke cluster IP).

Additionally, we have set the destination rules for INVITEs to the Avaya_IP:5060, yet this existing TCP session is still used (in addition to the INVITE send from the primary IP, to port 5060 on the Avaya.)

Is there a way to force the Brekeke servers to not reuse that existing TCP session from the Avaya?

Related Avaya documentation is Doc ID SOLN284625. Avaya recommends disabling TCP connection reuse and TCP keepalive. (Which is really funny, considering the connection begins with the Avaya, and the Avaya is also sending the TCP keepalive ACKs!)

I am going to try the following in advanced settings and see how if this issue continues tomorrow: The Avaya connections are the only connections using TCP, so impact from the change should be limited to only this issue.

Also set this at [Configuration]->[Advanced] page.

I'll try that as well - thank you!

Can you provide a reference to any documentation for the net.sip.tcp.connection.put.followable = false setting? I am interested to know specifically what change that enacts. The closest I can find in the documentation is net.sip.tls.connection.put.followable, with a description of "The flag for reaching a SIP UA even if the address/port of Contact-URI is different from the remote IP address/port. (default=true)".

Does setting "net.sip.tcp.connection.put.followable = false" mean the Brekeke will not try to reach the peer if the address port is different, and thus initiate a new connection on port 5060 as specified in the ruleset? If so, that is exactly what I want.

> thus initiate a new connection on port 5060 as specified in the ruleset?

It depends on how existing TCP connection was made but hopefully it solves the issue.

Prior to the version 3.8.6.4/501, its default was "false".

Unfortunately, those net.sip advanced settings actually exacerbated the issue. What resulted was a storm of TCP RSTs from the Brekeke to the Avaya and all the calls failing.

I then removed both of those directives from the Brekeke, because intermittent failure is better than total failure (at least from operations' perspective). Restarting Brekeke didn't help; even after restarting the SIP server was constantly just sending TCP resets to the Avaya connections. After reboot, the SIP sessions went right back to the 403 Forbidden (Unknown Far-End) error response form the Avaya.

What we ended up doing to solve this issue was to disable "Layer 3 Test" on the SIP trunks on the Avaya. This circumvented the initial TCP connection for the SIP OPTIONS check from the Avaya. Therefore, there would no longer be any existing TCP sessions from the Avaya to the Brekeke for the Brekeke to reuse. Instead of the Avaya using SIP OPTIONS packets to check if the Brekeke is up, it will just fall back to ICMP ping. Perfectly sufficient in this scenario. This seems to have solved the issue.

Upon further testing, net.sip.tcp.reuse = false was causing issues (all calls from Avaya fail, with error 404 from the Avaya), but net.sip.tcp.connection.put.followable = false does not appear to cause the same issues.

Our Avaya team found Avaya document SOLN2844625 about this specific issue. Of course, Avaya blames the issue on everyone else, not on their equipment having a bug in TCP stack implementation.

The recommendations from Avaya doc SOLN2844625 are to:
1) Disable TCP connection reuse (NOT what net.sip.tcp.reuse = false does at all! Net.sip.tcp.reuse = false controls the TCP socket port reuse, not session reuse!)
2) Disable TCP-keepalive, which is the default socket option used by Brekeke anyhow. This can be verified with netstat -lpno and note the timers section will show all zeros where Brekeke SIP server is listening, indicating there are no TCP keepalive timers for the Brekeke socket(s).
3) Change the TCP retry interval to 0. This would be a change at the OS level, and also would make the TCP stack timers way too aggressive for normal use.

So we ended up using the following Brekeke advanced directives: 1) net.sip.transport.follow.request=false should cause Brekeke to try to use a different transport than the initial request packet (http://wiki.brekeke.com/wiki/TCP-Support-Details)

2) net.sip.tcp.connection.put.followable=false was recommended in this thread by Harold. Harold mentioned this was the default prior to 3.8.6.4/501. Otherwise, I have not found any documentation available for this directive.

3) net.sip.tcp.keepalive.use=false is the default, but we set it explicitly anyhow to placate the Avaya team.

How about this? With this setting, the SIP Server closes TCP connection immediately just after associated SIP session is closed without waiting for subsequent SIP session.

If the above setting doesn't work, add the line below. FYI:
If you execute the "portlist" command at [Tools]->[Command] page, the list of current UDP/TCP ports what SIP Server manages will be shown.
https://docs.brekeke.com/sip/portlist

What does "net.sip.transport.share.connection=false" do, specifically?

net.sip.transport.share.connection is an experimental setting.

If the value is false, a TCP connection will not be shared with another SIP session. It means new SIP session always starts new TCP connection.

In that case, net.sip.transport.share.connection=false is exactly what we want here!

Harold,
How does net.sip.tcp.connection.put.followable=false effect session behavior if the TCP session was initiated by far end (not Brekeke SIP server)?

For example, Avaya PBX sends SIP options packet to check if Brekeke peer is up. Let us call this TCP session A (Avaya:ephemeral_port -> Brekeke:5060). With no special options set, upon a new call, Brekeke might re-use TCP session A and send a SIP INVITE (Brekeke:5060 -> Avaya:ephemeral_port). This is the situation I want to avoid at all costs (because Avaya has a TCP stack bug where it will reject all such calls - even though the Avaya is the cause of this situation!). So by setting net.sip.tcp.connection.put.followable=false, how does that change this scenario?

How specifically does net.sip.tcp.connection.put.followable=false differ from net.sip.transport.share.connection=false?

It depends on Contact header of the initial SIP request packet.
I suppose SIP OPTIONS packet sent from the Avaya has Contact header which points Avaya:5060.

Therefore, SIP Server re-uses the same TCP connection to reach to Avaya:5060 even if the remote port number is not 5060.

If you set "net.sip.tcp.connection.put.followable=false", SIP Server doesn't link Contact URI and the underlying TCP connection.

If you set "net.sip.transport.share.connection=false", SIP Sever doesn't share TCP connection even if destination IP address and port numbers are same.