PBX servers freezes / stops responding

1. Brekeke Product Name and Version:
Brekeke PBX, Version 3.3.8.1

2. Java version:
1.7.0_72 (Java HotSpot 64-Bit Server VM)

3. OS type and the version:
Windows Server 2008 R2

4. UA (phone), gateway or other hardware/software involved:
Behnke VoIP Intercom, SeeTec VoIP-Software Client, vSphere 5.5 U2. build 2001466

5. Your problem:
Our Brekeke SIP/PBX-Server occasionally stops responding between 03:00 am and 03:30 am. This happens more or less once a week. When the server hangs, I still get the login page, but after submitting my user credentials no further response is received.
When this problems occurs, the SIP service does not respond any more, no calls are possible.
Stopping the Brekeke service from the Windows service manager gives after 30 seconds an error message: The service did not respond to the stop request and was terminated.
After starting the Brekeke service again the SIP/PBX-Server starts, the clients register again and everythings works as expected.
Up to know we do not have any idea what causes this error. We have not found a correlation with other tasks running on the Windows server or the vSphere cluster. Switching off snapshots and backup of our windows server did not help.
It seems, that the windows patch level influences the fequency off the Brekeke failures.
Does anyone experience similar behaviour or can give me any advice how to solve this problem?

Kind regards, Stefan

Are there any incidents on Windows Event Logs?

Are you using Windows Automatic Update?
If so, let you disable it.

Are you using Java's Automatic Update?
If so, let you disable it.

janP wrote:Are there any incidents on Windows Event Logs?

Since I have implemented a surveillance script for the SIP service, which restarts the Brekeke service when the SIP service does not answer anymore, an entry stating the Brekeke service has terminated unexpectedly is written to the system log. No additional entries near the crash times are in the event log.

janP wrote:Are you using Windows Automatic Update?
If so, let you disable it.

Yes, but the service just checks for available updates and downloads them. It is configured as "Download updates but let me choose whether to install them". I have now changed the time for the update check in order to see if there is a correlation with our Brekeke hang up.

janP wrote:Are you using Java's Automatic Update?
If so, let you disable it.

No.

Did the issue start happening recently?

Does it happen between 03:00 am and 03:30 am always?

Are there any calls at that time? If so how many?

How many registered users does the PBX manage?

janP wrote:Did the issue start happening recently?

We had the same problem in summer. Applying Microsoft windows, java and Brekeke updates seemed to solve the issue. From August to October we had no problems with the Brekeke service. After updating our system in October (windows, java and brekeke) the service crashed more or less once a week.

janP wrote:Does it happen between 03:00 am and 03:30 am always?

Yes, as far as we can tell all crashes happened between 03:00 am and 03:30 am, at least all crashes have been in the early morning hours. To get the time, when the service stops responding, we checked the sip response every minute from another host.

janP wrote:Are there any calls at that time? If so how many?

No, I think there are no call at that time.

janP wrote:How many registered users does the PBX manage?

Our PBX manages 13 or 14 registered users. It just connects several door phones on our parking lots with two VoIP phones. There are no connections with other telephone service providers.

Can you check the CPU usage when the problem happens?

Do you have any scheduled processes which run around 3:00am?
Let you check Windows "Scheduled Tasks".

Does your surveillance script have ability to stop the PBX process?

janP wrote:Can you check the CPU usage when the problem happens?

We have plenty of CPU-power available on our server, the 5 minute averages of the CPU-usage for our server on the vSphere Cluster does not show high peaks when the problem happens.

janP wrote:Do you have any scheduled processes which run around 3:00am?
Let you check Windows "Scheduled Tasks".

Maybe the windows update check, where I changed the time from 03.00am to 02.00am. I am curious if the problem with PBX now happens earlier.
We tried to switch off all scheduled tasks and vSphrere activities for our server, but did not find a correlation with our problem.

janP wrote:Does your surveillance script have ability to stop the PBX process?

My first try with a "net stop "Brekeke PBX" command did not succeed, the service went into stopping state but did never finish. In order to restart the service I use a taskkill statement to end the processes and start them afterwards again using net start "Brekeke PBX".
The surveillance task runs locally on the PBX server with system privilidges.

Hi Stefan,

Could you send the following files to support@brekeke.com ?

- Brekeke PBX's backup file.

- log.log and systemlog files located in the following folder.
c:/Program Files/Brekeke/pbx/webapps/pbx/WEB-INF/work/logs

- Several recent log files located in the following folder.
c:/Program Files/Brekeke/pbx/logs


Danke,

Mike wrote: Could you send the following files to support@brekeke.com ?

- Brekeke PBX's backup file.

- log.log and systemlog files located in the following folder.
c:/Program Files/Brekeke/pbx/webapps/pbx/WEB-INF/work/logs

- Several recent log files located in the following folder.
c:/Program Files/Brekeke/pbx/logs

The files have been sent to Brekeke support.

Kind Regards,
Stefan

Hi Stefan,

Thank you for sending log files.
I asked the team to look at them.