[OpenSIPS-Devel] Hang due to "no more nonces can be generated"

[Alistair Cunningham]

Bodgan,

Thank you very much. I don't think the administrators of this machine 
would be happy if I put a subversion release on, so I think we'll need 
to wait for 1.4.3, which I see you're planning for the next few days. 
Until then, the 20 second timeout seems to be a working temporary fix.

Alistair Cunningham
+1 888 468 3111
+44 20 799 39 799
http://integrics.com/


Bogdan-Andrei Iancu wrote:
> Alistair,
> 
> The problem was found and fix, thanks to Anca. Please update from SVN.
> 
> This nonce checking does not introduce any limitation. With a 100K slot 
> and 30 seconds life time for nonces, you can do 3400 auth per second, 
> which is more than you will ever be able to handle in opensips.
> 
> Regards,
> Bogdan
> 
> Alistair Cunningham wrote:
>> Bodgen,
>>
>> Thank you for investigating this. I did try 60 seconds which did not 
>> help. I then tried 20 seconds and a nightly restart of OpenSIPS from 
>> crontab, and the problem has not occurred since. However, I'm 
>> concerned that this system is no-where near the largest we support and 
>> so 20 seconds may not suffice for large systems.
>>
>> I notice that Kamailio suffered from the same problem, and they 
>> introduced a "nonce_reuse" modparam:
>>
>> http://www.mail-archive.com/users@lists.kamailio.org/msg01303.html
>>
>> Would this be worth implementing for OpenSIPS? Is it safe from a 
>> security (e.g. replay attack) point of view?
>>
>> Alistair Cunningham
>> +1 888 468 3111
>> +44 20 799 39 799
>> http://integrics.com/
>>
>>
>> Bogdan-Andrei Iancu wrote:
>>> Hi Alistair,
>>>
>>> This is related to authentication. OpenSIPS is keeping state for all 
>>> the nonces it generates in order to avoid nonce re-usage.  The error 
>>> you get means that all the available slots for generating nonces are 
>>> used (by default are 100 000 of them), but no nonce was replied 
>>> (getting a response).
>>>
>>> Do you have such a large traffic that you may have more than 100 000 
>>> authentication requests at a time (without getting the responses yet) ?
>>>
>>> You may try to reduce the nonce lifetime and make the un-answered 
>>> ones to be released faster . see nonce_expire param:
>>>    http://www.opensips.org/html/docs/modules/1.4.x/auth.html#id2526655
>>>
>>> - try setting this to 30 seconds .
>>>
>>> I'm trying to figure out if in your case it is a simple problem of 
>>> load or it is a bug in the nonce reservation mechanism.
>>>
>>> Regards,
>>> Bogdan
>>>
>>> Alistair Cunningham wrote:
>>>> We've just had OpenSIPS 1.4.2 stop processing SIP packets and 
>>>> effectively hang. During this time, it logged the following many 
>>>> times to /var/log/daemon.log:
>>>>
>>>> ERROR:auth:build_auth_hf: no more nonces can be generated
>>>> ERROR:auth:challenge: failed to generate nonce
>>>>
>>>> Restarting OpenSIPS has temporarily cured it, but I expect the 
>>>> problem will come back.
>>>>
>>>> Another problem (probably unrelated) on the same machine was that 
>>>> when running "opensipsctl online", no output was produced and the 
>>>> following was logged to daemon.log:
>>>>
>>>> ERROR:core:create_mi_node: no more pkg mem
>>>> ERROR:mi_fifo:mi_fifo_server: command (ul_dump) processing failed
>>>>
>>>> I've since set the following in config.h:
>>>>
>>>> #define PKG_MEM_POOL_SIZE 10*1024*1024
>>>>
>>>> and this problem has gone away (opensipsctl online produces 1793 
>>>> lines of output), but it's unclear whether this will help with the 
>>>> nonce problem (I'm thinking probably not). In any case, may we 
>>>> please have either a config file option or a command line option to 
>>>> set PKG_MEM_POOL_SIZE without needing to patch the source code?
>>>>
>>>>   
>>>
>>>
>>>
>>
> 
> 
>