[OpenSIPS-Devel] Hang due to "no more nonces can be generated"

[Bogdan-Andrei Iancu]

Alistair,

The problem was found and fix, thanks to Anca. Please update from SVN.

This nonce checking does not introduce any limitation. With a 100K slot 
and 30 seconds life time for nonces, you can do 3400 auth per second, 
which is more than you will ever be able to handle in opensips.

Regards,
Bogdan

Alistair Cunningham wrote:
> Bodgen,
>
> Thank you for investigating this. I did try 60 seconds which did not 
> help. I then tried 20 seconds and a nightly restart of OpenSIPS from 
> crontab, and the problem has not occurred since. However, I'm 
> concerned that this system is no-where near the largest we support and 
> so 20 seconds may not suffice for large systems.
>
> I notice that Kamailio suffered from the same problem, and they 
> introduced a "nonce_reuse" modparam:
>
> http://www.mail-archive.com/users@lists.kamailio.org/msg01303.html
>
> Would this be worth implementing for OpenSIPS? Is it safe from a 
> security (e.g. replay attack) point of view?
>
> Alistair Cunningham
> +1 888 468 3111
> +44 20 799 39 799
> http://integrics.com/
>
>
> Bogdan-Andrei Iancu wrote:
>> Hi Alistair,
>>
>> This is related to authentication. OpenSIPS is keeping state for all 
>> the nonces it generates in order to avoid nonce re-usage.  The error 
>> you get means that all the available slots for generating nonces are 
>> used (by default are 100 000 of them), but no nonce was replied 
>> (getting a response).
>>
>> Do you have such a large traffic that you may have more than 100 000 
>> authentication requests at a time (without getting the responses yet) ?
>>
>> You may try to reduce the nonce lifetime and make the un-answered 
>> ones to be released faster . see nonce_expire param:
>>    http://www.opensips.org/html/docs/modules/1.4.x/auth.html#id2526655
>>
>> - try setting this to 30 seconds .
>>
>> I'm trying to figure out if in your case it is a simple problem of 
>> load or it is a bug in the nonce reservation mechanism.
>>
>> Regards,
>> Bogdan
>>
>> Alistair Cunningham wrote:
>>> We've just had OpenSIPS 1.4.2 stop processing SIP packets and 
>>> effectively hang. During this time, it logged the following many 
>>> times to /var/log/daemon.log:
>>>
>>> ERROR:auth:build_auth_hf: no more nonces can be generated
>>> ERROR:auth:challenge: failed to generate nonce
>>>
>>> Restarting OpenSIPS has temporarily cured it, but I expect the 
>>> problem will come back.
>>>
>>> Another problem (probably unrelated) on the same machine was that 
>>> when running "opensipsctl online", no output was produced and the 
>>> following was logged to daemon.log:
>>>
>>> ERROR:core:create_mi_node: no more pkg mem
>>> ERROR:mi_fifo:mi_fifo_server: command (ul_dump) processing failed
>>>
>>> I've since set the following in config.h:
>>>
>>> #define PKG_MEM_POOL_SIZE 10*1024*1024
>>>
>>> and this problem has gone away (opensipsctl online produces 1793 
>>> lines of output), but it's unclear whether this will help with the 
>>> nonce problem (I'm thinking probably not). In any case, may we 
>>> please have either a config file option or a command line option to 
>>> set PKG_MEM_POOL_SIZE without needing to patch the source code?
>>>
>>>   
>>
>>
>>
>