[OpenSIPS-Devel] Hang due to "no more nonces can be generated"

[Alistair Cunningham]

Bogdan,

I agree that reducing the registration rate would be a good idea. The 
limiting factor is not the technology, it's the administrative side of 
training customer service reps, end users, etc. This will be done, but 
is never a fast process!

Alistair Cunningham
+1 888 468 3111
+44 20 799 39 799
http://integrics.com/


Bogdan-Andrei Iancu wrote:
> Hi Alistair,
> 
> Thank you for your feedback - I had some investigation with Anca and we 
> think we found a bug in the algorithm for monitoring the used nonces. 
> I'm aware of the "nonce_reuse" param in kamilio, but I do not like the 
> approach of disabling something if it a bug there - we will try to fix 
> this issue asap. Nonce reusage is a security risk, a scenario quite easy 
> to exploit and this should not be disregarded in a production system.
> 
> Btw, you should reduce the registration rate  - make no sense to use 
> registration as keep alive mechanism when you have a  proofed  and 
> dedicated mechanism for this (try OPTIONS -or sipping). The load on your 
> system will dramatically reduce.
> 
> Regards,
> Bogdan
> 
> Alistair Cunningham wrote:
>> Bodgen,
>>
>> Thank you for investigating this. I did try 60 seconds which did not 
>> help. I then tried 20 seconds and a nightly restart of OpenSIPS from 
>> crontab, and the problem has not occurred since. However, I'm 
>> concerned that this system is no-where near the largest we support and 
>> so 20 seconds may not suffice for large systems.
>>
>> I notice that Kamailio suffered from the same problem, and they 
>> introduced a "nonce_reuse" modparam:
>>
>> http://www.mail-archive.com/users@lists.kamailio.org/msg01303.html
>>
>> Would this be worth implementing for OpenSIPS? Is it safe from a 
>> security (e.g. replay attack) point of view?
>>
>> Alistair Cunningham
>> +1 888 468 3111
>> +44 20 799 39 799
>> http://integrics.com/
>>
>>
>> Bogdan-Andrei Iancu wrote:
>>> Hi Alistair,
>>>
>>> This is related to authentication. OpenSIPS is keeping state for all 
>>> the nonces it generates in order to avoid nonce re-usage.  The error 
>>> you get means that all the available slots for generating nonces are 
>>> used (by default are 100 000 of them), but no nonce was replied 
>>> (getting a response).
>>>
>>> Do you have such a large traffic that you may have more than 100 000 
>>> authentication requests at a time (without getting the responses yet) ?
>>>
>>> You may try to reduce the nonce lifetime and make the un-answered 
>>> ones to be released faster . see nonce_expire param:
>>>    http://www.opensips.org/html/docs/modules/1.4.x/auth.html#id2526655
>>>
>>> - try setting this to 30 seconds .
>>>
>>> I'm trying to figure out if in your case it is a simple problem of 
>>> load or it is a bug in the nonce reservation mechanism.
>>>
>>> Regards,
>>> Bogdan
>>>
>>> Alistair Cunningham wrote:
>>>> We've just had OpenSIPS 1.4.2 stop processing SIP packets and 
>>>> effectively hang. During this time, it logged the following many 
>>>> times to /var/log/daemon.log:
>>>>
>>>> ERROR:auth:build_auth_hf: no more nonces can be generated
>>>> ERROR:auth:challenge: failed to generate nonce
>>>>
>>>> Restarting OpenSIPS has temporarily cured it, but I expect the 
>>>> problem will come back.
>>>>
>>>> Another problem (probably unrelated) on the same machine was that 
>>>> when running "opensipsctl online", no output was produced and the 
>>>> following was logged to daemon.log:
>>>>
>>>> ERROR:core:create_mi_node: no more pkg mem
>>>> ERROR:mi_fifo:mi_fifo_server: command (ul_dump) processing failed
>>>>
>>>> I've since set the following in config.h:
>>>>
>>>> #define PKG_MEM_POOL_SIZE 10*1024*1024
>>>>
>>>> and this problem has gone away (opensipsctl online produces 1793 
>>>> lines of output), but it's unclear whether this will help with the 
>>>> nonce problem (I'm thinking probably not). In any case, may we 
>>>> please have either a config file option or a command line option to 
>>>> set PKG_MEM_POOL_SIZE without needing to patch the source code?
>>>>
>>>>   
>>>
>>>
>>>
>>
> 
> 
>