[OpenSIPS-Users] rtpengine errors: can't send command to a RTP proxy (22:Invalid argument)

Mon Jun 29 11:19:00 EST 2020

Hi, Ovidiu!

I agree we should match the two modules to use a common logic.
I believe we should backport this in 2.4 as well, since it doesn't work 
for all scenarios, therefore it is considered a bug.

I somehow doubt this should be a module's param, because it is only 
needed in certain circumstances, where IOV_MAX is not "visible" in the 
code. IMO, a decent, commonly accepted limit should be enough - so I 
believe the way you've done it for 3.0 and 3.1 is more than fine. Adding 
a parameter will only complicate provisioning and the gain of setting a 
custom value would be hard to quantify.

Probably a better approach would be to make sure the IOV_MAX is properly 
defined using something described here[1]. What do you think?

[1] 
https://stackoverflow.com/questions/27271801/c-the-permitted-maximum-of-the-iovcnt-argument-in-writev

Cheers,

Răzvan Crainea
OpenSIPS Core Developer
http://www.opensips-solutions.com

On 6/27/20 9:35 PM, Ovidiu Sas wrote:
> Fix implemented on trunk, 3.1 and 3.0.
> We should update the fix on rtpproxy to match rtpengine and move the
> OSIP_IOV_MAX define somewhere upper in the code tree to be visible by
> both rtpproxy and rtpengine modules.
> For trunk, maybe we should add a new param.
> 
> Should we backport this fix to 2.4?
> 
> Thanks,
> Ovidiu
> 
> On Fri, Jun 26, 2020 at 9:35 AM Ovidiu Sas <osas at voipembedded.com> wrote:
>>
>> Hello Razvan,
>>
>> On ubuntu we have a failure for more then 1024 buffers. There is something defined in some experimental headers (as __IOV_MAX), but the IOV_MAX define is not visible in the rtpengine module.
>>
>> And yes, there is an extra buffer. I guess the length of that extra buffer was zero and that’s why it didn’t create issues. But as soon as I concatenated the buffer, the issue showed up and I had an extra trailing string at the end of the rtpengine command that was sent on the wire.
>> After reducing  the number of buffers by one, all was good on both scenarios: sending with and without concatenated buffers.
>> I will prepare a fix for it and push it.
>>
>> -ovidiu
>>
>> On Fri, Jun 26, 2020 at 03:55 Răzvan Crainea <razvan at opensips.org> wrote:
>>>
>>> Hi, Ovidiu!
>>>
>>> So you're saying that the IOV_MAX is not explicitly defined, but it does
>>> fail after 1024 buffers, correct? If so, perhaps we should limit the
>>> number of buffers to 1024, if not already defined.
>>> I did notice the extra vcnt as well, but I though that was related to
>>> the fact that it allocates one extra iovec (at the headr) for the
>>> cookie, and sometime uses it, sometime doesn't. Nevertheless, it is
>>> consistent on both cases, UNIX & UDP, both have that vcnt + 1. Now I
>>> haven't checked whether this is correct or not, could you please confirm?
>>>
>>> Best regards,
>>>
>>> Răzvan Crainea
>>> OpenSIPS Core Developer
>>> http://www.opensips-solutions.com
>>>
>>> On 6/26/20 2:29 AM, Ovidiu Sas wrote:
>>>> It looks like the hidden IOV_MAX is set to 1024 in debian.
>>>> I tested a patch and all looks good.
>>>> It seems that there is an error in the code: writev has the wrong
>>>> number of iovcnt (should be one less).
>>>> I tested and all looks ok (the code in rtpproxy.c has the proper iovcnt).
>>>>
>>>> -ovidiu
>>>>
>>>> On Thu, Jun 25, 2020 at 1:03 PM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>
>>>>> We will need to add a param to control the max number of buffers.
>>>>>
>>>>> -ovidiu
>>>>>
>>>>> On Thu, Jun 25, 2020 at 1:02 PM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>>
>>>>>> It seems that when we have more than roughly 1000 buffers, the send fails.
>>>>>>
>>>>>> -ovidiu
>>>>>>
>>>>>> On Wed, Jun 24, 2020 at 8:54 AM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>>>
>>>>>>> Hello Razvan,
>>>>>>>
>>>>>>> The system is a debian buster one.
>>>>>>> I patched the code:
>>>>>>> #ifdef IOV_MAX
>>>>>>>       LM_NOTICE("IOV_MAX=[%d]\n", IOV_MAX);
>>>>>>> #else
>>>>>>>       LM_NOTICE("no IOV_MAX\n");
>>>>>>> #endif
>>>>>>>
>>>>>>> and I get:
>>>>>>> NOTICE:rtpengine:send_rtpe_command: no IOV_MAX
>>>>>>> in the logs.
>>>>>>>
>>>>>>> Then I patched the code again to check how many buffers are being used:
>>>>>>> The max so far was 73:
>>>>>>> LM_NOTICE("writev(rtpe_socks[node->idx], v , %d)\n", vcnt + 1);
>>>>>>> got me:
>>>>>>> NOTICE:rtpengine:send_rtpe_command: writev(rtpe_socks[node->idx], v , 73)
>>>>>>>
>>>>>>> I will continue to monitor the system to see if there is a correlation
>>>>>>> between the error and the number of buffers.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ovidiu
>>>>>>>
>>>>>>> On Wed, Jun 24, 2020 at 1:59 AM Răzvan Crainea <razvan at opensips.org> wrote:
>>>>>>>>
>>>>>>>> Hi, Ovidiu!
>>>>>>>>
>>>>>>>> I doubt this is a problem of OpenSIPS version, but rather of the OS you
>>>>>>>> are running on. I suspect that error comes from the fact that the bson
>>>>>>>> resulted has more than IOV_MAX elements, which if I recall correctly it
>>>>>>>> was 15 on some OSes.
>>>>>>>> We had a similar problem in rtpproy[1], where we had merged the buffers
>>>>>>>> in a single one just to pass over this limitation. Could you check if
>>>>>>>> you are facing a similar issue?
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/OpenSIPS/opensips/blob/master/modules/rtpproxy/rtpproxy.c#L2031
>>>>>>>>
>>>>>>>> Răzvan Crainea
>>>>>>>> OpenSIPS Core Developer
>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>
>>>>>>>> On 6/24/20 7:32 AM, Ovidiu Sas wrote:
>>>>>>>>> This is happening also on the latest 3.0.
>>>>>>>>> The weird thing is that opensips doesn't send anything to rtpengine.
>>>>>>>>> The first opensips/rtpengine exchange on the initial INVITE works ok,
>>>>>>>>> but the opensips/rtpengine exchange on the 200ok fails (no command is
>>>>>>>>> sent by opensips - confirmed by running ngrep on the loopback
>>>>>>>>> interface).
>>>>>>>>> On the next call, the initial offer works fine, but the answer fails
>>>>>>>>> due to no command issued by opensips.
>>>>>>>>>
>>>>>>>>> version: opensips 3.0.2 (x86_64/linux)
>>>>>>>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
>>>>>>>>> Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>>>>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>>>>>> poll method support: poll, epoll, sigio_rt, select.
>>>>>>>>> git revision: 3a8f6f137
>>>>>>>>> main.c compiled on 09:36:42 Jun 22 2020 with gcc 9
>>>>>>>>>
>>>>>>>>> -ovidiu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 10, 2020 at 2:49 PM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello all,
>>>>>>>>>>
>>>>>>>>>> I'm running opensips 3.1.0-beta (latest version) and experiencing
>>>>>>>>>> connectivity issues to the rtpengine daemon running on the same host:
>>>>>>>>>> ERROR:rtpengine:send_rtpe_command: can't send command to a RTP proxy
>>>>>>>>>> (22:Invalid argument)
>>>>>>>>>> ERROR:rtpengine:send_rtpe_command: timeout waiting reply from a RTP proxy
>>>>>>>>>> ERROR:rtpengine:send_rtpe_command: proxy <udp:127.0.0.1:2223> does not
>>>>>>>>>> respond, disable it
>>>>>>>>>> ERROR:rtpengine:rtpe_function_call: no available proxies
>>>>>>>>>>
>>>>>>>>>> After an opensips restart, everything comes to normal.
>>>>>>>>>> I was running previously opensips 3.1.0-dev and everything was working fine.
>>>>>>>>>>
>>>>>>>>>> The issue starts showing up after a few days with very little traffic.
>>>>>>>>>> Is there anyone experiencing this issue?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ovidiu
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> VoIP Embedded, Inc.
>>>>>>>>>> http://www.voipembedded.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at lists.opensips.org
>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> VoIP Embedded, Inc.
>>>>>>> http://www.voipembedded.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> VoIP Embedded, Inc.
>>>>>> http://www.voipembedded.com
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> VoIP Embedded, Inc.
>>>>> http://www.voipembedded.com
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opensips.org
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>> --
>> VoIP Embedded, Inc.
>> http://www.voipembedded.com
> 
> 
> 