[OpenSIPS-Users] rtpengine errors: can't send command to a RTP proxy (22:Invalid argument)

Tue Jun 30 07:07:28 EST 2020

Hi, Ovidiu!

I agree, the fact that no one reported it before is a bit interesting. 
On the other hand, not sure if >1000 buffers is expected, perhaps the 
flags you are using for rtpengine are increasing the number of buffers.

Best regards,

Răzvan Crainea
OpenSIPS Core Developer
http://www.opensips-solutions.com

On 6/29/20 9:43 PM, Ovidiu Sas wrote:
> Hi Razvan,
> 
> I have updated the rtpproxy module and everything is backported down to 2.4.
> 
> Detecting IOV_MAX would be kind of an overkill ...
> The error logs were improved and the error is now searchable in the
> mailing list. If everyone stumble across it, it should be easy to
> troubleshoot.
> It seems that IOV_MAX will be properly defined in the future (it is
> defined in some experimental headers) and the detection of it will
> become obsolete.
> We could move the definition of OSIP_IOV_MAX into a common header file
> for both rtpengine and rtpproxy module.
> 
> I'm still puzzled as to why this was not reported before for the
> rtpengine module. It is quite easy to reproduce with very low traffic.
> 
> Thanks,
> Ovidiu
> 
> On Mon, Jun 29, 2020 at 7:20 AM Răzvan Crainea <razvan at opensips.org> wrote:
>>
>> Hi, Ovidiu!
>>
>> I agree we should match the two modules to use a common logic.
>> I believe we should backport this in 2.4 as well, since it doesn't work
>> for all scenarios, therefore it is considered a bug.
>>
>> I somehow doubt this should be a module's param, because it is only
>> needed in certain circumstances, where IOV_MAX is not "visible" in the
>> code. IMO, a decent, commonly accepted limit should be enough - so I
>> believe the way you've done it for 3.0 and 3.1 is more than fine. Adding
>> a parameter will only complicate provisioning and the gain of setting a
>> custom value would be hard to quantify.
>>
>> Probably a better approach would be to make sure the IOV_MAX is properly
>> defined using something described here[1]. What do you think?
>>
>> [1]
>> https://stackoverflow.com/questions/27271801/c-the-permitted-maximum-of-the-iovcnt-argument-in-writev
>>
>> Cheers,
>>
>> Răzvan Crainea
>> OpenSIPS Core Developer
>> http://www.opensips-solutions.com
>>
>> On 6/27/20 9:35 PM, Ovidiu Sas wrote:
>>> Fix implemented on trunk, 3.1 and 3.0.
>>> We should update the fix on rtpproxy to match rtpengine and move the
>>> OSIP_IOV_MAX define somewhere upper in the code tree to be visible by
>>> both rtpproxy and rtpengine modules.
>>> For trunk, maybe we should add a new param.
>>>
>>> Should we backport this fix to 2.4?
>>>
>>> Thanks,
>>> Ovidiu
>>>
>>> On Fri, Jun 26, 2020 at 9:35 AM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>
>>>> Hello Razvan,
>>>>
>>>> On ubuntu we have a failure for more then 1024 buffers. There is something defined in some experimental headers (as __IOV_MAX), but the IOV_MAX define is not visible in the rtpengine module.
>>>>
>>>> And yes, there is an extra buffer. I guess the length of that extra buffer was zero and that’s why it didn’t create issues. But as soon as I concatenated the buffer, the issue showed up and I had an extra trailing string at the end of the rtpengine command that was sent on the wire.
>>>> After reducing  the number of buffers by one, all was good on both scenarios: sending with and without concatenated buffers.
>>>> I will prepare a fix for it and push it.
>>>>
>>>> -ovidiu
>>>>
>>>> On Fri, Jun 26, 2020 at 03:55 Răzvan Crainea <razvan at opensips.org> wrote:
>>>>>
>>>>> Hi, Ovidiu!
>>>>>
>>>>> So you're saying that the IOV_MAX is not explicitly defined, but it does
>>>>> fail after 1024 buffers, correct? If so, perhaps we should limit the
>>>>> number of buffers to 1024, if not already defined.
>>>>> I did notice the extra vcnt as well, but I though that was related to
>>>>> the fact that it allocates one extra iovec (at the headr) for the
>>>>> cookie, and sometime uses it, sometime doesn't. Nevertheless, it is
>>>>> consistent on both cases, UNIX & UDP, both have that vcnt + 1. Now I
>>>>> haven't checked whether this is correct or not, could you please confirm?
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Răzvan Crainea
>>>>> OpenSIPS Core Developer
>>>>> http://www.opensips-solutions.com
>>>>>
>>>>> On 6/26/20 2:29 AM, Ovidiu Sas wrote:
>>>>>> It looks like the hidden IOV_MAX is set to 1024 in debian.
>>>>>> I tested a patch and all looks good.
>>>>>> It seems that there is an error in the code: writev has the wrong
>>>>>> number of iovcnt (should be one less).
>>>>>> I tested and all looks ok (the code in rtpproxy.c has the proper iovcnt).
>>>>>>
>>>>>> -ovidiu
>>>>>>
>>>>>> On Thu, Jun 25, 2020 at 1:03 PM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>>>
>>>>>>> We will need to add a param to control the max number of buffers.
>>>>>>>
>>>>>>> -ovidiu
>>>>>>>
>>>>>>> On Thu, Jun 25, 2020 at 1:02 PM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>>>>
>>>>>>>> It seems that when we have more than roughly 1000 buffers, the send fails.
>>>>>>>>
>>>>>>>> -ovidiu
>>>>>>>>
>>>>>>>> On Wed, Jun 24, 2020 at 8:54 AM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello Razvan,
>>>>>>>>>
>>>>>>>>> The system is a debian buster one.
>>>>>>>>> I patched the code:
>>>>>>>>> #ifdef IOV_MAX
>>>>>>>>>        LM_NOTICE("IOV_MAX=[%d]\n", IOV_MAX);
>>>>>>>>> #else
>>>>>>>>>        LM_NOTICE("no IOV_MAX\n");
>>>>>>>>> #endif
>>>>>>>>>
>>>>>>>>> and I get:
>>>>>>>>> NOTICE:rtpengine:send_rtpe_command: no IOV_MAX
>>>>>>>>> in the logs.
>>>>>>>>>
>>>>>>>>> Then I patched the code again to check how many buffers are being used:
>>>>>>>>> The max so far was 73:
>>>>>>>>> LM_NOTICE("writev(rtpe_socks[node->idx], v , %d)\n", vcnt + 1);
>>>>>>>>> got me:
>>>>>>>>> NOTICE:rtpengine:send_rtpe_command: writev(rtpe_socks[node->idx], v , 73)
>>>>>>>>>
>>>>>>>>> I will continue to monitor the system to see if there is a correlation
>>>>>>>>> between the error and the number of buffers.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ovidiu
>>>>>>>>>
>>>>>>>>> On Wed, Jun 24, 2020 at 1:59 AM Răzvan Crainea <razvan at opensips.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi, Ovidiu!
>>>>>>>>>>
>>>>>>>>>> I doubt this is a problem of OpenSIPS version, but rather of the OS you
>>>>>>>>>> are running on. I suspect that error comes from the fact that the bson
>>>>>>>>>> resulted has more than IOV_MAX elements, which if I recall correctly it
>>>>>>>>>> was 15 on some OSes.
>>>>>>>>>> We had a similar problem in rtpproy[1], where we had merged the buffers
>>>>>>>>>> in a single one just to pass over this limitation. Could you check if
>>>>>>>>>> you are facing a similar issue?
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/OpenSIPS/opensips/blob/master/modules/rtpproxy/rtpproxy.c#L2031
>>>>>>>>>>
>>>>>>>>>> Răzvan Crainea
>>>>>>>>>> OpenSIPS Core Developer
>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>
>>>>>>>>>> On 6/24/20 7:32 AM, Ovidiu Sas wrote:
>>>>>>>>>>> This is happening also on the latest 3.0.
>>>>>>>>>>> The weird thing is that opensips doesn't send anything to rtpengine.
>>>>>>>>>>> The first opensips/rtpengine exchange on the initial INVITE works ok,
>>>>>>>>>>> but the opensips/rtpengine exchange on the 200ok fails (no command is
>>>>>>>>>>> sent by opensips - confirmed by running ngrep on the loopback
>>>>>>>>>>> interface).
>>>>>>>>>>> On the next call, the initial offer works fine, but the answer fails
>>>>>>>>>>> due to no command issued by opensips.
>>>>>>>>>>>
>>>>>>>>>>> version: opensips 3.0.2 (x86_64/linux)
>>>>>>>>>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
>>>>>>>>>>> Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>>>>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>>>>>>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>>>>>>>> poll method support: poll, epoll, sigio_rt, select.
>>>>>>>>>>> git revision: 3a8f6f137
>>>>>>>>>>> main.c compiled on 09:36:42 Jun 22 2020 with gcc 9
>>>>>>>>>>>
>>>>>>>>>>> -ovidiu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:49 PM Ovidiu Sas <osas at voipembedded.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm running opensips 3.1.0-beta (latest version) and experiencing
>>>>>>>>>>>> connectivity issues to the rtpengine daemon running on the same host:
>>>>>>>>>>>> ERROR:rtpengine:send_rtpe_command: can't send command to a RTP proxy
>>>>>>>>>>>> (22:Invalid argument)
>>>>>>>>>>>> ERROR:rtpengine:send_rtpe_command: timeout waiting reply from a RTP proxy
>>>>>>>>>>>> ERROR:rtpengine:send_rtpe_command: proxy <udp:127.0.0.1:2223> does not
>>>>>>>>>>>> respond, disable it
>>>>>>>>>>>> ERROR:rtpengine:rtpe_function_call: no available proxies
>>>>>>>>>>>>
>>>>>>>>>>>> After an opensips restart, everything comes to normal.
>>>>>>>>>>>> I was running previously opensips 3.1.0-dev and everything was working fine.
>>>>>>>>>>>>
>>>>>>>>>>>> The issue starts showing up after a few days with very little traffic.
>>>>>>>>>>>> Is there anyone experiencing this issue?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Ovidiu
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> VoIP Embedded, Inc.
>>>>>>>>>>>> http://www.voipembedded.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> VoIP Embedded, Inc.
>>>>>>>>> http://www.voipembedded.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> VoIP Embedded, Inc.
>>>>>>>> http://www.voipembedded.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> VoIP Embedded, Inc.
>>>>>>> http://www.voipembedded.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opensips.org
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>> --
>>>> VoIP Embedded, Inc.
>>>> http://www.voipembedded.com
>>>
>>>
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> 
> 
> 