<div dir="auto">That's a given the question is can it mess up the active server if the passive one was updated already? I'm not planning on leaving it like that I just like to do the upgrades slowly, one at a time and test it and only then to upgrade the second one.<div dir="auto"><br></div><div dir="auto">Scott</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 19, 2022, 09:26 Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Schneur,<br>
<br>
It is strongly recommend that all OpenSIPS nodes in a cluster to have <br>
the same version.<br>
<br>
Best regards,<br>
<br>
Bogdan-Andrei Iancu<br>
<br>
OpenSIPS Founder and Developer<br>
<a href="https://www.opensips-solutions.com" rel="noreferrer noreferrer" target="_blank">https://www.opensips-solutions.com</a><br>
OpenSIPS eBootcamp 2021<br>
<a href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/" rel="noreferrer noreferrer" target="_blank">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a><br>
<br>
On 1/18/22 6:08 PM, Schneur Rosenberg wrote:<br>
> Hi, it seems like it was fixed in 3.2, I will have to migrate all my<br>
> servers, I use binary replication will it break if one server is<br>
> running 2.4 and the other 3.2? its a active/passive setup so I will<br>
> take down one at a time and upgrade it, I'm just worried what will<br>
> happen while one is 3.2 and the second one is still 2.4, in the past I<br>
> disabled the replication while I was doing the updates and I'm<br>
> wondering if its necessary.<br>
><br>
> thanks<br>
> Scott (Schneur)<br>
><br>
> On Fri, Dec 17, 2021 at 6:21 PM Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org" target="_blank" rel="noreferrer">bogdan@opensips.org</a>> wrote:<br>
>> While trying to reproduce (as I failed to do so), I noticed you mentioned this is on version 2.4.11, right ? As I was testing on 3.2 without getting the leak.<br>
>><br>
>> Could you try on 3.2/3.1 ? Keep in mind 2.4 is not maintained anymore :(<br>
>><br>
>> Regards,<br>
>><br>
>> Bogdan-Andrei Iancu<br>
>><br>
>> OpenSIPS Founder and Developer<br>
>> <a href="https://www.opensips-solutions.com" rel="noreferrer noreferrer" target="_blank">https://www.opensips-solutions.com</a><br>
>> OpenSIPS eBootcamp 2021<br>
>> <a href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/" rel="noreferrer noreferrer" target="_blank">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a><br>
>><br>
>> On 12/17/21 1:40 PM, Schneur Rosenberg wrote:<br>
>><br>
>> Thanks Bogdan!, this is my entire local_route, all my dst_uri's are IP only.<br>
>><br>
>> On Fri, Dec 17, 2021, 12:36 Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org" target="_blank" rel="noreferrer">bogdan@opensips.org</a>> wrote:<br>
>>> Hi Schneur,<br>
>>><br>
>>> I suspect that the leaking mk_proxy is related to the changing of the<br>
>>> RURI in local route. Let me test your snippet. BTW, is that the whole<br>
>>> processing you do in local route? is the $rd (from LB) a FQDN or<br>
>>> straight IP ?<br>
>>><br>
>>> Regards,<br>
>>><br>
>>> Bogdan-Andrei Iancu<br>
>>><br>
>>> OpenSIPS Founder and Developer<br>
>>> <a href="https://www.opensips-solutions.com" rel="noreferrer noreferrer" target="_blank">https://www.opensips-solutions.com</a><br>
>>> OpenSIPS eBootcamp 2021<br>
>>> <a href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/" rel="noreferrer noreferrer" target="_blank">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a><br>
>>><br>
>>> On 12/16/21 9:43 AM, Schneur Rosenberg wrote:<br>
>>>> Hi Bogdan<br>
>>>><br>
>>>> I think I found the issue, I recently added these lines of code,<br>
>>>> because of a probing issue I was having, I just searched from my<br>
>>>> previous tickets and I see that you have warned me about the<br>
>>>> implications but for some reason I never read the message.<br>
>>>><br>
>>>> Here is the old ticket<br>
>>>> <a href="https://www.mail-archive.com/users@lists.opensips.org/msg43301.html" rel="noreferrer noreferrer" target="_blank">https://www.mail-archive.com/users@lists.opensips.org/msg43301.html</a><br>
>>>> the reason I'm using INVITE to probe is because I want the servers<br>
>>>> that were probed not only to respond but also check if the database is<br>
>>>> working, I did it this way because I had cases where mysql crashed but<br>
>>>> my asterisk servers were still responding to the probe but all of the<br>
>>>> calls just hung, so I do a invite and it does a DB lookup and it will<br>
>>>> only return a positive message if it was able to query the DB, do you<br>
>>>> have a better solution? at the time I set it up I couldn't run a query<br>
>>>> on receipt of a OPTIONS but perhaps I didn't look good enough :-),<br>
>>>> either way can I do anything to make sure this code doesn't leak<br>
>>>> memory? this probing has worked for years until I needed the Contact<br>
>>>> header.<br>
>>>><br>
>>>> local_route {<br>
>>>> if (is_method("INVITE")&& $fU=="pingTest"){<br>
>>>> $ru="sip:s@"+$rd ;<br>
>>>> append_hf("Contact: <sip:pingTest@$fd:5060>\r\n");<br>
>>>> exit;<br>
>>>> }<br>
>>>> }<br>
>>>><br>
>>>> On Fri, Dec 10, 2021 at 2:16 PM Schneur Rosenberg<br>
>>>> <<a href="mailto:rosenberg11219@gmail.com" target="_blank" rel="noreferrer">rosenberg11219@gmail.com</a>> wrote:<br>
>>>>> Hi Bogdan,<br>
>>>>><br>
>>>>> I did it on a backup server, its also leaking memory but at a slower<br>
>>>>> pace, I'm attaching the logs when running kill -SIGUSR1 on the pid<br>
>>>>> that's growing in size, it still has available memory, I hop this will<br>
>>>>> give you a clue.<br>
>>>>><br>
>>>>> Here is a pastbin to the loggs <a href="https://pastebin.com/KJVb9Y75" rel="noreferrer noreferrer" target="_blank">https://pastebin.com/KJVb9Y75</a><br>
>>>>><br>
>>>>> On Fri, Dec 10, 2021 at 11:00 AM Schneur Rosenberg<br>
>>>>> <<a href="mailto:rosenberg11219@gmail.com" target="_blank" rel="noreferrer">rosenberg11219@gmail.com</a>> wrote:<br>
>>>>>> Thank you, does this reduce performance? can I leave it enabled on a<br>
>>>>>> production machine? I will wait for the memory leak to be apparent and<br>
>>>>>> I'll post the result.<br>
>>>>>><br>
>>>>>><br>
>>>>>> On Thu, Dec 9, 2021 at 12:31 PM Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org" target="_blank" rel="noreferrer">bogdan@opensips.org</a>> wrote:<br>
>>>>>>> Hi Schneur,<br>
>>>>>>><br>
>>>>>>> Just follow the<br>
>>>>>>> <a href="https://www.opensips.org/Documentation/TroubleShooting-OutOfMem" rel="noreferrer noreferrer" target="_blank">https://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a> and<br>
>>>>>>> provide the dump. This is the only way to investigate this.<br>
>>>>>>><br>
>>>>>>> Regards,<br>
>>>>>>><br>
>>>>>>> Bogdan-Andrei Iancu<br>
>>>>>>><br>
>>>>>>> OpenSIPS Founder and Developer<br>
>>>>>>> <a href="https://www.opensips-solutions.com" rel="noreferrer noreferrer" target="_blank">https://www.opensips-solutions.com</a><br>
>>>>>>> OpenSIPS eBootcamp 2021<br>
>>>>>>> <a href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/" rel="noreferrer noreferrer" target="_blank">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a><br>
>>>>>>><br>
>>>>>>> On 12/8/21 12:14 PM, Schneur Rosenberg wrote:<br>
>>>>>>>> I just noticed that process 88 runs the timer handler, perhaps this<br>
>>>>>>>> might shed light on whats going on.<br>
>>>>>>>><br>
>>>>>>>> opensipsctl fifo ps<br>
>>>>>>>> Process:: ID=88 PID=5327 Type=Timer handler<br>
>>>>>>>><br>
>>>>>>>> On Wed, Dec 8, 2021 at 10:55 AM Schneur Rosenberg<br>
>>>>>>>> <<a href="mailto:rosenberg11219@gmail.com" target="_blank" rel="noreferrer">rosenberg11219@gmail.com</a>> wrote:<br>
>>>>>>>>> Now a few hours later this is what I'm getting<br>
>>>>>>>>> Dec 8 09:50:13 /sbin/opensips[21699]: ERROR:nathelper:nh_timer: out<br>
>>>>>>>>> of pkg memory<br>
>>>>>>>>> Dec 8 09:50:16 /sbin/opensips[21699]: WARNING:core:fm_malloc: not<br>
>>>>>>>>> enough continuous free pkg memory (3024 bytes left, need 5128),<br>
>>>>>>>>> attempting defragmentation... please increase the "-M" command line<br>
>>>>>>>>> parameter!<br>
>>>>>>>>> Dec 8 09:50:16 /sbin/opensips[21699]: ERROR:core:fm_malloc: not<br>
>>>>>>>>> enough free pkg memory (3024 bytes left, need 5128), please increase<br>
>>>>>>>>> the "-M" command line parameter!<br>
>>>>>>>>><br>
>>>>>>>>> Here is the last 20 package memory max_used_size<br>
>>>>>>>>> pkmem:70-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:71-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:72-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:73-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:74-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:75-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:76-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:77-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:78-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:79-max_used_size:: 1009584<br>
>>>>>>>>> pkmem:80-max_used_size:: 1044752<br>
>>>>>>>>> pkmem:81-max_used_size:: 1075552<br>
>>>>>>>>> pkmem:82-max_used_size:: 1116848<br>
>>>>>>>>> pkmem:83-max_used_size:: 1117456<br>
>>>>>>>>> pkmem:84-max_used_size:: 1102640<br>
>>>>>>>>> pkmem:85-max_used_size:: 1306992<br>
>>>>>>>>> pkmem:86-max_used_size:: 1706304<br>
>>>>>>>>> pkmem:87-max_used_size:: 2507000<br>
>>>>>>>>> pkmem:88-max_used_size:: 4194264<br>
>>>>>>>>> pkmem:89-max_used_size:: 1009584<br>
>>>>>>>>><br>
>>>>>>>>> And here is the real used size, you can see that process 88 maxed out<br>
>>>>>>>>> pkmem:69-real_used_size:: 975528<br>
>>>>>>>>> pkmem:70-real_used_size:: 978016<br>
>>>>>>>>> pkmem:71-real_used_size:: 989592<br>
>>>>>>>>> pkmem:72-real_used_size:: 951416<br>
>>>>>>>>> pkmem:73-real_used_size:: 982496<br>
>>>>>>>>> pkmem:74-real_used_size:: 965744<br>
>>>>>>>>> pkmem:75-real_used_size:: 959424<br>
>>>>>>>>> pkmem:76-real_used_size:: 949472<br>
>>>>>>>>> pkmem:77-real_used_size:: 983080<br>
>>>>>>>>> pkmem:78-real_used_size:: 961400<br>
>>>>>>>>> pkmem:79-real_used_size:: 977808<br>
>>>>>>>>> pkmem:80-real_used_size:: 978928<br>
>>>>>>>>> pkmem:81-real_used_size:: 1009936<br>
>>>>>>>>> pkmem:82-real_used_size:: 1110760<br>
>>>>>>>>> pkmem:83-real_used_size:: 1116720<br>
>>>>>>>>> pkmem:84-real_used_size:: 1096568<br>
>>>>>>>>> pkmem:85-real_used_size:: 1300592<br>
>>>>>>>>> pkmem:86-real_used_size:: 1699648<br>
>>>>>>>>> pkmem:87-real_used_size:: 2501096<br>
>>>>>>>>> pkmem:88-real_used_size:: 4191280<br>
>>>>>>>>> pkmem:89-real_used_size:: 882528<br>
>>>>>>>>><br>
>>>>>>>>> On Tue, Dec 7, 2021 at 7:53 PM Schneur Rosenberg<br>
>>>>>>>>> <<a href="mailto:rosenberg11219@gmail.com" target="_blank" rel="noreferrer">rosenberg11219@gmail.com</a>> wrote:<br>
>>>>>>>>>> Hi, lately I'm getting these errors in my logs.<br>
>>>>>>>>>><br>
>>>>>>>>>> ERROR:core:fm_malloc: not enough free pkg memory (1792 bytes left,<br>
>>>>>>>>>> need 2184), please increase the "-M" command line para<br>
>>>>>>>>>> meter!<br>
>>>>>>>>>><br>
>>>>>>>>>> CRITICAL:core:hostent_cpy: pkg memory allocation failure<br>
>>>>>>>>>><br>
>>>>>>>>>> ERROR:nathelper:nh_timer: out of pkg memory<br>
>>>>>>>>>><br>
>>>>>>>>>> ERROR:core:fm_malloc: not enough free pkg memory (5952 bytes left,<br>
>>>>>>>>>> need 5408), please increase the "-M" command line para<br>
>>>>>>>>>> meter!<br>
>>>>>>>>>><br>
>>>>>>>>>> I was on version 2.4.8 and I upgraded to 2.4.11 and I'm monitoring the<br>
>>>>>>>>>> max_used_size of the package memory, a few hours later I see that 2<br>
>>>>>>>>>> processes keep on getting bigger, so far the rest are pretty stable, I<br>
>>>>>>>>>> have 90 processes and 87 and 88 are growing.<br>
>>>>>>>>>><br>
>>>>>>>>>> here you can see the last few processes, OpenSIPS set aside 4 mb per process.<br>
>>>>>>>>>><br>
>>>>>>>>>> pkmem:80-max_used_size:: 1009584<br>
>>>>>>>>>> pkmem:81-max_used_size:: 1009584<br>
>>>>>>>>>> pkmem:82-max_used_size:: 1009584<br>
>>>>>>>>>> pkmem:83-max_used_size:: 1009584<br>
>>>>>>>>>> pkmem:84-max_used_size:: 1009584<br>
>>>>>>>>>> pkmem:85-max_used_size:: 1009584<br>
>>>>>>>>>> pkmem:86-max_used_size:: 1143608<br>
>>>>>>>>>> pkmem:87-max_used_size:: 1323256<br>
>>>>>>>>>> pkmem:88-max_used_size:: 1831928<br>
>>>>>>>>>> pkmem:89-max_used_size:: 1009584<br>
>>>>>>>>>><br>
>>>>>>>>>> Any hints where to start looking besides the solutions fund here.<br>
>>>>>>>>>><br>
>>>>>>>>>> <a href="https://www.opensips.org/Documentation/TroubleShooting-OutOfMem" rel="noreferrer noreferrer" target="_blank">https://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a><br>
>>>>>>>>>><br>
>>>>>>>>>> thank you<br>
>>>>>>>>>> Scott<br>
>>>>>>>> _______________________________________________<br>
>>>>>>>> Users mailing list<br>
>>>>>>>> <a href="mailto:Users@lists.opensips.org" target="_blank" rel="noreferrer">Users@lists.opensips.org</a><br>
>>>>>>>> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
</blockquote></div>