[OpenSIPS-Users] opensips will not start while loading huge blacklist
Bogdan-Andrei Iancu
bogdan at opensips.org
Tue Oct 2 09:18:08 EDT 2018
Hi,
Taking a closer look at the module (to be honest I do not user it, as
drouting does a similar job) I see the module is loading the data into
memory. It is doing this during the init stage, holding back the entire
OpenSIPS startup.
Module doing similar heavy db load were changed to do the load
post-startup. Let patch the module to decouple the loading from the
startup, so that OpenSIPS will start without delays.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:
> OpenSIPs is loading the global blacklist into a digit trie structure
> into shared memory on startup (or reload_blacklist command). That
> causes the startup delay.
>
> - Jon
>
> On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit <basit.engg at gmail.com
> <mailto:basit.engg at gmail.com>> wrote:
>
> Hi Bogdan,
>
> I don't think so its translating to DB query on run time, instead
> its caching the blacklist.
> I am saying so because opensips response is quick, where as when I
> query manually from DB, it take some time even prefix col. is indexed.
> Moreover, while opensips restart i get following warnings in syslog:
>
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:timer_ticker: timer task <nh-timer> already scheduled
> for 990 ms (now 17920 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:timer_ticker: timer task <dlg-reinvite-pinger>
> already scheduled for 990 ms (now 17920 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:timer_ticker: timer task <dlg-options-pinger> already
> scheduled for 990 ms (now 17920 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:timer_ticker: timer task <dlg-timer> already
> scheduled for 990 ms (now 17920 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:timer_ticker: timer task <tm-timer> already scheduled
> for 990 ms (now 17920 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:timer_ticker: timer task <blcore-expire> already
> scheduled for 990 ms (now 17920 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:utimer_ticker: utimer task <tm-utimer> already
> scheduled for 100 ms (now 18020 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:utimer_ticker: utimer task <tm-utimer> already
> scheduled for 100 ms (now 18120 ms), it may overlap..
> Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
> WARNING:core:utimer_ticker: utimer task <tm-utimer> already
> scheduled for 100 ms (now 18220 ms), it may overlap..
> *Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
> INFO:userblacklist:reload_sources: got 1386901 entries from
> 'globalblacklist' *
> Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
> WARNING:core:handle_timer_job: utimer job <tm-utimer> has a
> 18170000 us delay in execution
> Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
> WARNING:core:handle_timer_job: timer job <nh-timer> has a 17280000
> us delay in execution
> Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
> WARNING:core:handle_timer_job: timer job <dlg-reinvite-pinger> has
> a 17280000 us delay in execution
>
> Once globalblacklist loaded *1386901 entries* , opensips behave
> normally.
> When I will add more records to globalblacklist, opensips wouldn't
> start due to service timeout.
> One way might be to increase service timeout to allow opensips
> keep loading the entries but it may take ages.
>
> --
> regards,
>
> abdul basit
>
> On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu
> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
>
> Hi Abdul,
>
> OK, but what is the blacklist related operation you do in
> startup route ?? As far as I know, the blacklist module is not
> doing any kind of caching, so any check you do against the
> blacklists will translate into a DB query, right ?
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/01/2018 06:58 PM, Abdul Basit wrote:
>> Yes. That's right.
>> The code I am using is as below. globalblacklist table has
>> complete number as prefixwith and whitelist status.
>>
>> route {
>> .....
>> if ( !check_blacklist("globalblacklist") ) {
>> xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not
>> Allowed. Dropping call.\n");
>> sl_send_reply("403", "Forbidden");
>> exit;
>> }
>> else {
>> xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
>> route(10);
>> exit;
>> }
>> .....
>> }
>>
>>
>> --
>> regards,
>>
>> abdul basit
>>
>>
>> On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu
>> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
>>
>> Hi Abdul,
>>
>> IF I understand correctly, you use startup route to load
>> (from DB?) some blacklists into localcache ?
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 09/28/2018 04:10 AM, Abdul Basit wrote:
>>> Hi team,
>>>
>>> I have a opensips 2.4 deployed having 4 children and
>>> with mysql configured for loading user blacklist to
>>> opensips local cache on opensips startup.
>>> Opensips work fine on filtering the call based on user
>>> blacklist records loaded into local cache.
>>>
>>> While blacklist was growing upto 1.3m records, opensips
>>> startup was taking ~60 sec on VM of 16GB RAM and 6 CPU.
>>> I increased shared memory to *6144* and package memory
>>> to *256.*
>>> Filter works fine here yet. Call load is not that much
>>> only blacklist entries are growing.
>>>
>>> But now blacklist has grown to very huge number, up to
>>> 10m records in mysql. Opensips is not starting
>>> successfully and service startup requests timeout after
>>> huge CPU spikes while trying to populate local cache
>>> from mysql.
>>> I increased shared memory to *12288* and package memory
>>> to *1024* but no success yet.
>>>
>>> I indexed mysql blacklist table but no luck.
>>>
>>> Please advise what settings I should add to opensips for
>>> successful startup.
>>>
>>> --
>>> regards,
>>>
>>> abdul basit
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20181002/2e5daa84/attachment-0001.html>
More information about the Users
mailing list