[OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module 1.6.4
Bogdan-Andrei Iancu
bogdan at voice-system.ro
Tue Dec 21 14:28:42 CET 2010
Hi Anton,
Is there any chance to get access to that machine to check the core file
? that will be the simplest way...otherwise I will try to do some remote
debugging with you.
Regards,
Bogdan
Anton Zagorskiy wrote:
> Hi.
>
> drouting module 1.6.4 crashes too like 1.6.3
> drouting.c line 1005: rt_info->pgwl[0].pgw == 0x1 (invalid pointer)
> with fork=no all is ok.
>
>
>
>
>
>
> WBR, Anton Zagorskiy
> VoIP Developer, Oyster Telecom
> Phone.: +7 812 601-0666
> Fax: +7 812 601-0593
> a.zagorskiy at oyster-telecom.ru
> www.oyster-telecom.ru
>
>
>
>
>> -----Original Message-----
>> From: devel-bounces at lists.opensips.org [mailto:devel-
>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>> Sent: Monday, December 20, 2010 6:01 PM
>> To: 'OpenSIPS devel mailling list'
>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module
>> 1.6.3
>>
>> No, there is 0 in the column probe_mode
>>
>>
>>
>>
>>
>>
>> WBR, Anton Zagorskiy
>> VoIP Developer, Oyster Telecom
>> Phone.: +7 812 601-0666
>> Fax: +7 812 601-0593
>> a.zagorskiy at oyster-telecom.ru
>> www.oyster-telecom.ru
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: devel-bounces at lists.opensips.org [mailto:devel-
>>> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
>>> Sent: Monday, December 20, 2010 5:36 PM
>>> To: OpenSIPS devel mailling list
>>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module
>>> 1.6.3
>>>
>>> Do you use probing in DR ?
>>>
>>> Anton Zagorskiy wrote:
>>>
>>>> It seems that openSIPS crashes only when fork=yes.
>>>> When I set fork=no drouting works well and openSIPS doesn't crash.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> WBR, Anton Zagorskiy
>>>> VoIP Developer, Oyster Telecom
>>>> Phone.: +7 812 601-0666
>>>> Fax: +7 812 601-0593
>>>> a.zagorskiy at oyster-telecom.ru
>>>> www.oyster-telecom.ru
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: devel-bounces at lists.opensips.org [mailto:devel-
>>>>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>>>>> Sent: Monday, December 20, 2010 1:07 PM
>>>>> To: 'OpenSIPS devel mailling list'
>>>>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module
>>>>> 1.6.3
>>>>>
>>>>> Hi.
>>>>>
>>>>> Please see memory debug http://pastebin.com/JKzyya5L
>>>>> I don't know how to interpret it but there is no 'overlap' word.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> WBR, Anton Zagorskiy
>>>>> VoIP Developer, Oyster Telecom
>>>>> Phone.: +7 812 601-0666
>>>>> Fax: +7 812 601-0593
>>>>> a.zagorskiy at oyster-telecom.ru
>>>>> www.oyster-telecom.ru
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: devel-bounces at lists.opensips.org [mailto:devel-
>>>>>> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
>>>>>> Sent: Friday, December 17, 2010 9:13 PM
>>>>>> To: OpenSIPS devel mailling list
>>>>>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting
>>>>>>
>> module
>>
>>>>>> 1.6.3
>>>>>>
>>>>>> Hi Anton,
>>>>>>
>>>>>> have you tried to enable to memory debugger to see if any memory
>>>>>> overlapping is reported ?
>>>>>>
>>>>>> Regards,
>>>>>> Bogdan
>>>>>>
>>>>>>
>>>>>> Anton Zagorskiy wrote:
>>>>>>
>>>>>>
>>>>>>> Hello.
>>>>>>>
>>>>>>> I've made some debug on this crash.
>>>>>>> The problem is dereferencing invalid pointer (0x1)
>>>>>>>
>>>>>>> This happens in the function drouting.c::static int
>>>>>>>
>>>>>>>
>>>>> do_routing(struct
>>>>>
>>>>>
>>>>>>> sip_msg* msg, dr_group_t *drg, int sort_order)
>>>>>>> Line 1005: if( (rt_info->pgwl[i].pgw->flags &
>>>>>>>
>>>>>>>
>>>>> DR_DST_STAT_DSBL_FLAG)
>>>>>
>>>>>
>>>>>> == 0 )
>>>>>>
>>>>>>
>>>>>>> In my case i == 0, and rt_info->pgwl[0].pgw is 0x1
>>>>>>>
>>>>>>> In fact, in the routing.c::build_rt_info(..) rt_info-
>>>>>>>
>>> pgwl[0].pgw
>>>
>>>>>> obtains
>>>>>>
>>>>>>
>>>>>>> correct address - lines from 170 to 180.
>>>>>>>
>>>>>>> So, something changes that pointer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> WBR, Anton Zagorskiy
>>>>>>> VoIP Developer, Oyster Telecom
>>>>>>> Phone.: +7 812 601-0666
>>>>>>> Fax: +7 812 601-0593
>>>>>>> a.zagorskiy at oyster-telecom.ru
>>>>>>> www.oyster-telecom.ru
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: users-bounces at lists.opensips.org
>>>>>>> [mailto:users-bounces at lists.opensips.org] On Behalf Of Anton
>>>>>>>
>>>>>>>
>>>>>> Zagorskiy
>>>>>>
>>>>>>
>>>>>>> Sent: Thursday, December 16, 2010 6:14 PM
>>>>>>> To: 'OpenSIPS users mailling list'
>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>
>>>>>>> I've made a simple config, and openSIPS still crashes.
>>>>>>>
>>>>>>> Please see my config here http://pastebin.com/mfabxGJJ
>>>>>>> And please see a full log file http://pastebin.com/0VXQ6vwT
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> WBR, Anton Zagorskiy
>>>>>>> VoIP Developer, Oyster Telecom
>>>>>>> Phone.: +7 812 601-0666
>>>>>>> Fax: +7 812 601-0593
>>>>>>> a.zagorskiy at oyster-telecom.ru
>>>>>>> www.oyster-telecom.ru
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>>>>>>>> Sent: Thursday, December 16, 2010 11:54 AM
>>>>>>>> To: 'OpenSIPS users mailling list'
>>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> What about investigation my crash?
>>>>>>>> Should I send any additional information?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>>>>>>>>> Sent: Tuesday, December 14, 2010 5:01 PM
>>>>>>>>> To: 'OpenSIPS users mailling list'
>>>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>>>
>>>>>>>>> Yes, openSIPS crashed because of there are "INFO:core:sig_usr:
>>>>>>>>>
>>>>>>>>>
>>>>>> signal
>>>>>>
>>>>>>
>>>>>>>>> 15 received" messages in a log file.
>>>>>>>>> I've tested on new VM with CentOS 5.5 with sources from svn
>>>>>>>>>
>>>>>>>>>
>>>>> stable
>>>>>
>>>>>
>>>>>>>>> release
>>>>>>>>>
>>>>>>>>> Please see bt and bt full results in the attach.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Code:
>>>>>>>>>
>>>>>>>>> route
>>>>>>>>> {
>>>>>>>>> [skipped]
>>>>>>>>>
>>>>>>>>> if (!do_routing())
>>>>>>>>> {
>>>>>>>>> xlog("*** !!! route[invite]: No rules found for
>>>>>>>>>
>>>>>>>>>
>>>>> do_routing().
>>>>>
>>>>>
>>>>>>>>> STOP");
>>>>>>>>> send_reply("503", "Error");
>>>>>>>>> exit;
>>>>>>>>> };
>>>>>>>>> xlog("*** route[invite]: do_routing() passed");
>>>>>>>>> xlog("*** route[invite]: dr_attrs: $avp(s:dr_attrs); ruri:
>>>>>>>>> $avp(s:dr_ruri)");
>>>>>>>>> route(main_route);
>>>>>>>>>
>>>>>>>>> [skipped]
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> route[main_route]
>>>>>>>>> {
>>>>>>>>> xlog("*** +++ route[main_route] has started");
>>>>>>>>>
>>>>>>>>> if (is_method("INVITE"))
>>>>>>>>> {
>>>>>>>>> t_on_branch("2");
>>>>>>>>> t_on_reply("2");
>>>>>>>>> t_on_failure("1");
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> if (!t_relay())
>>>>>>>>> {
>>>>>>>>> sl_reply_error();
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> xlog("*** --- route[main_route] has finished"); }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> WBR, Anton Zagorskiy
>>>>>>>>> VoIP Developer, Oyster Telecom
>>>>>>>>> Phone.: +7 812 601-0666
>>>>>>>>> Fax: +7 812 601-0593
>>>>>>>>> a.zagorskiy at oyster-telecom.ru
>>>>>>>>> www.oyster-telecom.ru
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
>>>>>>>>>> Sent: Tuesday, December 14, 2010 4:25 PM
>>>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>>>>
>>>>>>>>>> Hi Anton,
>>>>>>>>>>
>>>>>>>>>> Why do you say it crashes ? do you have a corefile ?
>>>>>>>>>>
>>>>>>>>>> your logs do not show any t_relay() after
>>>>>>>>>>
>> do_routing()....could
>>
>>>>>> you
>>>>>>
>>>>>>
>>>>>>>>>> post the relevant snip of your cfg?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bogdan
>>>>>>>>>>
>>>>>>>>>> Anton Zagorskiy wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hi!
>>>>>>>>>>>
>>>>>>>>>>> I'm trying to use drouting module, but something says to me
>>>>>>>>>>>
>>>>>>>>>>>
>>>>> that
>>>>>
>>>>>
>>>>>>>> I
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> using it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> wrong because openSIPS goes to crash.
>>>>>>>>>>>
>>>>>>>>>>> I've inserted following values in the DB:
>>>>>>>>>>>
>>>>>>>>>>> 1. dr_gateways:
>>>>>>>>>>>
>>>>>>>>>>> gwid(2)
>>>>>>>>>>> type(1)
>>>>>>>>>>> address(192.168.0.10)
>>>>>>>>>>> strip(0)
>>>>>>>>>>> pri_prefix(1000)
>>>>>>>>>>> attrs("")
>>>>>>>>>>> probe_mode(0)
>>>>>>>>>>> description(Gateway1)
>>>>>>>>>>>
>>>>>>>>>>> 2. dr_groups:
>>>>>>>>>>> id(1)
>>>>>>>>>>> username(100)
>>>>>>>>>>> domain(mydomain.com)
>>>>>>>>>>> groupid(1)
>>>>>>>>>>> description(All-Test)
>>>>>>>>>>>
>>>>>>>>>>> 3. dr_gw_lists:
>>>>>>>>>>> id(2)
>>>>>>>>>>> gwlist(2)
>>>>>>>>>>> description(Test)
>>>>>>>>>>>
>>>>>>>>>>> 4. dr_rules:
>>>>>>>>>>> ruleid(1)
>>>>>>>>>>> groupid(1)
>>>>>>>>>>> prefix(89112599077)
>>>>>>>>>>> timerec("")
>>>>>>>>>>> priority(1)
>>>>>>>>>>> routeid("")
>>>>>>>>>>> gwlist(#2)
>>>>>>>>>>> description(Test)
>>>>>>>>>>> attrs("")
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So, I'm calling from 100 at mydomain.com to 89112599077.
>>>>>>>>>>> I'm expecting that this will be routed to 192.168.0.10 with
>>>>>>>>>>>
>> R-
>>
>>>>> URI
>>>>>
>>>>>
>>>>>>>>>>> 100089112599077 at .. Right?
>>>>>>>>>>>
>>>>>>>>>>> But when I'm calling do_routing() the same INVITE is passed
>>>>>>>>>>>
>> to
>>
>>>>>>>>>>>
>>>>>>>> the
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> main
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> route block and I'm again calling do_routing (Nothing
>>>>>>>>>>>
>> changed
>>
>>>>> in
>>>>>
>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> script
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> :)) and opensips is crashing...
>>>>>>>>>>>
>>>>>>>>>>> Log file hasn't helpful messages:
>>>>>>>>>>>
>>>>>>>>>>> [8031]: *** route[invite]: call do_routing()
>>>>>>>>>>> [8031]: DBG:core:db_new_result: allocate 28 bytes for result
>>>>>>>>>>>
>>>>>>>>>>>
>>>>> set
>>>>>
>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>> 0x81ddb78
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_get_columns: 1 columns
>>>>>>>>>>>
>> returned
>>
>>>>>>>>>>>
>>>>>>>> from
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> the query
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> [8031]: DBG:core:db_allocate_columns: allocate 16 bytes for
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> result
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> columns
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> at 0x81ddba4
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_get_columns:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> RES_NAMES(0x81ddba8)[0]=[groupid]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_get_columns: use DB_INT result
>>>>>>>>>>>
>>>>>>>>>>>
>>>>> type
>>>>>
>>>>>
>>>>>>>>>>> [8031]: DBG:core:db_allocate_rows: allocate 28 bytes for
>>>>>>>>>>>
>>> result
>>>
>>>>>>>>>>>
>>>>>>>>> rows
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> values at 0x81ddbd0
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_str2val: converting INT [1]
>>>>>>>>>>> [8031]: DBG:core:db_free_columns: freeing result columns at
>>>>>>>>>>> 0x81ddba4
>>>>>>>>>>> [8031]: DBG:core:db_free_rows: freeing 1 rows
>>>>>>>>>>> [8031]: DBG:core:db_free_row: freeing row values at
>>>>>>>>>>>
>> 0x81ddbd8
>>
>>>>>>>>>>> [8031]: DBG:core:db_free_rows: freeing rows at 0x81ddbd0
>>>>>>>>>>> [8031]: DBG:core:db_free_result: freeing result set at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>> 0x81ddb78
>>>>>
>>>>>
>>>>>>>>>>> [8031]: DBG:drouting:do_routing: using dr group 1
>>>>>>>>>>> [8031]: DBG:drouting:internal_check_rt: found rgid 1 (rule
>>>>>>>>>>>
>>> list
>>>
>>>>>>>>>>>
>>>>>>>>>> 0xb60c0d0c)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> [8034]: DBG:tm:utimer_routine: timer routine:4,tl=0xb60c34b4
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> next=(nil),
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> timeout=11900000
>>>>>>>>>>> [8034]: DBG:tm:retransmission_handler:
>>>>>>>>>>>
>> retransmission_handler
>>
>>> :
>>>
>>>>>>>>>>>
>>>>>>>>>> request
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> resending (t=0xb60c3368, INVITE si ... )
>>>>>>>>>>> [8034]: DBG:tm:set_timer: relative timeout is 2000000
>>>>>>>>>>> [8034]: DBG:tm:insert_timer_unsafe: [5]: 0xb60c34b4
>>>>>>>>>>>
>> (13900000)
>>
>>>>>>>>>>> [8034]: DBG:tm:retransmission_handler:
>>>>>>>>>>>
>> retransmission_handler
>>
>>> :
>>>
>>>>>>>>>>>
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> [8032]: DBG:core:parse_msg: SIP Request:
>>>>>>>>>>> [8032]: DBG:core:parse_msg: method: <INVITE> (# The same
>>>>>>>>>>>
>>>>>>>>>>>
>>>>> INVITE
>>>>>
>>>>>
>>>>>>>>> is
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> coming!)
>>>>>>>>>>>
>>>>>>>>>>> What I'm doing wrong?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at lists.opensips.org
>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at lists.opensips.org
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at lists.opensips.org
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Event - expo, conf, social, bootcamp
>>>>>> 2 - 4 February 2011, ITExpo, Miami, USA
>>>>>> www.voice-system.ro
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at lists.opensips.org
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at lists.opensips.org
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel at lists.opensips.org
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>
>>>>
>>>>
>>> --
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Event - expo, conf, social, bootcamp
>>> 2 - 4 February 2011, ITExpo, Miami, USA
>>> www.voice-system.ro
>>>
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel at lists.opensips.org
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>
>> _______________________________________________
>> Devel mailing list
>> Devel at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>
>
>
> _______________________________________________
> Devel mailing list
> Devel at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>
--
Bogdan-Andrei Iancu
OpenSIPS Event - expo, conf, social, bootcamp
2 - 4 February 2011, ITExpo, Miami, USA
www.voice-system.ro
More information about the Devel
mailing list