[OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module 1.6.4

Bogdan-Andrei Iancu bogdan at voice-system.ro
Tue Dec 21 14:28:42 CET 2010


Hi Anton,

Is there any chance to get access to that machine to check the core file 
? that will be the simplest way...otherwise I will try to do some remote 
debugging with you.

Regards,
Bogdan

Anton Zagorskiy wrote:
> Hi.
>
> drouting module 1.6.4 crashes too like 1.6.3
> drouting.c line 1005: rt_info->pgwl[0].pgw == 0x1 (invalid pointer)
> with fork=no all is ok.
>
>
>
>
>
>
> WBR, Anton Zagorskiy
> VoIP Developer, Oyster Telecom
> Phone.: +7 812 601-0666
> Fax: +7 812 601-0593
> a.zagorskiy at oyster-telecom.ru
> www.oyster-telecom.ru
>
>
>
>   
>> -----Original Message-----
>> From: devel-bounces at lists.opensips.org [mailto:devel-
>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>> Sent: Monday, December 20, 2010 6:01 PM
>> To: 'OpenSIPS devel mailling list'
>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module
>> 1.6.3
>>
>> No, there is 0 in the column probe_mode
>>
>>
>>
>>
>>
>>
>> WBR, Anton Zagorskiy
>> VoIP Developer, Oyster Telecom
>> Phone.: +7 812 601-0666
>> Fax: +7 812 601-0593
>> a.zagorskiy at oyster-telecom.ru
>> www.oyster-telecom.ru
>>
>>
>>
>>     
>>> -----Original Message-----
>>> From: devel-bounces at lists.opensips.org [mailto:devel-
>>> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
>>> Sent: Monday, December 20, 2010 5:36 PM
>>> To: OpenSIPS devel mailling list
>>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module
>>> 1.6.3
>>>
>>> Do you use probing in DR ?
>>>
>>> Anton Zagorskiy wrote:
>>>       
>>>> It seems that openSIPS crashes only when fork=yes.
>>>> When I set fork=no drouting works well and openSIPS doesn't crash.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> WBR, Anton Zagorskiy
>>>> VoIP Developer, Oyster Telecom
>>>> Phone.: +7 812 601-0666
>>>> Fax: +7 812 601-0593
>>>> a.zagorskiy at oyster-telecom.ru
>>>> www.oyster-telecom.ru
>>>>
>>>>
>>>>
>>>>
>>>>         
>>>>> -----Original Message-----
>>>>> From: devel-bounces at lists.opensips.org [mailto:devel-
>>>>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>>>>> Sent: Monday, December 20, 2010 1:07 PM
>>>>> To: 'OpenSIPS devel mailling list'
>>>>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting module
>>>>> 1.6.3
>>>>>
>>>>> Hi.
>>>>>
>>>>> Please see memory debug http://pastebin.com/JKzyya5L
>>>>> I don't know how to interpret it but there is no 'overlap' word.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> WBR, Anton Zagorskiy
>>>>> VoIP Developer, Oyster Telecom
>>>>> Phone.: +7 812 601-0666
>>>>> Fax: +7 812 601-0593
>>>>> a.zagorskiy at oyster-telecom.ru
>>>>> www.oyster-telecom.ru
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: devel-bounces at lists.opensips.org [mailto:devel-
>>>>>> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
>>>>>> Sent: Friday, December 17, 2010 9:13 PM
>>>>>> To: OpenSIPS devel mailling list
>>>>>> Subject: Re: [OpenSIPS-Devel] FW: [OpenSIPS-Users] drouting
>>>>>>             
>> module
>>     
>>>>>> 1.6.3
>>>>>>
>>>>>> Hi Anton,
>>>>>>
>>>>>> have you tried to enable to memory debugger to see if any memory
>>>>>> overlapping is reported ?
>>>>>>
>>>>>> Regards,
>>>>>> Bogdan
>>>>>>
>>>>>>
>>>>>> Anton Zagorskiy wrote:
>>>>>>
>>>>>>             
>>>>>>> Hello.
>>>>>>>
>>>>>>> I've made some debug on this crash.
>>>>>>> The problem is dereferencing invalid pointer (0x1)
>>>>>>>
>>>>>>> This happens in the function drouting.c::static int
>>>>>>>
>>>>>>>               
>>>>> do_routing(struct
>>>>>
>>>>>           
>>>>>>> sip_msg* msg, dr_group_t *drg, int sort_order)
>>>>>>> Line 1005: if( (rt_info->pgwl[i].pgw->flags &
>>>>>>>
>>>>>>>               
>>>>> DR_DST_STAT_DSBL_FLAG)
>>>>>
>>>>>           
>>>>>> == 0 )
>>>>>>
>>>>>>             
>>>>>>> In my case i == 0, and rt_info->pgwl[0].pgw is 0x1
>>>>>>>
>>>>>>> In fact, in the routing.c::build_rt_info(..) rt_info-
>>>>>>>               
>>> pgwl[0].pgw
>>>       
>>>>>> obtains
>>>>>>
>>>>>>             
>>>>>>> correct address - lines from 170 to 180.
>>>>>>>
>>>>>>> So, something changes that pointer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> WBR, Anton Zagorskiy
>>>>>>> VoIP Developer, Oyster Telecom
>>>>>>> Phone.: +7 812 601-0666
>>>>>>> Fax: +7 812 601-0593
>>>>>>> a.zagorskiy at oyster-telecom.ru
>>>>>>> www.oyster-telecom.ru
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: users-bounces at lists.opensips.org
>>>>>>> [mailto:users-bounces at lists.opensips.org] On Behalf Of Anton
>>>>>>>
>>>>>>>               
>>>>>> Zagorskiy
>>>>>>
>>>>>>             
>>>>>>> Sent: Thursday, December 16, 2010 6:14 PM
>>>>>>> To: 'OpenSIPS users mailling list'
>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>
>>>>>>> I've made a simple config, and openSIPS still crashes.
>>>>>>>
>>>>>>> Please see my config here http://pastebin.com/mfabxGJJ
>>>>>>> And please see a full log file http://pastebin.com/0VXQ6vwT
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> WBR, Anton Zagorskiy
>>>>>>> VoIP Developer, Oyster Telecom
>>>>>>> Phone.: +7 812 601-0666
>>>>>>> Fax: +7 812 601-0593
>>>>>>> a.zagorskiy at oyster-telecom.ru
>>>>>>> www.oyster-telecom.ru
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> -----Original Message-----
>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>>>>>>>> Sent: Thursday, December 16, 2010 11:54 AM
>>>>>>>> To: 'OpenSIPS users mailling list'
>>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> What about investigation my crash?
>>>>>>>> Should I send any additional information?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Anton Zagorskiy
>>>>>>>>> Sent: Tuesday, December 14, 2010 5:01 PM
>>>>>>>>> To: 'OpenSIPS users mailling list'
>>>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>>>
>>>>>>>>> Yes, openSIPS crashed because of there are "INFO:core:sig_usr:
>>>>>>>>>
>>>>>>>>>                   
>>>>>> signal
>>>>>>
>>>>>>             
>>>>>>>>> 15 received" messages in a log file.
>>>>>>>>> I've tested on new VM with CentOS 5.5 with sources from svn
>>>>>>>>>
>>>>>>>>>                   
>>>>> stable
>>>>>
>>>>>           
>>>>>>>>> release
>>>>>>>>>
>>>>>>>>> Please see bt and bt full results in the attach.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Code:
>>>>>>>>>
>>>>>>>>> route
>>>>>>>>> {
>>>>>>>>> 	[skipped]
>>>>>>>>>
>>>>>>>>> 	if (!do_routing())
>>>>>>>>> 	{
>>>>>>>>> 	    xlog("*** !!! route[invite]: No rules found for
>>>>>>>>>
>>>>>>>>>                   
>>>>> do_routing().
>>>>>
>>>>>           
>>>>>>>>> STOP");
>>>>>>>>> 	    send_reply("503", "Error");
>>>>>>>>> 	    exit;
>>>>>>>>> 	};
>>>>>>>>> 	xlog("*** route[invite]: do_routing() passed");
>>>>>>>>> 	xlog("*** route[invite]: dr_attrs: $avp(s:dr_attrs); ruri:
>>>>>>>>> $avp(s:dr_ruri)");
>>>>>>>>> 	route(main_route);
>>>>>>>>>
>>>>>>>>> 	[skipped]
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> route[main_route]
>>>>>>>>> {
>>>>>>>>>     xlog("*** +++ route[main_route] has started");
>>>>>>>>>
>>>>>>>>>     if (is_method("INVITE"))
>>>>>>>>>     {
>>>>>>>>> 	t_on_branch("2");
>>>>>>>>> 	t_on_reply("2");
>>>>>>>>> 	t_on_failure("1");
>>>>>>>>>     };
>>>>>>>>>
>>>>>>>>>     if (!t_relay())
>>>>>>>>>     {
>>>>>>>>> 	sl_reply_error();
>>>>>>>>>     };
>>>>>>>>>
>>>>>>>>>     xlog("*** --- route[main_route] has finished"); }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> WBR, Anton Zagorskiy
>>>>>>>>> VoIP Developer, Oyster Telecom
>>>>>>>>> Phone.: +7 812 601-0666
>>>>>>>>> Fax: +7 812 601-0593
>>>>>>>>> a.zagorskiy at oyster-telecom.ru
>>>>>>>>> www.oyster-telecom.ru
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
>>>>>>>>>> Sent: Tuesday, December 14, 2010 4:25 PM
>>>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>>>> Subject: Re: [OpenSIPS-Users] drouting module 1.6.3 with b2b
>>>>>>>>>>
>>>>>>>>>> Hi Anton,
>>>>>>>>>>
>>>>>>>>>> Why do you say it crashes ? do you have a corefile ?
>>>>>>>>>>
>>>>>>>>>> your logs do not show any t_relay() after
>>>>>>>>>>                     
>> do_routing()....could
>>     
>>>>>> you
>>>>>>
>>>>>>             
>>>>>>>>>> post the relevant snip of your cfg?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bogdan
>>>>>>>>>>
>>>>>>>>>> Anton Zagorskiy wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> Hi!
>>>>>>>>>>>
>>>>>>>>>>> I'm trying to use drouting module, but something says to me
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>> that
>>>>>
>>>>>           
>>>>>>>> I
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> using it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> wrong because openSIPS goes to crash.
>>>>>>>>>>>
>>>>>>>>>>> I've inserted following values in the DB:
>>>>>>>>>>>
>>>>>>>>>>> 1. dr_gateways:
>>>>>>>>>>>
>>>>>>>>>>> gwid(2)
>>>>>>>>>>> type(1)
>>>>>>>>>>> address(192.168.0.10)
>>>>>>>>>>> strip(0)
>>>>>>>>>>> pri_prefix(1000)
>>>>>>>>>>> attrs("")
>>>>>>>>>>> probe_mode(0)
>>>>>>>>>>> description(Gateway1)
>>>>>>>>>>>
>>>>>>>>>>> 2. dr_groups:
>>>>>>>>>>> id(1)
>>>>>>>>>>> username(100)
>>>>>>>>>>> domain(mydomain.com)
>>>>>>>>>>> groupid(1)
>>>>>>>>>>> description(All-Test)
>>>>>>>>>>>
>>>>>>>>>>> 3. dr_gw_lists:
>>>>>>>>>>> id(2)
>>>>>>>>>>> gwlist(2)
>>>>>>>>>>> description(Test)
>>>>>>>>>>>
>>>>>>>>>>> 4. dr_rules:
>>>>>>>>>>> ruleid(1)
>>>>>>>>>>> groupid(1)
>>>>>>>>>>> prefix(89112599077)
>>>>>>>>>>> timerec("")
>>>>>>>>>>> priority(1)
>>>>>>>>>>> routeid("")
>>>>>>>>>>> gwlist(#2)
>>>>>>>>>>> description(Test)
>>>>>>>>>>> attrs("")
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So, I'm calling from 100 at mydomain.com to 89112599077.
>>>>>>>>>>> I'm expecting that this will be routed to 192.168.0.10 with
>>>>>>>>>>>                       
>> R-
>>     
>>>>> URI
>>>>>
>>>>>           
>>>>>>>>>>> 100089112599077 at .. Right?
>>>>>>>>>>>
>>>>>>>>>>> But when I'm calling do_routing() the same INVITE is passed
>>>>>>>>>>>                       
>> to
>>     
>>>>>>>>>>>                       
>>>>>>>> the
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> main
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> route block and I'm again calling do_routing (Nothing
>>>>>>>>>>>                       
>> changed
>>     
>>>>> in
>>>>>
>>>>>           
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> script
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> :)) and opensips is crashing...
>>>>>>>>>>>
>>>>>>>>>>> Log file hasn't helpful messages:
>>>>>>>>>>>
>>>>>>>>>>> [8031]: *** route[invite]: call do_routing()
>>>>>>>>>>> [8031]: DBG:core:db_new_result: allocate 28 bytes for result
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>> set
>>>>>
>>>>>           
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>>> 0x81ddb78
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_get_columns: 1 columns
>>>>>>>>>>>                       
>> returned
>>     
>>>>>>>>>>>                       
>>>>>>>> from
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> the query
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> [8031]: DBG:core:db_allocate_columns: allocate 16 bytes for
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>> result
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> columns
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> at 0x81ddba4
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_get_columns:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> RES_NAMES(0x81ddba8)[0]=[groupid]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_get_columns: use DB_INT result
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>> type
>>>>>
>>>>>           
>>>>>>>>>>> [8031]: DBG:core:db_allocate_rows: allocate 28 bytes for
>>>>>>>>>>>                       
>>> result
>>>       
>>>>>>>>>>>                       
>>>>>>>>> rows
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> values at 0x81ddbd0
>>>>>>>>>>> [8031]: DBG:db_mysql:db_mysql_str2val: converting INT [1]
>>>>>>>>>>> [8031]: DBG:core:db_free_columns: freeing result columns at
>>>>>>>>>>> 0x81ddba4
>>>>>>>>>>> [8031]: DBG:core:db_free_rows: freeing 1 rows
>>>>>>>>>>> [8031]: DBG:core:db_free_row: freeing row values at
>>>>>>>>>>>                       
>> 0x81ddbd8
>>     
>>>>>>>>>>> [8031]: DBG:core:db_free_rows: freeing rows at 0x81ddbd0
>>>>>>>>>>> [8031]: DBG:core:db_free_result: freeing result set at
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>> 0x81ddb78
>>>>>
>>>>>           
>>>>>>>>>>> [8031]: DBG:drouting:do_routing: using dr group 1
>>>>>>>>>>> [8031]: DBG:drouting:internal_check_rt: found rgid 1 (rule
>>>>>>>>>>>                       
>>> list
>>>       
>>>>>>>>>>>                       
>>>>>>>>>> 0xb60c0d0c)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> [8034]: DBG:tm:utimer_routine: timer routine:4,tl=0xb60c34b4
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> next=(nil),
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> timeout=11900000
>>>>>>>>>>> [8034]: DBG:tm:retransmission_handler:
>>>>>>>>>>>                       
>> retransmission_handler
>>     
>>> :
>>>       
>>>>>>>>>>>                       
>>>>>>>>>> request
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> resending (t=0xb60c3368, INVITE si ... )
>>>>>>>>>>> [8034]: DBG:tm:set_timer: relative timeout is 2000000
>>>>>>>>>>> [8034]: DBG:tm:insert_timer_unsafe: [5]: 0xb60c34b4
>>>>>>>>>>>                       
>> (13900000)
>>     
>>>>>>>>>>> [8034]: DBG:tm:retransmission_handler:
>>>>>>>>>>>                       
>> retransmission_handler
>>     
>>> :
>>>       
>>>>>>>>>>>                       
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>>> [8032]: DBG:core:parse_msg: SIP Request:
>>>>>>>>>>> [8032]: DBG:core:parse_msg:  method:  <INVITE> (# The same
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>> INVITE
>>>>>
>>>>>           
>>>>>>>>> is
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>>> coming!)
>>>>>>>>>>>
>>>>>>>>>>> What I'm doing wrong?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at lists.opensips.org
>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at lists.opensips.org
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at lists.opensips.org
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> --
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Event - expo, conf, social, bootcamp
>>>>>> 2 - 4 February 2011, ITExpo, Miami,  USA
>>>>>> www.voice-system.ro
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at lists.opensips.org
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>>>
>>>>>>             
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at lists.opensips.org
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>>
>>>>>           
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel at lists.opensips.org
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>>
>>>>
>>>>         
>>> --
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Event - expo, conf, social, bootcamp
>>> 2 - 4 February 2011, ITExpo, Miami,  USA
>>> www.voice-system.ro
>>>
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel at lists.opensips.org
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>>       
>> _______________________________________________
>> Devel mailing list
>> Devel at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>     
>
>
> _______________________________________________
> Devel mailing list
> Devel at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>   


-- 
Bogdan-Andrei Iancu
OpenSIPS Event - expo, conf, social, bootcamp
2 - 4 February 2011, ITExpo, Miami,  USA
www.voice-system.ro




More information about the Devel mailing list