Hey Bogdan,<div>I've been working on this issue and just wanted to update the group with my results. Keep in mind, this is a torture test.</div><div><br></div><div>So there were a number of issues, some DB, some opensips script.</div>
<div><br></div><div>Firstly, the opensips script:</div><div>I'm doing call limiting using the dialog module. The call limits are by source IP address. I store a key/value pair in memcache for IP/Call Limits, this way, I don't have to look up the calling limits every time I get a call (yay memcache!).</div>
<div><br></div><div>Well, this is what I found out. And I'm not sure if maybe I'm doing something wrong to cause this.. The first time OpenSIPs hits the DB to discover the call limits for an IP (using avp_db_query) the DB returns an INTEGER and stores it to memcache. Limits are checked and then the call is allowed or rejected. Works fine.. no errors.. Any future calls (within the memcache timeout) produce "type mismatch" error in the avp_check operation that checks the call counts. This is because when cache_fetch retrieves the account limits, it pulls the limits as a STRING, not an INT. So I had to do a type conversion. BTW, I couldn't find anything when I searched for "type casting" with opensips. So for those who are curious, look up {<a href="http://s.int">s.int</a>}.</div>
<div><br></div><div>Calls, and the logic worked with that type error in there, but it spit out a lot of error messages in the syslog. Since I put the type cast in there, no more errors. And amazingly, I've about quadrupled my performance!</div>
<div><br></div><div>DB Issues:</div><div>In addition, I think there is some define locking contention issues with MySQL/InnoDB with the ACC writes and my rating script reading. I have to run my rating script every minute, or else, I'll get horribly backlogged with calls to rate. My rating script can only process something like 800 calls per second.. So if opensips is sustaining 200CPS @ 10 sec calls and I run my rating script once every 60 seconds.. well you can see there is a finite limit to the CPS I'll be able to achieve solely based on how quickly I can rate the calls (which honestly I wasn't expecting). </div>
<div><br></div><div>To solve, some of my locking issues with MySQL, before my rating script wakes up, I create a copy of the acc table into a MySQL temp table (with an index of course, very important). Then process the copy. This solved a ton of issues. Although, I'm not sure how "sane" it is.</div>
<div><br></div><div>Some questions for the group:</div><div>I know there are a lot of issues regarding pretending OpenSIPS can act like a B2BUA and do proper billing. I use it for billing, and feel that it's reasonably accurate. The acc module, as it is, is hard to work with (well, for me at least). I have to do this fancy logic to group acc records into a single unified call record that lists the call, duration, pdd, etc. The way I'm doing it, works really well for me, but:</div>
<div>1. It's an external process to opensips</div><div>2. It has to play in the same tables as opensips, causing locking issues</div><div>3. Because acc logs actions are based on received signaling, there are *many* db hits. </div>
<div><br></div><div>What I imagine, is perhaps a "cdr" module based on dialog callbacks. Similar to the acc module but in general only write records on dialog completions and optionally can write records on dialog establishments and intra-dialog events (180, 183, 3XX, etc). End result being that a single dialog would have a single record. Yes, I understand that this requires more processing inside of opensips and probably limits the overall capacity, but I think it's a well justified tradeoff.</div>
<div><br></div><div>Thoughts?</div><div>-Brett</div><div><br></div><div><br></div><div><br></div><div><br><br><div class="gmail_quote">On Wed, Apr 22, 2009 at 2:38 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi Brett,<br>
<br>
I think the problem is at DB level - what you can try to do (just to spot the problem) is to stop whatever other apps/procs that are using the opensips DB and let only opensips runing on it. Redo the test and see if you get the same behaviour.<div class="im">
<br>
<br>
Regards,<br>
Bogdan<br>
<br>
Brett Nemeroff wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Bogdan,<br>
I no longer get crashes. However the opensips process hangs pretty badly while the DB operations are going on. I've tried to rewrite my queries to do more small queries rather than longer slow ones. <br>
So what I'm doing, I'm using sipp performing calls at 30CPS lasting 10 seconds (to generate a lot of call records).<br>
<br>
While this is running, I run my rating script, which gathers unique callid. smashes records together into a cdr record.<br>
<br>
My database engine is InnoDB and I'm using transactions. I'm not actually getting to a commit in any of this.<br>
<br>
So while my script is running. I see on the UAS side of sipp, it stops receiving calls, and starts performing retransmissions. I've verified with tshark that packets are hitting opensips, but not getting a reply.<br>
<br>
I have 20 children running. Am I doing something wrong?<br>
<br>
Thanks for your help,<br>
Brett<br>
<br>
<br></div><div class="im">
On Wed, Apr 8, 2009 at 7:54 AM, Bogdan-Andrei Iancu <<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>>> wrote:<br>
<br>
Both.<br>
<br>
Brett Nemeroff wrote:<br>
<br>
Is that on the 1_5 branch or trunk?<br>
<br>
<br>
On Wed, Apr 8, 2009 at 7:45 AM, Bogdan-Andrei Iancu<br>
<<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>><br></div><div><div></div><div class="h5">
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>>>> wrote:<br>
<br>
Hi Brett,<br>
<br>
thanks to your logs, I spoted the problem. The fix is<br>
available on<br>
SVN.<br>
<br>
<br>
Thanks and regards,<br>
Bogdan<br>
<br>
Brett Nemeroff wrote:<br>
<br>
Bogdan,<br>
For what it's worth, I've updated to latest 1_5 tonight<br>
(about<br>
20 minutes ago) and I still am having problems. Full out<br>
crashes as well.<br>
<br>
I rewrote my queries so I'd have a bunch of little<br>
(select *<br>
from acc where callid=X) kinds of queries. Of course,<br>
there is<br>
a lot of DB activity while this happens. Crashes start to<br>
happen within seconds of the DB activity ramping up.<br>
<br>
For grins, I slowed my queries down to ensure I only<br>
did one<br>
query per second (in my database, not opensips).. after<br>
about<br>
15-20 queries (different each time really) opensips<br>
would just<br>
crash.<br>
<br>
I have acc and sip_trace loaded up, sip_trace isn't<br>
active for<br>
these calls. Also potentially relevant, my acc table is an<br>
InnoDB table.<br>
<br>
Now if I slowed my call volume to 1CPS and keep the<br>
queries at<br>
1 QPS, it seemed to be happier, but still crashes<br>
eventually.<br>
<br>
-Brett<br>
<br>
<br>
<br>
On Mon, Apr 6, 2009 at 11:27 AM, Bogdan-Andrei Iancu<br>
<<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>>><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>>>>> wrote:<br>
<br>
Hi Brett,<br>
<br>
it looks like the DB connections are dropped and<br>
reconnect is<br>
taking place (this are the errors about). But to<br>
find out<br>
the real<br>
cause, I can enable some more logs to spot the<br>
reason for<br>
re-connect...<br>
<br>
I will do it later as right now I'm in the middle of<br>
some DB<br>
debugging and I'm afraid of mixing different patches and<br>
what goes<br>
on SVN :)<br>
<br>
Regards,<br>
Bogdan<br>
<br>
Brett Nemeroff wrote:<br>
<br>
Hi All,<br>
So I'm doing some load testing with sipp on my<br>
opensips 1.5<br>
system. I just checked out (like 2 hours ago,<br>
the 1.5<br>
branch<br>
from SVN). Everything works just fine, until I<br>
run some<br>
rating scripts on my database (perl scripts<br>
accessing the<br>
mysql db directly). While my scripts are<br>
running, I see the<br>
UAS in sipp retransmitting the 200 OKs and the<br>
following gets<br>
printed to the syslog:<br>
<a href="http://www.pastebin.ca/1381169" target="_blank">http://www.pastebin.ca/1381169</a><br>
<br>
As soon as my perl script is done, the 200OKs stop<br>
retransmitting...<br>
My PERL script isn't doing anything terribly<br>
unusual,<br>
however,<br>
it is performing the queries inside of a<br>
transaction,<br>
including a "SELECT/DELETE * FROM acc WHERE "<br>
kind of<br>
clause.<br>
<br>
Any ideas as to what is causing this? I'm afraid<br>
I may be<br>
losing call records..<br>
<br>
-Brett<br>
<br>
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>><br>
<mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>>><br>
<mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>><br>
<mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>>>><br>
<br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
<br>
<br>
<br>
<br>
</div></div></blockquote>
<br>
</blockquote></div><br></div>