[OpenSIPS-Users] Opensips 2.2.2 memory issue?
Vasilios Tzanoudakis
vasilios.tzanoudakis at voiceland.gr
Tue Apr 4 09:50:09 EDT 2017
Dear Team,
We faced the same problem again today after 5 days of heavy traffic.
Opensips Had exact the same behaviour like the first time above.
All Opensips processes were running but had no response. I have pasted
debugs from all listeners and rtpproxy daemons.
Here are all the gdb traces from opensips : https://pastebin.com/ix1puaqt
and here are the rtpproxy 2.0.0 debugs : https://pastebin.com/x9sdb5df
Thank you in advance
*Vasilios Tzanoudakis* | Technology Director
t. +30-212-222-8003 | f. +30-212-222-8001
2, Klisthenous Str., Metamorfosi, 144 52, Attica, Greece
vasilios.tzanoudakis at voiceland.gr | www.voiceland.gr
On Wed, Mar 29, 2017 at 12:25 PM, Vasilios Tzanoudakis <
vasilios.tzanoudakis at voiceland.gr> wrote:
> Dear all,
>
> After 11 days os usage Opensips process was not responding at all.
>
> This is the second time I am facing this kind of issue.
> I am sending you all the information below:
>
> System has 4 CPU Cores / 4 GB Ram
>
> Opensips version 2.2.2
>
> *Run parameters : *
> S_MEMORY=2048
> P_MEMORY=1024
>
> *uname -a*
> Linux 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
>
> *Compiled with the following flags:*
>
> PKG_MALLOC
> SHM_MMAP
> USE_MCAST
> DISABLE_NAGLE
> STATISTICS
> HAVE_RESOLV_RES
> F_MALLOC
> F_MALLOC_OPTIMIZATIONS
> NO_DEBUG
>
> *and with the following extra modules : *
>
> db_mysql
> dialplan
> regex
> snmpstats
>
> *Loaded modules in opensips.cfg*
>
> loadmodule "proto_udp.so"
> loadmodule "mi_datagram.so"
> loadmodule "signaling.so"
> loadmodule "sl.so"
> loadmodule "tm.so"
> loadmodule "rr.so"
> loadmodule "maxfwd.so"
> loadmodule "sipmsgops.so"
> loadmodule "mi_fifo.so"
> loadmodule "uri.so"
> loadmodule "db_mysql.so"
> loadmodule "usrloc.so"
> loadmodule "registrar.so"
> loadmodule "auth.so"
> loadmodule "auth_db.so"
> loadmodule "dialog.so"
> loadmodule "nathelper.so"
> loadmodule "rtpproxy.so"
> loadmodule "drouting.so"
> loadmodule "uac.so"
>
> */usr/local/opensips2/sbin/opensipsctl fifo ps*
>
> Process:: ID=0 PID=1843 Type=attendant
> Process:: ID=1 PID=1845 Type=MI FIFO
> Process:: ID=2 PID=1846 Type=MI Datagram
> Process:: ID=3 PID=1847 Type=time_keeper
> Process:: ID=4 PID=1849 Type=timer
> Process:: ID=5 PID=1850 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=6 PID=1851 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=7 PID=1852 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=8 PID=1853 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=9 PID=1854 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=10 PID=1855 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=11 PID=1856 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=12 PID=1857 Type=SIP receiver udp:X.X.X.X:65100
> Process:: ID=13 PID=1859 Type=Timer handler
>
> *gdb /usr/local/opensips2/sbin/opensips 1851*
>
> Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging
> symbols found)...done.
> Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
> Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2...Reading
> symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_dns-2.19.so..
> .done.
> done.
> Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2
> 0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd,
> word=0x636f732e3379786f) at wordexp.c:222
> 222 wordexp.c: No such file or directory.
>
>
> *(gdb) bt*
> #0 0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd,
> word=0x636f732e3379786f) at wordexp.c:222
> #1 parse_backtick (ifs_white=0x722f7261762f0001 <error: *Cannot access
> memory at address* 0x722f7261762f0001>, ifs=0x7fae24ccc960 "",
> pwordexp=0x838fd4 <buf+916>, flags=0, offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x1, word_length=0x7ffc32c06060,
> word=0x7ffc32c060c0) at wordexp.c:2147
> #2 wordexp (words=<optimized out>, pwordexp=<optimized out>, flags=0) at
> wordexp.c:2359
> #3 0x00007ffc32c06220 in ?? ()
> #4 0x0000000000000000 in ?? ()
>
>
> *(gdb) bt full*
> #0 0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd,
> word=0x636f732e3379786f) at wordexp.c:222
> No locals.
> #1 parse_backtick (ifs_white=0x722f7261762f0001 <error: *Cannot access
> memory at address *0x722f7261762f0001>, ifs=0x7fae24ccc960 "",
> pwordexp=0x838fd4 <buf+916>, flags=0, offset=0x7ffc00000000,
> words=<optimized out>, max_length=0x1, word_length=0x7ffc32c06060,
> word=0x7ffc32c060c0) at wordexp.c:2147
> squoting = <optimized out>
> comm_length = 8622050
> *comm = 0x6 <error: Cannot access memory at address 0x6>*
> error = <optimized out>
> comm_maxlen = 8700952
> #2 wordexp (words=<optimized out>, pwordexp=<optimized out>, flags=0) at
> wordexp.c:2359
> words_offset = 0
> word_length = 0
> max_length = 140388071391744
> word = 0x7ffc32c065c8 ""
> error = <optimized out>
> ifs = 0x7fae24ccc960 ""
> ifs_white = "\000\000\000"
> old_word = {we_wordc = 0, we_wordv = 0x0, we_offs = 0}
> #3 0x00007ffc32c06220 in ?? ()
> No symbol table info available.
> #4 0x0000000000000000 in ?? ()
> No symbol table info available.
>
> *ldd -v /usr/local/opensips2/sbin/opensips*
> linux-vdso.so.1 => (0x00007fff96f49000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f75c45de000)
> libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
> (0x00007f75c43c3000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f75c3ffa000)
> /lib64/ld-linux-x86-64.so.2 (0x000055e9cad86000)
>
> Version information:
> /usr/local/opensips2/sbin/opensips:
> libresolv.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libresolv.so.2
> libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2
> libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
> /lib/x86_64-linux-gnu/libdl.so.2:
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
> libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
> /lib/x86_64-linux-gnu/libresolv.so.2:
> libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
> libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
> /lib/x86_64-linux-gnu/libc.so.6:
> ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
>
> *# free -m*
> total used free shared buffers cached
> Mem: 3951 2842 1109 28 66 2156
> -/+ buffers/cache: 619 3332
> Swap: 4091 90 4001
>
> *# vmstat *
> procs -----------memory---------- ---swap-- -----io---- -system--
> ------cpu-----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa st
> 2 0 92396 1136328 68576 2207768 0 0 5 14 10 2 1 2
> 97 0 0
>
> *# w*
> 11:43:18 up 11 days, 23:16, 1 user, load average: 2.46, 2.39, 2.36
>
> *# netstat -ulnp*
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> udp 426496 0 X.X.X.X:65100 0.0.0.0:*
> 1843/opensips
>
> also after restarting opensips process system started to receive traffic
> again BUT I got the following errors on syslog regarding rtpproxy
>
> Mar 29 09:57:32 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:09 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:10 rtp1 /usr/local/opensips2/sbin/opensips[1857]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:12 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/opensips[1850]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/opensips[1851]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
> Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/opensips[1853]:
> ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply from rtp
> proxy
>
> and I had to reboot rtpproxy processes too for traffic to pass through
> without problems.
> I don't know if this is another problem with rtpproxy but I am just
> mentioning it.
>
> thank you in advance
>
>
> *Vasilios Tzanoudakis* | Technology Director
> t. +30-212-222-8003 <+30%2021%202222%208003> | f. +30-212-222-8001
> <+30%2021%202222%208001>
> 2, Klisthenous Str., Metamorfosi, 144 52, Attica, Greece
> vasilios.tzanoudakis at voiceland.gr | www.voiceland.gr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170404/2d7e477d/attachment-0001.html>
More information about the Users
mailing list