[OpenSIPS-Devel] [OpenSIPS-Users] python module - bug and questions
Maxim Sobolev
sobomax at sippysoft.com
Tue Apr 25 15:57:33 EDT 2017
Robert, what you have described is how essentially python module works. We
init the interpreter and then the opensips forks, and in each of its
children the mod_init() python function exported by your code is called,
which returns instance whose method can be invoked via script. So at the
end of the day you end up with as many independent processes as there are
opensips children.
Here for example what we have in our handler:
from OpenSIPS import LM_ERR
from ser import SerHandler
class OpenSIPSHandler(SerHandler):
def www_authenticate(self, msg):
...
return rval
def mod_init():
inst = OpenSIPSHandler()
return inst
-Max
On Tue, Apr 25, 2017 at 10:25 AM, Mundkowsky, Robert <rmundkowsky at ets.org>
wrote:
> Thanks for responding. Thought you would just advise me to use specific
> compiler flags first.
>
>
>
> Do you want me to use your jig (testing setup) to make a crash?
>
> Or do you want me to add python code to your git repo so you can run it
> using your jig?
>
>
>
>
>
> What you are doing with code is pretty complex for me and makes my brain
> want to explode.
>
>
>
> My understanding is:
>
> 1) Python threads are not real threads, instead only one python
> thread runs at a time (per the global interpreter lock)
>
>
>
> 2) You are creating a Python thread (that uses one Python
> interpreter) in the main openSIPS process that likely loads the openSIPS
> user’s Python module to be used
>
> 3) Then, I assume, openSIPS main process forks children processes
>
> a. I assume Python thread and interpreter is treated as data and
> code that is copied to all children processes. Note if it were a real
> Linux thread then it would not be copied to children processes.
>
> b. You then use the copied Python thread in all children processes
> as if it is a single Python thread across the main and child processes or
> as if it were a single Python thread per each main or child process
>
>
>
> I guess this might work, because if the Python thread is truly copied then
> you really have a different Python thread and interpreter per main and
> child processes. But Python documentation (https://docs.python.org/2/c-
> api/init.html#c.PyThreadState) states “When extending or embedding
> Python, there is no way to inform Python of additional (non-Python) locks
> that need to be acquired before or reset after a fork”, which makes me
> think what you are doing will not work. This is because any Python modules
> someone might use (e.g. MySQLdb) might use a lock in the main process that
> can’t get cleared in the child process.
>
>
>
> Might be wiser to just start one Python thread and interpreter in each
> child process I assume is kicked off via child_init. Then you would still
> have the module loaded once per child. Although I am not sure if main
> process needs a Python thread, but probably not.
>
>
>
>
>
> Robert Mundkowsky
>
>
>
> *From:* Maxim Sobolev [mailto:sobomax at sippysoft.com]
> *Sent:* Monday, April 24, 2017 10:09 PM
> *To:* Mundkowsky, Robert <rmundkowsky at ets.org>
> *Cc:* OpenSIPS devel mailling list <devel at lists.opensips.org>;
> Bogdan-Andrei Iancu <bogdan at opensips.org>
> *Subject:* RE: [OpenSIPS-Users] python module - bug and questions
>
>
>
> Robert sorry for the delay. What could be useful if you can produce chunk
> of python routing code (could be NOP on the functional side) that crashes
> repeatedly on our voiptest jig (https://github.com/sippy/voiptests). The
> travis is setup to build pull requests, so that you can inject your code
> and let it rip. It uses similar configiration of linux / python.
>
>
>
> -Max
>
>
>
> On Apr 21, 2017 9:32 AM, "Mundkowsky, Robert" <rmundkowsky at ets.org> wrote:
>
> Python 2.7.13 on Ubuntu 16.04.2 LTS.
>
>
>
> How do you link using pthreads?
>
>
>
> Do you add flags to Makefile, or will setting “USE_PTHREAD_MUTEX” and
> “USE_POSIX_SEM” via menuconfig do it?
>
>
>
> Since openSIPS is not multithread, seems like it would be a lot cleaner to
> not use threads at all on C/C++. Then threads could still be used on
> Python side if needed.
>
>
>
> Robert Mundkowsky
>
>
>
> *From:* Maxim Sobolev [mailto:sobomax at sippysoft.com]
> *Sent:* Friday, April 21, 2017 11:57 AM
> *To:* Bogdan-Andrei Iancu <bogdan at opensips.org>
> *Cc:* OpenSIPS users mailling list <users at lists.opensips.org>;
> Mundkowsky, Robert <rmundkowsky at ets.org>
> *Subject:* Re: [OpenSIPS-Users] python module - bug and questions
>
>
>
> Robert, what Bogdan says is essentially correct. The OpenSIPS itself is
> not using any threads AFAIK, therefore python module code is kept as simple
> as possible. Now back to the original question: we use quite a lot of
> python code in our routing and some of the python modules that are running
> are actually creating threads on their own and it appears to be pretty
> stable. What OS / python version are you using? What could be different in
> our case is that we link our OpenSIPS binary with pthreads always (which
> may or may not be the case on your build), so there might be some vital
> threads runtime infrastructure that is not getting initialized in your case.
>
>
>
> -Max
>
>
>
>
>
> On Fri, Apr 21, 2017 at 1:25 AM, Bogdan-Andrei Iancu <bogdan at opensips.org>
> wrote:
>
> Hi Robert,
>
> The only question I can answer is 1) - OpenSIPS it is a multi-process
> application (and not using threads).
>
> How the python module is design (from threading perspective), I do not
> know - maybe Maxim, the author of this module can help with this.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
>
>
> OpenSIPS Summit May 2017 Amsterdam
>
> http://www.opensips.org/events/Summit-2017Amsterdam.html
>
> On 04/19/2017 08:43 PM, Mundkowsky, Robert wrote:
>
>
>
> Hi,
>
>
>
> This email should likely go to your other dev forum, but I don’t have
> access yet.
>
>
>
> I am using openSIPS 2.2.3 on Ubuntu 16.04.2 LTS.
>
>
>
> I am using the python module. The python script called works fine when
> called from outside openSIPS and it works fine when called inside openSIPS,
> if it is triggered once at a time.
>
>
>
> But if the python script is triggered twice by two phone calls or more or
> even one call after another in short order, then there are weird errors
> which show up in different places in the code like:
>
>
>
> ERROR:python:python_handle_exception: #011TypeError: an integer is
> required
>
>
>
> I am guessing the openSIP python module has some problem/s related to
> threading.
>
>
>
>
>
> Questions:
>
> 1) I am guessing that OpenSIPS uses a multi-threaded architecture?
>
> 2) Is the python module meant to be a single thread? Or
> multi-threaded?
>
> a. If it is meant to be single threaded then why use python library
> thread functions? Just use python library without multi-threaded stuff.
>
> b. If it is meant to be multi-threaded then why is there only one
> call to PyThreadState_New and myThreadState is a global used everywhere?
>
> i. I
> would think a thread per call to python_exec would make more sense and make
> the code easier to understand?
>
> 3) Also why is there no clean up code (PyThreadState_Clear,
> PyThreadState_Delete, Py_Finalize)?
>
> a. I am guessing the idea is you do not need clean up, because it
> only happens when openSIPS is turned off.
>
>
>
> Robert
>
>
>
>
>
> For reference:
>
> https://www.codeproject.com/articles/11805/embedding-python-in-c-c-part-i
>
> http://www.awasu.com/weblog/embedding-python/threads/
>
> http://stackoverflow.com/questions/26061298/python-multi-thread-multi-
> interpreter-c-api
>
>
>
>
> ------------------------------
>
> This e-mail and any files transmitted with it may contain privileged or
> confidential information. It is solely for use by the individual for whom
> it is intended, even if addressed incorrectly. If you received this e-mail
> in error, please notify the sender; do not disclose, copy, distribute, or
> take any action in reliance on the contents of this information; and delete
> it from your system. Any other use of this e-mail is prohibited.
>
>
>
> Thank you for your compliance.
> ------------------------------
>
>
>
> _______________________________________________
>
> Users mailing list
>
> Users at lists.opensips.org
>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>
>
>
>
> --
>
> Maksym Sobolyev
> Sippy Software, Inc.
> Internet Telephony (VoIP) Experts
> Tel (Canada): +1-778-783-0474 <(778)%20783-0474>
> Tel (Toll-Free): +1-855-747-7779 <(855)%20747-7779>
> Fax: +1-866-857-6942 <(866)%20857-6942>
> Web: http://www.sippysoft.com
> MSN: sales at sippysoft.com
> Skype: SippySoft
>
>
> ------------------------------
>
> This e-mail and any files transmitted with it may contain privileged or
> confidential information. It is solely for use by the individual for whom
> it is intended, even if addressed incorrectly. If you received this e-mail
> in error, please notify the sender; do not disclose, copy, distribute, or
> take any action in reliance on the contents of this information; and delete
> it from your system. Any other use of this e-mail is prohibited.
>
>
>
> Thank you for your compliance.
> ------------------------------
>
>
> ------------------------------
>
> This e-mail and any files transmitted with it may contain privileged or
> confidential information. It is solely for use by the individual for whom
> it is intended, even if addressed incorrectly. If you received this e-mail
> in error, please notify the sender; do not disclose, copy, distribute, or
> take any action in reliance on the contents of this information; and delete
> it from your system. Any other use of this e-mail is prohibited.
>
> Thank you for your compliance.
> ------------------------------
>
--
Maksym Sobolyev
Sippy Software, Inc.
Internet Telephony (VoIP) Experts
Tel (Canada): +1-778-783-0474
Tel (Toll-Free): +1-855-747-7779
Fax: +1-866-857-6942
Web: http://www.sippysoft.com
MSN: sales at sippysoft.com
Skype: SippySoft
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20170425/0e523cd0/attachment-0001.html>
More information about the Devel
mailing list