GeezerVison

mpmprof uses argparse e.g. #128

Thanks @bbengfort! . Do you think it is possible to have only one executable file, i.e., merge mpmprof into mprof?

@fabianp yes, it's possible; just not very straightforward. I'll take a crack at it.

@fabianp ok, I've pushed a version that does it - would you mind taking a look and running your tests/examples?

Great work @bbengfort ! The code looks great, let me try it on some of my own problems. In the meantime, you can add your name to the Authors part of the README :-)

@fabianp thanks! I've updated the README with the example and usage and also added max value markers to the plot for the largest child.

If I run the example with mprof run -M multiprocessing_example.py then it fails with an _pickle.PicklingError exception. It works however with mprof run -M python multiprocessing_example.py (not the python after -M). However, both should give the same result as the python mode should be picked up by the .py extension.

Unfortunately, I think that the pickle error is a product of multiprocessing library: functions can only be pickled if they're defined at the top level and when you run it by creating a subprocess in mprof, the example function is no longer at the top level and therefore cannot be pickled.

I can see if I can change the example (though that may be difficult), but I also wanted to note that the problem exists even with the original version of the mprof, so it's not related to the code changes I proposed here. It's likely that any user that uses the multiprocessing library would have to use the python command to ensure their code is serialized/forked correctly.

http://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error

For the pickling, what I find strange is that both commands (with and without the "python" word in mprof) should be identical, as per lines 217-223 of mprof. Not sure why multiprocessing fails only on one of these cases since they should be executed the same way.

So I see what you mean, however there is a subtle difference. The args to Popen if options.python = True are:

['python', '-m', 'memory_profiler', '--timestamp', '-o', 'mprofile_20170321080505.dat', 'examples/multiprocessing_example.py']

Whereas the args to Popen if passed as an executable are

`['python', 'examples/multiprocessing_example.py']` 

In the first case, __main__ is memory_profiler.__main__ and examples/multiprocessing_example.py is run with execfile() on L1108 or exec() on L1119. This is correctly implemented (as discussed here) by embedding the exec into its own namespace (ns), thus making the exec'd file the equivalent of an import -- as a result, the functions inside examples/multiprocessing_example.py are not at the top level (but are nested under ns) and therefore cannot be pickled.

So that took me a while to figure out, but I think it makes sense. What do you think?

Thanks for the explanation and your time. I've merged the PR and added a small patch (52c5788) so that the example can be run without the python keyword.

Thanks!

Sure thing, let me know if there is anything else I can help with; sorry that this feature took so long!