Python multiprocessing Queue memory management
Python multiprocessing Queue memory management
Say I have the Main processes and 2 additional processes A and B. In this program A is supposed to send data to B. If we have some code like this:
Main
A
B
from multiprocessing import Process, Queue
def process_a(iterable, q):
for x in iterable:
q.put(x)
def process_b(q):
while some_condition():
x = q.get()
iterable = some_iterable()
q = Queue()
pa = Process(target=process_a, args=(iterable, q))
pb = Process(target=process_b, args=(q,))
pa.start()
pb.start()
pa.join()
pb.join()
given that the Queue q was created in the Main process, does the data flow like this?
q
A => Main => B
If so, is there a way to have a Queue initialized on B and passed to A such that data goes directly from A to B skipping Main?
Queue
B
A
A
B
Main
1 Answer
1
given that the Queue q was created in the Main process, does the data flow like this?
A => Main => B
No. As explained in the docs, a Queue is just an auto-synchronizing wrapper around a Pipe. When you pass a Queue to a child, you're just passing that Pipe and some locks.
Queue
Pipe
Queue
Pipe
And the Pipe is just a wrapper around an operating system pipe. When you pass a Pipe to a child, you're just passing the pipe's file descriptor/handle.
Pipe
Pipe
Ignoring the locks, process A is basically just writing to a pipe, and process B is just reading from it.
The locks do make things a bit more complicated (and may also mean that process A spins up a hidden background thread), but they still don't involve the main process at all.
Unless the main process calls a method on a queue, it has nothing to do with that queue at all.
Thanks for contributing an answer to Stack Overflow!
But avoid …
To learn more, see our tips on writing great answers.
Required, but never shown
Required, but never shown
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
