Coding With Fun
Home Docker Django Node.js Articles Python pip guide FAQ Policy

How to get realtime output from subprocess in python?


Asked by Saul Mullen on Dec 12, 2021 FAQ



If the subprocess will be a Python process, you could do this before the call: Or alternatively pass this in the env argument to Popen. Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like: See also here about stdbuf or other options. (See also here for the same answer.) I ran into the same problem awhile back.
Moreover,
It is very easy to run a sub-process in python using its subprocess module. A problem occurs when you need to process the output of a long-running process as soon as it is emitted: the output is usually buffered. It gets more complicated if you want to monitor several subprocesses in parallel.
Subsequently, You need to use the subprocess Python module which allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. Another option is to use operating system interfaces provided by Python such as: os.system.
Consequently,
The timeout argument is passed to Popen.communicate (). If the timeout expires, the child process will be killed and waited for. The TimeoutExpired exception will be re-raised after the child process has terminated. The input argument is passed to Popen.communicate () and thus to the subprocess’s stdin.
Similarly,
The p.communicate () has a few problems: The data read is buffered in memory, so do not use this method if the data size is large or unlimited. It will block next statement till external command is completed i.e. you will not get real time output from the command.