Ruby Programming/Running Multiple Processes

Running Multiple Processes

There are several ways to run external commands in Ruby.

output = `command here` # gives you back full stdout

stdout_and_stdin = IO.popen("command here")

require 'popen3' # require 'open3' in 1.9

input,output,error,running_thread_on_19_or_greater = Open3.popen3("command here")

# or the same:
Open3.popen3("command here") do |stdin, stdout, stderr|
  # ...
end

pid = fork { puts 'in child process' } # posix platforms only

pid = Process.spawn "ls" # 1.9.x only
pid = Process.daemon "ls" # 1.9.x only, basically does a spawn and a disassociate on that pid.

Process.spawn (1.9 only) has many options.

Accessing PID's of running processes

For many of these commands there is no way to get the pid of a running child process while it is still running. For fork you get it immediately via $? as well as the fork return value, as well as for Process.{spawn,daemon} where the PID is the return value.

If you want the pid and I/O of a running sub-process, you'll have to use fork and redirect the child processes I/O to previously created pipe's, or jruby users have an available

 pid,input,output,error = IO.popen4("ls") #jruby only

method or use the #pid method if it is available, ex:

input, output, error, thread_if_on_19 = Open3.popen3 "ls"
pid = thread.pid
io = IO.popen("ls")
pid = io.pid

The PID is not accessible for the system and backtick calls while they are running (since $? is only available on a per-thread basis, and they haven't finished).

In reality all that $? means is "tell me what my child process was that most recently finished."

Note that on 1.8 Open3.popen3 doesn't have the PID available. If you want the pid in 1.8 in Linux, you'll probably need to create some pipes, fork a new process, and redirects its pipes to the ones you created. For windows users, though, if you want the PID for an open3 call in 1.8 windows you'll want to use a helper gem or IO.popen4 on jruby.

If you wanted to start a sub process without spawning a new thread in 1.9 (like popen3 but without its extra thread) you could use Process.spawn or create pipes, fork a process, and redirects its IO to the pipes et al.

If you want to start a sub process without spawning a new thread or creating IO objects in windows 1.8, you'll need to use a helper gem, like the win32-process gem.

Note well that if you start a process with access to its stdout/stdin/stderr streams, if you do not read from those streams in some way the process can *block* after it fills up a stream's buffer. I.e. you must read from them if they give a lot of output.

Reading piece-wise from a stream

This code seems to work:

 while !out.eof?
   print out.read 1024    
  end          
 Process.wait out.pid  
}

Though you probably don't have to wait for the PID at the end.

You could also just read the full stream output thus:

  print out.read

Reading and writing to an external process

You can use popen3 if you want to read and write to an external process. You can do it with popen as well by using the open stream type of "r+" (not "rw").

Note also that with popen that by default on windows it opens up all your file streams in "ascii" mode. To set them as binary mode, if desired, call #binmode on each descriptor that comes back to you (thanks imagemagick guys for the example).

Ex:

IO.popen("ruby", "r+") do |pipe|
  pipe.puts "puts 10**6"
  pipe.puts "__END__"
  pipe.gets
end

Only writing to a command/using encoding

Most of the other examples here expose both a read and write pipe to the child process. Sometimes you want to have just one or the other. HEre is a way.

just_stdin_to_process = open("|process_name", "w")

You can also use this same method to set an encoding for strings coming in from a process (there may be other ways, as well), in 1.9.

just_stdin_to_process = open("|process_name", "r:UTF-8")

It might work to open like this, too

stdin_and_out = IO.popen(c, "w") # outputs to stdout, stderr, but you can write to it

Windows: how to run ruby without opening up a command window

In general, if you start an app using rubyw.exe instead of ruby.exe, your ruby app's stdout/stderr is piped to (windows' equivalent of) /dev/null, so no command window appears.

However, if, within that app, you make a call out to system("something_else.exe"), it will then pop up its own console for input/output, in case it needs any (unless it also has an equivalent of rubyw.exe available for you).

See http://www.ruby-forum.com/topic/213521 for a list of possible solutions. NB that the require 'win32/system' snippet is to use the win32-system gem, which you'd need to install first. There doesn't seem to be a way to do it with the stdlib currently.

Another option is to use ffi to call CreateProcess directly, to essentially the same effect: https://gist.github.com/rdp/8229520 (originally from https://gist.github.com/jarib/280865 so the childprocess gem might do some of this for you or you could roll your own).

Another option to rubyw.exe is to run it using ruby.exe but in a "minimized" window to lessen output: http://www.justskins.com/forums/winapp-without-console-window-97080.html

win32-open3 gem appears to be another option: http://stackoverflow.com/questions/12684463/ruby-system-call-on-windows-without-a-popup-command-prompt

jrubyw also seems to be able to run them without opening a new command prompt window (jrubyw.exe).

Also in windows you may be able to spawn a separate background process using "start.exe" like system("start my_command.exe") ref: http://stackoverflow.com/a/3840737/32453

Chaining processes

So say you want to chain some processes together in ruby, the equivalent of bashes `a | b | c`?

Well first, you could run just that: `a | b | c` and it will hand the string over to bash and let it do all the redirection and return you the output of stdout. You can even run popen with chained commands, and it will do the same.

IO.popen("ls | grep unins").read # same thing as `ls | grep unins`

Or you can roll your own. The basic way is that you want to open some pipe's, then fork, within the child redirect stdin/stdout to the appropriate pipe (in our example case, that would be two pipes per connection, so total 6), then exec the desired command within each child process. phew!

Ruby has a built-in "shell" class. See also [1] section "method 8". [2] shows the basic syntax it uses internally for its redirection, I believe.

The gist of it is something like

pipe_me_in, pipe_peer_out = IO.pipe
pipe_peer_in, pipe_me_out = IO.pipe

fork do
  STDIN.reopen(pipe_peer_in)
  STDOUT.reopen(pipe_peer_out)
  Kernel.exec("echo 33")
  # this line is never executed because exec moves the process
end
pipe_peer_out.close
# file handles have to all be closed in order for the "read" method, below, to be able 
# to know that it's done reading data, so it can return. 
#See also http://devver.wordpress.com/2009/10/22/beware-of-pipe-duplication-in-subprocesses/
pipe_me_in.read

1.9's Process.spawn with its :stdout and :stderr simplified syntax might make this even easier, and hopefully possible even in windows. For 1.8 windows you could probably use the win32/process gem which also provides simplified :stdout and :stderr redirection.

See also 1.9's Open3.pipeline_start [3]

External Links

Here is a good background (find the section "Running Multiple Processes").

Here has some good reference links.