Run long compute jobs on Jupyter

Typically we use Jupyter Notebooks to execute code in rather small steps - cell per cell.

But sometimes there is some heavy computational lifting involved and we want the code to run while we are away.

How can we make sure that a long running job does not break or stop when we close the Jupyter browser window?

Why unwatched jobs break

A Jupyter Notebook by default writes output to the "stdout" stream - or "stderr" for error handling.

When the browser window is closed these streams disappear.

Even though the code is still running on the Jupyter server, it cannot write to these streams, anymore.

As soon as any output gets written, the script breaks. So when we go back to the notebook with our browser, we see a failed job.

The solution

We can suppress any output to stdout or stderr. This keeps our script running.

There are many ways to achieve this. If you have something simple in mind, try it.

It might just be good enough.

A very thorough way, however, would be this:

import contextlib as cxt
import os

print('Here we can still print.')
with (
  cxt.redirect_stdout(fo := open(os.devnull, 'w')),
  cxt.redirect_stderr(fe := open(os.devnull, 'w')),
):
  print("This will not get printed.")

  # call your long-running job here!

  fo.close()
  fe.close()

print('Here we can print again.')

But I need the output!

We might want to preserve the output for later analysis or debugging.

So we can just as well save the output to text files:

import contextlib as cxt
import os

print('Here we can still print.')
with (
  cxt.redirect_stdout(fo := open('./my-saved-stdout.txt', 'w')),
  cxt.redirect_stderr(fe := open('./my-saved-stderr.txt', 'w')),
):
  print("This will get written to ./my-saved-stdout.txt")

  # call your long-running job here!

  fo.close()
  fe.close()

print('Here we can print again.')

Full test example

Before you invest your time, you might want to test this. If you run the code below you can see how it works.

Put the code in a Jupyter Notebook. Run the entire notebook.

While the execution is still ongoing, close the browser window.

After >10minutes go back to the Notebook and see if the job finished and the output got redirected to "my-saved-stdout.txt".

import time
import os
import contextlib as cxt

def run_and_print_for_n_minutes(n):
    for minute in range(n):
        print(f"Running for {minute} minutes.")
        time.sleep(60)

print('Here we can still print.')

with (
  cxt.redirect_stdout(fo := open('./my-saved-stdout.txt', 'w')),
  cxt.redirect_stderr(fe := open('./my-saved-stderr.txt', 'w')),
):
  print("This will get written to ./my-saved-stdout.txt")

  # call your long-running job here!
  run_and_print_for_n_minutes(10)
    
  fo.close()
  fe.close()