Getting started with Jupyter - internals

Jupyter notebook is a web based application that uses web socket to talk to ipython, execute python (or other languages like Julia and Haskell). It is really popular when it comes to executing machine learning kernels.

Say for example, you trying to use numpy to read off certain csv files and runs mathplot, you can easily do so using IPython.

Great! So how does it work?

First you need to setup jupyter notebook. I assumed that you have installed Anaconda in your system.

First, lets setup our virtual environment
virtualenv notebook
Now activate our newly created environment.
activate notebook
pip install --upgrade setuptools pip
git clone
cd notebook
pip install -e .

From diagram below, we can see notebook depends on jupyter_client (purple) to talk to ipython (yellow)

At the end of the cell execute request, results will be display / render on notebook. Communication is achieve via web socket and all the code is in javascript and not python. Will consume alot of time, if you start looking at the wrong place.

To see  what happens when a user submit a python code say "print("testing testing "")"", code,

Take a look at notebook/js/services/kernel/kernel.js, in particular

Executing  a piece of code. 

1. execute function,
2. send_shell_message,
3. kernel._send - sends code over for execution.

Response  flow (notebook/js/outputarea.js.

1. kernel._handle_output_message - receive response from ipython execution.

2. OutputArea.prototype.append_stream function - basically renders output to a div.

When you're exposed to the code above, you will be fine and able to understand the internals of jupyter.

Returning code execution results 

User code execution results are obtain from IOPUB. Yeap, IOPUB, not SHELL socket.

You can examine it by looking at a javascript  notebook/js/codecell.js under the method called "CodeCell.prototype.get_callbacks"

In fact, jupyter console uses the same approach.

Any responses from ipython -> jupiter client -> notebook gets display here.

What happens if you do a plot? The results will be return as embedded image like what you would normally do with based64 image encoder for your web page.


Popular posts from this blog

ionic2 cordova build android - Unable resolve gradle 2.2.3

OpenCover code coverage for .Net Core

Using Custom DLL with IronPython / Scripts