Jupyter Kernel: How to Speak in Another Language
9 likes2,392 views
The document discusses how to create a Jupyter kernel. It explains that kernels use ZeroMQ sockets to communicate with clients via the Jupyter messaging protocol. Native kernels are implemented from scratch while wrapper kernels are built from an existing interpreter using Python. The document provides examples of existing kernels like IJulia and the Python ipykernel. It also outlines the steps to build a wrapper kernel and mentions several other kernel types.
More Related Content
What's hot (19)
Viewers also liked (17)
Similar to Jupyter Kernel: How to Speak in Another Language (20)
More from Wey-Han Liaw (13)
Recently uploaded (20)
Jupyter Kernel: How to Speak in Another Language
- 1. Jupyter Kernel How to Speak in Another Language
- 2. How to Make a Jupyter Kernel:
- 3. 1: 5 ømq sockets
- 5. 3. Holy Shit, YOU RUINED EVERYTHING, LIKE EVERYTHING YOU DO
- 6. 4. This Kernel Is Like Your Life
- 7. 5. Begin a Lot of Things, Left Undone
- 8. 6. And Nobody Loves You
- 10. Agenda • Jupyter, IPython, notebook, console, client, kernel • The Interactive Computing Protocol • Implementing a Kernel • Live Demo
- 11. About this guy • 廖偉涵 Adrian Liaw • github.com/adrianliaw • High School Homeschooler • A Pythonista • Student of Udacity’s Data Analyst Nanodegree
- 12. About this guy • Amateur Pianist • Classical Music Lover • Coffee Shop Lover
- 15. IPYTHON NOTEBOOK WEB-BASED NOTEBOOK ENVIRONMENT
- 16. The Big Split™
- 18. Jupyter Notebook (With R Kernel)
- 19. -Project Jupyter “Project Jupyter was born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages.”
- 23. CLIENT KERNEL Execute “10 + 3”
- 26. What’s Inside IPython • The Interactive IPython Shell (No ØMQ) • Magic Commands • Auto Word Completer • Beautiful Traceback • Shell History Management
- 27. Uh, And Kernel?
- 28. IPyKernel • Jupyter’s “Kernel 0” • ØMQ Handlers • github.com/ipython/ipykernel
- 29. What’s Inside Jupyter • Web-based Notebook Interface & nbconvert • Interactive Qt Console • Terminal-based Console • Spec for Notebook Document Format • Spec for Interactive Computing Protocol
- 31. Interactive Computing Protocol • i.e. Jupyter Messaging Protocol • Communication Between Kernel and Client • Based on ØMQ and JSON
- 32. DEALER ROUTER CLIENT KERNEL Shell SUB PUBIOPub DEALER ROUTER stdin DEALER ROUTER Control REQ REP Heartbeat
- 33. Connection File { "transport": "tcp", "ip": "127.0.0.1", "shell_port": 56166, "iopub_port": 56167, "stdin_port": 56168, "control_port": 56169, "hb_port": 56170, "signature_scheme": "hmac-sha256", "key": "<hashed key>" }
- 34. Message Format { 'header' : { 'msg_id' : str, 'username' : str, 'session' : str, 'date': str, 'msg_type' : str, 'version' : '5.0', }, 'parent_header' : dict, 'metadata' : dict, 'content' : dict, }
- 36. { "header": { "msg_type": "kernel_info_request", ... }, "content": {}, } CLIENT KERNEL Shell
- 37. CLIENT KERNEL Shell { "header": { "msg_type": "kernel_info_reply", ... }, "parent_header": {...}, "content": { "banner": "Python 3.5.1 ... IPython 4.0.0 ...", "implementation": "ipython", "implementation_version": "4.0.0", "language_info": { "name": "python", "version": "3.5.1", ... }, }, }
- 38. CLIENT KERNELShell { "header": { "msg_type": "execute_request", ... }, "content": { "code": "print('HAI')", "silent": False, "store_history": True, }, }
- 39. CLIENT KERNELIOPub { "header": {"msg_type": "stream", ...}, "parent_header": {...}, "content": { "name": "stdout", "text": "HAIn", }, }
- 40. CLIENT KERNELShell { "header": { "msg_type": "execute_reply", ... }, "parent_header": {...}, "content": { "execution_count": 1, "status": "ok", "user_expressions": {}, }, }
- 41. CLIENT KERNELShell { "header": {"msg_type": "execute_request", ...}, "content": { "code": "pd.read_csv('foo.csv')", "silent": False, "store_history": True, }, }
- 42. CLIENT KERNELIOPub { "header": {"msg_type": "execute_result", ...}, "parent_header": {...}, "content": { "execution_count": 5, "data": { "text/plain": " col1 col2n A 3n....", "text/html": "<div><table>...</table></div>", }, }, }
- 43. CLIENT KERNELShell { "header": { "msg_type": "execute_reply", ... }, "parent_header": {...}, "content": { "execution_count": 5, "status": "ok", "user_expressions": {}, }, }
- 46. CLIENT KERNELShell { "header": {"msg_type": "execute_request", ...}, "content": { "code": "read.csv('table.csv')", "silent": False, "store_history": True, }, }
- 47. CLIENT KERNELIOPub { "header": {"msg_type": "execute_result", ...}, "parent_header": {...}, "content": { "execution_count": 1, "data": { "text/plain": " col1 col2n A 3n....", "text/html": "<table>...</table>", }, }, }
- 48. CLIENT KERNELShell { "header": { "msg_type": "execute_reply", ... }, "parent_header": {...}, "content": { "execution_count": 1, "status": "ok", "user_expressions": {}, }, }
- 49. MAKING A KERNEL
- 50. Kernel Types • Native Kernel • Python Wrapper Kernel • REPL Wrapper Kernel
- 51. IRKernel 11. zmqctx <<- zmq.ctx.new() 12. sockets <<- list( 13. hb = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$REP), 14. iopub = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$PUB), 15. control = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER), 16. stdin = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER), 17. shell = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER))
- 52. IRKernel IRuby IHaskell cling IJulia IScala Lua Kernel Clojure-Kernel ITorch IJavascript ipykernel
- 53. Wrapper Kernel • Python-driven • ipykernel.kernelbase.Kernel • http://bit.ly/WrapperKernel
- 54. from ipykernel.kernelbase import Kernel class MyKernel(Kernel): def do_execute def do_complete def do_history def do_shutdown
- 55. Hy Kernel Redis Kernel Xonsh C Kernel Matlab Kernel SAS Kernel
- 56. REPL Wrapper from pexpect.replwrap import REPLWrapper
- 57. Octave Kernel Jupyter Powershell Bash Kernel Mongo Kernel Scilab Kernel
- 58. LIVE DEMO
- 59. Q&A
- 60. Reference • P.3, P.4, P.17, P.20, P.50 images: Project Jupyter • P.6, P.7, P.8 images: Molg H. • P.16: Project Jupyter • P.11 image: Toomore Chiang • P.12 image: Virginia Cheng • P.13 image: ipython • P.45 image: R