Some musings on design architecture so far:
Update on the server side
The first week, I started by asking the question, how do I make Dnotebook work like Jupyter, meaning bash command works so I can run yarn add package in the browser, execute it server-side in Nodejs, and get real-time output logs in the browser.
I decided to go with the express HTTP stream. Nodejs has great support for streams, so I would basically start a server, send commands via HTTP request, open a stream response, capture logs from stdout and stderr, and continuously write back a response to the UI. This is basically done at the moment. see https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/routes/code_routes.ts
Now for the next task, I wanted to execute JS code. Executing JS code one time and returning the result is pretty straightforward, we can use an eval for that. But what I needed is a way to execute each cell in the same context. I.e All cells from the notebook will share the same variables-the typical notebook behavior So what do I do here?
Well, interestingly Nodejs has the VM API, which you can use to compile and execute code in a given context. So basically what I did was start a server, initialize a context, then send chunks of code to be executed in that context. Because the server keeps running between execution, the context is kept alive! This was implemented here: https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/runner/nodejs.ts
So with the core functionality done, I decided to add some more features.
First, I added an inline transpiler, meaning you can write any flavor of JS, and mid-way before it gets executed, it is transpiled using Babel to a compatible version that can run on the VM. See implementation here:
|
let transformedJsCode = transformSync(code, { |
So what next? Well, a lot of things still has to be done on the server-side, some of the important issues I’m yet to figure out is:
-
How to track the state of cells and know when they finish execution? This is quite tricky given the async nature of JS
-
Managing multiple child processes-Kernels in Jupyter notebooks-so I can easily start and stop them from the UI
-
Security of code process
Update on the UI
At the moment, it supports full editing of Markdown, JS, HTML, bash, and JSON. That means, syntax highlighting, code snippets, autocomplete, and execution all work at the moment. I used AceEditor for this. See implementation https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/frontend/components/CellEditor/index.tsx
So what’s next here? Well, the next major task we want to tackle in the UI is:
I will keep updating this comment as we add new features and encounter weird issues. This will be used to create a detailed architecture document before release.