Working on CoCalc#

Once you have a project ready to go, you can get to work.

Exploring#

To start doing things immediately, simply connect to our CoCalc project, and open a Jupyter notebook such as Demo.ipynb. The first time you will be asked to “Select a Kernel”, and the Python 3 (Anaconda 2021) kernel will provide the full set of Anaconda tools. That’s it!

Note

If you have installed the project (as discussed below), then you should also see an associated kernel (Python 3 (wsu-pwg-demo-2022) in this case). Selecting this will allow you to play with the carefully controlled environment for that project instead.

Doing Research#

Quickly using Jupyter notebooks is a great way to get started, but once you figure some things out and start writing real code, it is time to migrate this code to a package, version control it, test it, etc. When we are ready to do this, we create a project as discussed earlier, then clone this to CoCalc to start working:

ssh ccpwg
mkdir .repositories
cd .repositories
git clone git@gitlab.com:mforbes/wsu-python-working-group-demo-2022.git
cd wsu-python-working-group-demo-2022
make init   # May take a while...

This will get everything up and running in our CoCalc project. By our group convention, we put our repositories in ~/.repositories and then symlink them to the top level as needed so we can keep the top level clean:

cd ~
ln -s .repositories/wsu-python-working-group-demo-2022 .

A top level README.md file can be used to help people find their way.

Part of make init is to setup a kernel (named wsu-pwg-demon-2022 in this case) that can be accessed from the notebook interface. This allows you to use the carefully crafted environment for your project.

Warning

We have transitioned to using MyST Markdown files for all of our notebooks, with automatic synchronization provided by Jupytext. Unfortunately, the collaborative CoCalc does not yet support Jupyext (but the Classic Notebook Server does).

What this means is that, if you edit Jupyter notebooks (*.ipynb files) with the CoCalc interface, or if you update the MyST Markdown (*.md files) from the version control (or directly), then you must synchronize these. This can be done by running:

make sync

in the project directory. This will synchronize all paired *.md files with all paired *.ipynb files.

In order to pair files, I recommend simply copying a file like Demonstration.md, but you can also pair it through the Classic Notebook Server, or you can do it in a terminal:

jupytext --set-formats ipynb,md:myst *.ipynb

Hopefully issue 4715 will be resolved soon. Even more exciting: the CoCalc team is planning on making the Markdown Editor fully features so it will run the documents as notebooks!

Notebooks vs Modules#

Over the years I have come to understand that Notebooks are great for exploring and showing people how to use your code, but lousy for actually developing code. Instead, you should use a full-featured editor to edit code, organize your code into a proper package and modules, and test it, demanding at least high levels of coverage.

Setting up the framework for doing all this is not too hard, but this is the main point of our project. It comes with sample code src/wsu_pwg_demo_2022/example.py and corresponding tests tests/test_example.py that start with 100% coverage:

~$ cd wsu-python-working-group-demo-2022
~/wsu-python-working-group-demo-2022$ make test
...
Required test coverage of 85% reached. Total coverage: 100.00%
===================== 2 passed in 0.27s ======================

Migrate your code from your notebooks to your package, testing it thoroughly as you go along. Once you get your tests passing, make sure to add and commit them to your VCS.

Change your notebooks from a place to explore code into a place to demonstrate how to use your code. Then incorporate these into your documentation so you have a record. Push your code to GitLab, and ensure that the documentation can be bult.

Pick and Learn a Powerful Editor

I generally use Emacs or Vi, but these have steep learning curves. VS Code seems to be a good compromise. Whatever you use, make sure it is powerful enough (and spend the time to learn it well enough) that you can:

  • Perform regexp search and replace over multiple files.

  • Enable automatic syntax checking and code linting. (These tools save hours of time.)