Rhino WIP: On Python Environments

TLDR

In the next build of Rhino WIP,

• Python environments are now set up using python -m venv module and therefore have their own python and pip binaries. This change should NOT affect how your scripts are run. You should see improvement in package install and dependency handling.

• Rhino can maintain different named environments for executing python scripts

• Only one named environment can be used for a Rhino session. You will need to restart Rhino if you want to switch to a different environment. Multiple environments for a single session of Rhino is NOT supported (If you like to know more read below)

• Advanced → Open Python Shell menu now opens a shell that is executing in the active named environment and provides better access. You can run python, pip list, pip install, pip uninstall and other pip commands more easily without having to provide --target argument.

For the curious, here is a detailed explanation of why only one named environment can be used in Rhino.

Environments in Rhino

Two packages with different versions (e.g. numpy v1, and numpy v2) can not be loaded in the same running python process. It’s just not designed that way.

Virtual environments (commonly known as venv) are a solution to this problem. They keep conflicting packages installed in separate folders. However, they still do not load in the same running python process. So each environment ends up having its own python. That is why you need to activate the environment before using it. This ensures your shell finds the python inside the environment first and not the one installed globally on your system.

Separate environments in Rhino (using # venv) allows installing groups of packages in separate containers for scripts. Imagine having scripts for completely separate projects each with their own requirements; one might need torch for an AI model and others might need a different combination of numpy, scipy, matplotlib, etc for another task.

This, however, still does NOT allow loading two environments at the same time. Here I attempt to explain why pure virtual environments are close to impossible at the moment, in the hope of providing a bit more clarity and making this post a reference for future communications.

Let’s assume this slide represents on instance of Rhino, running on your machine with a document open. When ScriptEditor is loaded, we load a fork of Python.Net library that provides access to Python 3 to/from dotnet environment that Rhino is running on:

Python.Net in turn loads Python 3 (libpython3.*) inside of Rhino. This is Python 3!

Python 3 immediately creates the main interpretter that runs your python code. It stores all your loaded modules into sys.modules and provides globals and all the other python features. Python.Net also sets up clr modules and other necessary types for dotnet interoperability:

Here is where we set up the default environment that you can install your packages into:

Hopefully this visual makes it clear why another environment loaded in the same process can cause package conflicts since multiple versions of the same package are loaded in the same python and as we mentioned before, python is not designed for this:

Some packages (like numpy) have their own internal native libraries that get loaded and can cause conflicts as well:

Sub-Interpreters

One possible solution that we have explored is creating what is called a “Sub-Interpreter” inside of Python 3. Sub-interpreters are almost fully independent of the main interpreter. In theory we can load a completely separate set of packages in that interpreter without conflicting with the main:

HOWEVER,

Neither Python.Net nor numpy support sub-interpreters at the moment which makes them largely unusable until these issues are fixed:

• Python.Net - Support for creating python sub-interpreters
• numpy - Please support subinterpreters

We can take a look at this once the issues above are resolved.

3 posts - 1 participant

Read full topic