Generating New Build#

Author: Erik Inqwersen
Date: April 09, 2024

In this tutorial, we’ll walk through the process of generating a new build of the wip package and uploading it to Data Bricks. This process is essential for adding to Data Bricks the latest version of the wip package that contains new features or fixes.

Prerequisites#

Before proceeding, ensure that you have the following installed and configured:

Python 3.9 or newer
Poetry for package management
Access to the Data Bricks cluster with the necessary permissions to upload packages

Updating the Package Version#

First, decide on the type of version update you want to apply:

patch: new package version contains backward-compatible bug fixes.
minor: new package version contains backward-compatible functionalities.
major: new package version contains changes that make the new version incompatible with the old version.

Note

Understanding “Backward-Compatible”

When a new version of a package is described as “backward-compatible”, it means that the update introduces new features, improvements, or bug fixes that do not interfere with the package’s existing functionality. Applications or dependencies that rely on the previous version of the package should continue to work seamlessly with the new version without requiring any changes.

Open a terminal and navigate to your project directory.
Use the command below to update your package version accordingly:
```
poetry version [patch | minor | major]
```
Replace [patch | minor | major] with your chosen version type. For example, poetry version minor will increment the minor version number.

Important

The last command will update the version number inside pyproject.toml. However, the command poetry version [patch | minor | major] does not automatically update the __version__ attribute from src/wip/__init__.py. Before continuing to the next step, make sure you’ve replaced the __version__ attribute value from __init__.py to match the new version that was generated.

Generating a New Package Build#

After updating the version, the next step is to generate a build of your package.

Still in your project directory, execute the following command to build your package:
poetry build
This command generates a new build of your package. By default, it creates both a source archive (tar.gz) and a wheel file (.whl) in the ./dist directory.

Uploading to Data Bricks Cluster Packages#

To upload your newly generated .whl file to the Data Bricks cluster, follow these steps:

Locate the .whl file inside the ./dist directory of your project.
Log in to the Data Bricks workspace.
On the left-side pane, click on the "Compute" option and then select the Process_Optimization_Pelletizing cluster to navigate to the cluster’s details page.
Click on the Libraries tab and then on the Install New button, located on the top-right corner of the screen.
Select DBFS as the library source, and then Python Whl as the Library Type.
Upload the .whl file from your ./dist directory.

Conclusion#

After the new version of the wip package finishes installing on the cluster, you can verify that Data Bricks is using the new version of the package by creating a new notebook and executing the following Python code:

import wip
print(wip.__version__)

The above code should print the new version of wip package you’ve just installed.

Warning

If a notebook started running before the new version of wip was installed to the cluster, remember to Detach and Reattach Cluster” for the wip package changes to take effect.