https://docs.databricks.com/en/dev-tools/databricks-connect/python/install.html 1. Install the correct Python (minor) version as found on your Databricks cluster. 2. Set up a Cluster-specific venv with that version. 3. **Uninstall pyspark from the venv if you already installed it, to make sure the right pyspark gets installed.** 4. Install databricks-connect with the version matching you Cluster's version into the venv. 1. `pip3 install "databricks-connect==14.3.*"` 2. `poetry add databricks-connect@~14.3` 5. Verify that your local config points at the right instance so you load the right `DEFAULT` settings from `~/.databrickscfg` by running `databricks-connect test` 6. Start up a notebook and set up your `spark` context: ```python from databricks.connect import DatabricksSession spark = DatabricksSession.builder.remote().getOrCreate() ``` ## `.databrickscfg` settings ``` [DEFAULT] host = https://$WORKSPACE_ID.azuredatabricks.net cluster_id = $CLUSTER_ID token = $SECRET_TOKEN ``` (No quotes needed.)