- Add base classes for ML and refine code base (#1031, #1049)
- Capture dotnet application error stack trace (#1047)
- JvmBridge semaphore fix (#1061)
- None
- Switch to 1ES hosted pools on main (#1021)
- None
- Broadcast variables do not work with dotnet-interactive (#561)
- UDFs defined using class objects with closures does not work with dotnet-interactive (#619)
- In dotnet-interactive blocking Spark methods that require external threads to unblock them does not work. ie
StreamingQuery.AwaitTerminationrequiresStreamingQuery.Stopto unblock (#736)
The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker.
| Oldest compatible Microsoft.Spark.Worker version |
|---|
| v2.0.0 |
The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with.
| Oldest compatible .NET for Apache Spark release version |
|---|
| v2.0.0 |
The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:
| Spark Version | microsoft-spark JAR |
|---|---|
| 2.4.0 | microsoft-spark-2-4_2.11-2.1.1.jar |
| 2.4.1 | |
| 2.4.3 | |
| 2.4.4 | |
| 2.4.5 | |
| 2.4.6 | |
| 2.4.7 | |
| 2.4.2 | Not supported |
| 3.0.0 | microsoft-spark-3-0_2.12-2.1.1.jar |
| 3.0.1 | |
| 3.0.2 | |
| 3.1.1 | microsoft-spark-3-1_2.12-2.1.1.jar |
| 3.1.2 | |
| 3.2.0 | microsoft-spark-3-2_2.12-2.1.1.jar |
| 3.2.1 |
The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with:
| Delta Version | Microsoft.Spark.Extensions.Delta |
|---|---|
| 0.1.0 | 2.1.1 |
| 0.2.0 | |
| 0.3.0 | |
| 0.4.0 | |
| 0.5.0 | |
| 0.6.0 | |
| 0.6.1 | |
| 0.7.0 | |
| 0.8.0 | |
| 1.0.0 | |
| 1.1.0 |
The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with:
| Hyperspace Version | Microsoft.Spark.Extensions.Hyperspace |
|---|---|
| 0.1.0 | 2.1.1 |
| 0.2.0 | |
| 0.3.0 | |
| 0.4.0 |