Faster multi-threaded statistics collection Previously, if schema evolution happened with the from_avro connector, the new columns would return null. You can now enable your pipelines to restart when evolved records are detected. from_avro with schema registry connector adds support for schema evolution Stickier cache assignment improves consistency across runs and reduces data moved during rebalancing operations. The algorithm improves disk usage and partition assignment across nodes, with faster assignment both initially and after cluster scaling events. The Spark scheduler now uses a new disk caching algorithm. See Shallow clone for Unity Catalog tables. You can now use shallow clone with Unity Catalog external tables. Shallow clone for Unity Catalog external tables (Public Preview) See Add tables with deletion vectors to a share, Read tables with deletion vectors enabled, and Read tables with deletion vectors enabled. On Databricks Runtime 14.1, they can only perform batch queries. Delta Sharing: Recipients can perform batch, CDF, and streaming queries on shared tables with deletion vectors (Public Preview)ĭelta Sharing recipients can now perform batch, CDF, and streaming queries on shared tables that use deletion vectors. See Write conflicts with row-level concurrency. Row-level concurrency is enabled by default on Delta tables with deletion vectors enabled. Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering. Row-level concurrency reduces conflicts between concurrent write operations by detecting changes at the row-level. Row-level concurrency is Generally Available and on by default See Use foreachBatch to write to arbitrary data sinks and Monitoring Structured Streaming queries on Azure Databricks. You can now use the foreachBatch() and StreamingListener APIs with Structured Streaming in shared clusters. foreachBatch and StreamingListener support You can now use the from_avro, to_avro, from_protobuf, and to_protobuf Python functions with Schema Registry in shared clusters. Supporting Schema Registry for protobuf and Avro related functions in shared clusters When encountering an unreadable file in a table, these commands now fail even if these options are specified. The DML commands DELETE, UPDATE, and MERGE INTO no longer respect the read options ignoreCorruptFiles and ignoreMissingFiles. Support for Scala scalar user-defined functions on shared clusters (Public Preview)įixed corrupt file handling in DML commands.Pushdown filters in the DeltaSource on Delta files.Faster multi-threaded statistics collection.from_avro with schema registry connector adds support for schema evolution.Shallow clone for Unity Catalog external tables (Public Preview).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |