Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .

What's Changed

Below is a list of some major changes that we don't want you to miss.

Exciting New Features ✨

meta

  • add snapshot_id codec support (#8005)

planner

  • support update ast and planner (#7925)

query

  • jsonb parser optimize (#7947)
  • impl externalLocation for create table (#7789)
  • use common_jsonb::compare to compare variants (#8027)

storage

  • accept SESSION_TOKEN for AWS temporary credentials (#7946)

cluster

  • experimental distributed eval index (#7867)

new expression

  • migrate retention to v2 (#7952)
  • support constructing array and CAST(... AS VARIANT) (#7781)

Code Refactor 🎉

settings

  • add prefix "format_" for format related settings (#7960)

new expression

  • reorder comparision funciton priority (#7991)

unit tests

  • use goldenfile in tests on system tables (#7978 & #7982)

Thoughtful Bug Fix 🔧

legacy parser

  • use unicode_segmentation to truncate INSERT statement (#8011)

planner

  • find smallest column for pruning unused columns (#7955 & #7962
  • union needs more than one coercion type (#8007)

processor

  • try fix cannot kill optimize table (#7959)
  • try fix cannot kill drop table (#7963)

storage

  • shrink min max index (#7958)

new expression

  • fix the bug in logic expression or and add test cases (#7966)

News

Let's take a look at what's new at Datafuse Labs & Databend each week.

Better Index in Databend

In the past, Databend used a Bloom Filter (Bitmap Index) to check if a key was exists. Databend has enabled Bloom Index at the block level (#6639) and delivered an 8x read performance improvement in certain scenarios (index / data ~= 10%). Due to the implementation policy, it can take up very large amounts of storage space and has poor performance when it comes to point queries.

Now, Databend is making a number of improvements to enhance the insert and read capabilities of large data sets. Some of this work revolves around the index.

We introduced the Xor Fliter to replace the Bloom Filter (#7870), which in some scenarios gives about twice the performance improvement and requires very little data to be scanned. Recent work has also included distributed index pruning (#7867) and local parallel execution of pruning (especially the index pruning) (#7893) , which we believe will further improve cpu and network utilisation and hence performance.

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors

Thanks a lot to the contributors for their excellent work this week.

andylokandyariesdevilb41shBohuTANGChasen-ZhangClSlaid
andylokandyariesdevilb41shBohuTANGChasen-ZhangClSlaid
dantengskydrmingdrmerjunaireleiyskymergify[bot]PsiACE
dantengskydrmingdrmerjunaireleiyskymergify[bot]PsiACE
RinChanNOWWWsandfleesundy-liXuanwoxudong963youngsofun
RinChanNOWWWsandfleesundy-liXuanwoxudong963youngsofun
zenriazhang2014zhyass
zenriazhang2014zhyass

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.