This week in Databend #62
Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .
Below is a list of some major changes that we don't want you to miss.
Exciting New Features ✨
- add snapshot_id codec support (#8005)
- support update ast and planner (#7925)
- jsonb parser optimize (#7947)
- impl externalLocation for create table (#7789)
common_jsonb::compareto compare variants (#8027)
SESSION_TOKENfor AWS temporary credentials (#7946)
- experimental distributed eval index (#7867)
Code Refactor 🎉
- add prefix "format_" for format related settings (#7960)
- reorder comparison function priority (#7991)
Thoughtful Bug Fix 🔧
- use unicode_segmentation to truncate
- find smallest column for pruning unused columns (#7955 & #7962
- union needs more than one coercion type (#8007)
- shrink min max index (#7958)
- fix the bug in logic expression
orand add test cases (#7966)
Let's take a look at what's new at Datafuse Labs & Databend each week.
Better Index in Databend
In the past, Databend used a Bloom Filter (Bitmap Index) to check if a key was exists. Databend has enabled Bloom Index at the block level (#6639) and delivered an 8x read performance improvement in certain scenarios (
index / data ~= 10%). Due to the implementation policy, it can take up very large amounts of storage space and has poor performance when it comes to point queries.
Now, Databend is making a number of improvements to enhance the insert and read capabilities of large data sets. Some of this work revolves around the index.
We introduced the Xor Fliter to replace the Bloom Filter (#7870), which in some scenarios gives about twice the performance improvement and requires very little data to be scanned. Recent work has also included distributed index pruning (#7867) and local parallel execution of pruning (especially the index pruning) (#7893) , which we believe will further improve cpu and network utilisation and hence performance.
You can check the changelogs of Databend nightly to learn about our latest developments.
Thanks a lot to the contributors for their excellent work this week.
Please join the DatafuseLabs Community if you are interested in Databend.
We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.
You can submit issues for any problems you find. We also highly appreciate any of your pull requests.
- Databend Website
- Weekly (A weekly newsletter about Databend)
- GitHub Discussions (Feature/Bug reports, Contributions)
- Twitter (Get the news fast)
- Slack Channel (For live discussion with the Community)
- I'm feeling lucky (Pick up a good first issue now!)