This week in Databend #68
Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .
What's Changed
Below is a list of some major changes that we don't want you to miss.
Exciting New Features ✨
metrics
- add metrics for query detail (#8800)
multiple catalog
- multiple catalog config (#8743)
query
- adjust max io requests when read block data to avoid oom (#8726)
- change
PrecommitBlock
serde fromserde_json
tobincode
(#8726) - support parallel final aggregator, 4X faster! (#8577)
- parallel merge for distribute query (#8811)
storage
new expression
- add calc domain for comparison (#8754)
Code Refactor 🎉
io
- replace
NestedCheckpointReader
withCursor
(#8716)
handler
- use
FieldEncoder
to encode data (#8733)
format
- refactor with
FieldEncoder
(#8778)
query
- unified hashtable interface (#8681)
storage
- compact block (#8679)
new expression
Thoughtful Bug Fix 🔧
handler
- correct databend types to mysql types (#8745)
functions
l_col like r_col
will generate a hashmap based onr_col
, ifr_col
is huge, it will be oom kill (#8737)
News
Let's take a look at what's new at Datafuse Labs & Databend each week.
Shuffle Policy for Partitions
For cache affinity, we consider some strategies when re-shuffle partitions in plan_fragemnt.rs::redistribute_source_fragment
, default kind is Seq
.
pub enum PartitionsShuffleKind {
// Bind the Partition to executor one by one with order.
Seq,
// Bind the Partition to executor by partition.hash()%executor_nums order.
Mod,
// Bind the Partition to executor by partition.rand() order.
Rand,
}
Learn More
Databend x Rust China Hackathon 2022
The first Rust China Hackathon Online is here! The theme of this year's Hackathon is Rust for Fun and we look forward to working with you to unleash the possibilities of innovation with Rust.
As a co-organiser of this year's hackathon, Databend is sponsoring an enterprise track where participants can explore the appeal of cloud-native data warehouses by creating work around Databend components or the Databend ecosystem.
Learn More
Issues
Meet issues you may be interested in and try to solve it.
Switch to nextest in CI
cargo-nextest
is a next-generation test runner for Rust projects.
We have noticed that it is very much faster than cargo test
on many projects. However, there are still some challenges in applying this to Databend. For example, for different types of tests, the number of threads has to be adjusted to ensure that the tests are performed quickly and correctly. Also, some of the tests may need to be rewritten to get better results.
Issue 4376: switch to nexttest in ci
If you find it interesting, try to solve it or participate in discussions and PR reviews. Or you can click on https://link.databend.rs/i-m-feeling-lucky to pick up a good first issue, good luck!
Changelogs
You can check the changelogs of Databend nightly to learn about our latest developments.
- v0.8.115-nightly
- v0.8.114-nightly
- v0.8.113-nightly
- v0.8.112-nightly
- v0.8.111-nightly
- v0.8.110-nightly
- v0.8.109-nightly
- v0.8.108-nightly
- v0.8.107-nightly
- v0.8.106-nightly
Contributors
Thanks a lot to the contributors for their excellent work this week.
andylokandy | BohuTANG | ClSlaid | dantengsky | dependabot[bot] | everpcpc |
lichuang | mergify[bot] | RinChanNOWWW | sandflee | soyeric128 | sundy-li |
TCeason | wubx | Xuanwo | xudong963 | youngsofun | zhang2014 |
ZhiHanZ | zhyass |
Meet Us
Please join the DatafuseLabs Community if you are interested in Databend.
We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.
You can submit issues for any problems you find. We also highly appreciate any of your pull requests.
- Databend Website
- Weekly (A weekly newsletter about Databend)
- GitHub Discussions (Feature/Bug reports, Contributions)
- Twitter (Get the news fast)
- Slack Channel (For live discussion with the Community)