Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .

What's Changed

Below is a list of some major changes that we don't want you to miss.

Exciting New Features ✨

metrics

  • add metrics for query detail (#8800)

multiple catalog

  • multiple catalog config (#8743)

query

  • adjust max io requests when read block data to avoid oom (#8726)
  • change PrecommitBlock serde from serde_json to bincode (#8726)
  • support parallel final aggregator, 4X faster! (#8577)
  • parallel merge for distribute query (#8811)

storage

  • shuffle segments during distributed pruning (#8793)
  • add shuffle policy for Partitions (#8814)

new expression

  • add calc domain for comparison (#8754)

Code Refactor 🎉

io

  • replace NestedCheckpointReader with Cursor (#8716)

handler

  • use FieldEncoder to encode data (#8733)

format

  • refactor with FieldEncoder (#8778)

query

  • unified hashtable interface (#8681)

storage

new expression

  • refine domain (#8755)
  • allow sparse column id in chunk (#8789)

Thoughtful Bug Fix 🔧

handler

  • correct databend types to mysql types (#8745)

functions

  • l_col like r_col will generate a hashmap based on r_col, if r_col is huge, it will be oom kill (#8737)

News

Let's take a look at what's new at Datafuse Labs & Databend each week.

Shuffle Policy for Partitions

For cache affinity, we consider some strategies when re-shuffle partitions in plan_fragemnt.rs::redistribute_source_fragment, default kind is Seq.

pub enum PartitionsShuffleKind {
    // Bind the Partition to executor one by one with order.
    Seq,
    // Bind the Partition to executor by partition.hash()%executor_nums order.
    Mod,
    // Bind the Partition to executor by partition.rand() order.
    Rand,
}

Learn More

Databend x Rust China Hackathon 2022

The first Rust China Hackathon Online is here! The theme of this year's Hackathon is Rust for Fun and we look forward to working with you to unleash the possibilities of innovation with Rust.

As a co-organiser of this year's hackathon, Databend is sponsoring an enterprise track where participants can explore the appeal of cloud-native data warehouses by creating work around Databend components or the Databend ecosystem.

Learn More

Issues

Meet issues you may be interested in and try to solve it.

Switch to nextest in CI

cargo-nextest is a next-generation test runner for Rust projects.

We have noticed that it is very much faster than cargo test on many projects. However, there are still some challenges in applying this to Databend. For example, for different types of tests, the number of threads has to be adjusted to ensure that the tests are performed quickly and correctly. Also, some of the tests may need to be rewritten to get better results.

Issue 4376: switch to nexttest in ci

If you find it interesting, try to solve it or participate in discussions and PR reviews. Or you can click on https://link.databend.rs/i-m-feeling-lucky to pick up a good first issue, good luck!

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors

Thanks a lot to the contributors for their excellent work this week.

andylokandyBohuTANGClSlaiddantengskydependabot[bot]everpcpc
andylokandyBohuTANGClSlaiddantengskydependabot[bot]everpcpc
lichuangmergify[bot]RinChanNOWWWsandfleesoyeric128sundy-li
lichuangmergify[bot]RinChanNOWWWsandfleesoyeric128sundy-li
TCeasonwubxXuanwoxudong963youngsofunzhang2014
TCeasonwubxXuanwoxudong963youngsofunzhang2014
ZhiHanZzhyass
ZhiHanZzhyass

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.