Databend is an open source elastic and reliable Modern Cloud Data Warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

RFC

meta

  • add raft store fail metrics (#6927)
  • metasrv unittest logs tracing event with customized formatter (#6874)

storage

  • enable bloom filter index (#6639)
  • support query hive partition table (#6906)

RBAC

  • add auth role to jwt (#6829)

format

  • pass FileSplit instead of Vec (#6873)

new expression

  • make chunk support scalar values (#6918)
  • migrate quote, reverse and ascii (#6907)
  • migrate trim functions to new expression framework (#6921)

Improvement

  • Dedicate See you again to the old planner (#6895)
  • Remove unused reload config (#6933)

new expression

  • add NullableColumn and NullableColumnBuilder (#6867)
  • use Scalar to store constant in Expr (#6923)

Build/Testing/CI

Bug fixes

  • don't expand null scalar to column (#6834)
  • fix mistake using try_cast for cast (#6879)
  • fix session drop early in clickhouse handler (#6888)
  • fix binder create table (#6899)
  • fix mysql return 'Empty Set' when result set is empty (#6841)
  • fix case expr with case operator equal (#6950)
  • fix cannot kill query in cluster mod (#6954)

Tips

Let's learn a weekly tip from Databend.

Call for Migrating Functions to the New Expression Framework

If you are interested in typed type system, or maybe you'd like to try your hand at a database project, take a look at how Databend does it.

We are now trying to migrate some old functions to the new expression framework, would you like to try it out?

Background

Recently Databend is working on a new expression framework that will bring some interesting features.

  • Type checking.
  • Type-safe downcast.
  • Enum-dispatched columns.
  • Generic types.

How To

Legacy functions are settle in common/functions/src/scalars. The task is to migrate all of them to common/functions-v2/src/scalars/.

Usually you can reuse the logic of the previous implementation, it just needs some rewriting to make it fit the new way.

Similarly, the legacy tests in common/functions/tests/it/scalars/ should also be migrated to common/functions-v2/tests/it/scalars/.

The new tests will be written using goldenfile, so you can easily generate test cases without a lot of painful handwriting.

Example

A unary function OCTET_LENGTH can be defined using 6 lines incommon/functions-v2/src/scalars/strings.rs.

OCTET_LENGTH will return the length of a string in bytes.

registry.register_1_arg::<StringType, NumberType<u64>, _, _>(
    "octet_length",
    FunctionProperty::default(),
    |_| None,
    |val| val.len() as u64,
);

LENGTH is a synonym for OCTET_LENGTH.

We can easily define function aliases with one line.

registry.register_aliases("octet_length", &["length"]);

Next, let's write some tests to make sure it works correctly.

Edit common/functions-v2/tests/it/scalars/string.rs.

fn test_octet_length(file: &mut impl Write) {
    run_ast(file, "octet_length('latin')", &[]);
    run_ast(file, "octet_length(NULL)", &[]);
    run_ast(file, "length(a)", &[(
        "a",
        DataType::String,
        build_string_column(&["latin", "кириллица", "кириллица and latin"]),
    )]);
}

Register it in the test_string function,

#[test]
fn test_string() {
    let mut mint = Mint::new("tests/it/scalars/testdata");
    let file = &mut mint.new_goldenfile("string.txt").unwrap();

    ...
    test_octet_length(file);
    ...
}

Next, let's try to generate these test cases from the command line.

REGENERATE_GOLDENFILES=1 cargo test -p common-functions-v2 --test it

Well done, we did it.

Learn More

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors

Thanks a lot to the contributors for their excellent work this week.

andylokandyariesdevilb41shBohuTANGdantengskydrmingdrmer
andylokandyariesdevilb41shBohuTANGdantengskydrmingdrmer
flaneur2020gaoxingeleiyskylichuangmergify[bot]PsiACE
flaneur2020gaoxingeleiyskylichuangmergify[bot]PsiACE
RinChanNOWWWsandfleesoyeric128sundy-liTCeasonXuanwo
RinChanNOWWWsandfleesoyeric128sundy-liTCeasonXuanwo
xudong963ygf11youngsofunZeaLoVezhang2014zhyass
xudong963ygf11youngsofunZeaLoVezhang2014zhyass

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.