Databend Weekly

Databend is an open source elastic and reliable Modern Cloud Data Warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

RFC

recluster (#6905)

meta

add raft store fail metrics (#6927)
metasrv unittest logs tracing event with customized formatter (#6874)

storage

enable bloom filter index (#6639)
support query hive partition table (#6906)

RBAC

add auth role to jwt (#6829)

format

pass FileSplit instead of Vec (#6873)

new expression

make chunk support scalar values (#6918)
migrate quote, reverse and ascii (#6907)
migrate trim functions to new expression framework (#6921)

Improvement

Dedicate See you again to the old planner (#6895)
Remove unused reload config (#6933)

new expression

add NullableColumn and NullableColumnBuilder (#6867)
use Scalar to store constant in Expr (#6923)

Build/Testing/CI

migrate all stateless test to logic test (progress of migration)
add mini ontime dataset to stateful test (#6931)

Bug fixes

don't expand null scalar to column (#6834)
fix mistake using try_cast for cast (#6879)
fix session drop early in clickhouse handler (#6888)
fix binder create table (#6899)
fix mysql return 'Empty Set' when result set is empty (#6841)
fix case expr with case operator equal (#6950)
fix cannot kill query in cluster mod (#6954)

Tips

Let's learn a weekly tip from Databend.

Call for Migrating Functions to the New Expression Framework

If you are interested in typed type system, or maybe you'd like to try your hand at a database project, take a look at how Databend does it.

We are now trying to migrate some old functions to the new expression framework, would you like to try it out?

Background

Recently Databend is working on a new expression framework that will bring some interesting features.

Type checking.
Type-safe downcast.
Enum-dispatched columns.
Generic types.

How To

Legacy functions are settle in common/functions/src/scalars. The task is to migrate all of them to common/functions-v2/src/scalars/.

Usually you can reuse the logic of the previous implementation, it just needs some rewriting to make it fit the new way.

Similarly, the legacy tests in common/functions/tests/it/scalars/ should also be migrated to common/functions-v2/tests/it/scalars/.

The new tests will be written using goldenfile, so you can easily generate test cases without a lot of painful handwriting.

Example

A unary function OCTET_LENGTH can be defined using 6 lines incommon/functions-v2/src/scalars/strings.rs.

OCTET_LENGTH will return the length of a string in bytes.

registry.register_1_arg::<StringType, NumberType<u64>, _, _>(
    "octet_length",
    FunctionProperty::default(),
    |_| None,
    |val| val.len() as u64,
);

LENGTH is a synonym for OCTET_LENGTH.

We can easily define function aliases with one line.

registry.register_aliases("octet_length", &["length"]);

Next, let's write some tests to make sure it works correctly.

Edit common/functions-v2/tests/it/scalars/string.rs.

fn test_octet_length(file: &mut impl Write) {
    run_ast(file, "octet_length('latin')", &[]);
    run_ast(file, "octet_length(NULL)", &[]);
    run_ast(file, "length(a)", &[(
        "a",
        DataType::String,
        build_string_column(&["latin", "кириллица", "кириллица and latin"]),
    )]);
}

#[test]
fn test_string() {
    let mut mint = Mint::new("tests/it/scalars/testdata");
    let file = &mut mint.new_goldenfile("string.txt").unwrap();

    ...
    test_octet_length(file);
    ...
}

Next, let's try to generate these test cases from the command line.

REGENERATE_GOLDENFILES=1 cargo test -p common-functions-v2 --test it

Well done, we did it.

Learn More

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors

Thanks a lot to the contributors for their excellent work this week.


andylokandy	ariesdevil	b41sh	BohuTANG	dantengsky	drmingdrmer


flaneur2020	gaoxinge	leiysky	lichuang	mergify[bot]	PsiACE


RinChanNOWWW	sandflee	soyeric128	sundy-li	TCeason	Xuanwo


xudong963	ygf11	youngsofun	ZeaLoVe	zhang2014	zhyass

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Databend #53