This week in Databend #34
Databend is an open source elastic and reliable Modern Cloud Data Warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.
Big changes
Below is a list of some major changes that we don't want you to miss.
Features
- databend-query: support Semi-structured variant data type by @b41sh, (#4348)
- databend-query: support stage list & stage streaming upload by @sundy-li, (#4472 & #4477)
- databend-query: integrate fuse table with new processor by @zhang2014, (#4444)
- **databend-query: add support for alter table rename statement ** by @kevinw66, (#4532)
- *: add bootstrap_tenant procedure for tenant management by @Junnplus, (#4530)
- http handler: support server-side-session by @youngsofun, (#4538)
- metactl: dump data from a running metasrv by @light4, (#4473)
- common-functions: support
PARSE_JSON
/TRY_PARSE_JSON
function by @b41sh, (#4534)
Improvement
- common-functions: simplify function creator && adaptor by @sundy-li, (#4485)
- common-management: add escape for management key by @BlackSmith96, (#4363)
- databend-query: refine count optimization by @dantengsky, (#4508)
- *: adapt reqsign and separate IO runtime by @Xuanwo, (#4481)
Performance Improvement
- datavalues: Simd selected for BooleanColumn by @LiuYuHui, (#4484)
- databend-query: enable new processor by default(standalone mode) by @zhang2014, (#4486)
Build/Test/CI
- add musl support & release musl compiled binaries by @Xuanwo & @ZhiHanZ, (#4520 & #4535)
- drop databend-benchmark and old perf tool, add benchmark solution with hyperfine by @PsiACE, (#4545)
Bug fixes
- databend-query: fix groupby single string in new processor by @sundy-li, (#4475)
- clickhouse handler: to_clickhouse_block always convert to full column if constant by @sundy-li, (#4514)
Tips
Let's learn a weekly tip from Databend.
How to Benchmark with Hyperfine
Databend recommends using hyperfine to perform benchmarking via the ClickHouse/MySQL client. With a simple script, we can run benchmark easily:
#!/bin/bash
WARMUP=3
RUN=10
export script="hyperfine -w $WARMUP -r $RUN"
script=""
function run() {
port=$1
sql=$2
result=$3
script="hyperfine -w $WARMUP -r $RUN"
while read SQL; do
n="-n \"$SQL\" "
s="echo \"$SQL\" | mysql -h127.0.0.1 -P$port -uroot -s"
script="$script '$n' '$s'"
done <<< $(cat $sql)
script="$script --export-markdown $result"
echo $script | bash -x
}
run "$1" "$2" "$3"
For details, please read databend.rs - How to Benchmark with Hyperfine
We've also updated some of Databend's performance, if you're interested in that, check out the following articles:
- Databend Vectorized Engine Performance
- Databend On Amazon S3 Performance
- Databend On Wasabi Performance
Changlogs
You can check the changelogs of Databend nightly to learn about our latest developments.
- v0.6.100-nightly
- v0.6.99-nightly
- v0.6.98-nightly
- v0.6.97-nightly
- v0.6.96-nightly
- v0.6.95-nightly
- v0.6.94-nightly
Ecosystem/Upstream
From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.
- reqsign - Signing API requests without effort: minimal dependencies and support AWS & Azure Storage, by @Xuanwo
Meet Us
Please join the DatafuseLabs Community if you are interested in Databend.
We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.
You can submit issues for any problems you find. We also highly appreciate any of your pull requests.