Databend Weekly

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

databend-meta: add rename_table for meta api: remove old table record and add a new one in a sled-tree-transaction. by @Junnplus, (#4288)
databend-query: support COPY from external location(S3): load files from a storage location (Amazon S3). by @BohuTANG, (#4170 & #4241)
bendctl: add package purge subcommand by @linyihai, (#4245)

Improvement

databend-query: use database_id/table_id as data file prefix: no more having all parquet files in one big directory. by @dantengsky, (#4273)
databend-query: support aggregate sum/avg booleans by @everpcpc, (#4237)
databend-query: zero extra cost of async trait: use GAT feature instead of #[async_trait] in some hot path. by @sundy-li, (#4269)
databend-query: implement new processor for system tables & github & memory & null engine by @zhang2014, (#4166 & #4272)

Build/Test/CI

With change to wiremock from httpmock and a series of other PRs, @PsiACE removed ~60 dependencies for clippy and test.

Bug fixes

dal_context: use ObserveReader to calculate metrics: calculate the read cost time correctly. by @Xuanwo, (#4298)

Tips

Let's learn a weekly tip from Databend.

Loading files from S3 External Location

Benefit from #4170 & #4241, Databend now has the ability to load CSV format files from the s3 external location.

Here's an example of reading 5 lines from a file:

COPY INTO ontime FROM 's3://databend-external/t_ontime/t_ontime.csv'
    CREDENTIALS=(aws_key_id='<your-key-id>' aws_secret_key='<your-secret-key>')
    FILE_FORMAT = (type = "CSV" field_delimiter = '\t'  record_delimiter = '\n' skip_header = 1)
    SIZE_LIMIT=5; /* only read 5 rows */

/* Check. */
SELECT * FROM ontime;

Note To learn more, check out:

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

Upstream

arrow2 #871: Improved performance of filter performance via Simd selection [3x] by @sundy-li.

OpenDAL announced v0.1

We announced that Open Data Access Layer is in v0.1. Let's look at what's different together.

Observe reader support (opendal#92)
Object Stream (opendal#69)
List support (opendal#76)
Auto Region (opendal#59)

Welcome to use OpenDAL(github, crates.io) to connect your data and applications.

Big changes are happening at OpenRaft

Although no new releases have been made recently, we have seen some important changes with @schreter's help.

Refactor storage APIs to allow more clear ownership of data. (openraft#199)
Make NodeId type configurable via RaftTypeConfig. (openraft#220)

Feel free to visit Insights/Pulse to observe all changes, and it would be great if you would like to try it out in advance.

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Databend #31