This week in Databend #7

2021-09-15

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don’t want you to miss.

Refactor

rename datafuse to databend by Databend Authors
refactor datasources by @BohuTANG. (#1808, #1813, #1817 etc.)
refactor and add mTLS client auth for http service by @ZhiHanZ (#1771)

Features

add LocalKVStore: a KVApi impl backed with a sled tree by @drmingdrmer. (#1754)
initial work for saving table meta in obj storage by @dantengsky. (#1694)
add Interval support by @junli1026 (#1752)
add toStartOfYear,toStartOfISOYear by @dust1 (#1826)
add toYYYYMMDDhhmmss by @ygf11 (#1767)
add a basic credits table by @PsiACE (#1778)

Bug fixes

fix count function with not null column by @zhang2014. (#1794)
fix insert remote table of parquet format with DateTime types by @sundy-li. (#1765)
fix unary arithmetics function err fix unary arithmetics function err by @zhyass. (#1842)

RFCs

RFC: New SQL planner framework by @leiysky. (#1837

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

cargo-license #49: bump deps by @PsiACE

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Datafuse #6

2021-09-08

Datafuse is an open source elastic and scalable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don’t want you to miss.

remove DataType::Utf8 by @zhyass. (#1606)
support system.engines table by @b41sh (#1755, #1758)
support system.config table by @GrapeBaBa (#1738)
refactor Aggregator for improve performance and prepare refactor transform_aggregator_final by @zhang2014. (#1701)
support user namespace API by @BohuTANG. (#1727)
add timezone support for DateTime32 by @sundy-li. (#1750)
refactor all warp services to axum by @ZhiHanZ. (#1702) & @PsiACE (#1716, #1753)
streaming put for DAL by @dantengsky. (#1707)
support substring(string, start, length) by @tangwz. (#1710 & datafuse-extras/sqlparser-rs#6)
add date functions
- yesterday/tomorrow/timeSlot() by @dust1 (#1673, #1715)
- toYYYYMM by @ygf11 (#1726, #1742)

Changelogs

You can check the changelogs of Datafuse nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. The Datafuse team is also committed to contributing to the Rust ecosystem and upstream dependencies.

arrow2 #346: Added support for compute to BinaryArray by @zhyass

Meet Us

Please join the Datafuse Community if you are interested in Datafuse.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code. You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Datafuse #5

2021-09-01

Datafuse is an open source elastic and scalable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don’t want you to miss.

custom logic types on arrow2 by @sundy-li. (#1648)
add a basic lru memory/disk cache by @PsiACE (#1600).
use custom HashMap for HashMethodKind::Serializer by @zhang2014. (#1500)
a series of refactors on catalog by @BohuTANG. (#1642, #1643, #1644)
cross compile for arm-based architecture by @ZhiHanZ. (#1689)
refactor some warp services to axum by @ZhiHanZ. (#1654)

Changelogs

You can check the changelogs of Datafuse nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. The Datafuse team is also committed to contributing to the Rust ecosystem and upstream dependencies.

arrow2 #344: Bumped tonic and prost in flight by @PsiACE

Thanks

Thanks to your contribution, datafuse becomes better!

@jyz0309: Add v1/logs http api (#1657)
@b41sh: support aggregate function STDDEV_POP() (#1671)
@ygf11:support where option in show databases (#1510) and add doc for SHOW DATABASES WHERE/LIKE (#1615)
@cppcoffee: use arrow2 bitmap to improve memory usage (#1638)
@GrapeBaBa: improve error msg for address in use (#1645) and fix mysqlhandler log bug (#1646), etc.

Meet Us

Please join the Datafuse Community if you are interested in Datafuse.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code. You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Datafuse #4

2021-08-25

Datafuse is an open source elastic and scalable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don’t want you to miss.

IDX-1428: common indexing by @BohuTANG. (#1437)
add windowFunnel function by @sundy-li. (#1519)
add if function by @zhyass. (#1505)
refactor SQL parser (WIP) by @leiysky. (#1477 & #1523 & #1505)
Object Store Data Access Layer by @dantengsky. (#1442)
making distributed systems more robust by @drmingdrmer. (#1569 #1546 etc.)

Changelogs

You can check the changelogs of Datafuse nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. The Datafuse team is also committed to contributing to the Rust ecosystem and upstream dependencies.

sqlparser-rs #312: Add fuzzer based on honggfuzz: Generate some random bytes for the fuzz test. by @PsiACE
hashbrown #261: Upgrade rand to 0.8.3: Use seed_from_u64 instead of from_seed to avoid cross-platform issues. by @PsiACE

Thanks

Thanks to your contribution, datafuse becomes better!

@jyz0309:Add parser used time metric (#1586) and add optimize histogram metric (#1524)
@mapleFU: Support SHOW TABLES LIKE pattern and Where expr (#1518) and add SHOW TABLES LIKE pattern and Where expr to doc (#1541)
@dust1: Add Support SHOW TABLES FROM statament (#1555)

Meet Us

Please join the Datafuse Community if you are interested in Datafuse.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code. You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Datafuse #3

2021-08-18

Datafuse is an open source elastic and scalable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don’t want you to miss.

add ValuesSource and CsvSource by @sundy-li. (#1484)
introducing a custom HashMap to try to improve group by performance by @zhang2014. (#1443)
cli: change to clap from structopt by @ZhiHanZ. ([#1491]https://github.com/datafuselabs/datafuse/pull/1491))
implement Contains operation by @jyizheng. (#1379)
a series of new features make the storage layer better by @drmingdrmer. (#1461 #1479 #1439 etc.)
fix panic while reading data parts by @dantengsky. (#1489)
add datafuse design whitepaper to website by @BohuTANG. (#1490 & #1493)

Changelogs

You can check the changelogs of Datafuse nightly to learn about our latest developments.

Thanks

Thanks to your contribution, datafuse becomes better!

@junli1026:Fix read_opt_scalar and add test (#1446) and add test for array builder (#1468)
@yjhmelody: Fix variadic_arguments of LogicFunction (#1486)
@koushiro: Use #[from] attribute of thiserror::Error proc macro, and remove some boilerplate code (#1507)
@jackwener: Send slack notification when performance tests standalone start, fail or success (#1450)

Meet Us

Please join the Datafuse Community if you are interested in Datafuse.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code. You can submit issues for any problems you find. We also highly appreciate any of your pull requests.