Databend Weekly

Databend is an open source elastic and reliable Modern Cloud Data Warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

support correlated subquery (#5593)
support select ... at (#5617 & #5628)
exchange protocol version with client (#5645)
add decompress support for COPY INTO and streaming loading (#5655)

new parser

implement string unescape (#5638)
allow mysql-style hex number and single-item array (#5654)

stage related

copy into stage support (#5579)
add system.stages table and show stages (#5581)

call functions

add call stats functions (#5646)
add call function system$search_tables (#5643)

new planner

support explain for new planner (#5587)
support tuple in new planner (#5640)
support cross join (#5715)

Variant related

support variant group by (#5694)
support variant order by (#5668)

Improvement

refine table optimize (#5589)
parse integer to u64 (#5692)
support FixedKey u128, u256, u512 in group query (#5678)
introduce ScalarEvaluator to evaluate Scalar (#5689)
utilize with HashMethodFixedKeys in hash join (#5693)
record count of tables for a tenant in KV space (#5708)

Build / Testing / CI

add mock module, add test of out retention time data (#5707)

Bug fixes

retry while meeting error during load_credential (#5590)
deny the root login from others host (#5588)

Tips

Let's learn a weekly tip from Databend.

`COPY INTO` Databend with `COMPRESSION` Option

After PR 5655, Databend started to support decompression reads in COPY INTO and streaming leading.

Loading Compressed Files from Amazon S3

Try to load data from a gzip compressed csv and insert into mytable.

COPY INTO mytable
  FROM s3://mybucket/data.csv.gz
  credentials=(aws_key_id='<AWS_ACCESS_KEY_ID>' aws_secret_key='<AWS_SECRET_ACCESS_KEY>')
  FILE_FORMAT = (type = "CSV" field_delimiter = ',' record_delimiter = '\n' 
  skip_header = 1 compression = GZIP) size_limit=10;

COMPRESSION Option

The COMPRESSION option is a string that represents the compression algorithm.

| Values        | Notes                                                           |
| ------------- | --------------------------------------------------------------- |
| `AUTO`        | Auto detect compression via file extensions                     |
| `GZIP`        |                                                                 |
| `BZ2`         |                                                                 |
| `BROTLI`      | Must be specified if loading/unloading Brotli-compressed files. |
| `ZSTD`        | Zstandard v0.8 (and higher) is supported.                       |
| `DEFLATE`     | Deflate-compressed files (with zlib header, RFC1950).           |
| `RAW_DEFLATE` | Deflate-compressed files (without any header, RFC1951).         |
| `NONE`        | Indicates that the files have not been compressed.              |

Learn more:

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors

Thanks a lot to the contributors for their excellent work this week.


andylokandy	ariesdevil	b41sh	BohuTANG	Chasen-Zhang	dantengsky


devillove084	drmingdrmer	everpcpc	flaneur2020	hanyisong	junnplus


leiysky	lichuang	mergify[bot]	PsiACE	RinChanNOWWW	ryanrussell


soyeric128	sundy-li	TCeason	Xuanwo	xudong963	youngsofun


ZeaLoVe	zhyass

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Databend #44