Databend is an open source elastic and reliable Modern Cloud Data Warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

  • support correlated subquery (#5593)
  • support select ... at (#5617 & #5628)
  • exchange protocol version with client (#5645)
  • add decompress support for COPY INTO and streaming loading (#5655)

new parser

  • implment string unescape (#5638)
  • allow mysql-style hex number and single-item array (#5654)

stage related

  • copy into stage support (#5579)
  • add system.stages table and show stages (#5581)

call functions

  • add call stats functions (#5646)
  • add call function system$search_tables (#5643)

new planner

  • support explain for new planner (#5587)
  • support tuple in new planner (#5640)
  • support cross join (#5715)

Variant related

  • support variant group by (#5694)
  • support variant order by (#5668)

Improvement

  • refine table optimize (#5589)
  • parse integer to u64 (#5692)
  • support FixedKey u128, u256, u512 in group query (#5678)
  • introduce ScalarEvaluator to evaluate Scalar (#5689)
  • utilize with HashMethodFixedKeys in hash join (#5693)
  • record count of tables for a tenant in KV space (#5708)

Build / Testing / CI

  • add mock module, add test of out retention time data (#5707)

Bug fixes

  • retry while meeting error during load_credential (#5590)
  • deny the root login from others host (#5588)

Tips

Let's learn a weekly tip from Databend.

COPY INTO Databend with COMPRESSION Option

After PR 5655, Databend started to support decompression reads in COPY INTO and streaming leading.

Loading Compressed Files from Amazon S3

Try to load data from a gzip compressed csv and insert into mytable.

COPY INTO mytable
  FROM s3://mybucket/data.csv.gz
  credentials=(aws_key_id='<AWS_ACCESS_KEY_ID>' aws_secret_key='<AWS_SECRET_ACCESS_KEY>')
  FILE_FORMAT = (type = "CSV" field_delimiter = ',' record_delimiter = '\n' 
  skip_header = 1 compression = GZIP) size_limit=10;

COMPRESSION Option

The COMPRESSION option is a string that represents the compression algorithm.

| Values        | Notes                                                           |
| ------------- | --------------------------------------------------------------- |
| `AUTO`        | Auto detect compression via file extensions                     |
| `GZIP`        |                                                                 |
| `BZ2`         |                                                                 |
| `BROTLI`      | Must be specified if loading/unloading Brotli-compressed files. |
| `ZSTD`        | Zstandard v0.8 (and higher) is supported.                       |
| `DEFLATE`     | Deflate-compressed files (with zlib header, RFC1950).           |
| `RAW_DEFLATE` | Deflate-compressed files (without any header, RFC1951).         |
| `NONE`        | Indicates that the files have not been compressed.              |

Learn more:

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors

Thanks a lot to the contributors for their excellent work this week.

andylokandyariesdevilb41shBohuTANGChasen-Zhangdantengsky
andylokandyariesdevilb41shBohuTANGChasen-Zhangdantengsky
devillove084drmingdrmereverpcpcflaneur2020hanyisongjunnplus
devillove084drmingdrmereverpcpcflaneur2020hanyisongjunnplus
leiyskylichuangmergify[bot]PsiACERinChanNOWWWryanrussell
leiyskylichuangmergify[bot]PsiACERinChanNOWWWryanrussell
soyeric128sundy-liTCeasonXuanwoxudong963youngsofun
soyeric128sundy-liTCeasonXuanwoxudong963youngsofun
ZeaLoVezhyass
ZeaLoVezhyass

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.