Databend is an open source elastic and reliable Modern Cloud Data Warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

logging

  • implement RFC The New Logging (#6845)

meta

  • add grant and revoke object API in ShareApi (#6724)
  • show share api (#6790)
  • add get_share_grant_objects API in ShareApi (#6798)

http handle

  • http handler return session state (#6846)

processor

  • implement explain fragments (#6851)
  • support distributed subquery in new cluster framework (#6666)

new planner

  • support order by expression (#6725)
  • enable delete stmt (#6768)
  • implement distributed query (#6440)
  • support push down predicates to storage (#6842)

storage

  • add support for COPY from https (#6691)
  • construct leaf column statistics (#6731)
  • support read nested columns (#6612)

new expression

  • support float32, float64 and Map(T) datatype (#6711 & #6838)
  • add serializable expression (#6712)
  • support user-defined CAST and TRY_CAST (#6663)
  • migrate Boolean functions to new expression framework (#6763)
  • migrate some String functions to new expression framework (progress of migration #6766)

Improvement

  • purge mapping data in DB/table GC (#6753)
  • fuzz with afl (#6793)
  • make auto-nullable and auto-vectorization independent (#6797)
  • refactor pipeline builder (#6820)

new planner

  • make PRESIGN works on old planner by forwarding (#6713)
  • forward COPY and STAGE to new planner entirely (#6853)
  • migrate more new planners to be enabled (#6716)

Build/Testing/CI

Bug fixes

  • fix uncorrelated scalar subquery returns error results (#6720)
  • fix bug in FileSplitter skip header (#6732)
  • fix oom when returning large results in clickhouse tcp handler (#6789)
  • Any/Exists subquery in projection (#6809)

Tips

Let's learn a weekly tip from Databend.

COPY INTO <table> FROM REMOTE FILES

After #6691 has been merged, Databend now supports loading data into a table from one or more remote files by their URL.

Syntax

COPY INTO [<database>.]<table_name>
FROM 'https://<site>/<directory>/<filename>'
[ FILE_FORMAT = ( TYPE = { CSV | JSON | PARQUET } [ formatTypeOptions ] ) ]

Example

This example loads data into the table ontime200 from the remote files ontime_2006_200.csv, ontime_2007_200.csv, and ontime_2008_200.csv:

copy into ontime200 from 'https://repo.databend.rs/dataset/stateful/ontime_200{6,7,8}_200.csv' FILE_FORMAT = (type = 'CSV' field_delimiter = ','  record_delimiter = '\n' skip_header = 1)

Of course, this example could also be written in the following form:

copy into ontime200 from 'https://repo.databend.rs/dataset/stateful/ontime_200[6-8]_200.csv' FILE_FORMAT = (type = 'CSV' field_delimiter = ','  record_delimiter = '\n' skip_header = 1)

Learn More

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors

Thanks a lot to the contributors for their excellent work this week.

andylokandyariesdevilaseadayb41shBohuTANGClSlaid
andylokandyariesdevilaseadayb41shBohuTANGClSlaid
dantengskyleiyskylichuangmergify[bot]PsiACEsoyeric128
dantengskyleiyskylichuangmergify[bot]PsiACEsoyeric128
sundy-liTCeasonTianLangStudioXuanwoxudong963ygf11
sundy-liTCeasonTianLangStudioXuanwoxudong963ygf11
youngsofunZeaLoVezhang2014zhyass
youngsofunZeaLoVezhang2014zhyass

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.