This week in Databend #66
Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .
What's Changed
Below is a list of some major changes that we don't want you to miss.
Exciting New Features ✨
RFC
- User Stage (#8519)
user stage
- add internal and legacy internal stage support (#8526)
- implement user stage support along with stateful tests (#8556)
multiple catalog
- implement multiple catalog AST (#8608)
hive
- hive predict pushdown (#8469)
- support substr && order by with null values compatible with hive functions (#8558)
rbac
- only display available roles in
SHOW ROLES
statement (#8553)
compatibility
- support mydumper dump data (#8500)
auth
- add common auth module and support file token on sharing endpoint (#8607)
format
hashtable
- adaptive string hash table (#7971)
meta
query
- implement logging format for databend-query (#8466)
Code Refactor 🎉
error handler
- add internal error and merge other not needed errors (#8581)
interpreter
- move
interpreter_common.rs
to common (#8593)
query
- remove legacy crate codes (#8412)
- avoid extra memcpy (#8569)
- migrate serializations to expression (#8582)
storage
- make block meta easier to be cloned (#8548)
new expression
- cast timestamp to variant string instead of variant int (#8580)
Thoughtful Bug Fix 🔧
compatibility
- fed mysqldump 5.7.16
/*!code
query (#8485)
planner
- select view column not exists (#8543)
query
- div nullable(Int) with null value should not err (#8495)
CURRENT ROLE
behavior withAUTH ROLE
is set (#8546)
storage
News
Let's take a look at what's new at Datafuse Labs & Databend each week.
String Adaptive Hash Table
String Adaptive Hash Table is one of the Databend community's projects in the Open Source Promotion Plan 2022. @usamoi helped us with this important work, which has now been merged into Databend repository and has resulted in a 20% - 50% performance improvement in certain scenarios.
String Adaptive Hash Table is designed for the key to be a string type, and different data structures are applied to optimise the insertion and reading based on the different lengths of the key.
Learn More
- PR | adaptive string hash table
- Paper | SAHA: A String Adaptive Hash Table for Analytical Databases
- GitHub - usamoi/saha
RFC: User Stage
Databend only supports named internal stage:
CREATE STAGE @my_stage;
COPY INTO my_table FROM @my_stage;
However, named internal stages are complex to be used in some cases. Especially for users who only use stages to load data. By supporting the user stage, they can copy data more efficiently:
COPY INTO my_table from @~;
Learn More
Issues
Meet issues you may be interested in and try to solve it.
Replace RwLock<HashMap>
and Mutex<HashMap>
by Using DashMap
DashMap is an implementation of a concurrent associative array/hashmap in Rust. it tries to be very simple to use and to be a direct replacement for RwLock<HashMap<K, V>>
.
Using DashMap will bring two advantages:
- Code will be cleaner, and
- Potential performance improvement
Issue 8601: Feature: replace RwLock
If you find it interesting, try to solve it or participate in discussions and PR reviews. Or you can click on https://link.databend.rs/i-m-feeling-lucky to pick up a good first issue, good luck!
Changelogs
You can check the changelogs of Databend nightly to learn about our latest developments.
- v0.8.99-nightly
- v0.8.98-nightly
- v0.8.97-nightly
- v0.8.96-nightly
- v0.8.95-nightly
- v0.8.94-nightly
- v0.8.93-nightly
- v0.8.92-nightly
- v0.8.91-nightly
Contributors
Thanks a lot to the contributors for their excellent work this week.
andylokandy | b41sh | BohuTANG | Chasen-Zhang | ClSlaid | dantengsky |
drmingdrmer | everpcpc | flaneur2020 | leiysky | lichuang | mergify[bot] |
sandflee | soyeric128 | sundy-li | TCeason | TszKitLo40 | usamoi |
wubx | Xuanwo | xudong963 | youngsofun | ZhiHanZ | zhyass |
Meet Us
Please join the DatafuseLabs Community if you are interested in Databend.
We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.
You can submit issues for any problems you find. We also highly appreciate any of your pull requests.
- Databend Website
- Weekly (A weekly newsletter about Databend)
- GitHub Discussions (Feature/Bug reports, Contributions)
- Twitter (Get the news fast)
- Slack Channel (For live discussion with the Community)