This week in Databend #30
Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.
Big changes
Below is a list of some major changes that we don't want you to miss.
Features
Improvement
- add create/drop role, allows grant to roles by @flaneur2020. (#4194)
- use listen addr when start grpc by @lichuang. (#4199)
- add role cache mgr by @flaneur2020. (#4165)
- migrate more functions for datavalue2
In particular, with the merging of #4200 rename datavalues2 to datavalues, we now have a whole new set of datavalues and have successfully migrated all the relevant code.
Bug fixes
- refactor join cluster process by @lichuang. (#4219)
- fix arithmetic with null type bug by @zhyass. (#4221)
- fix if bug with null column by @Veeupup. (#4207)
Tips
Let's learn a weekly tip from Databend.
How to eliminate OOM at build time
Databend has a large crate to manage and implement most of the functions. This means that the following error may be reported at build time:
(signal: 9, SIGKILL: kill) warning: build failed, waiting for other jobs to finish... error: build failed.
We observed that this phenomenon is mainly due to memory overflow during linking. This should be a problem that many giant rust projects may have to face, so let's see how to solve it.
- Use a better linker. Modern linkers like mold, which in addition to being faster, are also more memory friendly, can reduce this problem to some extent.
- Enable the newer symbol mangling scheme. In the latest nightly Rust, this means that
-C symbol-manging-version=v0
. It generates smaller symbols, and we observe a great memory reduction. - Consider allocate more virtual memory. Of course, this is definitely a valid approach, but it may require adding another dozen GiB.
Changelogs
You can check the changelogs of Databend nightly to learn about our latest developments.
Meet Us
Please join the DatafuseLabs Community if you are interested in Databend.
We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.
You can submit issues for any problems you find. We also highly appreciate any of your pull requests.