This week in Databend #27

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Happy Chinese New Year, 春节快乐,虎年大吉!

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

Improvement

  • cluster level global settings by @BohuTANG. (track in #3933, there have been 4 related PRs)

Bug fixes

Build / Test / CI

Databend has always wanted to build stable, highly reliable data warehouses, we welcome everyone to come along and help us improve code quality & test coverage.

Experimental

  • dal2: eliminate type parameters in DAL by @Xuanwo. (#4001).
  • other functions migrate to Function2 trait by @junli1026. (#4036, merged into datavalues-dev branch).

Tips

Let's learn a weekly tip from Databend.

Roadmap 2022

It's February 2022 and just in time for the Chinese New Year, let's take a look at Databend's roadmap for 2022.

Work in Progress

  • Query cluster track #747
  • RBAC Privileges #2793
  • New planner framework #1217
  • Database sharings #3430
  • Stage #2976
  • Index design #3711
  • Push-Based + Pull-Based processor
  • Continuous Benchmarking #3084
  • Databend cli #938
  • Online playground

For more information, please see databend.rs - Roadmap 2022 or issue - Roadmap 2022 .

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Databend #26

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

Improvement

Bug fixes

Experimental

Another key plan is that we are migrating functions to the new datavalues2 and if you are interested, you can track #3967 , which is a list of tasks.

Tips

Let's learn a weekly tip from Databend.

Analyze OnTime datasets with Databend on AWS EC2 and S3

Now, databend.rs has a learning section and will be followed by short lessons to help you understand how to use Databend better.

This article provides step-by-step instructions on how to deploy Databend on EC2 and S3 and perform analysis on the OnTime dataset.

Reporting Carrier On-Time Performance (1987-present) contains on-time arrival and departure data for non-stop domestic flights by month and year, by carrier and by origin and destination airport. Includes scheduled and actual departure and arrival times, canceled and diverted flights, taxi-out and taxi-in times, causes of delay and cancellation, air time, and non-stop distance.

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Databend #25

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

Improvement

Build/Test/CI

Bug fixes

Experimental

A series of refactorings will be carried out at datavalues-dev branch to complete the migration. See RFC - new datavalues system design.

Tips

Let's learn a weekly tip from Databend.

Build/Test Databend with Dev container

In #3853 , we have introduced a development container to make it easy for contributors to build and test Databend.

build binary artifacts

./scripts/setup/run_docker.sh  make build

run test

./scripts/setup/run_docker.sh  make test

debug or get into dev container

./scripts/setup/run_docker.sh /bin/bash

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Databend #24

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

Improvement

Bug fixes

Tips

Let's learn a weekly tip from Databend.

New datatype system design

We need to redesign the datatype system because current implementation had some shortcomes.

Now, DataType is an enum type:

  • We must use specific type after matching.
  • We can't use it as generic argument.
  • It may involve some nested datatypes.
  • And hard to put attributes into it.

Want to find out how we will improve the DataType system? Please check the RFC - New datatype system design.

Plan to jump to specific implementation and track progress? Databend#3794 might be for you.

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

This week in Databend #23

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

Improvement

Performance Improvement

Bug fixes

Tips

Let's learn a weekly tip from Databend.

Tracing in Databend

Databend using Rust's tracing ecosystem tokio-tracing to do log and profile.

Distributed tracing with Jaeger

Jaeger, is a distributed tracing platform. It can be used for monitoring microservices-based distributed systems.

  • 4 steps to enable Jaeger monitor

    • build databend-query: cargo build --bin databend-query
    • run with DEBUG log level: LOG_LEVEL=DEBUG ./databend-query
    • start jaeger: docker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 jaegertracing/all-in-one:latest
    • Open http://127.0.0.1:16686/
  • Jaeger Tracing Show

jaeger-tracing-show

  • Read More: https://databend.rs/dev/development/tracing#distributed-tracing-with-jaeger

Explore and diagnose with tokio-console

tokio-console is a diagnostics and debugging tool for asynchronous Rust programs.

  • 3 steps to enable console subscriber

    • build databend-query with rustflags & features: RUSTFLAGS="--cfg tokio_unstable" cargo build --bin databend-query --features tokio-console
    • run with the log level of TRACE: LOG_LEVEL=TRACE databend-query
    • run tokio-console
  • Run tokio-console to explore databend-query

query-console

  • Read More: https://databend.rs/dev/development/tracing#explore-and-diagnose-with-tokio-console

Changelogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.