TiDB Weekly

News & blog posts

The query execution engine plays an important role in database system performance. Inspired by the paper MonetDB/X100: Hyper-Pipelining Query Execution, we began to employ vectorized execution in TiDB to improve query performance. Last week, we published an article that deep dives into why we need vectorized execution, how we implement it, how we managed to vectorize more than 360 functions along with community contributors, and our thoughts about the future.

The full article is here:

10x Performance Improvement for Expression Evaluation Made Possible by Vectorized Execution and the Community

Community

New contributors

tidb:

• Rustin-Liu
• dengsigua

tikv:

• ty666
• waynexia
• tw666

tispark:

• wfnuser

tidb-ansible:

• n0vad3v

Call for participation

• Vectorized Expression Working Group

  • Date or Time built-in functions (35/38)
  • Decimal built-in functions (26/31)
  • Int built-in functions (174/187)
  • JSON built-in functions (24/27)
  • Real built-in functions (47/49)
  • String built-in functions (100/108)
  • Duration built-in functions (26/34)
  • Last week, 13 PRs were landed in the expression component.

• Coprocessor SIG:

  • Improve the performance of Chunk Encoder
  • Implement more UDFs
  • Implement the new row format
  • Port BinaryJSON
  • Tracing
  • The maximum execution time of Per-request

• Planner SIG: Add rules for the cascades planner

• More issues to solve for TiDB

• More issues to solve for TiKV

Updates of the week

Progress

Last week, we landed 74 PRs in the TiDB repository, 21 PRs in the TiSpark repository, and 87 PRs in the TiKV and PD repositories.

Critical PRs

TiDB:

• Refactor GetClusterServerInfo to avoid executing the internal SQL statement to obtain the cluster topology
• Get GoVersion from the runtime variable instead of from the shell
• Raise the No database selected error for the GRANT statement
• Support retrieving the CPU/memory/mutex/block/allocs profile from PD via SQL statements
• Add the RPC server to get the information of other TiDB servers for diagnostics
• Show max-proc-keys and p95-proc-keys in the results of explain analyze
• Implement the non-block reading for Get and BatchGet under the large transaction protocol
• Unify the endian for the Chunk encode format
• Support a hint to force using an IndexMerge path

TiSpark:

• Support the data format and the columnar execution of the TiDB Chunk in Spark
• Register UDF for every sparkSession
• Fix a bug of encodeKey for the decimal type
• Disable pushing down the set, enum, and bit types

TiKV and PD:

• Improve TiKV performance: #5756 #5846 #6010 #6000 #6047
• Fix a Region merge bug
• Provide the diagnostics gRPC interface
• Support the Coprocessor cache protocol
• Support printing the SST files in Region properties for tikv-ctl
• Fix an unexpected error in PD when setting the configuration item to the same value
• Add the profiling API to PD

Upcoming TiDB event

Infra Meetup No.120 (in Chinese)

Date: December 7th, 2019

Location: Beike Building, Nanshan District, Guangdong Province