TiDB Weekly

News & blog posts

In recent years, building a large-scale distributed storage system has become a hot topic. Distributed consensus algorithms like Paxos and Raft are the focus of many technical articles. Last week, we published a post to share some of our firsthand experience in designing a large-scale distributed storage system based on the Raft consensus algorithm.

The full article is here:

Building a Large-scale Distributed Storage System Based on Raft

Community

New contributors

tidb:

• AerysNan
• Lanearth
• YourLovely
• zouhuan1215
• SilvaXiang
• catror
• someblue
• koushiro
• Baytwt
• silver–bullet

tikv:

• MaiCw4J
• AerysNan
• TommyCpp
• Renkai
• wangwangwar

pd:

• qinggniq
• hrbeuyz24

docs-cn:

• baiyuqing

Call for participation

• Vectorized Expression Working Group

  • Date or Time built-in functions (30/65)
  • Decimal built-in functions (20/31)
  • Int built-in functions (155/187)
  • JSON built-in functions (22/27)
  • Real built-in functions (46/49)
  • String built-in functions (89/133)
  • Duration built-in functions (17/45)
  • Last week, 43 PRs were landed in the expression component.

• Coprocessor SIG:

  • Improve the performance of Chunk Encoder
  • Implement more UDFs
  • Implement the new row format
  • Port BinaryJSON
  • Tracing
  • The maximum execution time of Per-request

• More issues to solve for TiDB

• More issues to solve for TiKV

Updates of the week

Progress

Last week, we landed 145 PRs in the TiDB repository, 4 PRs in the TiSpark repository, and 76 PRs in the TiKV and PD repositories.

Critical PRs

TiDB:

• Enable the push-down of 19 functions
• Do not set the statement priority when tidb_force_priority is set
• Allocate continuous RowID for the single INSERT statement
• Support the SQL bindings management
• Make the error message more readable when dropping the primary key
• Fix the bug that the cached store cannot be updated when the store address is changed and the original address is made offline
• Implement the non-block read when Coprocessor meets the lock of a large transaction

TiSpark:

• Fix TiKV DAGRequest's outputOffsets

TiKV and PD:

• Update the blob file format for the ZSTD dictionary compression
• Replace crc by crc32fast and crc64fast
• Reduce the size of tikv::storage::errors::Error
• Update RocksDB and Titan
• Return commit_ts in the commit response
• Fix the maximum receivable message size for the client
• Delay the Replica read when applying a snapshot
• Add more engine abstraction that works towards Peekable and Iterable for the snapshot
• Add the rollback reason for check_txn_status
• Encapsulate IOLimiter and remove the RocksDB dependency from sst_importer

Upcoming TiDB events

Apache Pulsar Meetup x PingCAP Infra Meetup

Date: November 30th, 2019

Location: PingCAP office in Beijing