跳到主要內容

發表文章

Distributed transactions

Table of contents [ hide ] Basic theory  CAP States that any distributed data store can provide only two of the following three guarantees. Consistency Every read receives the most recent write or an error. Availability Every request receives a (non-error) response, without the guarantee that it contains the most recent write. Partition tolerance The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes. Typical architecture of distributed systems When a network partition failure happens, it must be decided  whether to do one of the following: CP: cancel the operation and thus decrease the availability but ensure consistency AP: proceed with the operation and thus provide availability but risk inconsistency. BASE Basically-available, soft-state, eventual consistency. Base theory is the practical application of CAP theory, that is, under the premise of the existence of partitions and copies, through certain syste
張貼留言
閱讀完整內容
最近的文章

Kafka

Table of contents [ hide ] Kafka Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Why use a Message Queue like Kafka 1. Asynchronous The program can continue execution without waiting for I/O to complete, increasing throughput. 2. Decoupling It refers to reducing the dependencies between different parts of the system so that each component of the system can be developed, maintained, and evolved relatively independently. The main goal of decoupling is to reduce tight coupling between components to improve system flexibility, maintainability, and scalability. 3. Peak clipping Peak clipping is essential to delay user requests more, filter user access needs layer by layer, and follow the principle of "the number of requests that ultimately land on the database is as small as possible". Basic Concept 1. Client Includi
張貼留言
閱讀完整內容

ShardingSphere

Table of contents [ hide ]  ShardingSphere The distributed SQL transaction & query engine for data sharding, scaling, encryption, and more - on any database. ShardingJDBC ShardingSphere-JDBC is a lightweight Java framework that provides additional services at Java’s JDBC layer. ShardingProxy ShardingSphere-Proxy is a transparent database proxy, providing a database server that encapsulates database binary protocol to support heterogeneous languages. Core Concept 1. Virtual Database Provides a virtual database with sharding capabilities, allowing applications to be easily used as a single database 2. Real Database The database that stores real data in the shardingShereDatasource instance for use by ShardingSphere 3. Logic Table Tables used by the application 4. Real Table In the table that stores real data, the data structure is the same as the logical table. The application maintains the mapping between the logical table and the real table. All real tables map to ShardingSpher
張貼留言
閱讀完整內容

Mongo DB

Table of contents [ hide ] Mongo DB An open-source NoSQL file database. Due to the use of JSON document data storage structure, MongoDB has three advantages: flexibility, easy expansion, and high performance. Compared with traditional relational databases, users do not need to define the data structure in advance when using MongoDB to store data, and can freely add the Key/Value of the data. Basic concept SQL vs MongoDB Database vs Database Table vs Collection Row vs Document Column vs Field Index vs Index Primary key vs _id View vs View Table Joins vs $lookup Query Syntax Example SQL vs MongoDB a = 1 vs {a: 1} a<> 1 vs {a: {$ne: 1}} a > 1 vs {a: {$gt: 1}} a >= 1 vs {a: {$gte: 1}} a < 1 vs {a: {$lt: 1}} a <= 1 vs {a: {$lte: 1}} Aggregation  Pipeline Stages Description: Mongo DB vs SQL Filter criteria: $match vs where Projection: $project vs as Left outer join: $lookup vs left outer join Sort: $sort vs order by Group: $group vs group by Pagination: $skip/$limit vs
張貼留言
閱讀完整內容

Redis

Table of contents [ hide ]   Redis To improve response time and avoid the burden on the database, caching is a commonly used tool, and Redis is the most famous and best caching tool. Basic Operation SRING Save string, integer or double. LIST Linked list, we can prune nodes like list or queue. SET Like HashSet, the values ​​in the set are unique. HASH Often objects with multiple properties are saved. ZSET Same as linked hashmap, with orderliness. Advanced Operation Math operation. Element operations, such as union and heteroset. Key operations such as checking existence or getting all keys. Key expire operation. Backup RDB Snapshot of all data, fast recovery. AOF Operation log appending, recovery is slow. Backup thread operations save: the operation will stop the client threads. bgsave: the operation add new a thread to do backup. After redis4.0, you can use rewrite Aof with Rdb + Aof to reduce recovery time and save volume. Usage Single Testing or basic use of small machines. Mast
張貼留言
閱讀完整內容

Program design template

Table of contents [ hide ] Designing programs is a common job for every engineer, so templates help simplify the job of designing each program. Update History Let everyone know the latest version and update information. background Write down why we need this program and what is the background for building this program. Target Target functionality The goal of the program, what function to achieve, the main module and submodule, and these modules' relationship. Target performance Specific benchmarks such as QPS or milliseconds to evaluate programs. Target architecture Stability. Readability. Maintainability. Extendability. ... Others Target Overall design Design principles and thinking Explain how and why the program was designed. Overall architecture An overall architectural picture. Dependency Module dependencies on other modules or programs. Detail design Program flow design Program flow design diagram. API design The details of the API, and how to interact with the frontend
張貼留言
閱讀完整內容

I/O system

Table of contents [ hide ] I/O hardware Port: a connection point between I/O devices and the host. E.g.: user ports. Bus: a set of wires and a well-defined protocol that specifies messages sent over the wires. E.g.: PCI bus. Controller: a collection of electronics that can operate a port, a bus, or a device. A controller could have its own processor and memory. Etc. (e.g.: SCSI controller). Basic I/O Method (Port-mapped I/O) Each I/O port (device) is identified by the unique port address. Each I/O port consists of four registers (1~4bytes). Data-in register: read by the host to get input Data-out register: written by the host to send output Status register: read by the host to check I/O status Control register: written by the host to control the device The program interacts with an I/O port through special I/O instructions (different from mem. access). X86: IN, OUT I/O methods Categorization Depending on how to address a device: Port-mapped I/O. Use different address spaces for me
張貼留言
閱讀完整內容