| /rocksdb-6.9/docs/_posts/ |
| D | 2014-03-27-how-to-backup-rocksdb.markdown | 10 In RocksDB, we have implemented an easy way to backup your DB. Here is a simple example:
30 Restoring is also easy:
41 This code will restore the backup back to "/tmp/rocksdb". The second parameter is the location of l…
43 An alternative API for backups is to use BackupEngine directly:
62 Restoring with BackupEngine is similar to RestoreBackupableDB:
73 …is always calculated for any backuped file (including sst, log, and etc). It is used to make sure …
77 …is `RestoreDBFromBackup()` which takes a backup ID and restores that particular backup. Checksum i…
84 Let's say you want to backup your DB to HDFS. There is an option in `BackupableDBOptions` to set `b…
86 `BackupableDBOptions::info_log` is a Logger object that is used to print out LOG messages if not-nu…
92 …is false by default. When `flush_before_backup` is true, `BackupableDB` will first issue a memtabl…
[all …]
|
| D | 2018-11-21-delete-range.markdown | 27 product is removed, all these keys must be deleted.
49 If space reclamation time is important, or it is important that the deleted
77 A second solution we considered is appending to the manifest. This file is
80 is maintained in the background and fully read when the DB is opened. However,
88 tombstone is obsolete, and seqnum zeroing can cause a key
154 whether the key is deleted.
162 unsearchable form of A and converting it to the form of B, which is
165 key-space and the y-axis is the sequence number.*
174 …have a dedicated skiplist memtable, the complexity of inserting is O(log(T)), where T is the numbe…
183 tombstones. This operation is O(T\*log(T)) where T is the number of tombstones
[all …]
|
| D | 2019-03-08-format-version-4.markdown | 8 …is 2, the format of index and data blocks are the same: index blocks use the same key format of <`…
23 ### What is format_version 3?
24 …is not necessary for keys in the index blocks. In such cases, `format_version`=3 skips encoding th…
26 ### What is format_version 4?
27 …is set, which is used by the reader to know how to decode the index block. The format of each key…
35 where, k is key, v is value, and its encoding is in parenthesis.
|
| D | 2014-06-27-avoid-expensive-locks-in-get.markdown | 12 …strategy. Before reading data, it needs to grab the current version, which is encapsulated in a da…
24 The lock is necessary because pointer super_version_ may be updated, the corresponding SuperVersion…
27 …ook/rocksdb/raw/gh-pages/talks/2014-03-27-RocksDB-Meetup-Lei-Lockless-Get.pdf). This is outrageous!
32 …is a rare event comparable to millions of read requests. On the very first Get() request, each thr…
45 // slow path, cleanup of current super version is omitted
54 …is quite amazing. RocksDB can nicely [scale to 32 cores](https://github.com/facebook/rocksdb/raw/g…
59 …is shown in the assembly code. Mutex can keep threads spinning on CPU or even trigger thread conte…
64 …is referenced and cached in its thread local storage. All resources (e.g., memtables, files) which…
74 …InUse, that means a “sweep” was done and the reader thread is responsible for cleanup (this is exp…
|
| D | 2014-05-14-lock.markdown | 27 …ation is not expensive but if you are building a high throughput server with lots of reads, the lo…
35 …s takes several milliseconds. It can be even longer if file system journal is involved or I/Os are…
38 …is a chance that file write will wait for disk I/O to finish before finishing, even if fsync() is …
43 …hen an autovector is used as a stack variable, no malloc will be needed unless the pre-allocated b…
49 …is cleared. When the SST table needs to be read and its table handler is not in LRU cache, the tab…
51 …does not organize data by blocks. Data are located by memory addresses so no block cache is needed.
53 With all of those improvements, lock contention is not a bottleneck anymore, which is shown in our …
67 Sorry for the delay. We don’t have feature support for this scenario yet. Here is an example you ca…
77 By the way, the best way to ask those questions is in our [facebook group](https://www.facebook.com…
82 What all ACID properties is supported by RocksDB, only durability irrespective of single or batch w…
[all …]
|
| D | 2018-08-23-data-block-hash-index.markdown | 9 …compact hash table to the end of the data block for efficient indexing. It is backward compatible …
11 …int lookup code path, `DataBlockIter::Seek()`, is reduced by 21.8%, and the overall RocksDB throug…
17 …is disabled by default unless `BlockBasedTableOptions::data_block_index_type` is set to `data_bloc…
32 // #entries/#buckets. It is valid only when data_block_hash_index_type is
47 …ex the key to reduce the CPU overhead of the binary search. The hash index is just an array of poi…
54 We define hash table utilization ratio as the #keys/#buckets. If a utilization ratio is 0.5 and the…
56 …. Each bucket is a `uint8_t` (i.e. one byte). For a util ratio of 1, the space overhead is 1Byte …
82 …is enabled, cache miss also incurs data block decompression, which is CPU-consuming. Therefore the…
106 … the point lookup code path, DataBlockIter::Seek(), is reduced by 21.8% and the overall throughput…
108 …is not always purely cached. So we also evaluate the performance under different cache space press…
[all …]
|
| D | 2016-01-29-compaction_pri.markdown | 10 …is level-based compaction, which is an improved version of LevelDB's compaction algorithm. Page 9-…
14 Which file to pick to compact is an interesting question. LevelDB only uses one thread for compacti…
20 …is not the case, even if user updates are uniformly distributed across the whole key range. For in…
22 …me to the next level is the oldest, so the range will accumulated keys from incoming compactions f…
24 …always pick the file covers the oldest updates in the level, which usually is contains the densest…
30 …is the oldest. It means there is no incoming data for the range for the longest. Usually it is the…
34 …mpact delete keys into the last level, the sooner the disk space is reclaimed, so it is good for s…
36 …is more likely to be picked for compaction. The more number of deletes exceed inserts, the more li…
40 …xternal service checking modify time of all SST files. If any of the files is too old, we force th…
43 …t you start with `options.compaction_pri=kOldestSmallestSeqFirst` (note it is not the default one …
|
| D | 2017-12-19-write-prepared-txn.markdown | 8 …tic transactions is _WriteCommitted_, which means that the data is written to the DB, i.e., the me…
12 …is written to the memtable only after the transaction commits. This greatly simplifies the read pa…
16 …is composed of Write stage, where the transaction `::Put` is invoked, the prepare phase, where `::…
22 2) How do we figure if a key/value written by transaction Txn_w is in the read snapshot of the read…
25 …is called, it writes the in-memory write batch to the WAL (write-ahead log) as well as to the memt…
27 The _CommitCache_ is a lock-free data structure that caches the recent commit entries. Looking up t…
|
| D | 2014-04-21-indexing-sst-files-for-better-lookup-performance.markdown | 16 …is scheduled periodically to pick up files from an upper level and merges them with files from low…
18 …is that: after the LSM tree is built, an SST file's position in its level is fixed. Furthermore, i…
22 …is the candidate. Then key 80 is compared with its FileMetaData.smallest and FileMetaData.largest …
24 …is larger than file 1's FileMetaData.largest 200). A comparison with file 2's range shows the targ…
|
| D | 2017-08-25-flushwal.markdown | 8 …is called, the data is written to both memtable (to be flushed to SST files later) and the WAL (wr…
10 ### How much is the gain?
16 …L is first written to the application memory buffer. The buffer in the next step is "flushed" to O…
20 …:manual_wal_flush`. In that case, the WAL buffer is flushed when it is either full or `DB::FlushWA…
24 …ows the 1st phase (Prepare) to be run in parallel but after a commit group is formed performs the …
|
| D | 2015-11-16-analysis-file-read-latency-by-level.markdown | 10 …his approach, verifying effective of the OS page caching is challenging, because file system is a …
12 …alysis like it is especially important if users run RocksDB on hard drive disks, for the gap of la…
150 …vel 0 while issued 25 million reads from level 5. The latency distribution is also clearly shown a…
189 …is 421 micro seconds, with standard deviation 385. Median is 376, max value is 202 milliseconds. 0…
195 …SSD. One of the test is similar to what is mentioned in the wiki, TEST 4 : Random read , except th…
215 1. How is it that the count is around 7 billion when I have only inserted 1 billion hashes ? is the…
221 If I remember correctly, with db_bench, if you specify –num=1000000000 –threads=32, it is every thr…
230 this is the script on which I have used
240 … have read 200 million hashes in 2 and half days. Is there something which is missing in db_bench …
|
| D | 2014-03-27-how-to-persist-in-memory-rocksdb-database.markdown | 10 … keep their entire data in memory. Running in-memory database with RocksDB is easy -- just mount y…
16 …is written to two places - one is an in-memory data structure called memtable and second is write-…
18 Options::wal_dir is the directory where RocksDB stores write-ahead log files. If you configure this…
19 Options::WAL_ttl_seconds is the timeout when we delete the archived log files. If the timeout is no…
21 Let's assume Options::wal_dir is a directory on persistent storage and Options::WAL_ttl_seconds is …
|
| D | 2017-06-26-17-level-based-changes.markdown | 12 … which we enforce L0's file count limit may be unappealing. When the limit is reached, RocksDB int…
14 …is limited. Files at L0 or L1 may be locked due to involvement in pending L0->L1 or L1->L2 compact…
20 …k the largest file in the level. One special property of L0->L1 compaction is that files can overl…
24 This compaction pulls in every L0 and L1 file. This happens regardless of which L0 file is initiall…
30 …is the largest, and let's say the top two files are created after the compaction started. When the…
32 …is nothing we can do to eliminate L0 files. Second, L0->L1 compactions are relatively slow. As we …
40 Say the L1->L2 compaction started first. Now L0->L1 is prevented by the locked L1 file. In this cas…
42 …is this increases total compaction work, as we're now compacting files without contributing toward…
44 This feature is available since RocksDB 5.4.
60 …rt.html). Note the primary change between his March 17 and April 14 builds is the latter performs …
|
| D | 2014-09-12-cuckoo.markdown | 12 …is optimized for fast point lookups. The new format was built for applications which require very …
32 * The database is operated in read only mode
50 …_h1(k), ..., __hn__(k)_. We insert the key in the first hash location that is free. If all the loc…
52 …is naturally a shortest path problem in a directed graph where nodes are buckets of hash table and…
56 …is to minimize average (and maximum) number of hash functions required and hence the number of mem…
64 …* If the key is not present in first hash location, we jump to second hash location which may not …
67 …an optimal prefetching strategy - prefetching all hash locations for a key is wasteful. But prefet…
71 …is to insert more keys near first location. In case of collision in the first hash location - _h1(…
|
| D | 2015-10-27-getthreadlist.markdown | 10 …put / output size, elapsed time, the number of bytes it has written. Below is an example output of…
118 …is currently doing and the current status of the job. For instance, we can see thread 14071641616…
141 …eadList API is defined in [include/rocksdb/env.h](https://github.com/facebook/rocksdb/blob/master/…
164 // The name of the DB instance where the thread is currently
169 // The name of the column family where the thread is currently
174 // The operation (high-level action) that the current thread is involved.
180 // An integer showing the current stage where the thread is involved
189 // The state (lower-level action) that the current thread is involved.
|
| D | 2017-08-24-pinnableslice.markdown | 8 …ll copy the value. The memcpy overhead could be non-trivial when the value is large. The [new API]…
10 ### What is PinnableSlice?
12 …is being processed by the user, PinnableSlice, as its name suggests, has the data pinned in memory…
14 ### How good is it?
22 …is when reading large values from the block cache. There are however cases that it would still hav…
|
| D | 2015-01-16-migrating-from-leveldb-to-rocksdb-2.markdown | 10 …is to map the options for LevelDB to proper options for RocksDB. As of release 3.9 this can be aut…
62 The difference is:
84 …is handy is because a lot of individual options in RocksDB have moved to other structures in diffe…
112 …ocksDB from LevelDB and allows us to break down the various options across classes as it is needed.
|
| D | 2017-05-12-partitioned-index-filter.markdown | 12 …ndex/filter block of size 0.5/5MB is typical, which is much larger than the typical data block siz…
14 ### What is the big deal with large index/filter blocks?
16 …is occupying the space that could otherwise be used to cache 1000s of data blocks (of size 4KB). T…
18 …is not helping in reducing the IO cost. While a simple point lookup might need at most a couple of…
20 ## What is partitioned index/filters?
22 … SST file is partitioned into smaller blocks with an additional top-level index on them. When read…
28 In this example we have a DB of size 86G on HDD and emulate the small memory that is present to a n…
|
| /rocksdb-6.9/ |
| D | USERS.md | 9 4. Laser -- Laser is a high query throughput, low (millisecond) latency, key-value storage service …
33 DNANexus is using RocksDB to speed up processing of genomics data.
37 Iron.io is using RocksDB as a storage engine for their distributed queueing system.
41 Tango is using RocksDB as a graph storage to store all users' connection data and other social acti…
44 Turn is using RocksDB as a storage layer for their key/value store, serving at peak 2.4MM QPS out o…
67 …Code checker and URL helper uses RocksDB to store all the URLs where VWO's Smart Code is installed.
77 …ps://github.com/pingcap/tikv) is a GEO-replicated, high-performance, distributed, transactional ke…
92 LzLabs is using RocksDB as a storage engine in their multi-database distributed framework to store …
95 …rofanedb.gitlab.io/) is a database for Protocol Buffers, and uses RocksDB for storage. It is acces…
98 … is using RocksDB in the [IOTA Reference Implementation (IRI)](https://github.com/iotaledger/iri) …
[all …]
|
| D | HISTORY.md | 67 …x a regression bug that causes segfault when hash is used, max_open_files != -1 and total order se…
83 …ion is now created only if some memtables were actually trimmed. Second, trimming is only schedule…
145 …failure in MultiGet() when BlockBasedTableOptions::no_block_cache is true and there is no compress…
146 …roblem in BlockBasedTable::MultiGet() when compression is enabled and no compressed block cache is…
158 … where the compaction snapshot refresh feature is not disabled as advertised when `snap_refresh_na…
234 …etion of WALs is only enabled if DBOptions::wal_dir is not set, or explicitly set to db_name passe…
625 * Introduce WriteOptions.low_pri. If it is true, low priority writes will be throttled if the compa…
655 …leases the pinned resources that contain the value when it is destructed or when ::Reset() is call…
903 * options.hard_rate_limit is deprecated.
904 …ptions.level0_slowdown_writes_trigger is triggered, the way to slow down writes is changed to: wri…
[all …]
|
| D | COPYING | 6 Everyone is permitted to copy and distribute verbatim copies
7 of this license document, but changing it is not allowed.
14 software--to make sure the software is free for all its users. This
70 the term "modification".) Each licensee is addressed as "you".
75 is covered only if its contents constitute a work based on the
77 Whether that is true depends on what the Program does.
112 the Program is not required to print an announcement.)
218 integrity of the free software distribution system, which is
222 system; it is up to the author/donor to decide if he or she is willing
226 This section is intended to make thoroughly clear what is believed to
[all …]
|
| D | thirdparty.inc | 27 message(STATUS "GFLAGS library is enabled")
48 message(STATUS "GFLAGS library is disabled")
66 message(STATUS "SNAPPY library is enabled")
106 message(STATUS "LZ4 library is enabled")
127 message(STATUS "LZ4 library is disabled")
145 message(STATUS "ZLIB library is enabled")
166 message(STATUS "ZLIB library is disabled")
178 message(STATUS "XPRESS is enabled")
185 message(STATUS "XPRESS is disabled")
204 message(STATUS "ZSTD library is enabled")
[all …]
|
| /rocksdb-6.9/docs/_docs/ |
| D | faq.md | 8 Here is an ever-growing list of frequently asked questions around RocksDB
10 ## What is RocksDB?
12 … is an embeddable persistent key-value store for fast storage. RocksDB can also be the foundation …
24 ## What is RocksDB suitable for?
34 ## How big is RocksDB adoption?
36 …is an embedded storage engine that is used in a number of backend systems at Facebook. In the Face…
38 RocksDB is proving to be a useful component for a lot of other groups in the industry. For a list o…
40 ## How good is RocksDB as a database storage engine?
42 …ith MongoDB: [MongoRocks](https://github.com/mongodb-partners/mongo-rocks) is the RocksDB based st…
44 …is the RocksDB based storage engine for MySQL. Using RocksDB we have managed to achieve 2x better …
[all …]
|
| /rocksdb-6.9/utilities/ |
| D | env_librados.md | 2 …is a customized RocksDB Env to use RADOS as the backend file system of RocksDB. It overrides all f…
12 …read/write operation to the RADOS cluster assigned by config_path. Default pool is db_name+"_pool".
16 - write_buffer_size. This variable is the max buffer size for WritableFile. After reaching the buff…
18 …ure (dir_name/file_name), the format of wal_dir is "/dir_name"(CAN'T be "/dir1/dir2"). Default wal…
19 - wal_pool. Corresponding pool name for WAL files. Default value is db_name+"_wal_pool"
43 …LIBRADOS=1` under `rocksdb\`. The configure file used by env_librados_test is `../ceph/src/ceph.co…
53 My test environment is Ubuntu 14.04 in VirtualBox with 8 cores and 8G RAM. Following is the test re…
55 … order. The time of writing under default env is around 10s while the time of writing under EnvLib…
57 …g under default env drops to arround 1s. But the time of writing under EnvLibrados is not changed.
94 Following is the steps of configuration of MySQL.
|
| /rocksdb-6.9/docs/ |
| D | TEMPLATE-INFORMATION.md | 11 You can customize your homepage in three parts - the first in the homepage header, which is mostly …
13 …e second place for your homepage is in `index.md` which contains the bulk of the main content belo…
15 …is in the `_data/powered_by.yml` and `_data/powered_by_highlight.yml` files. Both these files comb…
17 …is setup your top level navigation bar. You can do this by editing `nav.yml` and keeping the exist…
|