| /rocksdb-6.9/ |
| D | HISTORY.md | 132 * Add a new Env::LoadEnv() overloaded function to return a shared_ptr to Env.
190 …places old Registra with ObjectRegistry to allow user to create custom object from string, also ad…
237 …tion `snap_refresh_nanos` (default to 0) to periodically refresh the snapshot list in compaction j…
277 * Adjust WriteBufferManager's dummy entry size to block cache from 1MB to 256KB.
480 * Added SstFileManager to the Java API to allow managing SST files across DB instances.
690 * Some fixes related to 2PC.
899 …to change_level, to allow users to move levels to lower levels if allowed. It can be used to migra…
904 …down_writes_trigger is triggered, the way to slow down writes is changed to: write rate to DB is l…
905 …tApproximateSizes() adds a parameter to allow the estimation to include data in mem table, with de…
923 …ytes_per_sync used to apply to both WAL and table files. As of 3.11 it applies only to table files…
[all …]
|
| D | COPYING | 13 License is intended to guarantee your freedom to share and change free
16 Foundation's software and to any other program whose authors commit to
29 anyone to deny you these rights or to ask you to surrender the rights.
83 notices that refer to this License and to the absence of any warranty;
100 part thereof, to be licensed as a whole at no charge to all third
125 your rights to work written entirely by you; rather, the intent is to
215 It is not the purpose of this section to induce you to infringe any
226 This section is intended to make thoroughly clear what is believed to
282 How to Apply These Terms to Your New Programs
285 possible use to the public, the best way to achieve this is to make it
[all …]
|
| D | WINDOWS_PORT.md | 10 …ber of CPUs and storage IOPS, to support IO-bound, in-memory and write-once workloads, most import…
12 …to improve the scalability, efficiency of platform and eventually benefit Bing end-user satisfacti…
14 These notes describe some decisions and changes we had to make with regards to porting RocksDB on W…
21 We strive to achieve the following goals:
27 * we would like to keep the port code inline with the master branch with no forking
35 …t it would be beneficial to merge the existing make-based build system and the new cmake-based bui…
45 * All posix specific headers were replaced to port/port.h which worked well
49 * in class member initialization were moved to a __ctors in some cases
50 …r` is not supported. We had to replace `std::numeric_limits<>::max/min()` to its C macros for cons…
51 * `constexpr` for functions was replaced to a template specialization (1 place)
[all …]
|
| D | DEFAULT_OPTIONS_HISTORY.md | 6 * Change the default of delayed slowdown value to 16MB/s and further increase the L0 stop condition…
10 * Options.level0_stop_writes_trigger default value changes from 24 to 32.
13 …ons.max_open_files changes from 5000 to -1. It improves performance, but users need to set file de…
14 …und_compactions to 1. When users set higher max_background_compactions but the write throughput is…
18 * options.write_buffer_size changes from 4MB to 64MB.
19 * options.target_file_size_base changes from 2MB to 64MB.
20 * options.max_bytes_for_level_base changes from 10MB to 256MB.
21 * options.soft_pending_compaction_bytes_limit changes from 0 (disabled) to 64GB.
22 * options.hard_pending_compaction_bytes_limit changes from 0 (disabled) to 256GB.
23 * table_cache_numshardbits changes from 4 to 6.
[all …]
|
| D | LICENSE.Apache | 19 "control" means (i) the power, direct or indirect, to cause the
28 including but not limited to software source code, documentation
33 not limited to compiled object code, generated documentation,
34 and conversions to other media types.
38 copyright notice that is included in or attached to the work
51 to that Work or Derivative Works thereof, that is intentionally
56 to the Licensor or its representatives, including but not limited to
70 copyright license to reproduce, prepare Derivative Works of,
104 excluding those notices that do not pertain to any part of
145 agreed to in writing, Licensor provides the Work (and each
[all …]
|
| /rocksdb-6.9/docs/_posts/ |
| D | 2018-11-21-delete-range.markdown | 23 to other nodes in the cluster.
29 When we decide what to improve, we try to find a use case that's common across
47 version at a lower level to reappear.
68 ## v1: Getting it to work
70 ### Where to persist them?
75 So, we decided to investigate segregated storage.
89 to go from above a range tombstone to below, i.e., disappearing. The upside is
93 tombstones to be aware of flush/compaction. An easy way to achieve this is put
145 scanned. Given these concerns we decided to defer collapsing work to the read
172 and applies to memtable. Logging to WAL always `fflush`es, and optionally
[all …]
|
| D | 2016-01-29-compaction_pri.markdown | 14 …to pick to compact is an interesting question. LevelDB only uses one thread for compaction and it …
16 …to multiple algorithms to choose from? Because there are different factors to consider when pickin…
20 …to the next level, it creates a hole. Over time, incoming compaction will fill data to the hole, b…
22 …ion, because a file being picked covers a range whose last compaction time to the next level is th…
24 …e where writes are uniformly distributed across the key space and you want to reduce write amplifi…
28 …to deeper levels will benefit write amplification, as well as space amplification. For example, if…
34 …slow down iterating over this area, because we still need to iterate those deleted keys just to ig…
36 …likely to be picked for compaction. The more number of deletes exceed inserts, the more likely it …
40 …to clean up old data to free up space. Picking files to compact may impact space efficiency. We do…
43 …ote it is not the default one for backward compatible reason). If you want to further optimize you…
[all …]
|
| D | 2014-03-27-how-to-backup-rocksdb.markdown | 2 title: How to backup RocksDB?
7 - /blog/191/how-to-backup-rocksdb/
10 In RocksDB, we have implemented an easy way to backup your DB. Here is a simple example:
23 delete backupable_db; // no need to also delete db
43 An alternative API for backups is to use BackupEngine directly:
62 Restoring with BackupEngine is similar to RestoreBackupableDB:
75 …to keep around only small number of backups. To delete old backups, just call `PurgeOldBackups(N)`…
84 …to backup your DB to HDFS. There is an option in `BackupableDBOptions` to set `backup_env`, which …
86 `BackupableDBOptions::info_log` is a Logger object that is used to print out LOG messages if not-nu…
99 …r methods to it: `CreateNewBackup()`, `GetBackupInfo()`, `PurgeOldBackups()`, `DeleteBackup()`. An…
[all …]
|
| D | 2014-03-27-how-to-persist-in-memory-rocksdb-database.markdown | 2 title: How to persist in-memory RocksDB database?
7 - /blog/245/how-to-persist-in-memory-rocksdb-database/
16 …to RocksDB is written to two places - one is an in-memory data structure called memtable and secon…
21 …to one day. To fully recover the DB, we also need to backup the current snapshot of the database (…
23 …ocess to avoid backing up log files, since they are already stored in persistent storage. To do th…
25 …to make sure that restoring the database doesn't delete log files in WAL directory. When restoring…
27 To reiterate, here's what you have to do:
32 * Set DB directory to tmpfs or ramfs mounted drive
36 * Set Options::wal_log to a directory on persistent storage
40 * Set Options::WAL_ttl_seconds to T seconds
[all …]
|
| D | 2014-04-21-indexing-sst-files-for-better-lookup-performance.markdown | 10 … goes through mutable memtable, list of immutable memtables, and SST files to look up the target k…
12 …d FileMetaData.largest) are mostly overlapped with each other. So it needs to look up every L0 fil…
16 …to pick up files from an upper level and merges them with files from lower level. As a result, key…
18 …to this problem is that: after the LSM tree is built, an SST file's position in its level is fixed…
22 …to look up key 80. A binary search based FileMetaData.largest tells you file 1 is the candidate. T…
24 …to get key 230. A binary search on level 1 locates to file 2 (this also implies key 230 is larger …
26 …to a range of files on level 2. For example, file 1 on level 1 points to file 3 (on level 2) on th…
|
| D | 2017-08-25-flushwal.markdown | 8 …to both memtable (to be flushed to SST files later) and the WAL (write-ahead log) if it is enabled…
16 …to the WAL is first written to the application memory buffer. The buffer in the next step is "flus…
20 …called by the user. The API also accepts a boolean argument should we want to sync right after the…
24 …to provide reliability. MySQL for example uses 2PC (two-phase commit) to write to both binlog as w…
|
| D | 2015-07-15-rocksdb-2015-h2-roadmap.markdown | 10 …nths, RocksDB team gets together to prioritize the work ahead of us. We just went through this exe…
16 …to integrate RocksDB as a storage engine for MySQL. This project is pretty important for us becaus…
24 2. Implement pessimistic concurrency control to support repeatable read isolation level in MyRocks
51 …to now, our mission was to build the best key-value store “for fast storage” (flash and in-memory)…
64 …king to our customers, there are couple of issues that keep reoccurring. We need to fix them to ma…
69 1. Iterate P99 can be high due to the presence of tombstones.
83 … debugging RocksDB issues. We plan to improve user experience when running RocksDB. The goal is to…
|
| D | 2014-05-14-lock.markdown | 10 In this post, we briefly introduce the recent improvements we did to RocksDB to improve the issue o…
25 This DB mutex used to be scalability bottleneck preventing us from scaling to more than 16 threads.…
31 …lace some reference counters to stc::atomic objects, so that decreasing reference count of an obje…
38 …to logs within mutex. There is a chance that file write will wait for disk I/O to finish before fi…
41 …to lock some shared data structures. Allocating can also be slow because we sometimes do expensive…
45 …to creating iterator of every live men table and SST table within the mutex and a merging iterator…
49 …to be evicted based on LRU, those information is cleared. When the SST table needs to be read and …
71 1. call `DB::GetLiveFiles()` to get a full list of the files.
72 1. for all the files except manifest, add a hardlink file in your new directory pointing to the ori…
75 …ry in another process to access those files. Please remember to delete the directory after reading…
[all …]
|
| D | 2018-08-23-data-block-hash-index.markdown | 9 …to the end of the data block for efficient indexing. It is backward compatible with the data base …
14 ### How to use it
29 // Set to kDataBlockBinaryAndHash to enable hash index
47 We implemented a hash map at the end of the block to index the key to reduce the CPU overhead of th…
54 …r a point lookup falls back to binary seek (fall back ratio) due to the collision. So a small util…
66 …to perform the comparison. Key string “16” and “0x10” is equal to each other as seen by this `Stri…
68 We add a new function member to the comparator interface:
77 …to use the hash index feature, one should 1) have a comparator that can never treat different keys…
89 …ts point lookup. We do not support range lookup. Range lookup request will fall back to BinarySeek.
101 For records not supported, the searching process will fall back to the traditional binary seek.
[all …]
|
| D | 2015-07-23-dynamic-level.markdown | 10 In this article, we follow up on the first part of an answer to one of the questions in our [AMA](h…
14 …ions)). In RocksDB we introduced parallelism and more configurable options to it but the main algo…
17 …to different sorted runs, called levels. Each level has a target size. Usually target size of lev…
20 …s its good space efficiency. We usually use the metric space amplification to measure the space ef…
23 …to the last level, the size of level will be equal as the size of last level before the compaction…
26 …to make sure the last level’s size is 1000GB, the same as the level’s size target? A user has to f…
29 …to be 10, and the DB size is 200GB. The target size of the last level is automatically set to be t…
|
| D | 2016-04-26-rocksdb-4-5-1-released.markdown | 23 …ill not be measured by default. If you want to have them enabled, you need to set Statistics::stat…
27 * ldb tool now supports operations to non-default column families.
28 …* Add kPersistedTier to ReadTier. This option allows Get and MultiGet to read only the persited da…
29 …tFileManager() in include/rocksdb/sst_file_manager.h to create a SstFileManager that can be used t…
41 …ite_buffer_number > 3, writes will be slowed down when writing to the last write buffer to delay a…
42 …* Introduce CompactionJobInfo::compaction_reason, this field include the reason to trigger the com…
44 * Increase default options.delayed_write_rate to 2MB/s.
45 …* Added a new parameter --path to ldb tool. --path accepts the name of either MANIFEST, SST or a W…
53 …s new member function called IgnoreSnapshots which allows CompactionFilter to be called even if th…
56 …ions_util.h. This function checks whether the input set of options is able to open the specified D…
|
| D | 2015-01-16-migrating-from-leveldb-to-rocksdb-2.markdown | 2 title: Migrating from LevelDB to RocksDB
7 - /blog/1811/migrating-from-leveldb-to-rocksdb-2/
10 …to migrate to using RocksDB, one problem you need to overcome is to map the options for LevelDB to…
84 …to other structures in different components. For example, block_size is not available in struct ro…
112 The LevelDBOptions utility can ease migration to RocksDB from LevelDB and allows us to break down t…
|
| D | 2017-12-19-write-prepared-txn.markdown | 8 …to provide isolation between the transactions. The default write policy in pessimistic transaction…
12 …to the memtable only after the transaction commits. This greatly simplifies the read path as any d…
16 …to commit the transaction if later is requested), and commit phase, where `::Commit` is invoked an…
20 These are the primary design questions that needs to be addressed:
25 …to the WAL (write-ahead log) as well as to the memtable(s) (one memtable per column family); We re…
27 … maintains some other data structures to cover the corner cases for transactions that takes abnorm…
|
| D | 2015-02-27-write-batch-with-index.markdown | 10 …ed to be enforced by concurrency controls on top of RocksDB; Atomicity can be implemented by commi…
14 …abases on top of RocksDB need to maintain an internal buffer for all the written keys, and when a …
16 …to a `WriteBatch` by calling `write_batch.Put("key1", "value1")` or `write_batch.Delete("key2")`, …
18 …to it. All the writes of the transaction go to the `WriteBatchWithIndex` first. When we commit the…
|
| D | 2014-06-27-avoid-expensive-locks-in-get.markdown | 12 RocksDB employs a multiversion concurrency control strategy. Before reading data, it needs to grab …
16 At the beginning of `GetImpl()`, it used to do this:
27 … throughput beyond 8 cores. Running 32 read threads on a 32-core CPU leads to [70% system CPU usag…
32 …to circumvent this problem by using [thread local storage](http://en.wikipedia.org/wiki/Thread-loc…
54 The result is quite amazing. RocksDB can nicely [scale to 32 cores](https://github.com/facebook/roc…
59 … in which all readers compete to access the critical area. Our approach prevents mutual competitio…
64 …resources (e.g., memtables, files) which belong to that version are frozen. A “supervisor” is requ…
69 (1) A reader thread uses CAS to acquire SuperVersion from its local storage and to put in a special…
74 (2) Upon completion of GetImpl(), the reader thread tries to return SuperVersion to local storage b…
|
| D | 2014-09-12-cuckoo.markdown | 12 …an, merge operator, etc. But, the existing RocksDB file formats were built to support range scan a…
18 * Reduce memory accesses per lookup to 1 or 2
21 * Get an end to end point lookup rate of at least 4 Mqps
46 In order to achieve high lookup speeds, we did multiple optimizations, including a cache friendly c…
50 …If all the locations are blocked, we try to move one of the colliding keys to a different location…
52 …to displace in order to accommodate the new key is naturally a shortest path problem in a directed…
56 Our goal is to minimize average (and maximum) number of hash functions required and hence the numbe…
64 …* If the key is not present in first hash location, we jump to second hash location which may not …
71 …to insert more keys near first location. In case of collision in the first hash location - _h1(k)_…
74 …alone can achieve 9.6 Mqps and we are working on improving it further. End to end RocksDB performa…
|
| /rocksdb-6.9/docs/ |
| D | LICENSE-DOCUMENTATION | 25 intended for use by those authorized to give the public
100 that applies to Your use of the Licensed Material.
106 g. Licensed Rights means the rights granted to You subject to the
114 i. Share means to provide material to the public by any means or
156 and to make technical modifications necessary to do so. The
194 extent necessary to allow You to exercise the Licensed
205 reserves any right to collect such royalties.
233 v. a URI or hyperlink to the Licensed Material to the
241 hyperlink to, this Public License.
261 apply to Your use of the Licensed Material:
[all …]
|
| D | CONTRIBUTING.md | 1 This provides guidance on how to contribute various content to `rocksdb.org`.
5 You should only have to do these one time.
7 - Rename this file to `CONTRIBUTING.md`.
9 - Rename `EXAMPLE-LICENSE` to `LICENSE`.
78 1. Push your changes to GitHub.
84 To add docs to the site....
88 1. Update `_data/nav_docs.yml` to add your new document to the navigation bar. Use the `docid` you …
90 1. Push your changes to GitHub.
103 1. If you want a visible link to that file, update `_data/nav.yml` to add a link to your new top-le…
105 …is not necessary if you just want to have a page that is linked to from another page, but not expo…
[all …]
|
| /rocksdb-6.9/tools/rdb/ |
| D | API.md | 28 # key (string) - Which key to get the value of.
41 # key (string) - Which key to associate the value with.
42 # value (string) - The value to associate with the key.
55 # key (string) - Which key to delete the value of..
64 # this method to check if a value existed before the call to `delete`.
90 # successfully written to the database, or false otherwise.
107 # In addition to the endpoints of the range, the method is overloaded to
110 # begin (string) - First key in the range to compact.
111 # end (string) - Last key in the range to compact.
146 changes to.
[all …]
|
| /rocksdb-6.9/docs/_docs/ |
| D | faq.md | 14 …to be scalable to run on servers with many CPU cores, to efficiently use fast storage, to support …
22 …rocess was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that …
29 * A spam detection application that needs fast access to big data sets.
30 * A graph-search query that needs to scan a data set in realtime.
31 * A cache data from Hadoop, thereby allowing applications to query Hadoop data in realtime.
36 …aph query engine part of the social graph infrastructure, is using RocksDB to store data. Parse ha…
38 RocksDB is proving to be a useful component for a lot of other groups in the industry. For a list o…
44 …to achieve 2x better compression and 10x less write amplification for our benchmarks compared to o…
48 …ocksDB for their use-cases. We would also like to engage with the academic community on topics rel…
|