| /rocksdb-6.9/docs/_posts/ |
| D | 2017-12-19-write-prepared-txn.markdown | 8 …in pessimistic transactions is _WriteCommitted_, which means that the data is written to the DB, i…
12 …in memory in the meanwhile. This makes memory a bottleneck for large transactions. The delay of th…
16 …in the read snapshot of the transaction. _WritePrepared_ would still have the issue of buffering t…
18 ### _WritePrepared_ in a nutshell
21 1) How do we identify the key/values in the DB with transactions that wrote them?
22 2) How do we figure if a key/value written by transaction Txn_w is in the read snapshot of the read…
25 …in a write batch object in memory. When 2PC `::Prepare` is called, it writes the in-memory write b…
27 …in the cache must be enough for almost all th transactions that commit in a timely manner. When ev…
30 Here we presents the improvements observed in MyRocks with sysbench and linkbench:
|
| D | 2018-11-21-delete-range.markdown | 11 ### Deletion patterns in LSM
23 to other nodes in the cluster.
44 in cases where readers must not see keys in deleted ranges. Further, it has the
57 completely contained in the deleted range. That saves write-amp because, in
94 them in the SST files themselves - but not in the data blocks, as explained for
105 *When there are range tombstones in an SST, they are segregated in a separate meta-block*
124 representation is always skiplist in order to minimize overhead in the usual
180 Iterating in a database with v1 range tombstones is usually slower than in a
195 used in production (e.g., scan-and-delete).
224 scan (if possible), and query each encountered key in this structure as in the old design,
[all …]
|
| D | 2014-09-12-cuckoo.markdown | 32 * The database is operated in read only mode
38 …r primary goal, compaction and build times were secondary. We may work on improving them in future.
41 Details for setting up the table format can be found in [GitHub](https://github.com/facebook/rocksd…
50 …_, we compute hashes of the key _h1(k), ..., __hn__(k)_. We insert the key in the first hash locat…
52 …in order to accommodate the new key is naturally a shortest path problem in a directed graph where…
54 To retrieve a key _k_, we compute hashes, _h1(k), ..., __hn__(k)_ and the key must be present in on…
56 …s 1.8 and maximum is 3. Around 44% of keys are accommodated in first hash location and 33% in seco…
61 We noticed the following two sub-optimal properties in original Cuckoo implementation:
64 … If the key is not present in first hash location, we jump to second hash location which may not b…
67 …* Because only 44% of keys are located in first cuckoo block, we couldn't have an optimal prefetch…
[all …]
|
| D | 2014-03-27-how-to-persist-in-memory-rocksdb-database.markdown | 2 title: How to persist in-memory RocksDB database?
7 - /blog/245/how-to-persist-in-memory-rocksdb-database/
10 …in-memory workloads. With growing RAM sizes and strict low-latency requirements, lots of applicati…
14 In this article we will explain how you can recover your in-memory RocksDB database even after a ma…
16 …in-memory data structure called memtable and second is write-ahead log. Write-ahead log can be use…
23 …ackup process to avoid backing up log files, since they are already stored in persistent storage. …
25 …in the backup, we need to make sure that restoring the database doesn't delete log files in WAL di…
|
| D | 2014-05-14-lock.markdown | 2 title: Reducing Lock Contention in RocksDB
25 …us from scaling to more than 16 threads. To address the issue, we improved RocksDB in several ways.
33 …in read queries. After consolidating reference counting to one single super version and removing t…
36 We identified in two situations, we might do disk I/O inside mutex and we removed them:
41 …in our case). Malloc sometimes is slow because it needs to lock some shared data structures. Alloc…
47 6. Deal with mutexes in LRU caches.
48 …in both of read and write operations. LRU caches are used in block cache and table cache. Both of …
72 1. for all the files except manifest, add a hardlink file in your new directory pointing to the ori…
77 By the way, the best way to ask those questions is in our [facebook group](https://www.facebook.com…
81 Will this consistency problem of RocksDB all occurs in case of single put/write?
[all …]
|
| D | 2017-02-17-bulkoad-ingest-sst-file.markdown | 12 - We create a new entry for the new key/value in our in-memory structure (memtable / SkipList by de…
13 …MB for example), we convert this memtable to a SST file, and put this file in level 0 of our LSM-T…
14 - Later, compaction will kick in and move data from level 0 to level 1, and then from level 1 to le…
23 This is bulk-loading, and in specific use-cases it allow users to achieve faster data loading and b…
44 You can find more details about how to generate SST files and ingesting them into RocksDB in this […
48 - Generating SST files in offline jobs in Hadoop, then downloading and ingesting the SST files into…
49 - Migrating shards between machines by dumping key-range in SST File and loading the file in a diff…
50 - Migrating from a different storage (InnoDB to RocksDB migration in MyRocks)
|
| D | 2017-08-25-flushwal.markdown | 8 …ks benefit from this API to gain higher write throughput with however no compromise in reliability.
12 …ing `DBOptions.concurrent_prepare`, MyRocks achieves 40% higher throughput in Sysbench's [update-n…
16 …uffer in the next step is "flushed" to OS buffer by calling fwrite syscall. The OS buffer is later…
24 …in place to provide reliability. MySQL for example uses 2PC (two-phase commit) to write to both bi…
26 …in recovering from some failure scenarios, MySQL can provide reliability without however needing a…
|
| D | 2017-06-26-17-level-based-changes.markdown | 10 …nlike files at lower levels) can span the entire key-range, a key might be in any file, thus reads…
12 …DB intentionally delays user writes. This slows down accumulation of files in L0, and frees up res…
14 …llelization is limited. Files at L0 or L1 may be locked due to involvement in pending L0->L1 or L1…
16 …, we added a new type of compaction, L0->L0. It quickly reduces file count in L0 and can be schedu…
20 …le in the level. One special property of L0->L1 compaction is that files can overlap in the input …
24 This compaction pulls in every L0 and L1 file. This happens regardless of which L0 file is initiall…
26 Users may insert their data less uniformly in the key-range. For example, a database may look like …
30 …rth L0 file and six rightmost L1 files are pulled in due to overlap. Notice this leaves the databa…
40 …also lets us do less work since we don't pull in any L1 files, whereas L0->L1 compaction would've …
54 Now, there can never be L0 files unreachable for L0->L0 due to L0->L1 selecting files in the middle…
[all …]
|
| D | 2019-03-08-format-version-4.markdown | 8 The data blocks in RocksDB consist of a sequence of key/values pairs sorted by key, where the pairs…
12 …ly reduces the index block size, in some cases around 4-5x. This frees more space in block cache, …
24 …sary for keys in the index blocks. In such cases, `format_version`=3 skips encoding the sequence n…
27 …in each restart interval. If used, `TableProperties::index_value_is_delta_encoded` is set, which i…
29 The index format in `format_version=4` would be as follows:
35 where, k is key, v is value, and its encoding is in parenthesis.
|
| D | 2014-06-27-avoid-expensive-locks-in-get.markdown | 2 title: Avoid Expensive Locks in Get()
7 - /blog/677/avoid-expensive-locks-in-get/
10 As promised in the previous [blog post](blog/2014/05/14/lock.html)!
12 … reading data, it needs to grab the current version, which is encapsulated in a data structure cal…
24 …_version_ may be updated, the corresponding SuperVersion may be deleted while Ref() is in progress.
27 …ns “this” pointer. However, this simple operation posed big challenges for in-memory workload and …
54 …ages/talks/2014-03-27-RocksDB-Meetup-Lei-Lockless-Get.pdf) and most CPU time is spent in user land.
59 …ies beyond what is shown in the assembly code. Mutex can keep threads spinning on CPU or even trig…
64 … once but can never come back again. SuperVersion is referenced and cached in its thread local sto…
69 (1) A reader thread uses CAS to acquire SuperVersion from its local storage and to put in a special…
[all …]
|
| D | 2014-09-12-new-bloom-filter-format.markdown | 12 …in-memory (all data stored in memory, files stored in tmpfs/ramfs, an [example](https://github.com…
19 …a bits array generated for a set of keys that could tell if an arbitrary key may exist in that set.
21 …s to the bloom filter block of SST file. If key may exist in filter, we goes into data block in SS…
25 …in SST file. It has complex structure (ref [here](https://github.com/facebook/rocksdb/wiki/Rocksdb…
27 Here's the work flow for checking original bloom filter in block based table:
35 New bloom filter creates filter for all keys in SST file and we name it "full filter". The data str…
|
| D | 2016-01-29-compaction_pri.markdown | 14 … files in round robin manner. We implemented multi-thread compaction in RocksDB by picking multipl…
20 …in the next level so we need to rewrite more data. For example, assume a file is 100MB, if an L2 f…
24 … effect. With this mode, we always pick the file covers the oldest updates in the level, which usu…
28 …in previous analysis. However, in many use cases, there are subset of keys that are frequently upd…
30 …coldest range first, we leave the hot ranges in the level. If your use case is to overwrite existi…
36 Our default compaction priority **kByCompensatedSize** considers the case. If number of deletes in …
40 …ity to optimize this case. In some of our use cases, we solved the problem in a different way: we …
51 …spot.com/2016/02/compaction-priority-in-rocksdb.html](http://smalldatum.blogspot.com/2016/02/compa…
|
| D | 2015-11-10-use-checkpoints-for-efficient-snapshots.markdown | 10 …in RocksDB which provides the ability to take a snapshot of a running RocksDB database in a separa…
15 …in the specified directory. If the snapshot is on the same filesystem as the original database, th…
30 …the CreateCheckpoint function creates a consistent snapshot of the database in the given directory.
45 Checkpoints are used for online backup in MyRocks. which is MySQL using RocksDB as the storage eng…
|
| D | 2016-09-28-rocksdb-4-11-2-released.markdown | 7 …st release is stable. In 4.11.2, we fixed several data corruption related bugs introduced in 4.9.0.
23 …* Fix data race condition related to hash index in block based table when putting indexes in the b…
47 …id deleting files in foreground when destroying iterators. Instead, a job is scheduled in high pri…
48 …ssociates data with non-default column families using information embedded in the SST/WAL files (4…
|
| D | 2015-02-27-write-batch-with-index.markdown | 10 …can be implemented by committing a transaction's writes with one write batch to RocksDB in the end.
14 …in the end of the transaction in one batch, you cannot get the updated value from RocksDB previous…
16 …e now have `WriteBatchWithIndex`. Users can put updates to WriteBatchIndex in the same way as to `…
18 …in the `WriteBatchWithIndex` and return the value if existing, by seeking and reading from an iter…
20 Using `WriteBatchWithIndex`, we successfully implemented read-your-own-writes in the RocksDB storag…
|
| D | 2015-10-27-getthreadlist.markdown | 118 … thread 140716416169728 is doing compaction on the `picachu` column family in database `db2`. In …
126 To enable thread-tracking of a rocksdb instance, simply set `enable_thread_tracking` to true in its…
129 // If true, then the status of the threads involved in this DB will
141 The GetThreadList API is defined in [include/rocksdb/env.h](https://github.com/facebook/rocksdb/blo…
153 the current status of a thread. The `ThreadStatus` structure, defined in
166 // does not involve in any DB operation.
171 // in any column family.
177 // The elapsed time in micros of the current thread operation.
181 // in the current operation.
185 // operation. Same field in op_properties[] might have different
[all …]
|
| /rocksdb-6.9/ |
| D | HISTORY.md | 92 …n caused by output of intra-L0 compaction on ingested file not being placed in correct order in L0.
157 * Fixed a number of data races in BlobDB.
241 * Add command `list_file_range_deletes` in ldb, which prints out tombstones in SST files.
407 * Remove sync point in Block destructor.
421 * Fix a bug in misreporting the estimated partition index size in properties block.
590 * Add `DB::VerifyChecksum()`, which verifies the checksums in all SST files in a running DB.
689 * Fix wrong results in a data race case in Get()
691 * Fix bugs of data corruption in direct I/O
991 * Full ColumnFamily support in RocksJava.
1139 are interested in.
[all …]
|
| D | LICENSE.Apache | 36 "Work" shall mean the work of authorship, whether in Source or
38 copyright notice that is included in or attached to the work
39 (an example is provided in the Appendix below).
72 Work and such Derivative Works in Source or Object form.
91 Work or Derivative Works thereof in any medium, with or without
92 modifications, and in Source or Object form, provided that You
101 (c) You must retain, in the Source form of any Derivative Works
129 the conditions stated in this License.
145 agreed to in writing, Licensor provides the Work (and each
184 the brackets!) The text should be enclosed in the appropriate
[all …]
|
| D | README.md | 16 making it especially suitable for storing multiple terabytes of data in a
23 The public interface is in `include/`. Callers should not include or
24 rely on the details of any other header files in this package. Those
27 Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/
31 …er both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found …
|
| D | WINDOWS_PORT.md | 24 * make all unit test pass both in debug and release builds.
25 * Note: latest introduction of SyncPoint seems to disable running db_test in Release.
40 … shortcomings of C++11 support in the current version of the MS compiler. Most or all of them are …
44 * Certain headers that are not present and not necessary on Windows were simply `#ifndef OS_WIN` in…
49 * in class member initialization were moved to a __ctors in some cases
52 * Union members that have non-trivial constructors were replaced to `char[]` in one place along wit…
90 …t to implement these two features because the hosting program as a rule has these two things in it.
91 We experienced no inconveniences debugging issues in the debugger or analyzing process dumps if nee…
104 * The parameters used, unless specified, were exactly the same as published in the GitHub Wiki page…
108 #### Test 1. Bulk Load of keys in Random Order
[all …]
|
| /rocksdb-6.9/docs/ |
| D | CONTRIBUTING.md | 12 - Make sure you update `title`, `description`, `tagline` and `gacode` (Google Analytics) in `./_con…
16 Most content is written in markdown. You name the file `something.md`, then have a header that look…
68 To modify a blog post, edit the appopriate markdown file in `./_posts/`.
74 1. Create your blog post in `./_posts/` in markdown (file extension `.md` or `.markdown`). See curr…
75 …You can add a `<!--truncate-->` tag in the middle of your post such that you show only the excerpt…
82 To modify docs, edit the appropriate markdown file in `./_docs/`.
87 - You can use folders in the `./_docs/` directory to organize your content if you want.
88 … document to the navigation bar. Use the `docid` you put in your doc markdown in as the `id` in th…
98 To modify a top-level page, edit the appropriate markdown file in `./top-level/`
100 If you want a top-level page (e.g., http://your-site.com/top-level.html) -- not in `/blog/` or `/do…
[all …]
|
| D | LICENSE-DOCUMENTATION | 20 and certain other rights specified in the public license below. The
26 permission to use material in ways otherwise restricted by
72 and in which the Licensed Material is translated, altered,
78 synched in timed relation with a moving image.
81 and Similar Rights in Your contributions to Adapted Material in
126 equivalent rights anywhere in the world.
141 in part; and
185 provided in Section 3(a)(1)(A)(i).
243 2. You may satisfy the conditions in Section 3(a)(1) in any
268 contents in a database in which You have Sui Generis Database
[all …]
|
| D | TEMPLATE-INFORMATION.md | 5 …, and `og_image.png` (used for Like button stories and Shares on Facbeook) in the `static` folder …
7 … can safely remove them (or you can safely leave them and not include them in your navigation - Je…
11 …in three parts - the first in the homepage header, which is mostly automatically derived from the …
13 …in `index.md` which contains the bulk of the main content below the header. This is all markdown i…
15 …in the `_data/powered_by.yml` and `_data/powered_by_highlight.yml` files. Both these files combine…
|
| /rocksdb-6.9/third-party/folly/folly/lang/ |
| D | Launder.h | 24 FOLLY_NODISCARD inline T* launder(T* in) noexcept { in launder() argument
27 return __builtin_launder(in); in launder()
31 __asm__("" : "+r"(in)); in launder()
32 return in; in launder()
37 return in; in launder()
|
| /rocksdb-6.9/java/ |
| D | HISTORY-JAVA.md | 6 …ctor support. To use such extractor, simply call useCappedPrefixExtractor in either Options or Co…
12 * MemEnv is now available in RocksJava via RocksMemEnv class.
13 * lz4 compression is now included in rocksjava static library when running `make rocksdbjavastatic`.
46 * Added RocksObject --- the base class of all RocksDB classes which holds some RocksDB resources in…
47 * Use environmental variable JAVA_HOME in Makefile for RocksJava
49 …b.Iterator to org.rocksdb.RocksIterator to avoid potential confliction with Java built-in Iterator.
81 * Avoid a potential byte-array-copy between c++ and Java in RocksDB.get.
82 * Added SizeUnit in org.rocksdb.util to store consts like KB and GB.
|