Home
last modified time | relevance | path

Searched refs:files (Results 1 – 25 of 168) sorted by relevance

1234567

/rocksdb-6.9/docs/_posts/
D2017-06-26-17-level-based-changes.markdown10 …vides an option to limit the number of L0 files, which bounds read-amplification. Since L0 files (…
12 …ys user writes. This slows down accumulation of files in L0, and frees up resources for compacting…
14 …files can span the entire key-range, compaction parallelization is limited. Files at L0 or L1 may …
16 …L0. It quickly reduces file count in L0 and can be scheduled even when L1 files are locked, unlike…
20 …ecial property of L0->L1 compaction is that files can overlap in the input level, so those overlap…
30 …files are created after the compaction started. When the compaction is picked, the fourth L0 file …
32 …files in L0 or L1 prevent us from parallelizing compactions. When locked files block L0->L1 compac…
40 …files within L0. This allows us to start the work for eliminating L0 files earlier. It also lets u…
48 …files. That span begins at the newest L0 file, and expands towards older files as long as they're …
50 … picking algorithm to start from the oldest file and expand towards newer files as long as they're…
[all …]
D2014-04-21-indexing-sst-files-for-better-lookup-performance.markdown7 - /blog/431/indexing-sst-files-for-better-lookup-performance/
10 …mutable memtable, list of immutable memtables, and SST files to look up the target key. SST files …
12 On level 0, files are sorted based on the time they are flushed. Their key range (as defined by Fil…
16 …files from an upper level and merges them with files from lower level. As a result, key/values are…
18 …file's position in its level is fixed. Furthermore, its order relative to files from the next leve…
22 …files and level 2 has 8 files. Now, we want to look up key 80. A binary search based FileMetaData.…
24 …e have derived hints that target key is in range between 200 and 300. Any files on level 2 that ca…
26 …files that point to a range of files on level 2. For example, file 1 on level 1 points to file 3 (…
D2014-03-27-how-to-backup-rocksdb.markdown41 …ackup back to "/tmp/rocksdb". The second parameter is the location of log files (In some DBs they …
73 …og, and etc). It is used to make sure files are kept sound in the file system. Checksum is also ve…
92 …files to the backup directory. Doing so will prevent log files from being copied to the backup dir…
111 2. Get live files (this includes table files, current and manifest file).
115 …files to the backup directory. Since table files are immutable and filenames unique, we don't copy…
119 …o false, we also need to copy log files to the backup directory. We call `GetSortedWalFiles()` and…
128 …files than `LATEST_BACKUP` claims there are. In that case, we will delete any backup newer than `L…
D2014-03-27-how-to-persist-in-memory-rocksdb-database.markdown18 Options::wal_dir is the directory where RocksDB stores write-ahead log files. If you configure this…
19 …d log files. If the timeout is non-zero, obsolete log files will be moved to `archive/` directory …
21 …ackup the current snapshot of the database (containing table and metadata files) with a frequency …
23 You should configure the backup process to avoid backing up log files, since they are already store…
25 …files in the backup, we need to make sure that restoring the database doesn't delete log files in …
54 …ing [PlainTable format](https://github.com/facebook/rocksdb/wiki/PlainTable-Format) for table files
D2016-01-29-compaction_pri.markdown10 …s key-range partitioned into many files. When size of a level exceeds its target size, we pick one…
14 …files in round robin manner. We implemented multi-thread compaction in RocksDB by picking multiple…
16 …se from? Because there are different factors to consider when picking the files, and we now don't …
20 …files in the next level so we need to rewrite more data. For example, assume a file is 100MB, if a…
40 …files to compact may impact space efficiency. We don't yet have a a compaction priority to optimiz…
D2015-11-10-use-checkpoints-for-efficient-snapshots.markdown15 …files will be hard-linked, otherwise SST files will be copied. The manifest and CURRENT files will…
40 …ne DB. When opened read/write, the SST files continue to be hard links and these links are removed…
D2017-02-17-bulkoad-ingest-sst-file.markdown2 title: Bulkloading by ingesting external SST files
44 …nerate SST files and ingesting them into RocksDB in this [wiki page](https://github.com/facebook/r…
48 - Generating SST files in offline jobs in Hadoop, then downloading and ingesting the SST files into…
/rocksdb-6.9/db/compaction/
Dcompaction.cc60 for (size_t i = 0; i < files.size() && files[i]; i++) { in TotalFileSize()
61 sum += files[i]->fd.GetFileSize(); in TotalFileSize()
82 if (inputs[i].files.empty()) { in GetBoundaryKeys()
87 for (const auto* f : inputs[i].files) { in GetBoundaryKeys()
134 const auto* f = inputs[i].files[j]; in PopulateWithAtomicBoundaries()
152 add_unit_boundary(inputs[i].files.size()); in PopulateWithAtomicBoundaries()
153 assert(inputs[i].files.size() == in PopulateWithAtomicBoundaries()
168 if (inputs[0].files.back() == file) { in IsBottommostLevel()
371 auto* f = files[level_ptrs->at(lvl)]; in KeyNotExistsBeyondOutputLevel()
431 for (auto f : input_level.files) { in CalculateTotalInputSize()
[all …]
Dcompaction_picker.cc31 for (size_t i = 0; i < files.size() && files[i]; i++) { in TotalCompensatedFileSize()
32 sum += files[i]->compensated_file_size; in TotalCompensatedFileSize()
324 if (files[i]->being_compacted) { in AreFilesInCompaction()
544 inputs->files = expanded_inputs.files; in SetupOtherInputs()
609 auto& files = inputs[level - start_level].files; in CompactRange() local
611 files.push_back(f); in CompactRange()
613 if (AreFilesInCompaction(files)) { in CompactRange()
679 inputs.files.resize(i + 1); in CompactRange()
711 inputs.files.swap(inputs_shrunk); in CompactRange()
1100 start_level_inputs->files.clear(); in PickFilesMarkedForCompaction()
[all …]
Dcompaction_job_test.cc311 compaction_level.files.insert(compaction_level.files.end(),
394 ASSERT_EQ(2U, files.size()); in TEST_F()
404 RunCompaction({files}, expected_results); in TEST_F()
424 RunCompaction({files}, expected_results); in TEST_F()
442 RunCompaction({files}, expected_results); in TEST_F()
464 RunCompaction({files}, expected_results); in TEST_F()
517 RunCompaction({files}, expected_results); in TEST_F()
541 RunCompaction({files}, expected_results); in TEST_F()
568 RunCompaction({files}, expected_results); in TEST_F()
604 RunCompaction({files}, expected_results); in TEST_F()
[all …]
Dcompaction.h55 std::vector<FileMetaData*> files; member
57 inline bool empty() const { return files.empty(); } in empty()
58 inline size_t size() const { return files.size(); } in size()
59 inline void clear() { files.clear(); } in clear()
60 inline FileMetaData* operator[](size_t i) const { return files[i]; }
146 return &inputs_[compaction_input_level].files; in inputs()
382 extern uint64_t TotalFileSize(const std::vector<FileMetaData*>& files);
Dcompaction_picker_fifo.cc22 uint64_t GetTotalFilesSize(const std::vector<FileMetaData*>& files) { in GetTotalFilesSize() argument
24 for (const auto& f : files) { in GetTotalFilesSize()
80 inputs[0].files.push_back(f); in PickTTLCompaction()
88 if (inputs[0].files.empty() || in PickTTLCompaction()
94 for (const auto& f : inputs[0].files) { in PickTTLCompaction()
178 inputs[0].files.push_back(f); in PickSizeCompaction()
Dcompaction_picker_level.cc147 start_level_inputs_.files = {level_file.second}; in PickExpiredTtlFiles()
160 start_level_inputs_.files.clear(); in PickExpiredTtlFiles()
180 start_level_inputs_.files = {level_file.second}; in PickFilesMarkedForPeriodicCompaction()
193 start_level_inputs_.files.clear(); in PickFilesMarkedForPeriodicCompaction()
268 start_level_inputs_.files = {level_and_file.second}; in SetupInitialFiles()
491 start_level_inputs_.files.push_back(f); in PickFileToCompact()
514 &output_level_inputs.files); in PickFileToCompact()
Dcompaction_picker_universal.cc196 void GetSmallestLargestSeqno(const std::vector<FileMetaData*>& files, in GetSmallestLargestSeqno() argument
200 for (FileMetaData* f : files) { in GetSmallestLargestSeqno()
727 inputs[0].files.push_back(picking_file); in PickCompactionToReduceSortedRuns()
729 auto& files = inputs[picking_sr.level - start_level].files; in PickCompactionToReduceSortedRuns() local
731 files.push_back(f); in PickCompactionToReduceSortedRuns()
865 start_level_inputs.files.clear(); in PickDeleteTriggeredCompaction()
872 start_level_inputs.files.push_back(f); in PickDeleteTriggeredCompaction()
989 inputs[0].files.push_back(f); in PickCompactionToOldest()
991 auto& files = inputs[picking_sr.level - start_level].files; in PickCompactionToOldest() local
993 files.push_back(f); in PickCompactionToOldest()
/rocksdb-6.9/db/
Ddb_info_dumper.cc29 std::vector<std::string> files; in DumpDBFileSummary() local
36 if (!env->GetChildren(dbname, &files).ok()) { in DumpDBFileSummary()
40 std::sort(files.begin(), files.end()); in DumpDBFileSummary()
41 for (const std::string& file : files) { in DumpDBFileSummary()
77 if (!env->GetChildren(db_path.path, &files).ok()) { in DumpDBFileSummary()
83 std::sort(files.begin(), files.end()); in DumpDBFileSummary()
84 for (const std::string& file : files) { in DumpDBFileSummary()
101 if (!env->GetChildren(options.wal_dir, &files).ok()) { in DumpDBFileSummary()
108 for (const std::string& file : files) { in DumpDBFileSummary()
Dfile_indexer_test.cc48 : kNumLevels(4), files(new std::vector<FileMetaData*>[kNumLevels]) {} in FileIndexerTest()
52 delete[] files; in ~FileIndexerTest()
59 files[level].push_back(f); in AddFile()
68 for (auto* f : files[i]) { in ClearFiles()
71 files[i].clear(); in ClearFiles()
90 std::vector<FileMetaData*>* files; member in ROCKSDB_NAMESPACE::FileIndexerTest
97 indexer->UpdateIndex(&arena, 0, files); in TEST_F()
117 indexer->UpdateIndex(&arena, kNumLevels, files); in TEST_F()
157 indexer->UpdateIndex(&arena, kNumLevels, files); in TEST_F()
198 indexer->UpdateIndex(&arena, kNumLevels, files); in TEST_F()
[all …]
Dimport_column_family_test.cc95 metadata.files.push_back( in TEST_F()
116 metadata.files.push_back( in TEST_F()
194 metadata.files.push_back( in TEST_F()
196 metadata.files.push_back( in TEST_F()
198 metadata.files.push_back( in TEST_F()
200 metadata.files.push_back( in TEST_F()
202 metadata.files.push_back( in TEST_F()
204 metadata.files.push_back( in TEST_F()
476 metadata.files.push_back( in TEST_F()
478 metadata.files.push_back( in TEST_F()
[all …]
Dwal_manager.cc48 Status WalManager::GetSortedWalFiles(VectorLogPtr& files) { in GetSortedWalFiles() argument
66 files.clear(); in GetSortedWalFiles()
71 s = GetSortedWalsOfType(archivedir, files, kArchivedLogFile); in GetSortedWalFiles()
81 if (!files.empty()) { in GetSortedWalFiles()
82 latest_archived_log_number = files.back()->LogNumber(); in GetSortedWalFiles()
87 files.reserve(files.size() + logs.size()); in GetSortedWalFiles()
90 files.push_back(std::move(log)); in GetSortedWalFiles()
163 std::vector<std::string> files; in PurgeObsoleteWALFiles() local
164 s = env_->GetChildren(archival_dir, &files); in PurgeObsoleteWALFiles()
175 for (auto& f : files) { in PurgeObsoleteWALFiles()
Dversion_set.cc72 const auto &b = file_level.files; in FindFileInRange()
125 files_(files), in FilePicker()
136 (void)files; in FilePicker()
774 size_t num = files.size(); in DoGenerateLevelFilesBrief()
791 f.fd = files[i]->fd; in DoGenerateLevelFilesBrief()
792 f.file_metadata = files[i]; in DoGenerateLevelFilesBrief()
2681 for (auto& file : files) { in SortFileByOverlappingRatio()
3010 const FdWithKeyRange* files = level_files_brief_[level].files; in GetOverlappingInputsRangeBinarySearch() local
3032 files); in GetOverlappingInputsRangeBinarySearch()
3056 std::upper_bound(files + start_index, files + num_files, end, cmp) - in GetOverlappingInputsRangeBinarySearch()
[all …]
Dwal_manager_test.cc167 std::vector<std::string> files; in GetLogDirSize() local
168 env->GetChildren(dir_path, &files); in GetLogDirSize()
169 for (auto& f : files) { in GetLogDirSize()
183 std::vector<std::string> files; in ListSpecificFiles() local
185 env->GetChildren(path, &files); in ListSpecificFiles()
188 for (size_t i = 0; i < files.size(); ++i) { in ListSpecificFiles()
189 if (ParseFileName(files[i], &number, &type)) { in ListSpecificFiles()
Ddb_test_util.cc1017 num_files += level.files.size(); in TotalLiveFiles()
1090 std::vector<std::string> files; in CountFiles() local
1179 env->GetChildren(path, files); in GetSstFiles()
1181 files->erase( in GetSstFiles()
1182 std::remove_if(files->begin(), files->end(), [](std::string name) { in GetSstFiles()
1186 }), files->end()); in GetSstFiles()
1190 std::vector<std::string> files; in GetSstFileCount() local
1381 std::vector<std::string> files; in GetAllSSTFiles() local
1383 for (auto& file_name : files) { in GetAllSSTFiles()
1401 std::vector<std::string> files; in ListTableFiles() local
[all …]
Dcompacted_db_impl.cc32 return static_cast<size_t>(std::lower_bound(files_.files, in FindFile()
33 files_.files + right, key, cmp) - files_.files); in FindFile()
42 files_.files[FindFile(key)].fd.table_reader->Get(options, lkey.internal_key(), in Get()
55 const FdWithKeyRange& f = files_.files[FindFile(key)]; in MultiGet()
Dfile_indexer.cc76 std::vector<FileMetaData*>* const files) { in UpdateIndex() argument
77 if (files == nullptr) { in UpdateIndex()
97 const auto& upper_files = files[level]; in UpdateIndex()
99 const auto& lower_files = files[level + 1]; in UpdateIndex()
140 static_cast<int32_t>(files[num_levels_ - 1].size()) - 1; in UpdateIndex()
/rocksdb-6.9/java/src/main/java/org/rocksdb/
DLevelMetaData.java17 private final SstFileMetaData[] files; field in LevelMetaData
23 final SstFileMetaData[] files) { in LevelMetaData() argument
26 this.files = files; in LevelMetaData()
53 public List<SstFileMetaData> files() { in files() method in LevelMetaData
54 return Arrays.asList(files); in files()
/rocksdb-6.9/table/
Dmock_table.cc72 auto it = file_system_.files.find(id); in NewTableReader()
73 if (it == file_system_.files.end()) { in NewTableReader()
102 file_system_.files.insert({id, std::move(file_contents)}); in CreateMockTable()
124 ASSERT_EQ(file_system_.files.size(), 1U); in AssertSingleFile()
125 ASSERT_EQ(file_contents, file_system_.files.begin()->second); in AssertSingleFile()
130 ASSERT_GE(file_system_.files.size(), 1U); in AssertLatestFile()
131 auto latest = file_system_.files.end(); in AssertLatestFile()

1234567