bcachefs: Use sort_nonatomic() instead of sort()

Fixes "task out to lunch" warnings during recovery on large machines
with lots of dirty data in the journal.

Signed-off-by: Kent Overstreet <kent.ov

bcachefs: Use sort_nonatomic() instead of sort()

Fixes "task out to lunch" warnings during recovery on large machines
with lots of dirty data in the journal.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: Kill btree_iter.trans

This was planned to be done ages ago, now finally completed; there are
places where we have quite a few btree_trans objects on the stack, so
this reduces stack usage

bcachefs: Kill btree_iter.trans

This was planned to be done ages ago, now finally completed; there are
places where we have quite a few btree_trans objects on the stack, so
this reduces stack usage somewhat.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: Change btree wb assert to runtime error

We just had a report of the assert for "btree in write buffer for
non-write buffer btree" popping during the 6.14 upgrade.

- 150TB filesystem, afte

bcachefs: Change btree wb assert to runtime error

We just had a report of the assert for "btree in write buffer for
non-write buffer btree" popping during the 6.14 upgrade.

- 150TB filesystem, after a reboot the upgrade was able to continue from
where it left off, so no major damage.

But with 6.14 about to come out we want to get this tracked down asap,
and need more data if other users hit this.

Convert the BUG_ON() to an emergency read-only, and print out btree, the
key itself, and stack trace from the original write buffer update (which
did not have this check before).

Reported-by: Stijn Tintel <[email protected]>
Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: Assert that btree write buffer only touches the right btrees

More asserts, more better.

Also, clean up the per-btree flags a bit.

Signed-off-by: Kent Overstreet <[email protected]

bcachefs: Assert that btree write buffer only touches the right btrees

More asserts, more better.

Also, clean up the per-btree flags a bit.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: trace_write_buffer_maybe_flush

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: btree_write_buffer_flush_seq() no longer closes journal

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: trivial btree write buffer refactoring

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: bch2_btree_write_buffer_flush_going_ro()

The write buffer needs to be specifically flushed when going RO: keys in
the journal that haven't yet been moved to the write buffer don't have a
j

bcachefs: bch2_btree_write_buffer_flush_going_ro()

The write buffer needs to be specifically flushed when going RO: keys in
the journal that haven't yet been moved to the write buffer don't have a
journal pin yet.

This fixes numerous syzbot bugs, all with symptoms of still doing writes
after we've got RO.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: silence silly kdoc warning

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: Eytzinger accumulation for accounting keys

The btree write buffer takes as input keys from the journal, sorts them,
deduplicates them, and flushes them back to the btree in sorted order.

bcachefs: Eytzinger accumulation for accounting keys

The btree write buffer takes as input keys from the journal, sorts them,
deduplicates them, and flushes them back to the btree in sorted order.

The disk space accounting rewrite is moving accounting to normal btree
keys, with update (in this case deltas) accumulated in the write buffer
and then flushed to the btree; but this is going to increase the number
of keys handled by the write buffer by perhaps as much as a factor of
3x-5x.

The overhead from copying around and sorting this many keys would cause
a significant performance regression, but: there is huge locality in
updates to accounting keys that we can take advantage of.

Instead of appending accounting keys to the list of keys to be sorted,
this patch adds an eytzinger search tree of recently seen accounting
keys. We look up the accounting key in the eytzinger search tree and
apply the delta directly, adding it if it doesn't exist, and
periodically prune the eytzinger tree of unused entries.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: btree write buffer knows how to accumulate bch_accounting keys

Teach the btree write buffer how to accumulate accounting keys - instead
of having the newer key overwrite the older key as w

bcachefs: btree write buffer knows how to accumulate bch_accounting keys

Teach the btree write buffer how to accumulate accounting keys - instead
of having the newer key overwrite the older key as we do with other
updates, we need to add them together.

Also, add a flag so that write buffer flush knows when journal replay is
finished flushing accounting, and teach it to hold accounting keys until
that flag is set.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: bch2_btree_write_buffer_maybe_flush()

Add a new helper for checking references to write buffer btrees, where
we need a flush before we definitively know we have an inconsistency.

Signed-o

bcachefs: bch2_btree_write_buffer_maybe_flush()

Add a new helper for checking references to write buffer btrees, where
we need a flush before we definitively know we have an inconsistency.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: iter/update/trigger/str_hash flag cleanup

Combine iter/update/trigger/str_hash flags into a single enum, and
x-macroize them for a to_text() function later.

These flags are all for a spec

bcachefs: iter/update/trigger/str_hash flag cleanup

Combine iter/update/trigger/str_hash flags into a single enum, and
x-macroize them for a to_text() function later.

These flags are all for a specific iter/key/update context, so it makes
sense to group them together - iter/update/trigger flags were already
given distinct bits, this cleans up and unifies that handling.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: Fix btree node merging on write buffer btrees

The btree write buffer flush fastpath that avoids the main transaction
commit path had the unfortunate side effect of not doing btree node
mer

bcachefs: Fix btree node merging on write buffer btrees

The btree write buffer flush fastpath that avoids the main transaction
commit path had the unfortunate side effect of not doing btree node
merging.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: Fix journal pins in btree write buffer

btree write buffer flush has two phases
- in natural key order, which is more efficient but may fail
- then in journal order

The journal order flu

bcachefs: Fix journal pins in btree write buffer

btree write buffer flush has two phases
- in natural key order, which is more efficient but may fail
- then in journal order

The journal order flush was assuming that keys were still correctly
ordered by journal sequence number - but due to coalescing by the
previous phase, we need an additional sort.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: Improve bch2_fatal_error()

error messages should always include __func__

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: jset_entry for loops declare loop iter

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: Fix journal_buf bitfield accesses

All jounal_buf bitfield updates must happen under the journal lock -
perhaps we should just switch these to atomic bit flags.

Signed-off-by: Kent Overstr

bcachefs: Fix journal_buf bitfield accesses

All jounal_buf bitfield updates must happen under the journal lock -
perhaps we should just switch these to atomic bit flags.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: Prep work for variable size btree node buffers

bcachefs btree nodes are big - typically 256k - and btree roots are
pinned in memory. As we're now up to 18 btrees, we now have significant
m

bcachefs: Prep work for variable size btree node buffers

bcachefs btree nodes are big - typically 256k - and btree roots are
pinned in memory. As we're now up to 18 btrees, we now have significant
memory overhead in mostly empty btree roots.

And in the future we're going to start enforcing that certain btree node
boundaries exist, to solve lock contention issues - analagous to XFS's
AGIs.

Thus, we need to start allocating smaller btree node buffers when we
can. This patch changes code that refers to the filesystem constant
c->opts.btree_node_size to refer to the btree node buffer size -
btree_buf_bytes() - where appropriate.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...

bcachefs: __bch2_journal_key_to_wb -> bch2_journal_key_to_wb_slowpath

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: wb_key_cmp -> wb_key_ref_cmp

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: btree_iter -> btree_path_idx_t

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: bch2_btree_path_make_mut() -> btree_path_idx_t

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: darray_for_each() now declares loop iter

Signed-off-by: Kent Overstreet <[email protected]>

bcachefs: Inline btree write buffer sort

The sort in the btree write buffer flush path is a very hot path, and
it's particularly performance sensitive since it's single threaded and
can block every

bcachefs: Inline btree write buffer sort

The sort in the btree write buffer flush path is a very hot path, and
it's particularly performance sensitive since it's single threaded and
can block every other thread on a multithreaded write workload.

It's well worth doing a sort with inlined cmp and swap functions.

Signed-off-by: Kent Overstreet <[email protected]>

show more ...