Changes in Makefile

49866ce7 - netfs: Add support for caching single monolithic objects such as AFS dirs

Mon, 16 Dec 2024 20:41:09 +0000

netfs: Add support for caching single monolithic objects such as AFS dirsAdd support for caching the content of a file that contains a singlemonolithic object that must be read/written with a single I/O operation,such as an AFS directory.Signed-off-by: David Howells Link: https://lore.kernel.org/r/20241216204124.3752367-20-dhowells@redhat.comcc: Jeff Layton cc: Marc Dionne cc: netfs@lists.linux.devcc: linux-afs@lists.infradead.orgcc: linux-fsdevel@vger.kernel.orgSigned-off-by: Christian Brauner List of files: /linux-6.15/fs/netfs/Makefile

751e213f - netfs: Split retry code out of fs/netfs/write_collect.c

Mon, 16 Dec 2024 20:40:57 +0000

netfs: Split retry code out of fs/netfs/write_collect.cSplit write-retry code out of fs/netfs/write_collect.c as it will becomemore elaborate when content crypto is introduced.Signed-off-by: David Howells Link: https://lore.kernel.org/r/20241216204124.3752367-8-dhowells@redhat.comcc: Jeff Layton cc: netfs@lists.linux.devcc: linux-fsdevel@vger.kernel.orgSigned-off-by: Christian Brauner List of files: /linux-6.15/fs/netfs/Makefile

06fa229c - netfs: Abstract out a rolling folio buffer implementation

Mon, 16 Dec 2024 20:40:55 +0000

netfs: Abstract out a rolling folio buffer implementationA rolling buffer is a series of folios held in a list of folio_queues. Newfolios and folio_queue structs may be inserted at the head simultaneouslywith spent ones being removed from the tail without the need for locking.The rolling buffer includes an iov_iter and it has to be careful managingthis as the list of folio_queues is extended such that an oops doesn'tincurred because the iterator was pointing to the end of a folio_queuesegment that got appended to and then removed.We need to use the mechanism twice, once for read and once for write, and,in future patches, we will use a second rolling buffer to handle bouncebuffering for content encryption.Signed-off-by: David Howells Link: https://lore.kernel.org/r/20241216204124.3752367-6-dhowells@redhat.comcc: Jeff Layton cc: netfs@lists.linux.devcc: linux-fsdevel@vger.kernel.orgSigned-off-by: Christian Brauner List of files: /linux-6.15/fs/netfs/Makefile

ee4cdf7b - netfs: Speed up buffered reading

Mon, 01 Jul 2024 23:40:22 +0000

netfs: Speed up buffered readingImprove the efficiency of buffered reads in a number of ways: (1) Overhaul the algorithm in general so that it's a lot more compact and split the read submission code between buffered and unbuffered versions. The unbuffered version can be vastly simplified. (2) Read-result collection is handed off to a work queue rather than being done in the I/O thread. Multiple subrequests can be processes simultaneously. (3) When a subrequest is collected, any folios it fully spans are collected and "spare" data on either side is donated to either the previous or the next subrequest in the sequence.Notes: (*) Readahead expansion is massively slows down fio, presumably because it causes a load of extra allocations, both folio and xarray, up front before RPC requests can be transmitted. (*) RDMA with cifs does appear to work, both with SIW and RXE. (*) PG_private_2-based reading and copy-to-cache is split out into its own file and altered to use folio_queue. Note that the copy to the cache now creates a new write transaction against the cache and adds the folios to be copied into it. This allows it to use part of the writeback I/O code.Signed-off-by: David Howells cc: Jeff Layton cc: netfs@lists.linux.devcc: linux-fsdevel@vger.kernel.orgLink: https://lore.kernel.org/r/20240814203850.2240469-20-dhowells@redhat.com/ # v2Signed-off-by: Christian Brauner List of files: /linux-6.15/fs/netfs/Makefile

2df86547 - netfs: Cut over to using new writeback code

Fri, 08 Mar 2024 12:36:05 +0000

netfs: Cut over to using new writeback codeCut over to using the new writeback code. The old code is #ifdef'd out orotherwise removed from compilation to avoid conflicts and will be removedin a future patch.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: Eric Van Hensbergen cc: Latchesar Ionkov cc: Dominique Martinet cc: Christian Schoenebeck cc: Marc Dionne cc: v9fs@lists.linux.devcc: linux-afs@lists.infradead.orgcc: netfs@lists.linux.devcc: linux-fsdevel@vger.kernel.org List of files: /linux-6.15/fs/netfs/Makefile

288ace2f - netfs: New writeback implementation

Mon, 18 Mar 2024 16:52:05 +0000

netfs: New writeback implementationThe current netfslib writeback implementation creates writeback requests ofcontiguous folio data and then separately tiles subrequests over the spacetwice, once for the server and once for the cache. This creates a fewissues: (1) Every time there's a discontiguity or a change between writing to only one destination or writing to both, it must create a new request. This makes it harder to do vectored writes. (2) The folios don't have the writeback mark removed until the end of the request - and a request could be hundreds of megabytes. (3) In future, I want to support a larger cache granularity, which will require aggregation of some folios that contain unmodified data (which only need to go to the cache) and some which contain modifications (which need to be uploaded and stored to the cache) - but, currently, these are treated as discontiguous.There's also a move to get everyone to use writeback_iter() to extractwritable folios from the pagecache. That said, currently writeback_iter()has some issues that make it less than ideal: (1) there's no way to cancel the iteration, even if you find a "temporary" error that means the current folio and all subsequent folios are going to fail; (2) there's no way to filter the folios being written back - something that will impact Ceph with it's ordered snap system; (3) and if you get a folio you can't immediately deal with (say you need to flush the preceding writes), you are left with a folio hanging in the locked state for the duration, when really we should unlock it and relock it later.In this new implementation, I use writeback_iter() to pump folios,progressively creating two parallel, but separate streams and cleaning upthe finished folios as the subrequests complete. Either or both streamscan contain gaps, and the subrequests in each stream can be of variablesize, don't need to align with each other and don't need to align with thefolios.Indeed, subrequests can cross folio boundaries, may cover several folios ora folio may be spanned by multiple folios, e.g.: +---+---+-----+-----+---+----------+Folios: | | | | | | | +---+---+-----+-----+---+----------+ +------+------+ +----+----+Upload: | | |.....| | | +------+------+ +----+----+ +------+------+------+------+------+Cache: | | | | | | +------+------+------+------+------+The progressive subrequest construction permits the algorithm to bepreparing both the next upload to the server and the next write to thecache whilst the previous ones are already in progress. Throttling can beapplied to control the rate of production of subrequests - and, in anycase, we probably want to write them to the server in ascending order,particularly if the file will be extended.Content crypto can also be prepared at the same time as the subrequests andrun asynchronously, with the prepped requests being stalled until thecrypto catches up with them. This might also be useful for transportcrypto, but that happens at a lower layer, so probably would be harder topull off.The algorithm is split into three parts: (1) The issuer. This walks through the data, packaging it up, encrypting it and creating subrequests. The part of this that generates subrequests only deals with file positions and spans and so is usable for DIO/unbuffered writes as well as buffered writes. (2) The collector. This asynchronously collects completed subrequests, unlocks folios, frees crypto buffers and performs any retries. This runs in a work queue so that the issuer can return to the caller for writeback (so that the VM can have its kswapd thread back) or async writes. (3) The retryer. This pauses the issuer, waits for all outstanding subrequests to complete and then goes through the failed subrequests to reissue them. This may involve reprepping them (with cifs, the credits must be renegotiated, and a subrequest may need splitting), and doing RMW for content crypto if there's a conflicting change on the server.[!] Note that some of the functions are prefixed with "new_" to avoidclashes with existing functions. These will be renamed in a later patchthat cuts over to the new algorithm.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: Eric Van Hensbergen cc: Latchesar Ionkov cc: Dominique Martinet cc: Christian Schoenebeck cc: Marc Dionne cc: v9fs@lists.linux.devcc: linux-afs@lists.infradead.orgcc: netfs@lists.linux.devcc: linux-fsdevel@vger.kernel.org List of files: /linux-6.15/fs/netfs/Makefile

153a9961 - netfs: Implement unbuffered/DIO write support

Mon, 21 Feb 2022 11:38:17 +0000

netfs: Implement unbuffered/DIO write supportImplement support for unbuffered writes and direct I/O writes. If thewrite is misaligned with respect to the fscrypt block size, then RMW cyclesare performed if necessary. DIO writes are a special case of unbufferedwrites with extra restriction imposed, such as block size alignmentrequirements.Also provide a field that can tell the code to add some extra space ontothe bounce buffer for use by the filesystem in the case of acontent-encrypted file.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comcc: linux-fsdevel@vger.kernel.orgcc: linux-mm@kvack.org List of files: /linux-6.15/fs/netfs/Makefile

016dc851 - netfs: Implement unbuffered/DIO read support

Fri, 14 Jan 2022 17:39:55 +0000

netfs: Implement unbuffered/DIO read supportImplement support for unbuffered and DIO reads in the netfs library,utilising the existing read helper code to do block splitting andindividual queuing. The code also handles extraction of the destinationbuffer from the supplied iterator, allowing async unbuffered reads to takeplace.The read will be split up according to the rsize setting and, if supplied,the ->clamp_length() method. Note that the next subrequest will be issuedas soon as issue_op returns, without waiting for previous ones to finish.The network filesystem needs to pause or handle queuing them if it doesn'twant to fire them all at the server simultaneously.Once all the subrequests have finished, the state will be assessed and theamount of data to be indicated as having being obtained will bedetermined. As the subrequests may finish in any order, if an intermediatesubrequest is short, any further subrequests may be copied into the bufferand then abandoned.In the future, this will also take care of doing an unbuffered read fromencrypted content, with the decryption being done by the library.Signed-off-by: David Howells cc: Jeff Layton cc: linux-cachefs@redhat.comcc: linux-fsdevel@vger.kernel.orgcc: linux-mm@kvack.org List of files: /linux-6.15/fs/netfs/Makefile

c38f4e96 - netfs: Provide func to copy data to pagecache for buffered write

Thu, 17 Jun 2021 12:09:21 +0000

netfs: Provide func to copy data to pagecache for buffered writeProvide a netfs write helper, netfs_perform_write() to buffer data to bewritten in the pagecache and mark the modified folios dirty.It will perform "streaming writes" for folios that aren't currentlyresident, if possible, storing data in partially modified folios that aremarked dirty, but not uptodate. It will also tag pages as belonging tofs-specific write groups if so directed by the filesystem.This is derived from generic_perform_write(), but doesn't use->write_begin() and ->write_end(), having that logic rolled in instead.Signed-off-by: David Howells cc: Jeff Layton cc: linux-cachefs@redhat.comcc: linux-fsdevel@vger.kernel.orgcc: linux-mm@kvack.org List of files: /linux-6.15/fs/netfs/Makefile

0e0f2dfe - netfs: Dispatch write requests to process a writeback slice

Tue, 29 Jun 2021 21:31:48 +0000

netfs: Dispatch write requests to process a writeback sliceDispatch one or more write reqeusts to process a writeback slice, where aslice is tailored more to logical block divisions within the file (such ascrypto blocks, an object layout or cache granules) than the protocol RPCmaximum capacity.The dispatch doesn't happen until throttling allows, at which point theentire writeback slice is processed and queued. A slice may be written tomultiple destinations (one or more servers and the local cache) and thewrites to each destination might be split up along different lines.The writeback slice holds the required folios pinned. An iov_iter isprovided in netfs_write_request that describes the buffer to be used. Thismay be part of the pagecache, may have auxiliary padding pages attached ormay be a bounce buffer resulting from crypto or compression. Consequently,the filesystem must not twiddle the folio markings directly.The following API is available to the filesystem: (1) The ->create_write_requests() method is called to ask the filesystem to create the requests it needs. This is passed the writeback slice to be processed. (2) The filesystem should then call netfs_create_write_request() to create the requests it needs. (3) Once a request is initialised, netfs_queue_write_request() can be called to dispatch it asynchronously, if not completed immediately. (4) netfs_write_request_completed() should be called to note the completion of a request. (5) netfs_get_write_request() and netfs_put_write_request() are provided to refcount a request. These take constants from the netfs_wreq_trace enum for logging into ftrace. (6) The ->free_write_request is method is called to ask the filesystem to clean up a request.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comcc: linux-fsdevel@vger.kernel.orgcc: linux-mm@kvack.org List of files: /linux-6.15/fs/netfs/Makefile

46ed60dc - netfs: Implement unbuffered/DIO vs buffered I/O locking

Wed, 11 Oct 2023 14:34:07 +0000

netfs: Implement unbuffered/DIO vs buffered I/O lockingBorrow NFS's direct-vs-buffered I/O locking into netfslib. Similar code isalso used in ceph.Modify it to have the correct checker annotations for i_rwsem lockacquisition/release and to return -ERESTARTSYS if waits are interrupted.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comcc: linux-fsdevel@vger.kernel.orgcc: linux-mm@kvack.org List of files: /linux-6.15/fs/netfs/Makefile

c9c4ff12 - netfs: Move pinning-for-writeback from fscache to netfs

Mon, 27 Nov 2023 13:58:07 +0000

netfs: Move pinning-for-writeback from fscache to netfsMove the resource pinning-for-writeback from fscache code to netfslib code.This is used to keep a cache backing object pinned whilst we have dirtypages on the netfs inode in the pagecache such that VM writeback will beable to reach it.Whilst we're at it, switch the parameters of netfs_unpin_writeback() tomatch ->write_inode() so that it can be used for that directly.Note that this mechanism could be more generically useful than that fornetwork filesystems. Quite often they have to keep around other resources(e.g. authentication tokens or network connections) until the writeback iscomplete.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comcc: linux-fsdevel@vger.kernel.orgcc: linux-mm@kvack.org List of files: /linux-6.15/fs/netfs/Makefile

915cd30c - netfs, fscache: Combine fscache with netfs

Mon, 20 Nov 2023 15:55:18 +0000

netfs, fscache: Combine fscache with netfsNow that the fscache code is moved to be colocated with the netfslib codeso that they combined into one module, do the combining.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: Christian Brauner cc: linux-fsdevel@vger.kernel.orgcc: linux-cachefs@redhat.comcc: linux-nfs@vger.kernel.org,cc: linux-erofs@lists.ozlabs.org List of files: /linux-6.15/fs/netfs/Makefile

47757ea8 - netfs, fscache: Move fs/fscache/* into fs/netfs/

Mon, 20 Nov 2023 15:29:09 +0000

netfs, fscache: Move fs/fscache/* into fs/netfs/There's a problem with dependencies between netfslib and fscache as eachwants to access some functions of the other. Deal with this by movingfs/fscache/* into fs/netfs/ and renaming those files to begin with"fscache-".For the moment, the moved files are changed as little as possible and anfscache module is still built. A subsequent patch will integrate them.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: Christian Brauner cc: linux-fsdevel@vger.kernel.orgcc: linux-cachefs@redhat.com List of files: /linux-6.15/fs/netfs/Makefile

85dd2c8f - netfs: Add a function to extract a UBUF or IOVEC into a BVEC iterator

Thu, 20 Oct 2022 15:16:32 +0000

netfs: Add a function to extract a UBUF or IOVEC into a BVEC iteratorAdd a function to extract the pages from a user-space supplied iterator(UBUF- or IOVEC-type) into a BVEC-type iterator, retaining the pages bygetting a pin on them (as FOLL_PIN) as we go.This is useful in three situations: (1) A userspace thread may have a sibling that unmaps or remaps the process's VM during the operation, changing the assignment of the pages and potentially causing an error. Retaining the pages keeps some pages around, even if this occurs; futher, we find out at the point of extraction if EFAULT is going to be incurred. (2) Pages might get swapped out/discarded if not retained, so we want to retain them to avoid the reload causing a deadlock due to a DIO from/to an mmapped region on the same file. (3) The iterator may get passed to sendmsg() by the filesystem. If a fault occurs, we may get a short write to a TCP stream that's then tricky to recover from.We don't deal with other types of iterator here, leaving it to othermechanisms to retain the pages (eg. PG_locked, PG_writeback and the pipelock).Signed-off-by: David Howells cc: Jeff Layton cc: Steve French cc: Shyam Prasad N cc: Rohith Surabattula cc: linux-cachefs@redhat.comcc: linux-cifs@vger.kernel.orgcc: linux-fsdevel@vger.kernel.orgSigned-off-by: Steve French List of files: /linux-6.15/fs/netfs/Makefile

b900f4b8 - netfs: Split some core bits out into their own file

Tue, 01 Mar 2022 15:25:00 +0000

netfs: Split some core bits out into their own fileSplit some core bits out into their own file. More bits will be added tothis file later.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comLink: https://lore.kernel.org/r/164623006934.3564931.17932680017894039748.stgit@warthog.procyon.org.uk/ # v1Link: https://lore.kernel.org/r/164678218407.1200972.1731208226140990280.stgit@warthog.procyon.org.uk/ # v2Link: https://lore.kernel.org/r/164692920944.2099075.11990502173226013856.stgit@warthog.procyon.org.uk/ # v3 List of files: /linux-6.15/fs/netfs/Makefile

16211268 - netfs: Split fs/netfs/read_helper.c

Tue, 01 Mar 2022 14:35:58 +0000

netfs: Split fs/netfs/read_helper.cSplit fs/netfs/read_helper.c into two pieces, one to deal with bufferedwrites and one to deal with the I/O mechanism.Changes=======ver #2) - Add kdoc reference to new file.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comLink: https://lore.kernel.org/r/164623005586.3564931.6149556072728481767.stgit@warthog.procyon.org.uk/ # v1Link: https://lore.kernel.org/r/164678217075.1200972.5101072043126828757.stgit@warthog.procyon.org.uk/ # v2Link: https://lore.kernel.org/r/164692919953.2099075.7156989585513833046.stgit@warthog.procyon.org.uk/ # v3 List of files: /linux-6.15/fs/netfs/Makefile

3be01750 - netfs: Rename read_helper.c to io.c

Mon, 07 Mar 2022 21:57:24 +0000

netfs: Rename read_helper.c to io.cRename the read_helper.c file to io.c before splitting out the bufferedread functions and some other bits.Changes=======ver #2) - Rename read_helper.c before splitting.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comLink: https://lore.kernel.org/r/164678216109.1200972.16567696909952495832.stgit@warthog.procyon.org.uk/ # v2Link: https://lore.kernel.org/r/164692918076.2099075.8120961172717347610.stgit@warthog.procyon.org.uk/ # v3 List of files: /linux-6.15/fs/netfs/Makefile

3a4a38e6 - netfs: Split netfs_io_* object handling out

Thu, 17 Feb 2022 13:30:38 +0000

netfs: Split netfs_io_* object handling outSplit netfs_io_* object handling out into a file that's going to containobject allocation, get and put routines.Signed-off-by: David Howells Reviewed-by: Jeff Layton cc: linux-cachefs@redhat.comLink: https://lore.kernel.org/r/164622995118.3564931.6089530629052064470.stgit@warthog.procyon.org.uk/ # v1Link: https://lore.kernel.org/r/164678197044.1200972.11511937252083343775.stgit@warthog.procyon.org.uk/ # v2Link: https://lore.kernel.org/r/164692894693.2099075.7831091294248735173.stgit@warthog.procyon.org.uk/ # v3 List of files: /linux-6.15/fs/netfs/Makefile

289af54c - netfs: Gather stats

Tue, 03 Nov 2020 11:32:41 +0000

netfs: Gather statsGather statistics from the netfs interface that can be exported through aseqfile. This is intended to be called by a later patch when viewing/proc/fs/fscache/stats.Signed-off-by: David Howells Reviewed-and-tested-by: Jeff Layton Tested-by: Dave Wysochanski Tested-By: Marc Dionne cc: Matthew Wilcox cc: linux-mm@kvack.orgcc: linux-cachefs@redhat.comcc: linux-afs@lists.infradead.orgcc: linux-nfs@vger.kernel.orgcc: linux-cifs@vger.kernel.orgcc: ceph-devel@vger.kernel.orgcc: v9fs-developer@lists.sourceforge.netcc: linux-fsdevel@vger.kernel.orgLink: https://lore.kernel.org/r/161118139247.1232039.10556850937548511068.stgit@warthog.procyon.org.uk/ # rfcLink: https://lore.kernel.org/r/161161034669.2537118.2761232524997091480.stgit@warthog.procyon.org.uk/ # v2Link: https://lore.kernel.org/r/161340397101.1303470.17581910581108378458.stgit@warthog.procyon.org.uk/ # v3Link: https://lore.kernel.org/r/161539539959.286939.6794352576462965914.stgit@warthog.procyon.org.uk/ # v4Link: https://lore.kernel.org/r/161653797700.2770958.5801990354413178228.stgit@warthog.procyon.org.uk/ # v5Link: https://lore.kernel.org/r/161789079281.6155.17141344853277186500.stgit@warthog.procyon.org.uk/ # v6 List of files: /linux-6.15/fs/netfs/Makefile