admin/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2025-12-07 20:06:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'for-6.19/block-20251201' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux

Browse Source 
Pull block updates from Jens Axboe:

 - Fix head insertion for mq-deadline, a regression from when priority
   support was added

 - Series simplifying and improving the ublk user copy code

 - Various ublk related cleanups

 - Fixup REQ_NOWAIT handling in loop/zloop, clearing NOWAIT when the
   request is punted to a thread for handling

 - Merge and then later revert loop dio nowait support, as it ended up
   causing excessive stack usage for when the inline issue code needs to
   dip back into the full file system code

 - Improve auto integrity code, making it less deadlock prone

 - Speedup polled IO handling, but manually managing the hctx lookups

 - Fixes for blk-throttle for SSD devices

 - Small series with fixes for the S390 dasd driver

 - Add support for caching zones, avoiding unnecessary report zone
   queries

 - MD pull requests via Yu:
      - fix null-ptr-dereference regression for dm-raid0
      - fix IO hang for raid5 when array is broken with IO inflight
      - remove legacy 1s delay to speed up system shutdown
      - change maintainer's email address
      - data can be lost if array is created with different lbs devices,
        fix this problem and record lbs of the array in metadata
      - fix rcu protection for md_thread
      - fix mddev kobject lifetime regression
      - enable atomic writes for md-linear
      - some cleanups

 - bcache updates via Coly
      - remove useless discard and cache device code
      - improve usage of per-cpu workqueues

 - Reorganize the IO scheduler switching code, fixing some lockdep
   reports as well

 - Improve the block layer P2P DMA support

 - Add support to the block tracing code for zoned devices

 - Segment calculation improves, and memory alignment flexibility
   improvements

 - Set of prep and cleanups patches for ublk batching support. The
   actual batching hasn't been added yet, but helps shrink down the
   workload of getting that patchset ready for 6.20

 - Fix for how the ps3 block driver handles segments offsets

 - Improve how block plugging handles batch tag allocations

 - nbd fixes for use-after-free of the configuration on device clear/put

 - Set of improvements and fixes for zloop

 - Add Damien as maintainer of the block zoned device code handling

 - Various other fixes and cleanups

* tag 'for-6.19/block-20251201' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (162 commits)
  block/rnbd: correct all kernel-doc complaints
  blk-mq: use queue_hctx in blk_mq_map_queue_type
  md: remove legacy 1s delay in md_notify_reboot
  md/raid5: fix IO hang when array is broken with IO inflight
  md: warn about updating super block failure
  md/raid0: fix NULL pointer dereference in create_strip_zones() for dm-raid
  sbitmap: fix all kernel-doc warnings
  ublk: add helper of __ublk_fetch()
  ublk: pass const pointer to ublk_queue_is_zoned()
  ublk: refactor auto buffer register in ublk_dispatch_req()
  ublk: add `union ublk_io_buf` with improved naming
  ublk: add parameter `struct io_uring_cmd *` to ublk_prep_auto_buf_reg()
  kfifo: add kfifo_alloc_node() helper for NUMA awareness
  blk-mq: fix potential uaf for 'queue_hw_ctx'
  blk-mq: use array manage hctx map instead of xarray
  ublk: prevent invalid access with DEBUG
  s390/dasd: Use scnprintf() instead of sprintf()
  s390/dasd: Move device name formatting into separate function
  s390/dasd: Remove unnecessary debugfs_create() return checks
  s390/dasd: Fix gendisk parent after copy pair swap
  ...
This commit is contained in:
Linus Torvalds
2025-12-03 19:26:18 -08:00
parent 0abcfd8983 d211a28035
commit cc25df3e2e
108 changed files with 3050 additions and 1588 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
Documentation/ABI/testing/sysfs-block-bcache
View File

@@ -106,13 +106,6 @@ Description:
will be discarded from the cache. Should not be turned off with
writeback caching enabled.
What: /sys/block/<disk>/bcache/discard
Date: November 2010
Contact: Kent Overstreet <kent.overstreet@gmail.com>
Description:
For a cache, a boolean allowing discard/TRIM to be turned off
or back on if the device supports it.
What: /sys/block/<disk>/bcache/bucket_size
Date: November 2010
Contact: Kent Overstreet <kent.overstreet@gmail.com>

13
Documentation/admin-guide/bcache.rst
View File

@@ -17,8 +17,7 @@ The latest bcache kernel code can be found from mainline Linux kernel:
It's designed around the performance characteristics of SSDs - it only allocates
in erase block sized buckets, and it uses a hybrid btree/log to track cached
extents (which can be anywhere from a single sector to the bucket size). It's
designed to avoid random writes at all costs; it fills up an erase block
sequentially, then issues a discard before reusing it.
designed to avoid random writes at all costs.
Both writethrough and writeback caching are supported. Writeback defaults to
off, but can be switched on and off arbitrarily at runtime. Bcache goes to
@@ -618,19 +617,11 @@ bucket_size
cache_replacement_policy
One of either lru, fifo or random.
discard
Boolean; if on a discard/TRIM will be issued to each bucket before it is
reused. Defaults to off, since SATA TRIM is an unqueued command (and thus
slow).
freelist_percent
Size of the freelist as a percentage of nbuckets. Can be written to to
increase the number of buckets kept on the freelist, which lets you
artificially reduce the size of the cache at runtime. Mostly for testing
purposes (i.e. testing how different size caches affect your hit rate), but
since buckets are discarded when they move on to the freelist will also make
the SSD's garbage collection easier by effectively giving it more reserved
space.
purposes (i.e. testing how different size caches affect your hit rate).
io_errors
Number of errors that have occurred, decayed by io_error_halflife.

61
Documentation/admin-guide/blockdev/zoned_loop.rst
View File

@@ -68,30 +68,43 @@ The options available for the add command can be listed by reading the
In more details, the options that can be used with the "add" command are as
follows.
================ ===========================================================
id Device number (the X in /dev/zloopX).
Default: automatically assigned.
capacity_mb Device total capacity in MiB. This is always rounded up to
the nearest higher multiple of the zone size.
Default: 16384 MiB (16 GiB).
zone_size_mb Device zone size in MiB. Default: 256 MiB.
zone_capacity_mb Device zone capacity (must always be equal to or lower than
the zone size. Default: zone size.
conv_zones Total number of conventioanl zones starting from sector 0.
Default: 8.
base_dir Path to the base directory where to create the directory
containing the zone files of the device.
Default=/var/local/zloop.
The device directory containing the zone files is always
named with the device ID. E.g. the default zone file
directory for /dev/zloop0 is /var/local/zloop/0.
nr_queues Number of I/O queues of the zoned block device. This value is
always capped by the number of online CPUs
Default: 1
queue_depth Maximum I/O queue depth per I/O queue.
Default: 64
buffered_io Do buffered IOs instead of direct IOs (default: false)
================ ===========================================================
=================== =========================================================
id Device number (the X in /dev/zloopX).
Default: automatically assigned.
capacity_mb Device total capacity in MiB. This is always rounded up
to the nearest higher multiple of the zone size.
Default: 16384 MiB (16 GiB).
zone_size_mb Device zone size in MiB. Default: 256 MiB.
zone_capacity_mb Device zone capacity (must always be equal to or lower
than the zone size. Default: zone size.
conv_zones Total number of conventioanl zones starting from
sector 0
Default: 8
base_dir Path to the base directory where to create the directory
containing the zone files of the device.
Default=/var/local/zloop.
The device directory containing the zone files is always
named with the device ID. E.g. the default zone file
directory for /dev/zloop0 is /var/local/zloop/0.
nr_queues Number of I/O queues of the zoned block device. This
value is always capped by the number of online CPUs
Default: 1
queue_depth Maximum I/O queue depth per I/O queue.
Default: 64
buffered_io Do buffered IOs instead of direct IOs (default: false)
zone_append Enable or disable a zloop device native zone append
support.
Default: 1 (enabled).
If native zone append support is disabled, the block layer
will emulate this operation using regular write
operations.
ordered_zone_append Enable zloop mitigation of zone append reordering.
Default: disabled.
This is useful for testing file systems file data mapping
(extents), as when enabled, this can significantly reduce
the number of data extents needed to for a file data
mapping.
=================== =========================================================
3) Deleting a Zoned Device
--------------------------

10
Documentation/admin-guide/md.rst
View File

@@ -238,6 +238,16 @@ All md devices contain:
the number of devices in a raid4/5/6, or to support external
metadata formats which mandate such clipping.
logical_block_size
Configure the array's logical block size in bytes. This attribute
is only supported for 1.x meta. Write the value before starting
array. The final array LBS uses the maximum between this
configuration and LBS of all combined devices. Note that
LBS cannot exceed PAGE_SIZE before RAID supports folio.
WARNING: Arrays created on new kernel cannot be assembled at old
kernel due to padding check, Set module parameter 'check_new_feature'
to false to bypass, but data loss may occur.
reshape_position
This is either ``none`` or a sector number within the devices of
the array where ``reshape`` is up to. If this is set, the three

13
MAINTAINERS
View File

@@ -4307,7 +4307,7 @@ F: Documentation/filesystems/befs.rst
F: fs/befs/
BFQ I/O SCHEDULER
M: Yu Kuai <yukuai3@huawei.com>
M: Yu Kuai <yukuai@fnnas.com>
L: linux-block@vger.kernel.org
S: Odd Fixes
F: Documentation/block/bfq-iosched.rst
@@ -4407,6 +4407,8 @@ F: block/
F: drivers/block/
F: include/linux/bio.h
F: include/linux/blk*
F: include/uapi/linux/blk*
F: include/uapi/linux/ioprio.h
F: kernel/trace/blktrace.c
F: lib/sbitmap.c
@@ -23908,7 +23910,7 @@ F: include/linux/property.h
SOFTWARE RAID (Multiple Disks) SUPPORT
M: Song Liu <song@kernel.org>
M: Yu Kuai <yukuai3@huawei.com>
M: Yu Kuai <yukuai@fnnas.com>
L: linux-raid@vger.kernel.org
S: Supported
Q: https://patchwork.kernel.org/project/linux-raid/list/
@@ -28371,6 +28373,13 @@ L: linux-kernel@vger.kernel.org
S: Maintained
F: arch/x86/kernel/cpu/zhaoxin.c
ZONED BLOCK DEVICE (BLOCK LAYER)
M: Damien Le Moal <dlemoal@kernel.org>
L: linux-block@vger.kernel.org
S: Maintained
F: block/blk-zoned.c
F: include/uapi/linux/blkzoned.h
ZONED LOOP DEVICE
M: Damien Le Moal <dlemoal@kernel.org>
R: Christoph Hellwig <hch@lst.de>

26
block/bio-integrity-auto.c
View File

@@ -29,7 +29,7 @@ static void bio_integrity_finish(struct bio_integrity_data *bid)
{
bid->bio->bi_integrity = NULL;
bid->bio->bi_opf &= ~REQ_INTEGRITY;
kfree(bvec_virt(bid->bip.bip_vec));
bio_integrity_free_buf(&bid->bip);
mempool_free(bid, &bid_pool);
}
@@ -110,8 +110,6 @@ bool bio_integrity_prep(struct bio *bio)
struct bio_integrity_data *bid;
bool set_flags = true;
gfp_t gfp = GFP_NOIO;
unsigned int len;
void *buf;
if (!bi)
return true;
@@ -152,19 +150,12 @@ bool bio_integrity_prep(struct bio *bio)
if (WARN_ON_ONCE(bio_has_crypt_ctx(bio)))
return true;
/* Allocate kernel buffer for protection data */
len = bio_integrity_bytes(bi, bio_sectors(bio));
buf = kmalloc(len, gfp);
if (!buf)
goto err_end_io;
bid = mempool_alloc(&bid_pool, GFP_NOIO);
if (!bid)
goto err_free_buf;
bio_integrity_init(bio, &bid->bip, &bid->bvec, 1);
bid->bio = bio;
bid->bip.bip_flags |= BIP_BLOCK_INTEGRITY;
bio_integrity_alloc_buf(bio, gfp & __GFP_ZERO);
bip_set_seed(&bid->bip, bio->bi_iter.bi_sector);
if (set_flags) {
@@ -176,23 +167,12 @@ bool bio_integrity_prep(struct bio *bio)
bid->bip.bip_flags |= BIP_CHECK_REFTAG;
}
if (bio_integrity_add_page(bio, virt_to_page(buf), len,
offset_in_page(buf)) < len)
goto err_end_io;
/* Auto-generate integrity metadata if this is a write */
if (bio_data_dir(bio) == WRITE && bip_should_check(&bid->bip))
blk_integrity_generate(bio);
else
bid->saved_bio_iter = bio->bi_iter;
return true;
err_free_buf:
kfree(buf);
err_end_io:
bio->bi_status = BLK_STS_RESOURCE;
bio_endio(bio);
return false;
}
EXPORT_SYMBOL(bio_integrity_prep);

48
block/bio-integrity.c
View File

@@ -14,6 +14,45 @@ struct bio_integrity_alloc {
struct bio_vec bvecs[];
};
static mempool_t integrity_buf_pool;
void bio_integrity_alloc_buf(struct bio *bio, bool zero_buffer)
{
struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
struct bio_integrity_payload *bip = bio_integrity(bio);
unsigned int len = bio_integrity_bytes(bi, bio_sectors(bio));
gfp_t gfp = GFP_NOIO | (zero_buffer ? __GFP_ZERO : 0);
void *buf;
buf = kmalloc(len, (gfp & ~__GFP_DIRECT_RECLAIM) |
__GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN);
if (unlikely(!buf)) {
struct page *page;
page = mempool_alloc(&integrity_buf_pool, GFP_NOFS);
if (zero_buffer)
memset(page_address(page), 0, len);
bvec_set_page(&bip->bip_vec[0], page, len, 0);
bip->bip_flags |= BIP_MEMPOOL;
} else {
bvec_set_page(&bip->bip_vec[0], virt_to_page(buf), len,
offset_in_page(buf));
}
bip->bip_vcnt = 1;
bip->bip_iter.bi_size = len;
}
void bio_integrity_free_buf(struct bio_integrity_payload *bip)
{
struct bio_vec *bv = &bip->bip_vec[0];
if (bip->bip_flags & BIP_MEMPOOL)
mempool_free(bv->bv_page, &integrity_buf_pool);
else
kfree(bvec_virt(bv));
}
/**
* bio_integrity_free - Free bio integrity payload
* @bio: bio containing bip to be freed
@@ -438,3 +477,12 @@ int bio_integrity_clone(struct bio *bio, struct bio *bio_src,
return 0;
}
static int __init bio_integrity_initfn(void)
{
if (mempool_init_page_pool(&integrity_buf_pool, BIO_POOL_SIZE,
get_order(BLK_INTEGRITY_MAX_SIZE)))
panic("bio: can't create integrity buf pool\n");
return 0;
}
subsys_initcall(bio_integrity_initfn);

1
block/bio.c
View File

@@ -253,6 +253,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,
bio->bi_write_hint = 0;
bio->bi_write_stream = 0;
bio->bi_status = 0;
bio->bi_bvec_gap_bit = 0;
bio->bi_iter.bi_sector = 0;
bio->bi_iter.bi_size = 0;
bio->bi_iter.bi_idx = 0;

12
block/blk-core.c
View File

@@ -662,13 +662,13 @@ static void __submit_bio(struct bio *bio)
* bio_list of new bios to be added. ->submit_bio() may indeed add some more
* bios through a recursive call to submit_bio_noacct. If it did, we find a
* non-NULL value in bio_list and re-enter the loop from the top.
* - In this case we really did just take the bio of the top of the list (no
* - In this case we really did just take the bio off the top of the list (no
* pretending) and so remove it from bio_list, and call into ->submit_bio()
* again.
*
* bio_list_on_stack[0] contains bios submitted by the current ->submit_bio.
* bio_list_on_stack[1] contains bios that were submitted before the current
* ->submit_bio, but that haven't been processed yet.
* ->submit_bio(), but that haven't been processed yet.
*/
static void __submit_bio_noacct(struct bio *bio)
{
@@ -743,8 +743,8 @@ void submit_bio_noacct_nocheck(struct bio *bio, bool split)
/*
* We only want one ->submit_bio to be active at a time, else stack
* usage with stacked devices could be a problem. Use current->bio_list
* to collect a list of requests submited by a ->submit_bio method while
* it is active, and then process them after it returned.
* to collect a list of requests submitted by a ->submit_bio method
* while it is active, and then process them after it returned.
*/
if (current->bio_list) {
if (split)
@@ -901,7 +901,7 @@ static void bio_set_ioprio(struct bio *bio)
*
* submit_bio() is used to submit I/O requests to block devices. It is passed a
* fully set up &struct bio that describes the I/O that needs to be done. The
* bio will be send to the device described by the bi_bdev field.
* bio will be sent to the device described by the bi_bdev field.
*
* The success/failure status of the request, along with notification of
* completion, is delivered asynchronously through the ->bi_end_io() callback
@@ -991,7 +991,7 @@ int iocb_bio_iopoll(struct kiocb *kiocb, struct io_comp_batch *iob,
* point to a freshly allocated bio at this point. If that happens
* we have a few cases to consider:
*
* 1) the bio is beeing initialized and bi_bdev is NULL. We can just
* 1) the bio is being initialized and bi_bdev is NULL. We can just
* simply nothing in this case
* 2) the bio points to a not poll enabled device. bio_poll will catch
* this and return 0

6
block/blk-iocost.c
View File

@@ -2334,10 +2334,8 @@ static void ioc_timer_fn(struct timer_list *timer)
else
usage_dur = max_t(u64, now.now - ioc->period_at, 1);
usage = clamp_t(u32,
DIV64_U64_ROUND_UP(usage_us * WEIGHT_ONE,
usage_dur),
1, WEIGHT_ONE);
usage = clamp(DIV64_U64_ROUND_UP(usage_us * WEIGHT_ONE, usage_dur),
1, WEIGHT_ONE);
/*
* Already donating or accumulated enough to start.

6
block/blk-lib.c
View File

@@ -87,11 +87,11 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector,
{
struct bio *bio = NULL;
struct blk_plug plug;
int ret;
int ret = 0;
blk_start_plug(&plug);
ret = __blkdev_issue_discard(bdev, sector, nr_sects, gfp_mask, &bio);
if (!ret && bio) {
__blkdev_issue_discard(bdev, sector, nr_sects, gfp_mask, &bio);
if (bio) {
ret = submit_bio_wait(bio);
if (ret == -EOPNOTSUPP)
ret = 0;

3
block/blk-map.c
View File

@@ -459,6 +459,8 @@ int blk_rq_append_bio(struct request *rq, struct bio *bio)
if (rq->bio) {
if (!ll_back_merge_fn(rq, bio, nr_segs))
return -EINVAL;
rq->phys_gap_bit = bio_seg_gap(rq->q, rq->biotail, bio,
rq->phys_gap_bit);
rq->biotail->bi_next = bio;
rq->biotail = bio;
rq->__data_len += bio->bi_iter.bi_size;
@@ -469,6 +471,7 @@ int blk_rq_append_bio(struct request *rq, struct bio *bio)
rq->nr_phys_segments = nr_segs;
rq->bio = rq->biotail = bio;
rq->__data_len = bio->bi_iter.bi_size;
rq->phys_gap_bit = bio->bi_bvec_gap_bit;
return 0;
}
EXPORT_SYMBOL(blk_rq_append_bio);

44
block/blk-merge.c
View File

@@ -302,6 +302,12 @@ static unsigned int bio_split_alignment(struct bio *bio,
return lim->logical_block_size;
}
static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,
struct bio_vec *bv)
{
return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len);
}
/**
* bio_split_io_at - check if and where to split a bio
* @bio: [in] bio to be split
@@ -319,8 +325,8 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
unsigned *segs, unsigned max_bytes, unsigned len_align_mask)
{
struct bio_vec bv, bvprv, *bvprvp = NULL;
unsigned nsegs = 0, bytes = 0, gaps = 0;
struct bvec_iter iter;
unsigned nsegs = 0, bytes = 0;
bio_for_each_bvec(bv, bio, iter) {
if (bv.bv_offset & lim->dma_alignment ||
@@ -331,12 +337,15 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
* If the queue doesn't support SG gaps and adding this
* offset would create a gap, disallow it.
*/
if (bvprvp && bvec_gap_to_prev(lim, bvprvp, bv.bv_offset))
goto split;
if (bvprvp) {
if (bvec_gap_to_prev(lim, bvprvp, bv.bv_offset))
goto split;
gaps |= bvec_seg_gap(bvprvp, &bv);
}
if (nsegs < lim->max_segments &&
bytes + bv.bv_len <= max_bytes &&
bv.bv_offset + bv.bv_len <= lim->min_segment_size) {
bv.bv_offset + bv.bv_len <= lim->max_fast_segment_size) {
nsegs++;
bytes += bv.bv_len;
} else {
@@ -350,6 +359,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
}
*segs = nsegs;
bio->bi_bvec_gap_bit = ffs(gaps);
return 0;
split:
if (bio->bi_opf & REQ_ATOMIC)
@@ -385,6 +395,7 @@ split:
* big IO can be trival, disable iopoll when split needed.
*/
bio_clear_polled(bio);
bio->bi_bvec_gap_bit = ffs(gaps);
return bytes >> SECTOR_SHIFT;
}
EXPORT_SYMBOL_GPL(bio_split_io_at);
@@ -721,6 +732,24 @@ static bool blk_atomic_write_mergeable_rqs(struct request *rq,
return (rq->cmd_flags & REQ_ATOMIC) == (next->cmd_flags & REQ_ATOMIC);
}
u8 bio_seg_gap(struct request_queue *q, struct bio *prev, struct bio *next,
u8 gaps_bit)
{
struct bio_vec pb, nb;
if (!bio_has_data(prev))
return 0;
gaps_bit = min_not_zero(gaps_bit, prev->bi_bvec_gap_bit);
gaps_bit = min_not_zero(gaps_bit, next->bi_bvec_gap_bit);
bio_get_last_bvec(prev, &pb);
bio_get_first_bvec(next, &nb);
if (!biovec_phys_mergeable(q, &pb, &nb))
gaps_bit = min_not_zero(gaps_bit, ffs(bvec_seg_gap(&pb, &nb)));
return gaps_bit;
}
/*
* For non-mq, this has to be called with the request spinlock acquired.
* For mq with scheduling, the appropriate queue wide lock should be held.
@@ -785,6 +814,9 @@ static struct request *attempt_merge(struct request_queue *q,
if (next->start_time_ns < req->start_time_ns)
req->start_time_ns = next->start_time_ns;
req->phys_gap_bit = bio_seg_gap(req->q, req->biotail, next->bio,
min_not_zero(next->phys_gap_bit,
req->phys_gap_bit));
req->biotail->bi_next = next->bio;
req->biotail = next->biotail;
@@ -908,6 +940,8 @@ enum bio_merge_status bio_attempt_back_merge(struct request *req,
if (req->rq_flags & RQF_ZONE_WRITE_PLUGGING)
blk_zone_write_plug_bio_merged(bio);
req->phys_gap_bit = bio_seg_gap(req->q, req->biotail, bio,
req->phys_gap_bit);
req->biotail->bi_next = bio;
req->biotail = bio;
req->__data_len += bio->bi_iter.bi_size;
@@ -942,6 +976,8 @@ static enum bio_merge_status bio_attempt_front_merge(struct request *req,
blk_update_mixed_merge(req, bio, true);
req->phys_gap_bit = bio_seg_gap(req->q, bio, req->bio,
req->phys_gap_bit);
bio->bi_next = req->bio;
req->bio = bio;

29
block/blk-mq-dma.c
View File

@@ -79,8 +79,7 @@ static bool blk_map_iter_next(struct request *req, struct blk_map_iter *iter,
static inline bool blk_can_dma_map_iova(struct request *req,
struct device *dma_dev)
{
return !((queue_virt_boundary(req->q) + 1) &
dma_get_merge_boundary(dma_dev));
return !(req_phys_gap_mask(req) & dma_get_merge_boundary(dma_dev));
}
static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)
@@ -93,8 +92,13 @@ static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)
static bool blk_dma_map_direct(struct request *req, struct device *dma_dev,
struct blk_dma_iter *iter, struct phys_vec *vec)
{
iter->addr = dma_map_page(dma_dev, phys_to_page(vec->paddr),
offset_in_page(vec->paddr), vec->len, rq_dma_dir(req));
unsigned int attrs = 0;
if (iter->p2pdma.map == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE)
attrs |= DMA_ATTR_MMIO;
iter->addr = dma_map_phys(dma_dev, vec->paddr, vec->len,
rq_dma_dir(req), attrs);
if (dma_mapping_error(dma_dev, iter->addr)) {
iter->status = BLK_STS_RESOURCE;
return false;
@@ -109,14 +113,18 @@ static bool blk_rq_dma_map_iova(struct request *req, struct device *dma_dev,
{
enum dma_data_direction dir = rq_dma_dir(req);
unsigned int mapped = 0;
unsigned int attrs = 0;
int error;
iter->addr = state->addr;
iter->len = dma_iova_size(state);
if (iter->p2pdma.map == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE)
attrs |= DMA_ATTR_MMIO;
do {
error = dma_iova_link(dma_dev, state, vec->paddr, mapped,
vec->len, dir, 0);
vec->len, dir, attrs);
if (error)
break;
mapped += vec->len;
@@ -143,7 +151,7 @@ static inline void blk_rq_map_iter_init(struct request *rq,
.bi_size = rq->special_vec.bv_len,
}
};
} else if (bio) {
} else if (bio) {
*iter = (struct blk_map_iter) {
.bio = bio,
.bvecs = bio->bi_io_vec,
@@ -151,7 +159,7 @@ static inline void blk_rq_map_iter_init(struct request *rq,
};
} else {
/* the internal flush request may not have bio attached */
*iter = (struct blk_map_iter) {};
*iter = (struct blk_map_iter) {};
}
}
@@ -163,6 +171,7 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev,
memset(&iter->p2pdma, 0, sizeof(iter->p2pdma));
iter->status = BLK_STS_OK;
iter->p2pdma.map = PCI_P2PDMA_MAP_NONE;
/*
* Grab the first segment ASAP because we'll need it to check for P2P
@@ -174,10 +183,6 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev,
switch (pci_p2pdma_state(&iter->p2pdma, dma_dev,
phys_to_page(vec.paddr))) {
case PCI_P2PDMA_MAP_BUS_ADDR:
if (iter->iter.is_integrity)
bio_integrity(req->bio)->bip_flags |= BIP_P2P_DMA;
else
req->cmd_flags |= REQ_P2PDMA;
return blk_dma_map_bus(iter, &vec);
case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
/*
@@ -352,7 +357,7 @@ bool blk_rq_integrity_dma_map_iter_start(struct request *req,
EXPORT_SYMBOL_GPL(blk_rq_integrity_dma_map_iter_start);
/**
* blk_rq_integrity_dma_map_iter_start - map the next integrity DMA segment for
* blk_rq_integrity_dma_map_iter_next - map the next integrity DMA segment for
* a request
* @req: request to map
* @dma_dev: device to map to

120
block/blk-mq-sched.c
View File

@@ -427,11 +427,25 @@ void blk_mq_free_sched_tags(struct elevator_tags *et,
kfree(et);
}
void blk_mq_free_sched_tags_batch(struct xarray *et_table,
void blk_mq_free_sched_res(struct elevator_resources *res,
struct elevator_type *type,
struct blk_mq_tag_set *set)
{
if (res->et) {
blk_mq_free_sched_tags(res->et, set);
res->et = NULL;
}
if (res->data) {
blk_mq_free_sched_data(type, res->data);
res->data = NULL;
}
}
void blk_mq_free_sched_res_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set)
{
struct request_queue *q;
struct elevator_tags *et;
struct elv_change_ctx *ctx;
lockdep_assert_held_write(&set->update_nr_hwq_lock);
@@ -444,15 +458,48 @@ void blk_mq_free_sched_tags_batch(struct xarray *et_table,
* concurrently.
*/
if (q->elevator) {
et = xa_load(et_table, q->id);
if (unlikely(!et))
ctx = xa_load(elv_tbl, q->id);
if (!ctx) {
WARN_ON_ONCE(1);
else
blk_mq_free_sched_tags(et, set);
continue;
}
blk_mq_free_sched_res(&ctx->res, ctx->type, set);
}
}
}
void blk_mq_free_sched_ctx_batch(struct xarray *elv_tbl)
{
unsigned long i;
struct elv_change_ctx *ctx;
xa_for_each(elv_tbl, i, ctx) {
xa_erase(elv_tbl, i);
kfree(ctx);
}
}
int blk_mq_alloc_sched_ctx_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set)
{
struct request_queue *q;
struct elv_change_ctx *ctx;
lockdep_assert_held_write(&set->update_nr_hwq_lock);
list_for_each_entry(q, &set->tag_list, tag_set_list) {
ctx = kzalloc(sizeof(struct elv_change_ctx), GFP_KERNEL);
if (!ctx)
return -ENOMEM;
if (xa_insert(elv_tbl, q->id, ctx, GFP_KERNEL)) {
kfree(ctx);
return -ENOMEM;
}
}
return 0;
}
struct elevator_tags *blk_mq_alloc_sched_tags(struct blk_mq_tag_set *set,
unsigned int nr_hw_queues, unsigned int nr_requests)
{
@@ -466,8 +513,7 @@ struct elevator_tags *blk_mq_alloc_sched_tags(struct blk_mq_tag_set *set,
else
nr_tags = nr_hw_queues;
et = kmalloc(sizeof(struct elevator_tags) +
nr_tags * sizeof(struct blk_mq_tags *), gfp);
et = kmalloc(struct_size(et, tags, nr_tags), gfp);
if (!et)
return NULL;
@@ -498,12 +544,33 @@ out:
return NULL;
}
int blk_mq_alloc_sched_tags_batch(struct xarray *et_table,
int blk_mq_alloc_sched_res(struct request_queue *q,
struct elevator_type *type,
struct elevator_resources *res,
unsigned int nr_hw_queues)
{
struct blk_mq_tag_set *set = q->tag_set;
res->et = blk_mq_alloc_sched_tags(set, nr_hw_queues,
blk_mq_default_nr_requests(set));
if (!res->et)
return -ENOMEM;
res->data = blk_mq_alloc_sched_data(q, type);
if (IS_ERR(res->data)) {
blk_mq_free_sched_tags(res->et, set);
return -ENOMEM;
}
return 0;
}
int blk_mq_alloc_sched_res_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set, unsigned int nr_hw_queues)
{
struct elv_change_ctx *ctx;
struct request_queue *q;
struct elevator_tags *et;
gfp_t gfp = GFP_NOIO | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY;
int ret = -ENOMEM;
lockdep_assert_held_write(&set->update_nr_hwq_lock);
@@ -516,39 +583,44 @@ int blk_mq_alloc_sched_tags_batch(struct xarray *et_table,
* concurrently.
*/
if (q->elevator) {
et = blk_mq_alloc_sched_tags(set, nr_hw_queues,
blk_mq_default_nr_requests(set));
if (!et)
ctx = xa_load(elv_tbl, q->id);
if (WARN_ON_ONCE(!ctx)) {
ret = -ENOENT;
goto out_unwind;
}
ret = blk_mq_alloc_sched_res(q, q->elevator->type,
&ctx->res, nr_hw_queues);
if (ret)
goto out_unwind;
if (xa_insert(et_table, q->id, et, gfp))
goto out_free_tags;
}
}
return 0;
out_free_tags:
blk_mq_free_sched_tags(et, set);
out_unwind:
list_for_each_entry_continue_reverse(q, &set->tag_list, tag_set_list) {
if (q->elevator) {
et = xa_load(et_table, q->id);
if (et)
blk_mq_free_sched_tags(et, set);
ctx = xa_load(elv_tbl, q->id);
if (ctx)
blk_mq_free_sched_res(&ctx->res,
ctx->type, set);
}
}
return -ENOMEM;
return ret;
}
/* caller must have a reference to @e, will grab another one if successful */
int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e,
struct elevator_tags *et)
struct elevator_resources *res)
{
unsigned int flags = q->tag_set->flags;
struct elevator_tags *et = res->et;
struct blk_mq_hw_ctx *hctx;
struct elevator_queue *eq;
unsigned long i;
int ret;
eq = elevator_alloc(q, e, et);
eq = elevator_alloc(q, e, res);
if (!eq)
return -ENOMEM;

40
block/blk-mq-sched.h
View File

@@ -19,18 +19,52 @@ void __blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx);
void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx);
int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e,
struct elevator_tags *et);
struct elevator_resources *res);
void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e);
void blk_mq_sched_free_rqs(struct request_queue *q);
struct elevator_tags *blk_mq_alloc_sched_tags(struct blk_mq_tag_set *set,
unsigned int nr_hw_queues, unsigned int nr_requests);
int blk_mq_alloc_sched_tags_batch(struct xarray *et_table,
int blk_mq_alloc_sched_res(struct request_queue *q,
struct elevator_type *type,
struct elevator_resources *res,
unsigned int nr_hw_queues);
int blk_mq_alloc_sched_res_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set, unsigned int nr_hw_queues);
int blk_mq_alloc_sched_ctx_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set);
void blk_mq_free_sched_ctx_batch(struct xarray *elv_tbl);
void blk_mq_free_sched_tags(struct elevator_tags *et,
struct blk_mq_tag_set *set);
void blk_mq_free_sched_tags_batch(struct xarray *et_table,
void blk_mq_free_sched_res(struct elevator_resources *res,
struct elevator_type *type,
struct blk_mq_tag_set *set);
void blk_mq_free_sched_res_batch(struct xarray *et_table,
struct blk_mq_tag_set *set);
/*
* blk_mq_alloc_sched_data() - Allocates scheduler specific data
* Returns:
* - Pointer to allocated data on success
* - NULL if no allocation needed
* - ERR_PTR(-ENOMEM) in case of failure
*/
static inline void *blk_mq_alloc_sched_data(struct request_queue *q,
struct elevator_type *e)
{
void *sched_data;
if (!e || !e->ops.alloc_sched_data)
return NULL;
sched_data = e->ops.alloc_sched_data(q);
return (sched_data) ?: ERR_PTR(-ENOMEM);
}
static inline void blk_mq_free_sched_data(struct elevator_type *e, void *data)
{
if (e && e->ops.free_sched_data)
e->ops.free_sched_data(data);
}
static inline void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx)
{

2
block/blk-mq-tag.c
View File

@@ -499,7 +499,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn,
int srcu_idx;
/*
* __blk_mq_update_nr_hw_queues() updates nr_hw_queues and hctx_table
* __blk_mq_update_nr_hw_queues() updates nr_hw_queues and queue_hw_ctx
* while the queue is frozen. So we can use q_usage_counter to avoid
* racing with it.
*/

152
block/blk-mq.c
View File

@@ -376,6 +376,7 @@ void blk_rq_init(struct request_queue *q, struct request *rq)
INIT_LIST_HEAD(&rq->queuelist);
rq->q = q;
rq->__sector = (sector_t) -1;
rq->phys_gap_bit = 0;
INIT_HLIST_NODE(&rq->hash);
RB_CLEAR_NODE(&rq->rb_node);
rq->tag = BLK_MQ_NO_TAG;
@@ -467,21 +468,26 @@ __blk_mq_alloc_requests_batch(struct blk_mq_alloc_data *data)
unsigned long tag_mask;
int i, nr = 0;
tag_mask = blk_mq_get_tags(data, data->nr_tags, &tag_offset);
if (unlikely(!tag_mask))
return NULL;
do {
tag_mask = blk_mq_get_tags(data, data->nr_tags - nr, &tag_offset);
if (unlikely(!tag_mask)) {
if (nr == 0)
return NULL;
break;
}
tags = blk_mq_tags_from_data(data);
for (i = 0; tag_mask; i++) {
if (!(tag_mask & (1UL << i)))
continue;
tag = tag_offset + i;
prefetch(tags->static_rqs[tag]);
tag_mask &= ~(1UL << i);
rq = blk_mq_rq_ctx_init(data, tags, tag);
rq_list_add_head(data->cached_rqs, rq);
nr++;
}
} while (data->nr_tags > nr);
tags = blk_mq_tags_from_data(data);
for (i = 0; tag_mask; i++) {
if (!(tag_mask & (1UL << i)))
continue;
tag = tag_offset + i;
prefetch(tags->static_rqs[tag]);
tag_mask &= ~(1UL << i);
rq = blk_mq_rq_ctx_init(data, tags, tag);
rq_list_add_head(data->cached_rqs, rq);
nr++;
}
if (!(data->rq_flags & RQF_SCHED_TAGS))
blk_mq_add_active_requests(data->hctx, nr);
/* caller already holds a reference, add for remainder */
@@ -668,6 +674,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf,
goto out_queue_exit;
}
rq->__data_len = 0;
rq->phys_gap_bit = 0;
rq->__sector = (sector_t) -1;
rq->bio = rq->biotail = NULL;
return rq;
@@ -723,7 +730,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
* If not tell the caller that it should skip this queue.
*/
ret = -EXDEV;
data.hctx = xa_load(&q->hctx_table, hctx_idx);
data.hctx = q->queue_hw_ctx[hctx_idx];
if (!blk_mq_hw_queue_mapped(data.hctx))
goto out_queue_exit;
cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
@@ -748,6 +755,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
rq = blk_mq_rq_ctx_init(&data, blk_mq_tags_from_data(&data), tag);
blk_mq_rq_time_init(rq, alloc_time_ns);
rq->__data_len = 0;
rq->phys_gap_bit = 0;
rq->__sector = (sector_t) -1;
rq->bio = rq->biotail = NULL;
return rq;
@@ -2674,6 +2682,8 @@ static void blk_mq_bio_to_request(struct request *rq, struct bio *bio,
rq->bio = rq->biotail = bio;
rq->__sector = bio->bi_iter.bi_sector;
rq->__data_len = bio->bi_iter.bi_size;
rq->phys_gap_bit = bio->bi_bvec_gap_bit;
rq->nr_phys_segments = nr_segs;
if (bio_integrity(bio))
rq->nr_integrity_segments = blk_rq_count_integrity_sg(rq->q,
@@ -3380,6 +3390,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src,
}
rq->nr_phys_segments = rq_src->nr_phys_segments;
rq->nr_integrity_segments = rq_src->nr_integrity_segments;
rq->phys_gap_bit = rq_src->phys_gap_bit;
if (rq->bio && blk_crypto_rq_bio_prep(rq, rq->bio, gfp_mask) < 0)
goto free_and_out;
@@ -3935,8 +3946,6 @@ static void blk_mq_exit_hctx(struct request_queue *q,
blk_free_flush_queue_callback);
hctx->fq = NULL;
xa_erase(&q->hctx_table, hctx_idx);
spin_lock(&q->unused_hctx_lock);
list_add(&hctx->hctx_list, &q->unused_hctx_list);
spin_unlock(&q->unused_hctx_lock);
@@ -3978,14 +3987,8 @@ static int blk_mq_init_hctx(struct request_queue *q,
hctx->numa_node))
goto exit_hctx;
if (xa_insert(&q->hctx_table, hctx_idx, hctx, GFP_KERNEL))
goto exit_flush_rq;
return 0;
exit_flush_rq:
if (set->ops->exit_request)
set->ops->exit_request(set, hctx->fq->flush_rq, hctx_idx);
exit_hctx:
if (set->ops->exit_hctx)
set->ops->exit_hctx(hctx, hctx_idx);
@@ -4374,7 +4377,7 @@ void blk_mq_release(struct request_queue *q)
kobject_put(&hctx->kobj);
}
xa_destroy(&q->hctx_table);
kfree(q->queue_hw_ctx);
/*
* release .mq_kobj and sw queue's kobject now because
@@ -4518,26 +4521,49 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx(
static void __blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
unsigned long i, j;
int i, j, end;
struct blk_mq_hw_ctx **hctxs = q->queue_hw_ctx;
if (q->nr_hw_queues < set->nr_hw_queues) {
struct blk_mq_hw_ctx **new_hctxs;
new_hctxs = kcalloc_node(set->nr_hw_queues,
sizeof(*new_hctxs), GFP_KERNEL,
set->numa_node);
if (!new_hctxs)
return;
if (hctxs)
memcpy(new_hctxs, hctxs, q->nr_hw_queues *
sizeof(*hctxs));
rcu_assign_pointer(q->queue_hw_ctx, new_hctxs);
/*
* Make sure reading the old queue_hw_ctx from other
* context concurrently won't trigger uaf.
*/
synchronize_rcu_expedited();
kfree(hctxs);
hctxs = new_hctxs;
}
for (i = 0; i < set->nr_hw_queues; i++) {
int old_node;
int node = blk_mq_get_hctx_node(set, i);
struct blk_mq_hw_ctx *old_hctx = xa_load(&q->hctx_table, i);
struct blk_mq_hw_ctx *old_hctx = hctxs[i];
if (old_hctx) {
old_node = old_hctx->numa_node;
blk_mq_exit_hctx(q, set, old_hctx, i);
}
if (!blk_mq_alloc_and_init_hctx(set, q, i, node)) {
hctxs[i] = blk_mq_alloc_and_init_hctx(set, q, i, node);
if (!hctxs[i]) {
if (!old_hctx)
break;
pr_warn("Allocate new hctx on node %d fails, fallback to previous one on node %d\n",
node, old_node);
hctx = blk_mq_alloc_and_init_hctx(set, q, i, old_node);
WARN_ON_ONCE(!hctx);
hctxs[i] = blk_mq_alloc_and_init_hctx(set, q, i,
old_node);
WARN_ON_ONCE(!hctxs[i]);
}
}
/*
@@ -4546,13 +4572,21 @@ static void __blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
*/
if (i != set->nr_hw_queues) {
j = q->nr_hw_queues;
end = i;
} else {
j = i;
end = q->nr_hw_queues;
q->nr_hw_queues = set->nr_hw_queues;
}
xa_for_each_start(&q->hctx_table, j, hctx, j)
blk_mq_exit_hctx(q, set, hctx, j);
for (; j < end; j++) {
struct blk_mq_hw_ctx *hctx = hctxs[j];
if (hctx) {
blk_mq_exit_hctx(q, set, hctx, j);
hctxs[j] = NULL;
}
}
}
static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
@@ -4588,8 +4622,6 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
INIT_LIST_HEAD(&q->unused_hctx_list);
spin_lock_init(&q->unused_hctx_lock);
xa_init(&q->hctx_table);
blk_mq_realloc_hw_ctxs(set, q);
if (!q->nr_hw_queues)
goto err_hctxs;
@@ -4983,27 +5015,28 @@ struct elevator_tags *blk_mq_update_nr_requests(struct request_queue *q,
* Switch back to the elevator type stored in the xarray.
*/
static void blk_mq_elv_switch_back(struct request_queue *q,
struct xarray *elv_tbl, struct xarray *et_tbl)
struct xarray *elv_tbl)
{
struct elevator_type *e = xa_load(elv_tbl, q->id);
struct elevator_tags *t = xa_load(et_tbl, q->id);
struct elv_change_ctx *ctx = xa_load(elv_tbl, q->id);
if (WARN_ON_ONCE(!ctx))
return;
/* The elv_update_nr_hw_queues unfreezes the queue. */
elv_update_nr_hw_queues(q, e, t);
elv_update_nr_hw_queues(q, ctx);
/* Drop the reference acquired in blk_mq_elv_switch_none. */
if (e)
elevator_put(e);
if (ctx->type)
elevator_put(ctx->type);
}
/*
* Stores elevator type in xarray and set current elevator to none. It uses
* q->id as an index to store the elevator type into the xarray.
* Stores elevator name and type in ctx and set current elevator to none.
*/
static int blk_mq_elv_switch_none(struct request_queue *q,
struct xarray *elv_tbl)
{
int ret = 0;
struct elv_change_ctx *ctx;
lockdep_assert_held_write(&q->tag_set->update_nr_hwq_lock);
@@ -5015,10 +5048,11 @@ static int blk_mq_elv_switch_none(struct request_queue *q,
* can't run concurrently.
*/
if (q->elevator) {
ctx = xa_load(elv_tbl, q->id);
if (WARN_ON_ONCE(!ctx))
return -ENOENT;
ret = xa_insert(elv_tbl, q->id, q->elevator->type, GFP_KERNEL);
if (WARN_ON_ONCE(ret))
return ret;
ctx->name = q->elevator->type->elevator_name;
/*
* Before we switch elevator to 'none', take a reference to
@@ -5029,9 +5063,14 @@ static int blk_mq_elv_switch_none(struct request_queue *q,
*/
__elevator_get(q->elevator->type);
/*
* Store elevator type so that we can release the reference
* taken above later.
*/
ctx->type = q->elevator->type;
elevator_set_none(q);
}
return ret;
return 0;
}
static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
@@ -5041,7 +5080,7 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
int prev_nr_hw_queues = set->nr_hw_queues;
unsigned int memflags;
int i;
struct xarray elv_tbl, et_tbl;
struct xarray elv_tbl;
bool queues_frozen = false;
lockdep_assert_held(&set->tag_list_lock);
@@ -5055,11 +5094,12 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
memflags = memalloc_noio_save();
xa_init(&et_tbl);
if (blk_mq_alloc_sched_tags_batch(&et_tbl, set, nr_hw_queues) < 0)
goto out_memalloc_restore;
xa_init(&elv_tbl);
if (blk_mq_alloc_sched_ctx_batch(&elv_tbl, set) < 0)
goto out_free_ctx;
if (blk_mq_alloc_sched_res_batch(&elv_tbl, set, nr_hw_queues) < 0)
goto out_free_ctx;
list_for_each_entry(q, &set->tag_list, tag_set_list) {
blk_mq_debugfs_unregister_hctxs(q);
@@ -5105,7 +5145,7 @@ switch_back:
/* switch_back expects queue to be frozen */
if (!queues_frozen)
blk_mq_freeze_queue_nomemsave(q);
blk_mq_elv_switch_back(q, &elv_tbl, &et_tbl);
blk_mq_elv_switch_back(q, &elv_tbl);
}
list_for_each_entry(q, &set->tag_list, tag_set_list) {
@@ -5116,9 +5156,9 @@ switch_back:
blk_mq_add_hw_queues_cpuhp(q);
}
out_free_ctx:
blk_mq_free_sched_ctx_batch(&elv_tbl);
xa_destroy(&elv_tbl);
xa_destroy(&et_tbl);
out_memalloc_restore:
memalloc_noio_restore(memflags);
/* Free the excess tags when nr_hw_queues shrink. */
@@ -5168,7 +5208,7 @@ int blk_mq_poll(struct request_queue *q, blk_qc_t cookie,
{
if (!blk_mq_can_poll(q))
return 0;
return blk_hctx_poll(q, xa_load(&q->hctx_table, cookie), iob, flags);
return blk_hctx_poll(q, q->queue_hw_ctx[cookie], iob, flags);
}
int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,

2
block/blk-mq.h
View File

@@ -84,7 +84,7 @@ static inline struct blk_mq_hw_ctx *blk_mq_map_queue_type(struct request_queue *
enum hctx_type type,
unsigned int cpu)
{
return xa_load(&q->hctx_table, q->tag_set->map[type].mq_map[cpu]);
return queue_hctx((q), (q->tag_set->map[type].mq_map[cpu]));
}
static inline enum hctx_type blk_mq_get_hctx_type(blk_opf_t opf)

27
block/blk-settings.c
View File

@@ -123,6 +123,19 @@ static int blk_validate_zoned_limits(struct queue_limits *lim)
return 0;
}
/*
* Maximum size of I/O that needs a block layer integrity buffer. Limited
* by the number of intervals for which we can fit the integrity buffer into
* the buffer size. Because the buffer is a single segment it is also limited
* by the maximum segment size.
*/
static inline unsigned int max_integrity_io_size(struct queue_limits *lim)
{
return min_t(unsigned int, lim->max_segment_size,
(BLK_INTEGRITY_MAX_SIZE / lim->integrity.metadata_size) <<
lim->integrity.interval_exp);
}
static int blk_validate_integrity_limits(struct queue_limits *lim)
{
struct blk_integrity *bi = &lim->integrity;
@@ -194,6 +207,14 @@ static int blk_validate_integrity_limits(struct queue_limits *lim)
(1U << bi->interval_exp) - 1);
}
/*
* The block layer automatically adds integrity data for bios that don't
* already have it. Limit the I/O size so that a single maximum size
* metadata segment can cover the integrity data for the entire I/O.
*/
lim->max_sectors = min(lim->max_sectors,
max_integrity_io_size(lim) >> SECTOR_SHIFT);
return 0;
}
@@ -467,12 +488,12 @@ int blk_validate_limits(struct queue_limits *lim)
return -EINVAL;
}
/* setup min segment size for building new segment in fast path */
/* setup max segment size for building new segment in fast path */
if (lim->seg_boundary_mask > lim->max_segment_size - 1)
seg_size = lim->max_segment_size;
else
seg_size = lim->seg_boundary_mask + 1;
lim->min_segment_size = min_t(unsigned int, seg_size, PAGE_SIZE);
lim->max_fast_segment_size = min_t(unsigned int, seg_size, PAGE_SIZE);
/*
* We require drivers to at least do logical block aligned I/O, but
@@ -535,6 +556,8 @@ int queue_limits_commit_update(struct request_queue *q,
{
int error;
lockdep_assert_held(&q->limits_lock);
error = blk_validate_limits(lim);
if (error)
goto out_unlock;

26
block/blk-sysfs.c
View File

@@ -143,21 +143,22 @@ queue_ra_store(struct gendisk *disk, const char *page, size_t count)
{
unsigned long ra_kb;
ssize_t ret;
unsigned int memflags;
struct request_queue *q = disk->queue;
ret = queue_var_store(&ra_kb, page, count);
if (ret < 0)
return ret;
/*
* ->ra_pages is protected by ->limits_lock because it is usually
* calculated from the queue limits by queue_limits_commit_update.
* The ->ra_pages change below is protected by ->limits_lock because it
* is usually calculated from the queue limits by
* queue_limits_commit_update().
*
* bdi->ra_pages reads are not serialized against bdi->ra_pages writes.
* Use WRITE_ONCE() to write bdi->ra_pages once.
*/
mutex_lock(&q->limits_lock);
memflags = blk_mq_freeze_queue(q);
disk->bdi->ra_pages = ra_kb >> (PAGE_SHIFT - 10);
WRITE_ONCE(disk->bdi->ra_pages, ra_kb >> (PAGE_SHIFT - 10));
mutex_unlock(&q->limits_lock);
blk_mq_unfreeze_queue(q, memflags);
return ret;
}
@@ -375,21 +376,18 @@ static ssize_t queue_nomerges_store(struct gendisk *disk, const char *page,
size_t count)
{
unsigned long nm;
unsigned int memflags;
struct request_queue *q = disk->queue;
ssize_t ret = queue_var_store(&nm, page, count);
if (ret < 0)
return ret;
memflags = blk_mq_freeze_queue(q);
blk_queue_flag_clear(QUEUE_FLAG_NOMERGES, q);
blk_queue_flag_clear(QUEUE_FLAG_NOXMERGES, q);
if (nm == 2)
blk_queue_flag_set(QUEUE_FLAG_NOMERGES, q);
else if (nm)
blk_queue_flag_set(QUEUE_FLAG_NOXMERGES, q);
blk_mq_unfreeze_queue(q, memflags);
return ret;
}
@@ -409,7 +407,6 @@ queue_rq_affinity_store(struct gendisk *disk, const char *page, size_t count)
#ifdef CONFIG_SMP
struct request_queue *q = disk->queue;
unsigned long val;
unsigned int memflags;
ret = queue_var_store(&val, page, count);
if (ret < 0)
@@ -421,7 +418,6 @@ queue_rq_affinity_store(struct gendisk *disk, const char *page, size_t count)
* are accessed individually using atomic test_bit operation. So we
* don't grab any lock while updating these flags.
*/
memflags = blk_mq_freeze_queue(q);
if (val == 2) {
blk_queue_flag_set(QUEUE_FLAG_SAME_COMP, q);
blk_queue_flag_set(QUEUE_FLAG_SAME_FORCE, q);
@@ -432,7 +428,6 @@ queue_rq_affinity_store(struct gendisk *disk, const char *page, size_t count)
blk_queue_flag_clear(QUEUE_FLAG_SAME_COMP, q);
blk_queue_flag_clear(QUEUE_FLAG_SAME_FORCE, q);
}
blk_mq_unfreeze_queue(q, memflags);
#endif
return ret;
}
@@ -446,11 +441,9 @@ static ssize_t queue_poll_delay_store(struct gendisk *disk, const char *page,
static ssize_t queue_poll_store(struct gendisk *disk, const char *page,
size_t count)
{
unsigned int memflags;
ssize_t ret = count;
struct request_queue *q = disk->queue;
memflags = blk_mq_freeze_queue(q);
if (!(q->limits.features & BLK_FEAT_POLL)) {
ret = -EINVAL;
goto out;
@@ -459,7 +452,6 @@ static ssize_t queue_poll_store(struct gendisk *disk, const char *page,
pr_info_ratelimited("writes to the poll attribute are ignored.\n");
pr_info_ratelimited("please use driver specific parameters instead.\n");
out:
blk_mq_unfreeze_queue(q, memflags);
return ret;
}
@@ -472,7 +464,7 @@ static ssize_t queue_io_timeout_show(struct gendisk *disk, char *page)
static ssize_t queue_io_timeout_store(struct gendisk *disk, const char *page,
size_t count)
{
unsigned int val, memflags;
unsigned int val;
int err;
struct request_queue *q = disk->queue;
@@ -480,9 +472,7 @@ static ssize_t queue_io_timeout_store(struct gendisk *disk, const char *page,
if (err || val == 0)
return -EINVAL;
memflags = blk_mq_freeze_queue(q);
blk_queue_rq_timeout(q, msecs_to_jiffies(val));
blk_mq_unfreeze_queue(q, memflags);
return count;
}

45
block/blk-throttle.c
View File

@@ -12,7 +12,6 @@
#include <linux/blktrace_api.h>
#include "blk.h"
#include "blk-cgroup-rwstat.h"
#include "blk-stat.h"
#include "blk-throttle.h"
/* Max dispatch from a group in 1 round */
@@ -22,9 +21,7 @@
#define THROTL_QUANTUM 32
/* Throttling is performed over a slice and after that slice is renewed */
#define DFL_THROTL_SLICE_HD (HZ / 10)
#define DFL_THROTL_SLICE_SSD (HZ / 50)
#define MAX_THROTL_SLICE (HZ)
#define DFL_THROTL_SLICE (HZ / 10)
/* A workqueue to queue throttle related work */
static struct workqueue_struct *kthrotld_workqueue;
@@ -41,12 +38,8 @@ struct throtl_data
/* Total Number of queued bios on READ and WRITE lists */
unsigned int nr_queued[2];
unsigned int throtl_slice;
/* Work for dispatching throttled bios */
struct work_struct dispatch_work;
bool track_bio_latency;
};
static void throtl_pending_timer_fn(struct timer_list *t);
@@ -451,7 +444,7 @@ static void throtl_dequeue_tg(struct throtl_grp *tg)
static void throtl_schedule_pending_timer(struct throtl_service_queue *sq,
unsigned long expires)
{
unsigned long max_expire = jiffies + 8 * sq_to_td(sq)->throtl_slice;
unsigned long max_expire = jiffies + 8 * DFL_THROTL_SLICE;
/*
* Since we are adjusting the throttle limit dynamically, the sleep
@@ -519,7 +512,7 @@ static inline void throtl_start_new_slice_with_credit(struct throtl_grp *tg,
if (time_after(start, tg->slice_start[rw]))
tg->slice_start[rw] = start;
tg->slice_end[rw] = jiffies + tg->td->throtl_slice;
tg->slice_end[rw] = jiffies + DFL_THROTL_SLICE;
throtl_log(&tg->service_queue,
"[%c] new slice with credit start=%lu end=%lu jiffies=%lu",
rw == READ ? 'R' : 'W', tg->slice_start[rw],
@@ -534,7 +527,7 @@ static inline void throtl_start_new_slice(struct throtl_grp *tg, bool rw,
tg->io_disp[rw] = 0;
}
tg->slice_start[rw] = jiffies;
tg->slice_end[rw] = jiffies + tg->td->throtl_slice;
tg->slice_end[rw] = jiffies + DFL_THROTL_SLICE;
throtl_log(&tg->service_queue,
"[%c] new slice start=%lu end=%lu jiffies=%lu",
@@ -545,7 +538,7 @@ static inline void throtl_start_new_slice(struct throtl_grp *tg, bool rw,
static inline void throtl_set_slice_end(struct throtl_grp *tg, bool rw,
unsigned long jiffy_end)
{
tg->slice_end[rw] = roundup(jiffy_end, tg->td->throtl_slice);
tg->slice_end[rw] = roundup(jiffy_end, DFL_THROTL_SLICE);
}
static inline void throtl_extend_slice(struct throtl_grp *tg, bool rw,
@@ -676,12 +669,12 @@ static inline void throtl_trim_slice(struct throtl_grp *tg, bool rw)
* sooner, then we need to reduce slice_end. A high bogus slice_end
* is bad because it does not allow new slice to start.
*/
throtl_set_slice_end(tg, rw, jiffies + tg->td->throtl_slice);
throtl_set_slice_end(tg, rw, jiffies + DFL_THROTL_SLICE);
time_elapsed = rounddown(jiffies - tg->slice_start[rw],
tg->td->throtl_slice);
DFL_THROTL_SLICE);
/* Don't trim slice until at least 2 slices are used */
if (time_elapsed < tg->td->throtl_slice * 2)
if (time_elapsed < DFL_THROTL_SLICE * 2)
return;
/*
@@ -692,7 +685,7 @@ static inline void throtl_trim_slice(struct throtl_grp *tg, bool rw)
* lower rate than expected. Therefore, other than the above rounddown,
* one extra slice is preserved for deviation.
*/
time_elapsed -= tg->td->throtl_slice;
time_elapsed -= DFL_THROTL_SLICE;
bytes_trim = throtl_trim_bps(tg, rw, time_elapsed);
io_trim = throtl_trim_iops(tg, rw, time_elapsed);
if (!bytes_trim && !io_trim)
@@ -702,7 +695,7 @@ static inline void throtl_trim_slice(struct throtl_grp *tg, bool rw)
throtl_log(&tg->service_queue,
"[%c] trim slice nr=%lu bytes=%lld io=%d start=%lu end=%lu jiffies=%lu",
rw == READ ? 'R' : 'W', time_elapsed / tg->td->throtl_slice,
rw == READ ? 'R' : 'W', time_elapsed / DFL_THROTL_SLICE,
bytes_trim, io_trim, tg->slice_start[rw], tg->slice_end[rw],
jiffies);
}
@@ -773,7 +766,7 @@ static unsigned long tg_within_iops_limit(struct throtl_grp *tg, struct bio *bio
jiffy_elapsed = jiffies - tg->slice_start[rw];
/* Round up to the next throttle slice, wait time must be nonzero */
jiffy_elapsed_rnd = roundup(jiffy_elapsed + 1, tg->td->throtl_slice);
jiffy_elapsed_rnd = roundup(jiffy_elapsed + 1, DFL_THROTL_SLICE);
io_allowed = calculate_io_allowed(iops_limit, jiffy_elapsed_rnd);
if (io_allowed > 0 && tg->io_disp[rw] + 1 <= io_allowed)
return 0;
@@ -799,9 +792,9 @@ static unsigned long tg_within_bps_limit(struct throtl_grp *tg, struct bio *bio,
/* Slice has just started. Consider one slice interval */
if (!jiffy_elapsed)
jiffy_elapsed_rnd = tg->td->throtl_slice;
jiffy_elapsed_rnd = DFL_THROTL_SLICE;
jiffy_elapsed_rnd = roundup(jiffy_elapsed_rnd, tg->td->throtl_slice);
jiffy_elapsed_rnd = roundup(jiffy_elapsed_rnd, DFL_THROTL_SLICE);
bytes_allowed = calculate_bytes_allowed(bps_limit, jiffy_elapsed_rnd);
/* Need to consider the case of bytes_allowed overflow. */
if ((bytes_allowed > 0 && tg->bytes_disp[rw] + bio_size <= bytes_allowed)
@@ -853,7 +846,7 @@ static void tg_update_slice(struct throtl_grp *tg, bool rw)
sq_queued(&tg->service_queue, rw) == 0)
throtl_start_new_slice(tg, rw, true);
else
throtl_extend_slice(tg, rw, jiffies + tg->td->throtl_slice);
throtl_extend_slice(tg, rw, jiffies + DFL_THROTL_SLICE);
}
static unsigned long tg_dispatch_bps_time(struct throtl_grp *tg, struct bio *bio)
@@ -1338,18 +1331,8 @@ static int blk_throtl_init(struct gendisk *disk)
if (ret) {
q->td = NULL;
kfree(td);
goto out;
}
if (blk_queue_nonrot(q))
td->throtl_slice = DFL_THROTL_SLICE_SSD;
else
td->throtl_slice = DFL_THROTL_SLICE_HD;
td->track_bio_latency = !queue_is_mq(q);
if (!td->track_bio_latency)
blk_stat_enable_accounting(q);
out:
blk_mq_unquiesce_queue(disk->queue);
blk_mq_unfreeze_queue(disk->queue, memflags);

988
block/blk-zoned.c
View File

File diff suppressed because it is too large Load Diff

23
block/blk.h
View File

@@ -11,8 +11,7 @@
#include <xen/xen.h>
#include "blk-crypto-internal.h"
struct elevator_type;
struct elevator_tags;
struct elv_change_ctx;
/*
* Default upper limit for the software max_sectors limit used for regular I/Os.
@@ -333,8 +332,8 @@ bool blk_bio_list_merge(struct request_queue *q, struct list_head *list,
bool blk_insert_flush(struct request *rq);
void elv_update_nr_hw_queues(struct request_queue *q, struct elevator_type *e,
struct elevator_tags *t);
void elv_update_nr_hw_queues(struct request_queue *q,
struct elv_change_ctx *ctx);
void elevator_set_default(struct request_queue *q);
void elevator_set_none(struct request_queue *q);
@@ -377,7 +376,7 @@ static inline bool bio_may_need_split(struct bio *bio,
if (bio->bi_vcnt != 1)
return true;
return bio->bi_io_vec->bv_len + bio->bi_io_vec->bv_offset >
lim->min_segment_size;
lim->max_fast_segment_size;
}
/**
@@ -489,9 +488,23 @@ static inline bool blk_req_bio_is_zone_append(struct request *rq,
void blk_zone_write_plug_bio_merged(struct bio *bio);
void blk_zone_write_plug_init_request(struct request *rq);
void blk_zone_append_update_request_bio(struct request *rq, struct bio *bio);
void blk_zone_mgmt_bio_endio(struct bio *bio);
void blk_zone_write_plug_bio_endio(struct bio *bio);
static inline void blk_zone_bio_endio(struct bio *bio)
{
/*
* Zone management BIOs may impact zone write plugs (e.g. a zone reset
* changes a zone write plug zone write pointer offset), but these
* operation do not go through zone write plugging as they may operate
* on zones that do not have a zone write
* plug. blk_zone_mgmt_bio_endio() handles the potential changes to zone
* write plugs that are present.
*/
if (op_is_zone_mgmt(bio_op(bio))) {
blk_zone_mgmt_bio_endio(bio);
return;
}
/*
* For write BIOs to zoned devices, signal the completion of the BIO so
* that the next write BIO can be submitted by zone write plugging.

80
block/elevator.c
View File

@@ -45,19 +45,6 @@
#include "blk-wbt.h"
#include "blk-cgroup.h"
/* Holding context data for changing elevator */
struct elv_change_ctx {
const char *name;
bool no_uevent;
/* for unregistering old elevator */
struct elevator_queue *old;
/* for registering new elevator */
struct elevator_queue *new;
/* holds sched tags data */
struct elevator_tags *et;
};
static DEFINE_SPINLOCK(elv_list_lock);
static LIST_HEAD(elv_list);
@@ -134,7 +121,7 @@ static struct elevator_type *elevator_find_get(const char *name)
static const struct kobj_type elv_ktype;
struct elevator_queue *elevator_alloc(struct request_queue *q,
struct elevator_type *e, struct elevator_tags *et)
struct elevator_type *e, struct elevator_resources *res)
{
struct elevator_queue *eq;
@@ -147,7 +134,8 @@ struct elevator_queue *elevator_alloc(struct request_queue *q,
kobject_init(&eq->kobj, &elv_ktype);
mutex_init(&eq->sysfs_lock);
hash_init(eq->hash);
eq->et = et;
eq->et = res->et;
eq->elevator_data = res->data;
return eq;
}
@@ -593,7 +581,7 @@ static int elevator_switch(struct request_queue *q, struct elv_change_ctx *ctx)
}
if (new_e) {
ret = blk_mq_init_sched(q, new_e, ctx->et);
ret = blk_mq_init_sched(q, new_e, &ctx->res);
if (ret)
goto out_unfreeze;
ctx->new = q->elevator;
@@ -617,7 +605,8 @@ out_unfreeze:
return ret;
}
static void elv_exit_and_release(struct request_queue *q)
static void elv_exit_and_release(struct elv_change_ctx *ctx,
struct request_queue *q)
{
struct elevator_queue *e;
unsigned memflags;
@@ -629,7 +618,7 @@ static void elv_exit_and_release(struct request_queue *q)
mutex_unlock(&q->elevator_lock);
blk_mq_unfreeze_queue(q, memflags);
if (e) {
blk_mq_free_sched_tags(e->et, q->tag_set);
blk_mq_free_sched_res(&ctx->res, ctx->type, q->tag_set);
kobject_put(&e->kobj);
}
}
@@ -640,11 +629,15 @@ static int elevator_change_done(struct request_queue *q,
int ret = 0;
if (ctx->old) {
struct elevator_resources res = {
.et = ctx->old->et,
.data = ctx->old->elevator_data
};
bool enable_wbt = test_bit(ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT,
&ctx->old->flags);
elv_unregister_queue(q, ctx->old);
blk_mq_free_sched_tags(ctx->old->et, q->tag_set);
blk_mq_free_sched_res(&res, ctx->old->type, q->tag_set);
kobject_put(&ctx->old->kobj);
if (enable_wbt)
wbt_enable_default(q->disk);
@@ -652,7 +645,7 @@ static int elevator_change_done(struct request_queue *q,
if (ctx->new) {
ret = elv_register_queue(q, ctx->new, !ctx->no_uevent);
if (ret)
elv_exit_and_release(q);
elv_exit_and_release(ctx, q);
}
return ret;
}
@@ -669,10 +662,10 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
lockdep_assert_held(&set->update_nr_hwq_lock);
if (strncmp(ctx->name, "none", 4)) {
ctx->et = blk_mq_alloc_sched_tags(set, set->nr_hw_queues,
blk_mq_default_nr_requests(set));
if (!ctx->et)
return -ENOMEM;
ret = blk_mq_alloc_sched_res(q, ctx->type, &ctx->res,
set->nr_hw_queues);
if (ret)
return ret;
}
memflags = blk_mq_freeze_queue(q);
@@ -693,11 +686,12 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
blk_mq_unfreeze_queue(q, memflags);
if (!ret)
ret = elevator_change_done(q, ctx);
/*
* Free sched tags if it's allocated but we couldn't switch elevator.
* Free sched resource if it's allocated but we couldn't switch elevator.
*/
if (ctx->et && !ctx->new)
blk_mq_free_sched_tags(ctx->et, set);
if (!ctx->new)
blk_mq_free_sched_res(&ctx->res, ctx->type, set);
return ret;
}
@@ -706,32 +700,29 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
* The I/O scheduler depends on the number of hardware queues, this forces a
* reattachment when nr_hw_queues changes.
*/
void elv_update_nr_hw_queues(struct request_queue *q, struct elevator_type *e,
struct elevator_tags *t)
void elv_update_nr_hw_queues(struct request_queue *q,
struct elv_change_ctx *ctx)
{
struct blk_mq_tag_set *set = q->tag_set;
struct elv_change_ctx ctx = {};
int ret = -ENODEV;
WARN_ON_ONCE(q->mq_freeze_depth == 0);
if (e && !blk_queue_dying(q) && blk_queue_registered(q)) {
ctx.name = e->elevator_name;
ctx.et = t;
if (ctx->type && !blk_queue_dying(q) && blk_queue_registered(q)) {
mutex_lock(&q->elevator_lock);
/* force to reattach elevator after nr_hw_queue is updated */
ret = elevator_switch(q, &ctx);
ret = elevator_switch(q, ctx);
mutex_unlock(&q->elevator_lock);
}
blk_mq_unfreeze_queue_nomemrestore(q);
if (!ret)
WARN_ON_ONCE(elevator_change_done(q, &ctx));
WARN_ON_ONCE(elevator_change_done(q, ctx));
/*
* Free sched tags if it's allocated but we couldn't switch elevator.
* Free sched resource if it's allocated but we couldn't switch elevator.
*/
if (t && !ctx.new)
blk_mq_free_sched_tags(t, set);
if (!ctx->new)
blk_mq_free_sched_res(&ctx->res, ctx->type, set);
}
/*
@@ -745,7 +736,6 @@ void elevator_set_default(struct request_queue *q)
.no_uevent = true,
};
int err;
struct elevator_type *e;
/* now we allow to switch elevator */
blk_queue_flag_clear(QUEUE_FLAG_NO_ELV_SWITCH, q);
@@ -758,8 +748,8 @@ void elevator_set_default(struct request_queue *q)
* have multiple queues or mq-deadline is not available, default
* to "none".
*/
e = elevator_find_get(ctx.name);
if (!e)
ctx.type = elevator_find_get(ctx.name);
if (!ctx.type)
return;
if ((q->nr_hw_queues == 1 ||
@@ -769,7 +759,7 @@ void elevator_set_default(struct request_queue *q)
pr_warn("\"%s\" elevator initialization, failed %d, falling back to \"none\"\n",
ctx.name, err);
}
elevator_put(e);
elevator_put(ctx.type);
}
void elevator_set_none(struct request_queue *q)
@@ -818,6 +808,7 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,
ctx.name = strstrip(elevator_name);
elv_iosched_load_module(ctx.name);
ctx.type = elevator_find_get(ctx.name);
down_read(&set->update_nr_hwq_lock);
if (!blk_queue_no_elv_switch(q)) {
@@ -828,6 +819,9 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,
ret = -ENOENT;
}
up_read(&set->update_nr_hwq_lock);
if (ctx.type)
elevator_put(ctx.type);
return ret;
}

27
block/elevator.h
View File

@@ -32,12 +32,36 @@ struct elevator_tags {
struct blk_mq_tags *tags[];
};
struct elevator_resources {
/* holds elevator data */
void *data;
/* holds elevator tags */
struct elevator_tags *et;
};
/* Holding context data for changing elevator */
struct elv_change_ctx {
const char *name;
bool no_uevent;
/* for unregistering old elevator */
struct elevator_queue *old;
/* for registering new elevator */
struct elevator_queue *new;
/* store elevator type */
struct elevator_type *type;
/* store elevator resources */
struct elevator_resources res;
};
struct elevator_mq_ops {
int (*init_sched)(struct request_queue *, struct elevator_queue *);
void (*exit_sched)(struct elevator_queue *);
int (*init_hctx)(struct blk_mq_hw_ctx *, unsigned int);
void (*exit_hctx)(struct blk_mq_hw_ctx *, unsigned int);
void (*depth_updated)(struct request_queue *);
void *(*alloc_sched_data)(struct request_queue *);
void (*free_sched_data)(void *);
bool (*allow_merge)(struct request_queue *, struct request *, struct bio *);
bool (*bio_merge)(struct request_queue *, struct bio *, unsigned int);
@@ -147,7 +171,6 @@ extern bool elv_attempt_insert_merge(struct request_queue *, struct request *,
struct list_head *);
extern struct request *elv_former_request(struct request_queue *, struct request *);
extern struct request *elv_latter_request(struct request_queue *, struct request *);
void elevator_init_mq(struct request_queue *q);
/*
* io scheduler registration
@@ -163,7 +186,7 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *page, size_t count);
extern bool elv_bio_merge_ok(struct request *, struct bio *);
struct elevator_queue *elevator_alloc(struct request_queue *,
struct elevator_type *, struct elevator_tags *);
struct elevator_type *, struct elevator_resources *);
/*
* Helper functions.

8
block/genhd.c
View File

@@ -90,7 +90,7 @@ bool set_capacity_and_notify(struct gendisk *disk, sector_t size)
(disk->flags & GENHD_FL_HIDDEN))
return false;
pr_info("%s: detected capacity change from %lld to %lld\n",
pr_info_ratelimited("%s: detected capacity change from %lld to %lld\n",
disk->disk_name, capacity, size);
/*
@@ -795,11 +795,11 @@ static void disable_elv_switch(struct request_queue *q)
* partitions associated with the gendisk, and unregisters the associated
* request_queue.
*
* This is the counter to the respective __device_add_disk() call.
* This is the counter to the respective device_add_disk() call.
*
* The final removal of the struct gendisk happens when its refcount reaches 0
* with put_disk(), which should be called after del_gendisk(), if
* __device_add_disk() was used.
* device_add_disk() was used.
*
* Drivers exist which depend on the release of the gendisk to be synchronous,
* it should not be deferred.
@@ -1265,7 +1265,7 @@ static const struct attribute_group *disk_attr_groups[] = {
*
* This function releases all allocated resources of the gendisk.
*
* Drivers which used __device_add_disk() have a gendisk with a request_queue
* Drivers which used device_add_disk() have a gendisk with a request_queue
* assigned. Since the request_queue sits on top of the gendisk for these
* drivers we also call blk_put_queue() for them, and we expect the
* request_queue refcount to reach 0 at this point, and so the request_queue

2
block/ioctl.c
View File

@@ -581,6 +581,7 @@ static int blkdev_common_ioctl(struct block_device *bdev, blk_mode_t mode,
case BLKGETDISKSEQ:
return put_u64(argp, bdev->bd_disk->diskseq);
case BLKREPORTZONE:
case BLKREPORTZONEV2:
return blkdev_report_zones_ioctl(bdev, cmd, arg);
case BLKRESETZONE:
case BLKOPENZONE:
@@ -691,6 +692,7 @@ long blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg)
/* Incompatible alignment on i386 */
case BLKTRACESETUP:
case BLKTRACESETUP2:
return blk_trace_ioctl(bdev, cmd, argp);
default:
break;

30
block/kyber-iosched.c
View File

@@ -409,30 +409,42 @@ static void kyber_depth_updated(struct request_queue *q)
static int kyber_init_sched(struct request_queue *q, struct elevator_queue *eq)
{
struct kyber_queue_data *kqd;
kqd = kyber_queue_data_alloc(q);
if (IS_ERR(kqd))
return PTR_ERR(kqd);
blk_stat_enable_accounting(q);
blk_queue_flag_clear(QUEUE_FLAG_SQ_SCHED, q);
eq->elevator_data = kqd;
q->elevator = eq;
kyber_depth_updated(q);
return 0;
}
static void *kyber_alloc_sched_data(struct request_queue *q)
{
struct kyber_queue_data *kqd;
kqd = kyber_queue_data_alloc(q);
if (IS_ERR(kqd))
return NULL;
return kqd;
}
static void kyber_exit_sched(struct elevator_queue *e)
{
struct kyber_queue_data *kqd = e->elevator_data;
int i;
timer_shutdown_sync(&kqd->timer);
blk_stat_disable_accounting(kqd->q);
}
static void kyber_free_sched_data(void *elv_data)
{
struct kyber_queue_data *kqd = elv_data;
int i;
if (!kqd)
return;
for (i = 0; i < KYBER_NUM_DOMAINS; i++)
sbitmap_queue_free(&kqd->domain_tokens[i]);
@@ -1004,6 +1016,8 @@ static struct elevator_type kyber_sched = {
.exit_sched = kyber_exit_sched,
.init_hctx = kyber_init_hctx,
.exit_hctx = kyber_exit_hctx,
.alloc_sched_data = kyber_alloc_sched_data,
.free_sched_data = kyber_free_sched_data,
.limit_depth = kyber_limit_depth,
.bio_merge = kyber_bio_merge,
.prepare_request = kyber_prepare_request,

129
block/mq-deadline.c
View File

@@ -71,7 +71,6 @@ struct io_stats_per_prio {
* present on both sort_list[] and fifo_list[].
*/
struct dd_per_prio {
struct list_head dispatch;
struct rb_root sort_list[DD_DIR_COUNT];
struct list_head fifo_list[DD_DIR_COUNT];
/* Position of the most recently dispatched request. */
@@ -84,6 +83,7 @@ struct deadline_data {
* run time data
*/
struct list_head dispatch;
struct dd_per_prio per_prio[DD_PRIO_COUNT];
/* Data direction of latest dispatched request. */
@@ -306,6 +306,19 @@ static bool started_after(struct deadline_data *dd, struct request *rq,
return time_after(start_time, latest_start);
}
static struct request *dd_start_request(struct deadline_data *dd,
enum dd_data_dir data_dir,
struct request *rq)
{
u8 ioprio_class = dd_rq_ioclass(rq);
enum dd_prio prio = ioprio_class_to_prio[ioprio_class];
dd->per_prio[prio].latest_pos[data_dir] = blk_rq_pos(rq);
dd->per_prio[prio].stats.dispatched++;
rq->rq_flags |= RQF_STARTED;
return rq;
}
/*
* deadline_dispatch_requests selects the best request according to
* read/write expire, fifo_batch, etc and with a start time <= @latest_start.
@@ -316,21 +329,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
{
struct request *rq, *next_rq;
enum dd_data_dir data_dir;
enum dd_prio prio;
u8 ioprio_class;
lockdep_assert_held(&dd->lock);
if (!list_empty(&per_prio->dispatch)) {
rq = list_first_entry(&per_prio->dispatch, struct request,
queuelist);
if (started_after(dd, rq, latest_start))
return NULL;
list_del_init(&rq->queuelist);
data_dir = rq_data_dir(rq);
goto done;
}
/*
* batches are currently reads XOR writes
*/
@@ -410,13 +411,7 @@ dispatch_request:
*/
dd->batching++;
deadline_move_request(dd, per_prio, rq);
done:
ioprio_class = dd_rq_ioclass(rq);
prio = ioprio_class_to_prio[ioprio_class];
dd->per_prio[prio].latest_pos[data_dir] = blk_rq_pos(rq);
dd->per_prio[prio].stats.dispatched++;
rq->rq_flags |= RQF_STARTED;
return rq;
return dd_start_request(dd, data_dir, rq);
}
/*
@@ -463,6 +458,14 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx)
enum dd_prio prio;
spin_lock(&dd->lock);
if (!list_empty(&dd->dispatch)) {
rq = list_first_entry(&dd->dispatch, struct request, queuelist);
list_del_init(&rq->queuelist);
dd_start_request(dd, rq_data_dir(rq), rq);
goto unlock;
}
rq = dd_dispatch_prio_aged_requests(dd, now);
if (rq)
goto unlock;
@@ -551,10 +554,10 @@ static int dd_init_sched(struct request_queue *q, struct elevator_queue *eq)
eq->elevator_data = dd;
INIT_LIST_HEAD(&dd->dispatch);
for (prio = 0; prio <= DD_PRIO_MAX; prio++) {
struct dd_per_prio *per_prio = &dd->per_prio[prio];
INIT_LIST_HEAD(&per_prio->dispatch);
INIT_LIST_HEAD(&per_prio->fifo_list[DD_READ]);
INIT_LIST_HEAD(&per_prio->fifo_list[DD_WRITE]);
per_prio->sort_list[DD_READ] = RB_ROOT;
@@ -658,7 +661,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
trace_block_rq_insert(rq);
if (flags & BLK_MQ_INSERT_AT_HEAD) {
list_add(&rq->queuelist, &per_prio->dispatch);
list_add(&rq->queuelist, &dd->dispatch);
rq->fifo_time = jiffies;
} else {
deadline_add_rq_rb(per_prio, rq);
@@ -725,8 +728,7 @@ static void dd_finish_request(struct request *rq)
static bool dd_has_work_for_prio(struct dd_per_prio *per_prio)
{
return !list_empty_careful(&per_prio->dispatch) ||
!list_empty_careful(&per_prio->fifo_list[DD_READ]) ||
return !list_empty_careful(&per_prio->fifo_list[DD_READ]) ||
!list_empty_careful(&per_prio->fifo_list[DD_WRITE]);
}
@@ -735,6 +737,9 @@ static bool dd_has_work(struct blk_mq_hw_ctx *hctx)
struct deadline_data *dd = hctx->queue->elevator->elevator_data;
enum dd_prio prio;
if (!list_empty_careful(&dd->dispatch))
return true;
for (prio = 0; prio <= DD_PRIO_MAX; prio++)
if (dd_has_work_for_prio(&dd->per_prio[prio]))
return true;
@@ -943,49 +948,39 @@ static int dd_owned_by_driver_show(void *data, struct seq_file *m)
return 0;
}
#define DEADLINE_DISPATCH_ATTR(prio) \
static void *deadline_dispatch##prio##_start(struct seq_file *m, \
loff_t *pos) \
__acquires(&dd->lock) \
{ \
struct request_queue *q = m->private; \
struct deadline_data *dd = q->elevator->elevator_data; \
struct dd_per_prio *per_prio = &dd->per_prio[prio]; \
\
spin_lock(&dd->lock); \
return seq_list_start(&per_prio->dispatch, *pos); \
} \
\
static void *deadline_dispatch##prio##_next(struct seq_file *m, \
void *v, loff_t *pos) \
{ \
struct request_queue *q = m->private; \
struct deadline_data *dd = q->elevator->elevator_data; \
struct dd_per_prio *per_prio = &dd->per_prio[prio]; \
\
return seq_list_next(v, &per_prio->dispatch, pos); \
} \
\
static void deadline_dispatch##prio##_stop(struct seq_file *m, void *v) \
__releases(&dd->lock) \
{ \
struct request_queue *q = m->private; \
struct deadline_data *dd = q->elevator->elevator_data; \
\
spin_unlock(&dd->lock); \
} \
\
static const struct seq_operations deadline_dispatch##prio##_seq_ops = { \
.start = deadline_dispatch##prio##_start, \
.next = deadline_dispatch##prio##_next, \
.stop = deadline_dispatch##prio##_stop, \
.show = blk_mq_debugfs_rq_show, \
static void *deadline_dispatch_start(struct seq_file *m, loff_t *pos)
__acquires(&dd->lock)
{
struct request_queue *q = m->private;
struct deadline_data *dd = q->elevator->elevator_data;
spin_lock(&dd->lock);
return seq_list_start(&dd->dispatch, *pos);
}
DEADLINE_DISPATCH_ATTR(0);
DEADLINE_DISPATCH_ATTR(1);
DEADLINE_DISPATCH_ATTR(2);
#undef DEADLINE_DISPATCH_ATTR
static void *deadline_dispatch_next(struct seq_file *m, void *v, loff_t *pos)
{
struct request_queue *q = m->private;
struct deadline_data *dd = q->elevator->elevator_data;
return seq_list_next(v, &dd->dispatch, pos);
}
static void deadline_dispatch_stop(struct seq_file *m, void *v)
__releases(&dd->lock)
{
struct request_queue *q = m->private;
struct deadline_data *dd = q->elevator->elevator_data;
spin_unlock(&dd->lock);
}
static const struct seq_operations deadline_dispatch_seq_ops = {
.start = deadline_dispatch_start,
.next = deadline_dispatch_next,
.stop = deadline_dispatch_stop,
.show = blk_mq_debugfs_rq_show,
};
#define DEADLINE_QUEUE_DDIR_ATTRS(name) \
{#name "_fifo_list", 0400, \
@@ -1008,9 +1003,7 @@ static const struct blk_mq_debugfs_attr deadline_queue_debugfs_attrs[] = {
{"batching", 0400, deadline_batching_show},
{"starved", 0400, deadline_starved_show},
{"async_depth", 0400, dd_async_depth_show},
{"dispatch0", 0400, .seq_ops = &deadline_dispatch0_seq_ops},
{"dispatch1", 0400, .seq_ops = &deadline_dispatch1_seq_ops},
{"dispatch2", 0400, .seq_ops = &deadline_dispatch2_seq_ops},
{"dispatch", 0400, .seq_ops = &deadline_dispatch_seq_ops},
{"owned_by_driver", 0400, dd_owned_by_driver_show},
{"queued", 0400, dd_queued_show},
{},

3
block/partitions/efi.c
View File

@@ -215,8 +215,7 @@ check_hybrid:
sz = le32_to_cpu(mbr->partition_record[part].size_in_lba);
if (sz != (uint32_t) total_sectors - 1 && sz != 0xFFFFFFFF)
pr_debug("GPT: mbr size in lba (%u) different than whole disk (%u).\n",
sz, min_t(uint32_t,
total_sectors - 1, 0xFFFFFFFF));
sz, (uint32_t)min(total_sectors - 1, 0xFFFFFFFF));
}
done:
return ret;

10
drivers/block/drbd/drbd_bitmap.c
View File

@@ -1210,7 +1210,7 @@ static int bm_rw(struct drbd_device *device, const unsigned int flags, unsigned
return err;
}
/**
/*
* drbd_bm_read() - Read the whole bitmap from its on disk location.
* @device: DRBD device.
*/
@@ -1221,7 +1221,7 @@ int drbd_bm_read(struct drbd_device *device,
return bm_rw(device, BM_AIO_READ, 0);
}
/**
/*
* drbd_bm_write() - Write the whole bitmap to its on disk location.
* @device: DRBD device.
*
@@ -1233,7 +1233,7 @@ int drbd_bm_write(struct drbd_device *device,
return bm_rw(device, 0, 0);
}
/**
/*
* drbd_bm_write_all() - Write the whole bitmap to its on disk location.
* @device: DRBD device.
*
@@ -1255,7 +1255,7 @@ int drbd_bm_write_lazy(struct drbd_device *device, unsigned upper_idx) __must_ho
return bm_rw(device, BM_AIO_COPY_PAGES, upper_idx);
}
/**
/*
* drbd_bm_write_copy_pages() - Write the whole bitmap to its on disk location.
* @device: DRBD device.
*
@@ -1272,7 +1272,7 @@ int drbd_bm_write_copy_pages(struct drbd_device *device,
return bm_rw(device, BM_AIO_COPY_PAGES, 0);
}
/**
/*
* drbd_bm_write_hinted() - Write bitmap pages with "hint" marks, if they have changed.
* @device: DRBD device.
*/

8
drivers/block/drbd/drbd_receiver.c
View File

@@ -1736,13 +1736,13 @@ read_in_block(struct drbd_peer_device *peer_device, u64 id, sector_t sector,
page = peer_req->pages;
page_chain_for_each(page) {
unsigned len = min_t(int, ds, PAGE_SIZE);
data = kmap(page);
data = kmap_local_page(page);
err = drbd_recv_all_warn(peer_device->connection, data, len);
if (drbd_insert_fault(device, DRBD_FAULT_RECEIVE)) {
drbd_err(device, "Fault injection: Corrupting data on receive\n");
data[0] = data[0] ^ (unsigned long)-1;
}
kunmap(page);
kunmap_local(data);
if (err) {
drbd_free_peer_req(device, peer_req);
return NULL;
@@ -1777,7 +1777,7 @@ static int drbd_drain_block(struct drbd_peer_device *peer_device, int data_size)
page = drbd_alloc_pages(peer_device, 1, 1);
data = kmap(page);
data = kmap_local_page(page);
while (data_size) {
unsigned int len = min_t(int, data_size, PAGE_SIZE);
@@ -1786,7 +1786,7 @@ static int drbd_drain_block(struct drbd_peer_device *peer_device, int data_size)
break;
data_size -= len;
}
kunmap(page);
kunmap_local(data);
drbd_free_pages(peer_device->device, page);
return err;
}

2
drivers/block/floppy.c
View File

@@ -329,7 +329,7 @@ static bool initialized;
* This default is used whenever the current disk size is unknown.
* [Now it is rather a minimum]
*/
#define MAX_DISK_SIZE 4 /* 3984 */
#define MAX_DISK_SIZE (PAGE_SIZE / 1024)
/*
* globals used by 'result()'

4
drivers/block/loop.c
View File

@@ -1908,6 +1908,10 @@ static void loop_handle_cmd(struct loop_cmd *cmd)
goto failed;
}
/* We can block in this context, so ignore REQ_NOWAIT. */
if (rq->cmd_flags & REQ_NOWAIT)
rq->cmd_flags &= ~REQ_NOWAIT;
if (cmd_blkcg_css)
kthread_associate_blkcg(cmd_blkcg_css);
if (cmd_memcg_css)

5
drivers/block/nbd.c
View File

@@ -1021,9 +1021,9 @@ static void recv_work(struct work_struct *work)
nbd_mark_nsock_dead(nbd, nsock, 1);
mutex_unlock(&nsock->tx_lock);
nbd_config_put(nbd);
atomic_dec(&config->recv_threads);
wake_up(&config->recv_wq);
nbd_config_put(nbd);
kfree(args);
}
@@ -2238,12 +2238,13 @@ again:
ret = nbd_start_device(nbd);
out:
mutex_unlock(&nbd->config_lock);
if (!ret) {
set_bit(NBD_RT_HAS_CONFIG_REF, &config->runtime_flags);
refcount_inc(&nbd->config_refs);
nbd_connect_reply(info, nbd->index);
}
mutex_unlock(&nbd->config_lock);
nbd_config_put(nbd);
if (put_dev)
nbd_put(nbd);

82
drivers/block/null_blk/main.c
View File

@@ -1129,26 +1129,28 @@ again:
return 0;
}
static int copy_to_nullb(struct nullb *nullb, struct page *source,
unsigned int off, sector_t sector, size_t n, bool is_fua)
static blk_status_t copy_to_nullb(struct nullb *nullb, void *source,
loff_t pos, size_t n, bool is_fua)
{
size_t temp, count = 0;
unsigned int offset;
struct nullb_page *t_page;
sector_t sector;
while (count < n) {
temp = min_t(size_t, nullb->dev->blocksize, n - count);
temp = min3(nullb->dev->blocksize, n - count,
PAGE_SIZE - offset_in_page(pos));
sector = pos >> SECTOR_SHIFT;
if (null_cache_active(nullb) && !is_fua)
null_make_cache_space(nullb, PAGE_SIZE);
offset = (sector & SECTOR_MASK) << SECTOR_SHIFT;
t_page = null_insert_page(nullb, sector,
!null_cache_active(nullb) || is_fua);
if (!t_page)
return -ENOSPC;
return BLK_STS_NOSPC;
memcpy_page(t_page->page, offset, source, off + count, temp);
memcpy_to_page(t_page->page, offset_in_page(pos),
source + count, temp);
__set_bit(sector & SECTOR_MASK, t_page->bitmap);
@@ -1156,41 +1158,34 @@ static int copy_to_nullb(struct nullb *nullb, struct page *source,
null_free_sector(nullb, sector, true);
count += temp;
sector += temp >> SECTOR_SHIFT;
pos += temp;
}
return 0;
return BLK_STS_OK;
}
static int copy_from_nullb(struct nullb *nullb, struct page *dest,
unsigned int off, sector_t sector, size_t n)
static void copy_from_nullb(struct nullb *nullb, void *dest, loff_t pos,
size_t n)
{
size_t temp, count = 0;
unsigned int offset;
struct nullb_page *t_page;
sector_t sector;
while (count < n) {
temp = min_t(size_t, nullb->dev->blocksize, n - count);
temp = min3(nullb->dev->blocksize, n - count,
PAGE_SIZE - offset_in_page(pos));
sector = pos >> SECTOR_SHIFT;
offset = (sector & SECTOR_MASK) << SECTOR_SHIFT;
t_page = null_lookup_page(nullb, sector, false,
!null_cache_active(nullb));
if (t_page)
memcpy_page(dest, off + count, t_page->page, offset,
temp);
memcpy_from_page(dest + count, t_page->page,
offset_in_page(pos), temp);
else
memzero_page(dest, off + count, temp);
memset(dest + count, 0, temp);
count += temp;
sector += temp >> SECTOR_SHIFT;
pos += temp;
}
return 0;
}
static void nullb_fill_pattern(struct nullb *nullb, struct page *page,
unsigned int len, unsigned int off)
{
memset_page(page, off, 0xff, len);
}
blk_status_t null_handle_discard(struct nullb_device *dev,
@@ -1234,34 +1229,39 @@ static blk_status_t null_handle_flush(struct nullb *nullb)
return errno_to_blk_status(err);
}
static int null_transfer(struct nullb *nullb, struct page *page,
unsigned int len, unsigned int off, bool is_write, sector_t sector,
static blk_status_t null_transfer(struct nullb *nullb, struct page *page,
unsigned int len, unsigned int off, bool is_write, loff_t pos,
bool is_fua)
{
struct nullb_device *dev = nullb->dev;
blk_status_t err = BLK_STS_OK;
unsigned int valid_len = len;
int err = 0;
void *p;
p = kmap_local_page(page) + off;
if (!is_write) {
if (dev->zoned)
if (dev->zoned) {
valid_len = null_zone_valid_read_len(nullb,
sector, len);
pos >> SECTOR_SHIFT, len);
if (valid_len && valid_len != len)
valid_len -= pos & (SECTOR_SIZE - 1);
}
if (valid_len) {
err = copy_from_nullb(nullb, page, off,
sector, valid_len);
copy_from_nullb(nullb, p, pos, valid_len);
off += valid_len;
len -= valid_len;
}
if (len)
nullb_fill_pattern(nullb, page, len, off);
memset(p + valid_len, 0xff, len);
flush_dcache_page(page);
} else {
flush_dcache_page(page);
err = copy_to_nullb(nullb, page, off, sector, len, is_fua);
err = copy_to_nullb(nullb, p, pos, len, is_fua);
}
kunmap_local(p);
return err;
}
@@ -1274,9 +1274,9 @@ static blk_status_t null_handle_data_transfer(struct nullb_cmd *cmd,
{
struct request *rq = blk_mq_rq_from_pdu(cmd);
struct nullb *nullb = cmd->nq->dev->nullb;
int err = 0;
blk_status_t err = BLK_STS_OK;
unsigned int len;
sector_t sector = blk_rq_pos(rq);
loff_t pos = blk_rq_pos(rq) << SECTOR_SHIFT;
unsigned int max_bytes = nr_sectors << SECTOR_SHIFT;
unsigned int transferred_bytes = 0;
struct req_iterator iter;
@@ -1288,18 +1288,18 @@ static blk_status_t null_handle_data_transfer(struct nullb_cmd *cmd,
if (transferred_bytes + len > max_bytes)
len = max_bytes - transferred_bytes;
err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset,
op_is_write(req_op(rq)), sector,
op_is_write(req_op(rq)), pos,
rq->cmd_flags & REQ_FUA);
if (err)
break;
sector += len >> SECTOR_SHIFT;
pos += len;
transferred_bytes += len;
if (transferred_bytes >= max_bytes)
break;
}
spin_unlock_irq(&nullb->lock);
return errno_to_blk_status(err);
return err;
}
static inline blk_status_t null_handle_throttled(struct nullb_cmd *cmd)
@@ -1949,7 +1949,7 @@ static int null_add_dev(struct nullb_device *dev)
.logical_block_size = dev->blocksize,
.physical_block_size = dev->blocksize,
.max_hw_sectors = dev->max_sectors,
.dma_alignment = dev->blocksize - 1,
.dma_alignment = 1,
};
struct nullb *nullb;

3
drivers/block/null_blk/null_blk.h
View File

@@ -143,7 +143,8 @@ int null_init_zoned_dev(struct nullb_device *dev, struct queue_limits *lim);
int null_register_zoned_dev(struct nullb *nullb);
void null_free_zoned_dev(struct nullb_device *dev);
int null_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data);
unsigned int nr_zones,
struct blk_report_zones_args *args);
blk_status_t null_process_zoned_cmd(struct nullb_cmd *cmd, enum req_op op,
sector_t sector, sector_t nr_sectors);
size_t null_zone_valid_read_len(struct nullb *nullb,

6
drivers/block/null_blk/zoned.c
View File

@@ -191,7 +191,7 @@ void null_free_zoned_dev(struct nullb_device *dev)
}
int null_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones, struct blk_report_zones_args *args)
{
struct nullb *nullb = disk->private_data;
struct nullb_device *dev = nullb->dev;
@@ -225,7 +225,7 @@ int null_report_zones(struct gendisk *disk, sector_t sector,
blkz.capacity = zone->capacity;
null_unlock_zone(dev, zone);
error = cb(&blkz, i, data);
error = disk_report_zone(disk, &blkz, i, args);
if (error)
return error;
}
@@ -242,7 +242,7 @@ size_t null_zone_valid_read_len(struct nullb *nullb,
{
struct nullb_device *dev = nullb->dev;
struct nullb_zone *zone = &dev->zones[null_zone_no(dev, sector)];
unsigned int nr_sectors = len >> SECTOR_SHIFT;
unsigned int nr_sectors = DIV_ROUND_UP(len, SECTOR_SIZE);
/* Read must be below the write pointer position */
if (zone->type == BLK_ZONE_TYPE_CONVENTIONAL ||

4
drivers/block/ps3disk.c
View File

@@ -85,10 +85,14 @@ static void ps3disk_scatter_gather(struct ps3_storage_device *dev,
struct bio_vec bvec;
rq_for_each_segment(bvec, req, iter) {
dev_dbg(&dev->sbd.core, "%s:%u: %u sectors from %llu\n",
__func__, __LINE__, bio_sectors(iter.bio),
iter.bio->bi_iter.bi_sector);
if (gather)
memcpy_from_bvec(dev->bounce_buf + offset, &bvec);
else
memcpy_to_bvec(&bvec, dev->bounce_buf + offset);
offset += bvec.bv_len;
}
}

15
drivers/block/rnbd/rnbd-proto.h
View File

@@ -24,7 +24,7 @@
#define RTRS_PORT 1234
/**
* enum rnbd_msg_types - RNBD message types
* enum rnbd_msg_type - RNBD message types
* @RNBD_MSG_SESS_INFO: initial session info from client to server
* @RNBD_MSG_SESS_INFO_RSP: initial session info from server to client
* @RNBD_MSG_OPEN: open (map) device request
@@ -47,10 +47,11 @@ enum rnbd_msg_type {
*/
struct rnbd_msg_hdr {
__le16 type;
/* private: */
__le16 __padding;
};
/**
/*
* We allow to map RO many times and RW only once. We allow to map yet another
* time RW, if MIGRATION is provided (second RW export can be required for
* example for VM migration)
@@ -78,6 +79,7 @@ static const __maybe_unused struct {
struct rnbd_msg_sess_info {
struct rnbd_msg_hdr hdr;
u8 ver;
/* private: */
u8 reserved[31];
};
@@ -89,6 +91,7 @@ struct rnbd_msg_sess_info {
struct rnbd_msg_sess_info_rsp {
struct rnbd_msg_hdr hdr;
u8 ver;
/* private: */
u8 reserved[31];
};
@@ -97,13 +100,16 @@ struct rnbd_msg_sess_info_rsp {
* @hdr: message header
* @access_mode: the mode to open remote device, valid values see:
* enum rnbd_access_mode
* @device_name: device path on remote side
* @dev_name: device path on remote side
*/
struct rnbd_msg_open {
struct rnbd_msg_hdr hdr;
u8 access_mode;
/* private: */
u8 resv1;
/* public: */
s8 dev_name[NAME_MAX];
/* private: */
u8 reserved[3];
};
@@ -155,6 +161,7 @@ struct rnbd_msg_open_rsp {
__le16 secure_discard;
u8 obsolete_rotational;
u8 cache_policy;
/* private: */
u8 reserved[10];
};
@@ -187,7 +194,7 @@ struct rnbd_msg_io {
* @RNBD_OP_DISCARD: discard sectors
* @RNBD_OP_SECURE_ERASE: securely erase sectors
* @RNBD_OP_WRITE_ZEROES: write zeroes sectors
*
* @RNBD_F_SYNC: request is sync (sync write or read)
* @RNBD_F_FUA: forced unit access
*/

3
drivers/block/rnull/rnull.rs
View File

@@ -17,8 +17,7 @@ use kernel::{
error::Result,
pr_info,
prelude::*,
sync::Arc,
types::ARef,
sync::{aref::ARef, Arc},
};
use pin_init::PinInit;

387
drivers/block/ublk_drv.c
View File

@@ -155,12 +155,13 @@ struct ublk_uring_cmd_pdu {
*/
#define UBLK_REFCOUNT_INIT (REFCOUNT_MAX / 2)
union ublk_io_buf {
__u64 addr;
struct ublk_auto_buf_reg auto_reg;
};
struct ublk_io {
/* userspace buffer address from io cmd */
union {
__u64 addr;
struct ublk_auto_buf_reg buf;
};
union ublk_io_buf buf;
unsigned int flags;
int res;
@@ -203,15 +204,12 @@ struct ublk_queue {
bool fail_io; /* copy of dev->state == UBLK_S_DEV_FAIL_IO */
spinlock_t cancel_lock;
struct ublk_device *dev;
struct ublk_io ios[];
struct ublk_io ios[] __counted_by(q_depth);
};
struct ublk_device {
struct gendisk *ub_disk;
char *__queues;
unsigned int queue_size;
struct ublksrv_ctrl_dev_info dev_info;
struct blk_mq_tag_set tag_set;
@@ -239,6 +237,8 @@ struct ublk_device {
bool canceling;
pid_t ublksrv_tgid;
struct delayed_work exit_work;
struct ublk_queue *queues[];
};
/* header of ublk_params */
@@ -265,7 +265,7 @@ static inline bool ublk_dev_is_zoned(const struct ublk_device *ub)
return ub->dev_info.flags & UBLK_F_ZONED;
}
static inline bool ublk_queue_is_zoned(struct ublk_queue *ubq)
static inline bool ublk_queue_is_zoned(const struct ublk_queue *ubq)
{
return ubq->flags & UBLK_F_ZONED;
}
@@ -368,7 +368,7 @@ static void *ublk_alloc_report_buffer(struct ublk_device *ublk,
}
static int ublk_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones, struct blk_report_zones_args *args)
{
struct ublk_device *ub = disk->private_data;
unsigned int zone_size_sectors = disk->queue->limits.chunk_sectors;
@@ -431,7 +431,7 @@ free_req:
if (!zone->len)
break;
ret = cb(zone, i, data);
ret = disk_report_zone(disk, zone, i, args);
if (ret)
goto out;
@@ -499,7 +499,7 @@ static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
iod->op_flags = ublk_op | ublk_req_build_flags(req);
iod->nr_sectors = blk_rq_sectors(req);
iod->start_sector = blk_rq_pos(req);
iod->addr = io->addr;
iod->addr = io->buf.addr;
return BLK_STS_OK;
}
@@ -781,7 +781,7 @@ static noinline void ublk_put_device(struct ublk_device *ub)
static inline struct ublk_queue *ublk_get_queue(struct ublk_device *dev,
int qid)
{
return (struct ublk_queue *)&(dev->__queues[qid * dev->queue_size]);
return dev->queues[qid];
}
static inline bool ublk_rq_has_data(const struct request *rq)
@@ -914,73 +914,6 @@ static const struct block_device_operations ub_fops = {
.report_zones = ublk_report_zones,
};
#define UBLK_MAX_PIN_PAGES 32
struct ublk_io_iter {
struct page *pages[UBLK_MAX_PIN_PAGES];
struct bio *bio;
struct bvec_iter iter;
};
/* return how many pages are copied */
static void ublk_copy_io_pages(struct ublk_io_iter *data,
size_t total, size_t pg_off, int dir)
{
unsigned done = 0;
unsigned pg_idx = 0;
while (done < total) {
struct bio_vec bv = bio_iter_iovec(data->bio, data->iter);
unsigned int bytes = min3(bv.bv_len, (unsigned)total - done,
(unsigned)(PAGE_SIZE - pg_off));
void *bv_buf = bvec_kmap_local(&bv);
void *pg_buf = kmap_local_page(data->pages[pg_idx]);
if (dir == ITER_DEST)
memcpy(pg_buf + pg_off, bv_buf, bytes);
else
memcpy(bv_buf, pg_buf + pg_off, bytes);
kunmap_local(pg_buf);
kunmap_local(bv_buf);
/* advance page array */
pg_off += bytes;
if (pg_off == PAGE_SIZE) {
pg_idx += 1;
pg_off = 0;
}
done += bytes;
/* advance bio */
bio_advance_iter_single(data->bio, &data->iter, bytes);
if (!data->iter.bi_size) {
data->bio = data->bio->bi_next;
if (data->bio == NULL)
break;
data->iter = data->bio->bi_iter;
}
}
}
static bool ublk_advance_io_iter(const struct request *req,
struct ublk_io_iter *iter, unsigned int offset)
{
struct bio *bio = req->bio;
for_each_bio(bio) {
if (bio->bi_iter.bi_size > offset) {
iter->bio = bio;
iter->iter = bio->bi_iter;
bio_advance_iter(iter->bio, &iter->iter, offset);
return true;
}
offset -= bio->bi_iter.bi_size;
}
return false;
}
/*
* Copy data between request pages and io_iter, and 'offset'
* is the start point of linear offset of request.
@@ -988,34 +921,35 @@ static bool ublk_advance_io_iter(const struct request *req,
static size_t ublk_copy_user_pages(const struct request *req,
unsigned offset, struct iov_iter *uiter, int dir)
{
struct ublk_io_iter iter;
struct req_iterator iter;
struct bio_vec bv;
size_t done = 0;
if (!ublk_advance_io_iter(req, &iter, offset))
return 0;
rq_for_each_segment(bv, req, iter) {
void *bv_buf;
size_t copied;
while (iov_iter_count(uiter) && iter.bio) {
unsigned nr_pages;
ssize_t len;
size_t off;
int i;
len = iov_iter_get_pages2(uiter, iter.pages,
iov_iter_count(uiter),
UBLK_MAX_PIN_PAGES, &off);
if (len <= 0)
return done;
ublk_copy_io_pages(&iter, len, off, dir);
nr_pages = DIV_ROUND_UP(len + off, PAGE_SIZE);
for (i = 0; i < nr_pages; i++) {
if (dir == ITER_DEST)
set_page_dirty(iter.pages[i]);
put_page(iter.pages[i]);
if (offset >= bv.bv_len) {
offset -= bv.bv_len;
continue;
}
done += len;
}
bv.bv_offset += offset;
bv.bv_len -= offset;
bv_buf = bvec_kmap_local(&bv);
if (dir == ITER_DEST)
copied = copy_to_iter(bv_buf, bv.bv_len, uiter);
else
copied = copy_from_iter(bv_buf, bv.bv_len, uiter);
kunmap_local(bv_buf);
done += copied;
if (copied < bv.bv_len)
break;
offset = 0;
}
return done;
}
@@ -1030,8 +964,9 @@ static inline bool ublk_need_unmap_req(const struct request *req)
(req_op(req) == REQ_OP_READ || req_op(req) == REQ_OP_DRV_IN);
}
static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req,
const struct ublk_io *io)
static unsigned int ublk_map_io(const struct ublk_queue *ubq,
const struct request *req,
const struct ublk_io *io)
{
const unsigned int rq_bytes = blk_rq_bytes(req);
@@ -1047,13 +982,13 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req,
struct iov_iter iter;
const int dir = ITER_DEST;
import_ubuf(dir, u64_to_user_ptr(io->addr), rq_bytes, &iter);
import_ubuf(dir, u64_to_user_ptr(io->buf.addr), rq_bytes, &iter);
return ublk_copy_user_pages(req, 0, &iter, dir);
}
return rq_bytes;
}
static int ublk_unmap_io(bool need_map,
static unsigned int ublk_unmap_io(bool need_map,
const struct request *req,
const struct ublk_io *io)
{
@@ -1068,7 +1003,7 @@ static int ublk_unmap_io(bool need_map,
WARN_ON_ONCE(io->res > rq_bytes);
import_ubuf(dir, u64_to_user_ptr(io->addr), io->res, &iter);
import_ubuf(dir, u64_to_user_ptr(io->buf.addr), io->res, &iter);
return ublk_copy_user_pages(req, 0, &iter, dir);
}
return rq_bytes;
@@ -1134,7 +1069,7 @@ static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req)
iod->op_flags = ublk_op | ublk_req_build_flags(req);
iod->nr_sectors = blk_rq_sectors(req);
iod->start_sector = blk_rq_pos(req);
iod->addr = io->addr;
iod->addr = io->buf.addr;
return BLK_STS_OK;
}
@@ -1233,45 +1168,65 @@ static inline void __ublk_abort_rq(struct ublk_queue *ubq,
}
static void
ublk_auto_buf_reg_fallback(const struct ublk_queue *ubq, struct ublk_io *io)
ublk_auto_buf_reg_fallback(const struct ublk_queue *ubq, unsigned tag)
{
unsigned tag = io - ubq->ios;
struct ublksrv_io_desc *iod = ublk_get_iod(ubq, tag);
iod->op_flags |= UBLK_IO_F_NEED_REG_BUF;
}
static bool ublk_auto_buf_reg(const struct ublk_queue *ubq, struct request *req,
struct ublk_io *io, unsigned int issue_flags)
enum auto_buf_reg_res {
AUTO_BUF_REG_FAIL,
AUTO_BUF_REG_FALLBACK,
AUTO_BUF_REG_OK,
};
static void ublk_prep_auto_buf_reg_io(const struct ublk_queue *ubq,
struct request *req, struct ublk_io *io,
struct io_uring_cmd *cmd,
enum auto_buf_reg_res res)
{
if (res == AUTO_BUF_REG_OK) {
io->task_registered_buffers = 1;
io->buf_ctx_handle = io_uring_cmd_ctx_handle(cmd);
io->flags |= UBLK_IO_FLAG_AUTO_BUF_REG;
}
ublk_init_req_ref(ubq, io);
__ublk_prep_compl_io_cmd(io, req);
}
static enum auto_buf_reg_res
__ublk_do_auto_buf_reg(const struct ublk_queue *ubq, struct request *req,
struct ublk_io *io, struct io_uring_cmd *cmd,
unsigned int issue_flags)
{
int ret;
ret = io_buffer_register_bvec(io->cmd, req, ublk_io_release,
io->buf.index, issue_flags);
ret = io_buffer_register_bvec(cmd, req, ublk_io_release,
io->buf.auto_reg.index, issue_flags);
if (ret) {
if (io->buf.flags & UBLK_AUTO_BUF_REG_FALLBACK) {
ublk_auto_buf_reg_fallback(ubq, io);
return true;
if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) {
ublk_auto_buf_reg_fallback(ubq, req->tag);
return AUTO_BUF_REG_FALLBACK;
}
blk_mq_end_request(req, BLK_STS_IOERR);
return false;
return AUTO_BUF_REG_FAIL;
}
io->task_registered_buffers = 1;
io->buf_ctx_handle = io_uring_cmd_ctx_handle(io->cmd);
io->flags |= UBLK_IO_FLAG_AUTO_BUF_REG;
return true;
return AUTO_BUF_REG_OK;
}
static bool ublk_prep_auto_buf_reg(struct ublk_queue *ubq,
struct request *req, struct ublk_io *io,
unsigned int issue_flags)
static void ublk_do_auto_buf_reg(const struct ublk_queue *ubq, struct request *req,
struct ublk_io *io, struct io_uring_cmd *cmd,
unsigned int issue_flags)
{
ublk_init_req_ref(ubq, io);
if (ublk_support_auto_buf_reg(ubq) && ublk_rq_has_data(req))
return ublk_auto_buf_reg(ubq, req, io, issue_flags);
enum auto_buf_reg_res res = __ublk_do_auto_buf_reg(ubq, req, io, cmd,
issue_flags);
return true;
if (res != AUTO_BUF_REG_FAIL) {
ublk_prep_auto_buf_reg_io(ubq, req, io, cmd, res);
io_uring_cmd_done(cmd, UBLK_IO_RES_OK, issue_flags);
}
}
static bool ublk_start_io(const struct ublk_queue *ubq, struct request *req,
@@ -1343,8 +1298,12 @@ static void ublk_dispatch_req(struct ublk_queue *ubq, struct request *req)
if (!ublk_start_io(ubq, req, io))
return;
if (ublk_prep_auto_buf_reg(ubq, req, io, issue_flags))
if (ublk_support_auto_buf_reg(ubq) && ublk_rq_has_data(req)) {
ublk_do_auto_buf_reg(ubq, req, io, io->cmd, issue_flags);
} else {
ublk_init_req_ref(ubq, io);
ublk_complete_io_cmd(io, req, UBLK_IO_RES_OK, issue_flags);
}
}
static void ublk_cmd_tw_cb(struct io_tw_req tw_req, io_tw_token_t tw)
@@ -1536,7 +1495,7 @@ static void ublk_queue_reinit(struct ublk_device *ub, struct ublk_queue *ubq)
*/
io->flags &= UBLK_IO_FLAG_CANCELED;
io->cmd = NULL;
io->addr = 0;
io->buf.addr = 0;
/*
* old task is PF_EXITING, put it now
@@ -2097,13 +2056,16 @@ static inline int ublk_check_cmd_op(u32 cmd_op)
static inline int ublk_set_auto_buf_reg(struct ublk_io *io, struct io_uring_cmd *cmd)
{
io->buf = ublk_sqe_addr_to_auto_buf_reg(READ_ONCE(cmd->sqe->addr));
struct ublk_auto_buf_reg buf;
if (io->buf.reserved0 || io->buf.reserved1)
buf = ublk_sqe_addr_to_auto_buf_reg(READ_ONCE(cmd->sqe->addr));
if (buf.reserved0 || buf.reserved1)
return -EINVAL;
if (io->buf.flags & ~UBLK_AUTO_BUF_REG_F_MASK)
if (buf.flags & ~UBLK_AUTO_BUF_REG_F_MASK)
return -EINVAL;
io->buf.auto_reg = buf;
return 0;
}
@@ -2125,7 +2087,7 @@ static int ublk_handle_auto_buf_reg(struct ublk_io *io,
* this ublk request gets stuck.
*/
if (io->buf_ctx_handle == io_uring_cmd_ctx_handle(cmd))
*buf_idx = io->buf.index;
*buf_idx = io->buf.auto_reg.index;
}
return ublk_set_auto_buf_reg(io, cmd);
@@ -2153,7 +2115,7 @@ ublk_config_io_buf(const struct ublk_device *ub, struct ublk_io *io,
if (ublk_dev_support_auto_buf_reg(ub))
return ublk_handle_auto_buf_reg(io, cmd, buf_idx);
io->addr = buf_addr;
io->buf.addr = buf_addr;
return 0;
}
@@ -2271,10 +2233,31 @@ static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr)
return 0;
}
static int __ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
struct ublk_io *io)
{
/* UBLK_IO_FETCH_REQ is only allowed before dev is setup */
if (ublk_dev_ready(ub))
return -EBUSY;
/* allow each command to be FETCHed at most once */
if (io->flags & UBLK_IO_FLAG_ACTIVE)
return -EINVAL;
WARN_ON_ONCE(io->flags & UBLK_IO_FLAG_OWNED_BY_SRV);
ublk_fill_io_cmd(io, cmd);
WRITE_ONCE(io->task, get_task_struct(current));
ublk_mark_io_ready(ub);
return 0;
}
static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
struct ublk_io *io, __u64 buf_addr)
{
int ret = 0;
int ret;
/*
* When handling FETCH command for setting up ublk uring queue,
@@ -2282,28 +2265,9 @@ static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
* FETCH, so it is fine even for IO_URING_F_NONBLOCK.
*/
mutex_lock(&ub->mutex);
/* UBLK_IO_FETCH_REQ is only allowed before dev is setup */
if (ublk_dev_ready(ub)) {
ret = -EBUSY;
goto out;
}
/* allow each command to be FETCHed at most once */
if (io->flags & UBLK_IO_FLAG_ACTIVE) {
ret = -EINVAL;
goto out;
}
WARN_ON_ONCE(io->flags & UBLK_IO_FLAG_OWNED_BY_SRV);
ublk_fill_io_cmd(io, cmd);
ret = ublk_config_io_buf(ub, io, cmd, buf_addr, NULL);
if (ret)
goto out;
WRITE_ONCE(io->task, get_task_struct(current));
ublk_mark_io_ready(ub);
out:
ret = __ublk_fetch(cmd, ub, io);
if (!ret)
ret = ublk_config_io_buf(ub, io, cmd, buf_addr, NULL);
mutex_unlock(&ub->mutex);
return ret;
}
@@ -2350,7 +2314,7 @@ static bool ublk_get_data(const struct ublk_queue *ubq, struct ublk_io *io,
*/
io->flags &= ~UBLK_IO_FLAG_NEED_GET_DATA;
/* update iod->addr because ublksrv may have passed a new io buffer */
ublk_get_iod(ubq, req->tag)->addr = io->addr;
ublk_get_iod(ubq, req->tag)->addr = io->buf.addr;
pr_devel("%s: update iod->addr: qid %d tag %d io_flags %x addr %llx\n",
__func__, ubq->q_id, req->tag, io->flags,
ublk_get_iod(ubq, req->tag)->addr);
@@ -2366,7 +2330,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
u16 buf_idx = UBLK_INVALID_BUF_IDX;
struct ublk_device *ub = cmd->file->private_data;
struct ublk_queue *ubq;
struct ublk_io *io;
struct ublk_io *io = NULL;
u32 cmd_op = cmd->cmd_op;
u16 q_id = READ_ONCE(ub_src->q_id);
u16 tag = READ_ONCE(ub_src->tag);
@@ -2487,7 +2451,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
out:
pr_devel("%s: complete: cmd op %d, tag %d ret %x io_flags %x\n",
__func__, cmd_op, tag, ret, io->flags);
__func__, cmd_op, tag, ret, io ? io->flags : 0);
return ret;
}
@@ -2575,9 +2539,6 @@ static struct request *ublk_check_and_get_req(struct kiocb *iocb,
size_t buf_off;
u16 tag, q_id;
if (!ub)
return ERR_PTR(-EACCES);
if (!user_backed_iter(iter))
return ERR_PTR(-EACCES);
@@ -2603,9 +2564,6 @@ static struct request *ublk_check_and_get_req(struct kiocb *iocb,
if (!req)
return ERR_PTR(-EINVAL);
if (!req->mq_hctx || !req->mq_hctx->driver_data)
goto fail;
if (!ublk_check_ubuf_dir(req, dir))
goto fail;
@@ -2662,9 +2620,13 @@ static const struct file_operations ublk_ch_fops = {
static void ublk_deinit_queue(struct ublk_device *ub, int q_id)
{
int size = ublk_queue_cmd_buf_size(ub);
struct ublk_queue *ubq = ublk_get_queue(ub, q_id);
int i;
struct ublk_queue *ubq = ub->queues[q_id];
int size, i;
if (!ubq)
return;
size = ublk_queue_cmd_buf_size(ub);
for (i = 0; i < ubq->q_depth; i++) {
struct ublk_io *io = &ubq->ios[i];
@@ -2676,57 +2638,76 @@ static void ublk_deinit_queue(struct ublk_device *ub, int q_id)
if (ubq->io_cmd_buf)
free_pages((unsigned long)ubq->io_cmd_buf, get_order(size));
kvfree(ubq);
ub->queues[q_id] = NULL;
}
static int ublk_get_queue_numa_node(struct ublk_device *ub, int q_id)
{
unsigned int cpu;
/* Find first CPU mapped to this queue */
for_each_possible_cpu(cpu) {
if (ub->tag_set.map[HCTX_TYPE_DEFAULT].mq_map[cpu] == q_id)
return cpu_to_node(cpu);
}
return NUMA_NO_NODE;
}
static int ublk_init_queue(struct ublk_device *ub, int q_id)
{
struct ublk_queue *ubq = ublk_get_queue(ub, q_id);
int depth = ub->dev_info.queue_depth;
gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO;
void *ptr;
struct ublk_queue *ubq;
struct page *page;
int numa_node;
int size;
/* Determine NUMA node based on queue's CPU affinity */
numa_node = ublk_get_queue_numa_node(ub, q_id);
/* Allocate queue structure on local NUMA node */
ubq = kvzalloc_node(struct_size(ubq, ios, depth), GFP_KERNEL,
numa_node);
if (!ubq)
return -ENOMEM;
spin_lock_init(&ubq->cancel_lock);
ubq->flags = ub->dev_info.flags;
ubq->q_id = q_id;
ubq->q_depth = ub->dev_info.queue_depth;
ubq->q_depth = depth;
size = ublk_queue_cmd_buf_size(ub);
ptr = (void *) __get_free_pages(gfp_flags, get_order(size));
if (!ptr)
/* Allocate I/O command buffer on local NUMA node */
page = alloc_pages_node(numa_node, gfp_flags, get_order(size));
if (!page) {
kvfree(ubq);
return -ENOMEM;
}
ubq->io_cmd_buf = page_address(page);
ubq->io_cmd_buf = ptr;
ub->queues[q_id] = ubq;
ubq->dev = ub;
return 0;
}
static void ublk_deinit_queues(struct ublk_device *ub)
{
int nr_queues = ub->dev_info.nr_hw_queues;
int i;
if (!ub->__queues)
return;
for (i = 0; i < nr_queues; i++)
for (i = 0; i < ub->dev_info.nr_hw_queues; i++)
ublk_deinit_queue(ub, i);
kvfree(ub->__queues);
}
static int ublk_init_queues(struct ublk_device *ub)
{
int nr_queues = ub->dev_info.nr_hw_queues;
int depth = ub->dev_info.queue_depth;
int ubq_size = sizeof(struct ublk_queue) + depth * sizeof(struct ublk_io);
int i, ret = -ENOMEM;
int i, ret;
ub->queue_size = ubq_size;
ub->__queues = kvcalloc(nr_queues, ubq_size, GFP_KERNEL);
if (!ub->__queues)
return ret;
for (i = 0; i < nr_queues; i++) {
if (ublk_init_queue(ub, i))
for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
ret = ublk_init_queue(ub, i);
if (ret)
goto fail;
}
@@ -3128,7 +3109,7 @@ static int ublk_ctrl_add_dev(const struct ublksrv_ctrl_cmd *header)
goto out_unlock;
ret = -ENOMEM;
ub = kzalloc(sizeof(*ub), GFP_KERNEL);
ub = kzalloc(struct_size(ub, queues, info.nr_hw_queues), GFP_KERNEL);
if (!ub)
goto out_unlock;
mutex_init(&ub->mutex);
@@ -3178,17 +3159,17 @@ static int ublk_ctrl_add_dev(const struct ublksrv_ctrl_cmd *header)
ub->dev_info.nr_hw_queues, nr_cpu_ids);
ublk_align_max_io_size(ub);
ret = ublk_init_queues(ub);
ret = ublk_add_tag_set(ub);
if (ret)
goto out_free_dev_number;
ret = ublk_add_tag_set(ub);
ret = ublk_init_queues(ub);
if (ret)
goto out_deinit_queues;
goto out_free_tag_set;
ret = -EFAULT;
if (copy_to_user(argp, &ub->dev_info, sizeof(info)))
goto out_free_tag_set;
goto out_deinit_queues;
/*
* Add the char dev so that ublksrv daemon can be setup.
@@ -3197,10 +3178,10 @@ static int ublk_ctrl_add_dev(const struct ublksrv_ctrl_cmd *header)
ret = ublk_add_chdev(ub);
goto out_unlock;
out_free_tag_set:
blk_mq_free_tag_set(&ub->tag_set);
out_deinit_queues:
ublk_deinit_queues(ub);
out_free_tag_set:
blk_mq_free_tag_set(&ub->tag_set);
out_free_dev_number:
ublk_free_dev_number(ub);
out_free_ub:

24
drivers/block/virtio_blk.c
View File

@@ -584,7 +584,8 @@ out:
static int virtblk_parse_zone(struct virtio_blk *vblk,
struct virtio_blk_zone_descriptor *entry,
unsigned int idx, report_zones_cb cb, void *data)
unsigned int idx,
struct blk_report_zones_args *args)
{
struct blk_zone zone = { };
@@ -650,12 +651,12 @@ static int virtblk_parse_zone(struct virtio_blk *vblk,
* The callback below checks the validity of the reported
* entry data, no need to further validate it here.
*/
return cb(&zone, idx, data);
return disk_report_zone(vblk->disk, &zone, idx, args);
}
static int virtblk_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb,
void *data)
unsigned int nr_zones,
struct blk_report_zones_args *args)
{
struct virtio_blk *vblk = disk->private_data;
struct virtio_blk_zone_report *report;
@@ -693,7 +694,7 @@ static int virtblk_report_zones(struct gendisk *disk, sector_t sector,
for (i = 0; i < nz && zone_idx < nr_zones; i++) {
ret = virtblk_parse_zone(vblk, &report->zones[i],
zone_idx, cb, data);
zone_idx, args);
if (ret)
goto fail_report;
@@ -1026,8 +1027,13 @@ static int init_vq(struct virtio_blk *vblk)
out:
kfree(vqs);
kfree(vqs_info);
if (err)
if (err) {
kfree(vblk->vqs);
/*
* Set to NULL to prevent freeing vqs again during freezing.
*/
vblk->vqs = NULL;
}
return err;
}
@@ -1598,6 +1604,12 @@ static int virtblk_freeze_priv(struct virtio_device *vdev)
vdev->config->del_vqs(vdev);
kfree(vblk->vqs);
/*
* Set to NULL to prevent freeing vqs again after a failed vqs
* allocation during resume. Note that kfree() already handles NULL
* pointers safely.
*/
vblk->vqs = NULL;
return 0;
}

160
drivers/block/zloop.c
View File

@@ -32,6 +32,8 @@ enum {
ZLOOP_OPT_NR_QUEUES = (1 << 6),
ZLOOP_OPT_QUEUE_DEPTH = (1 << 7),
ZLOOP_OPT_BUFFERED_IO = (1 << 8),
ZLOOP_OPT_ZONE_APPEND = (1 << 9),
ZLOOP_OPT_ORDERED_ZONE_APPEND = (1 << 10),
};
static const match_table_t zloop_opt_tokens = {
@@ -44,6 +46,8 @@ static const match_table_t zloop_opt_tokens = {
{ ZLOOP_OPT_NR_QUEUES, "nr_queues=%u" },
{ ZLOOP_OPT_QUEUE_DEPTH, "queue_depth=%u" },
{ ZLOOP_OPT_BUFFERED_IO, "buffered_io" },
{ ZLOOP_OPT_ZONE_APPEND, "zone_append=%u" },
{ ZLOOP_OPT_ORDERED_ZONE_APPEND, "ordered_zone_append" },
{ ZLOOP_OPT_ERR, NULL }
};
@@ -56,6 +60,8 @@ static const match_table_t zloop_opt_tokens = {
#define ZLOOP_DEF_NR_QUEUES 1
#define ZLOOP_DEF_QUEUE_DEPTH 128
#define ZLOOP_DEF_BUFFERED_IO false
#define ZLOOP_DEF_ZONE_APPEND true
#define ZLOOP_DEF_ORDERED_ZONE_APPEND false
/* Arbitrary limit on the zone size (16GB). */
#define ZLOOP_MAX_ZONE_SIZE_MB 16384
@@ -71,6 +77,8 @@ struct zloop_options {
unsigned int nr_queues;
unsigned int queue_depth;
bool buffered_io;
bool zone_append;
bool ordered_zone_append;
};
/*
@@ -92,6 +100,7 @@ struct zloop_zone {
unsigned long flags;
struct mutex lock;
spinlock_t wp_lock;
enum blk_zone_cond cond;
sector_t start;
sector_t wp;
@@ -108,6 +117,8 @@ struct zloop_device {
struct workqueue_struct *workqueue;
bool buffered_io;
bool zone_append;
bool ordered_zone_append;
const char *base_dir;
struct file *data_dir;
@@ -147,6 +158,7 @@ static int zloop_update_seq_zone(struct zloop_device *zlo, unsigned int zone_no)
struct zloop_zone *zone = &zlo->zones[zone_no];
struct kstat stat;
sector_t file_sectors;
unsigned long flags;
int ret;
lockdep_assert_held(&zone->lock);
@@ -172,16 +184,18 @@ static int zloop_update_seq_zone(struct zloop_device *zlo, unsigned int zone_no)
return -EINVAL;
}
spin_lock_irqsave(&zone->wp_lock, flags);
if (!file_sectors) {
zone->cond = BLK_ZONE_COND_EMPTY;
zone->wp = zone->start;
} else if (file_sectors == zlo->zone_capacity) {
zone->cond = BLK_ZONE_COND_FULL;
zone->wp = zone->start + zlo->zone_size;
zone->wp = ULLONG_MAX;
} else {
zone->cond = BLK_ZONE_COND_CLOSED;
zone->wp = zone->start + file_sectors;
}
spin_unlock_irqrestore(&zone->wp_lock, flags);
return 0;
}
@@ -225,6 +239,7 @@ unlock:
static int zloop_close_zone(struct zloop_device *zlo, unsigned int zone_no)
{
struct zloop_zone *zone = &zlo->zones[zone_no];
unsigned long flags;
int ret = 0;
if (test_bit(ZLOOP_ZONE_CONV, &zone->flags))
@@ -243,10 +258,12 @@ static int zloop_close_zone(struct zloop_device *zlo, unsigned int zone_no)
break;
case BLK_ZONE_COND_IMP_OPEN:
case BLK_ZONE_COND_EXP_OPEN:
spin_lock_irqsave(&zone->wp_lock, flags);
if (zone->wp == zone->start)
zone->cond = BLK_ZONE_COND_EMPTY;
else
zone->cond = BLK_ZONE_COND_CLOSED;
spin_unlock_irqrestore(&zone->wp_lock, flags);
break;
case BLK_ZONE_COND_EMPTY:
case BLK_ZONE_COND_FULL:
@@ -264,6 +281,7 @@ unlock:
static int zloop_reset_zone(struct zloop_device *zlo, unsigned int zone_no)
{
struct zloop_zone *zone = &zlo->zones[zone_no];
unsigned long flags;
int ret = 0;
if (test_bit(ZLOOP_ZONE_CONV, &zone->flags))
@@ -281,9 +299,11 @@ static int zloop_reset_zone(struct zloop_device *zlo, unsigned int zone_no)
goto unlock;
}
spin_lock_irqsave(&zone->wp_lock, flags);
zone->cond = BLK_ZONE_COND_EMPTY;
zone->wp = zone->start;
clear_bit(ZLOOP_ZONE_SEQ_ERROR, &zone->flags);
spin_unlock_irqrestore(&zone->wp_lock, flags);
unlock:
mutex_unlock(&zone->lock);
@@ -308,6 +328,7 @@ static int zloop_reset_all_zones(struct zloop_device *zlo)
static int zloop_finish_zone(struct zloop_device *zlo, unsigned int zone_no)
{
struct zloop_zone *zone = &zlo->zones[zone_no];
unsigned long flags;
int ret = 0;
if (test_bit(ZLOOP_ZONE_CONV, &zone->flags))
@@ -325,9 +346,11 @@ static int zloop_finish_zone(struct zloop_device *zlo, unsigned int zone_no)
goto unlock;
}
spin_lock_irqsave(&zone->wp_lock, flags);
zone->cond = BLK_ZONE_COND_FULL;
zone->wp = zone->start + zlo->zone_size;
zone->wp = ULLONG_MAX;
clear_bit(ZLOOP_ZONE_SEQ_ERROR, &zone->flags);
spin_unlock_irqrestore(&zone->wp_lock, flags);
unlock:
mutex_unlock(&zone->lock);
@@ -369,6 +392,7 @@ static void zloop_rw(struct zloop_cmd *cmd)
struct zloop_zone *zone;
struct iov_iter iter;
struct bio_vec tmp;
unsigned long flags;
sector_t zone_end;
int nr_bvec = 0;
int ret;
@@ -378,6 +402,11 @@ static void zloop_rw(struct zloop_cmd *cmd)
cmd->nr_sectors = nr_sectors;
cmd->ret = 0;
if (WARN_ON_ONCE(is_append && !zlo->zone_append)) {
ret = -EIO;
goto out;
}
/* We should never get an I/O beyond the device capacity. */
if (WARN_ON_ONCE(zone_no >= zlo->nr_zones)) {
ret = -EIO;
@@ -406,16 +435,31 @@ static void zloop_rw(struct zloop_cmd *cmd)
if (!test_bit(ZLOOP_ZONE_CONV, &zone->flags) && is_write) {
mutex_lock(&zone->lock);
if (is_append) {
sector = zone->wp;
cmd->sector = sector;
}
spin_lock_irqsave(&zone->wp_lock, flags);
/*
* Write operations must be aligned to the write pointer and
* fully contained within the zone capacity.
* Zone append operations always go at the current write
* pointer, but regular write operations must already be
* aligned to the write pointer when submitted.
*/
if (sector != zone->wp || zone->wp + nr_sectors > zone_end) {
if (is_append) {
/*
* If ordered zone append is in use, we already checked
* and set the target sector in zloop_queue_rq().
*/
if (!zlo->ordered_zone_append) {
if (zone->cond == BLK_ZONE_COND_FULL ||
zone->wp + nr_sectors > zone_end) {
spin_unlock_irqrestore(&zone->wp_lock,
flags);
ret = -EIO;
goto unlock;
}
sector = zone->wp;
}
cmd->sector = sector;
} else if (sector != zone->wp) {
spin_unlock_irqrestore(&zone->wp_lock, flags);
pr_err("Zone %u: unaligned write: sect %llu, wp %llu\n",
zone_no, sector, zone->wp);
ret = -EIO;
@@ -428,13 +472,19 @@ static void zloop_rw(struct zloop_cmd *cmd)
zone->cond = BLK_ZONE_COND_IMP_OPEN;
/*
* Advance the write pointer of sequential zones. If the write
* fails, the wp position will be corrected when the next I/O
* copmpletes.
* Advance the write pointer, unless ordered zone append is in
* use. If the write fails, the write pointer position will be
* corrected when the next I/O starts execution.
*/
zone->wp += nr_sectors;
if (zone->wp == zone_end)
zone->cond = BLK_ZONE_COND_FULL;
if (!is_append || !zlo->ordered_zone_append) {
zone->wp += nr_sectors;
if (zone->wp == zone_end) {
zone->cond = BLK_ZONE_COND_FULL;
zone->wp = ULLONG_MAX;
}
}
spin_unlock_irqrestore(&zone->wp_lock, flags);
}
rq_for_each_bvec(tmp, rq, rq_iter)
@@ -498,6 +548,10 @@ static void zloop_handle_cmd(struct zloop_cmd *cmd)
struct request *rq = blk_mq_rq_from_pdu(cmd);
struct zloop_device *zlo = rq->q->queuedata;
/* We can block in this context, so ignore REQ_NOWAIT. */
if (rq->cmd_flags & REQ_NOWAIT)
rq->cmd_flags &= ~REQ_NOWAIT;
switch (req_op(rq)) {
case REQ_OP_READ:
case REQ_OP_WRITE:
@@ -608,6 +662,35 @@ static void zloop_complete_rq(struct request *rq)
blk_mq_end_request(rq, sts);
}
static bool zloop_set_zone_append_sector(struct request *rq)
{
struct zloop_device *zlo = rq->q->queuedata;
unsigned int zone_no = rq_zone_no(rq);
struct zloop_zone *zone = &zlo->zones[zone_no];
sector_t zone_end = zone->start + zlo->zone_capacity;
sector_t nr_sectors = blk_rq_sectors(rq);
unsigned long flags;
spin_lock_irqsave(&zone->wp_lock, flags);
if (zone->cond == BLK_ZONE_COND_FULL ||
zone->wp + nr_sectors > zone_end) {
spin_unlock_irqrestore(&zone->wp_lock, flags);
return false;
}
rq->__sector = zone->wp;
zone->wp += blk_rq_sectors(rq);
if (zone->wp >= zone_end) {
zone->cond = BLK_ZONE_COND_FULL;
zone->wp = ULLONG_MAX;
}
spin_unlock_irqrestore(&zone->wp_lock, flags);
return true;
}
static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,
const struct blk_mq_queue_data *bd)
{
@@ -618,6 +701,16 @@ static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,
if (zlo->state == Zlo_deleting)
return BLK_STS_IOERR;
/*
* If we need to strongly order zone append operations, set the request
* sector to the zone write pointer location now instead of when the
* command work runs.
*/
if (zlo->ordered_zone_append && req_op(rq) == REQ_OP_ZONE_APPEND) {
if (!zloop_set_zone_append_sector(rq))
return BLK_STS_IOERR;
}
blk_mq_start_request(rq);
INIT_WORK(&cmd->work, zloop_cmd_workfn);
@@ -647,11 +740,12 @@ static int zloop_open(struct gendisk *disk, blk_mode_t mode)
}
static int zloop_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones, struct blk_report_zones_args *args)
{
struct zloop_device *zlo = disk->private_data;
struct blk_zone blkz = {};
unsigned int first, i;
unsigned long flags;
int ret;
first = disk_zone_no(disk, sector);
@@ -675,7 +769,9 @@ static int zloop_report_zones(struct gendisk *disk, sector_t sector,
blkz.start = zone->start;
blkz.len = zlo->zone_size;
spin_lock_irqsave(&zone->wp_lock, flags);
blkz.wp = zone->wp;
spin_unlock_irqrestore(&zone->wp_lock, flags);
blkz.cond = zone->cond;
if (test_bit(ZLOOP_ZONE_CONV, &zone->flags)) {
blkz.type = BLK_ZONE_TYPE_CONVENTIONAL;
@@ -687,7 +783,7 @@ static int zloop_report_zones(struct gendisk *disk, sector_t sector,
mutex_unlock(&zone->lock);
ret = cb(&blkz, i, data);
ret = disk_report_zone(disk, &blkz, i, args);
if (ret)
return ret;
}
@@ -783,6 +879,7 @@ static int zloop_init_zone(struct zloop_device *zlo, struct zloop_options *opts,
int ret;
mutex_init(&zone->lock);
spin_lock_init(&zone->wp_lock);
zone->start = (sector_t)zone_no << zlo->zone_shift;
if (!restore)
@@ -884,7 +981,6 @@ static int zloop_ctl_add(struct zloop_options *opts)
{
struct queue_limits lim = {
.max_hw_sectors = SZ_1M >> SECTOR_SHIFT,
.max_hw_zone_append_sectors = SZ_1M >> SECTOR_SHIFT,
.chunk_sectors = opts->zone_size,
.features = BLK_FEAT_ZONED,
};
@@ -936,6 +1032,9 @@ static int zloop_ctl_add(struct zloop_options *opts)
zlo->nr_zones = nr_zones;
zlo->nr_conv_zones = opts->nr_conv_zones;
zlo->buffered_io = opts->buffered_io;
zlo->zone_append = opts->zone_append;
if (zlo->zone_append)
zlo->ordered_zone_append = opts->ordered_zone_append;
zlo->workqueue = alloc_workqueue("zloop%d", WQ_UNBOUND | WQ_FREEZABLE,
opts->nr_queues * opts->queue_depth, zlo->id);
@@ -976,6 +1075,8 @@ static int zloop_ctl_add(struct zloop_options *opts)
lim.physical_block_size = zlo->block_size;
lim.logical_block_size = zlo->block_size;
if (zlo->zone_append)
lim.max_hw_zone_append_sectors = lim.max_hw_sectors;
zlo->tag_set.ops = &zloop_mq_ops;
zlo->tag_set.nr_hw_queues = opts->nr_queues;
@@ -1016,10 +1117,14 @@ static int zloop_ctl_add(struct zloop_options *opts)
zlo->state = Zlo_live;
mutex_unlock(&zloop_ctl_mutex);
pr_info("Added device %d: %u zones of %llu MB, %u B block size\n",
pr_info("zloop: device %d, %u zones of %llu MiB, %u B block size\n",
zlo->id, zlo->nr_zones,
((sector_t)zlo->zone_size << SECTOR_SHIFT) >> 20,
zlo->block_size);
pr_info("zloop%d: using %s%s zone append\n",
zlo->id,
zlo->ordered_zone_append ? "ordered " : "",
zlo->zone_append ? "native" : "emulated");
return 0;
@@ -1106,6 +1211,8 @@ static int zloop_parse_options(struct zloop_options *opts, const char *buf)
opts->nr_queues = ZLOOP_DEF_NR_QUEUES;
opts->queue_depth = ZLOOP_DEF_QUEUE_DEPTH;
opts->buffered_io = ZLOOP_DEF_BUFFERED_IO;
opts->zone_append = ZLOOP_DEF_ZONE_APPEND;
opts->ordered_zone_append = ZLOOP_DEF_ORDERED_ZONE_APPEND;
if (!buf)
return 0;
@@ -1215,6 +1322,21 @@ static int zloop_parse_options(struct zloop_options *opts, const char *buf)
case ZLOOP_OPT_BUFFERED_IO:
opts->buffered_io = true;
break;
case ZLOOP_OPT_ZONE_APPEND:
if (match_uint(args, &token)) {
ret = -EINVAL;
goto out;
}
if (token != 0 && token != 1) {
pr_err("Invalid zone_append value\n");
ret = -EINVAL;
goto out;
}
opts->zone_append = token;
break;
case ZLOOP_OPT_ORDERED_ZONE_APPEND:
opts->ordered_zone_append = true;
break;
case ZLOOP_OPT_ERR:
default:
pr_warn("unknown parameter or missing value '%s'\n", p);

25
drivers/md/bcache/alloc.c
View File

@@ -24,21 +24,18 @@
* Since the gens and priorities are all stored contiguously on disk, we can
* batch this up: We fill up the free_inc list with freshly invalidated buckets,
* call prio_write(), and when prio_write() finishes we pull buckets off the
* free_inc list and optionally discard them.
* free_inc list.
*
* free_inc isn't the only freelist - if it was, we'd often to sleep while
* priorities and gens were being written before we could allocate. c->free is a
* smaller freelist, and buckets on that list are always ready to be used.
*
* If we've got discards enabled, that happens when a bucket moves from the
* free_inc list to the free list.
*
* There is another freelist, because sometimes we have buckets that we know
* have nothing pointing into them - these we can reuse without waiting for
* priorities to be rewritten. These come from freed btree nodes and buckets
* that garbage collection discovered no longer had valid keys pointing into
* them (because they were overwritten). That's the unused list - buckets on the
* unused list move to the free list, optionally being discarded in the process.
* unused list move to the free list.
*
* It's also important to ensure that gens don't wrap around - with respect to
* either the oldest gen in the btree or the gen on disk. This is quite
@@ -118,8 +115,7 @@ void bch_rescale_priorities(struct cache_set *c, int sectors)
/*
* Background allocation thread: scans for buckets to be invalidated,
* invalidates them, rewrites prios/gens (marking them as invalidated on disk),
* then optionally issues discard commands to the newly free buckets, then puts
* them on the various freelists.
* then puts them on the various freelists.
*/
static inline bool can_inc_bucket_gen(struct bucket *b)
@@ -321,8 +317,7 @@ static int bch_allocator_thread(void *arg)
while (1) {
/*
* First, we pull buckets off of the unused and free_inc lists,
* possibly issue discards to them, then we add the bucket to
* the free list:
* then we add the bucket to the free list:
*/
while (1) {
long bucket;
@@ -330,14 +325,6 @@ static int bch_allocator_thread(void *arg)
if (!fifo_pop(&ca->free_inc, bucket))
break;
if (ca->discard) {
mutex_unlock(&ca->set->bucket_lock);
blkdev_issue_discard(ca->bdev,
bucket_to_sector(ca->set, bucket),
ca->sb.bucket_size, GFP_KERNEL);
mutex_lock(&ca->set->bucket_lock);
}
allocator_wait(ca, bch_allocator_push(ca, bucket));
wake_up(&ca->set->btree_cache_wait);
wake_up(&ca->set->bucket_wait);
@@ -412,7 +399,11 @@ long bch_bucket_alloc(struct cache *ca, unsigned int reserve, bool wait)
TASK_UNINTERRUPTIBLE);
mutex_unlock(&ca->set->bucket_lock);
atomic_inc(&ca->set->bucket_wait_cnt);
schedule();
atomic_dec(&ca->set->bucket_wait_cnt);
mutex_lock(&ca->set->bucket_lock);
} while (!fifo_pop(&ca->free[RESERVE_NONE], r) &&
!fifo_pop(&ca->free[reserve], r));

6
drivers/md/bcache/bcache.h
View File

@@ -447,8 +447,7 @@ struct cache {
* free_inc: Incoming buckets - these are buckets that currently have
* cached data in them, and we can't reuse them until after we write
* their new gen to disk. After prio_write() finishes writing the new
* gens/prios, they'll be moved to the free list (and possibly discarded
* in the process)
* gens/prios, they'll be moved to the free list.
*/
DECLARE_FIFO(long, free)[RESERVE_NR];
DECLARE_FIFO(long, free_inc);
@@ -467,8 +466,6 @@ struct cache {
*/
unsigned int invalidate_needs_gc;
bool discard; /* Get rid of? */
struct journal_device journal;
/* The rest of this all shows up in sysfs */
@@ -607,6 +604,7 @@ struct cache_set {
*/
atomic_t prio_blocked;
wait_queue_head_t bucket_wait;
atomic_t bucket_wait_cnt;
/*
* For any bio we don't skip we subtract the number of sectors from

8
drivers/md/bcache/bset.h
View File

@@ -327,9 +327,13 @@ struct btree_iter {
/* Fixed-size btree_iter that can be allocated on the stack */
struct btree_iter_stack {
struct btree_iter iter;
struct btree_iter_set stack_data[MAX_BSETS];
/* Must be last as it ends in a flexible-array member. */
TRAILING_OVERLAP(struct btree_iter, iter, data,
struct btree_iter_set stack_data[MAX_BSETS];
);
};
static_assert(offsetof(struct btree_iter_stack, iter.data) ==
offsetof(struct btree_iter_stack, stack_data));
typedef bool (*ptr_filter_fn)(struct btree_keys *b, const struct bkey *k);

53
drivers/md/bcache/btree.c
View File

@@ -89,8 +89,9 @@
* Test module load/unload
*/
#define MAX_GC_TIMES 100
#define MIN_GC_NODES 100
#define MAX_GC_TIMES_SHIFT 7 /* 128 loops */
#define GC_NODES_MIN 10
#define GC_SLEEP_MS_MIN 10
#define GC_SLEEP_MS 100
#define PTR_DIRTY_BIT (((uint64_t) 1 << 36))
@@ -371,7 +372,7 @@ static void do_btree_node_write(struct btree *b)
SET_PTR_OFFSET(&k.key, 0, PTR_OFFSET(&k.key, 0) +
bset_sector_offset(&b->keys, i));
if (!bch_bio_alloc_pages(b->bio, __GFP_NOWARN|GFP_NOWAIT)) {
if (!bch_bio_alloc_pages(b->bio, GFP_NOWAIT)) {
struct bio_vec *bv;
void *addr = (void *) ((unsigned long) i & ~(PAGE_SIZE - 1));
struct bvec_iter_all iter_all;
@@ -1578,29 +1579,29 @@ static unsigned int btree_gc_count_keys(struct btree *b)
static size_t btree_gc_min_nodes(struct cache_set *c)
{
size_t min_nodes;
size_t min_nodes = GC_NODES_MIN;
/*
* Since incremental GC would stop 100ms when front
* side I/O comes, so when there are many btree nodes,
* if GC only processes constant (100) nodes each time,
* GC would last a long time, and the front side I/Os
* would run out of the buckets (since no new bucket
* can be allocated during GC), and be blocked again.
* So GC should not process constant nodes, but varied
* nodes according to the number of btree nodes, which
* realized by dividing GC into constant(100) times,
* so when there are many btree nodes, GC can process
* more nodes each time, otherwise, GC will process less
* nodes each time (but no less than MIN_GC_NODES)
*/
min_nodes = c->gc_stats.nodes / MAX_GC_TIMES;
if (min_nodes < MIN_GC_NODES)
min_nodes = MIN_GC_NODES;
if (atomic_read(&c->search_inflight) == 0) {
size_t n = c->gc_stats.nodes >> MAX_GC_TIMES_SHIFT;
if (min_nodes < n)
min_nodes = n;
}
return min_nodes;
}
static uint64_t btree_gc_sleep_ms(struct cache_set *c)
{
uint64_t sleep_ms;
if (atomic_read(&c->bucket_wait_cnt) > 0)
sleep_ms = GC_SLEEP_MS_MIN;
else
sleep_ms = GC_SLEEP_MS;
return sleep_ms;
}
static int btree_gc_recurse(struct btree *b, struct btree_op *op,
struct closure *writes, struct gc_stat *gc)
@@ -1668,8 +1669,7 @@ static int btree_gc_recurse(struct btree *b, struct btree_op *op,
memmove(r + 1, r, sizeof(r[0]) * (GC_MERGE_NODES - 1));
r->b = NULL;
if (atomic_read(&b->c->search_inflight) &&
gc->nodes >= gc->nodes_pre + btree_gc_min_nodes(b->c)) {
if (gc->nodes >= (gc->nodes_pre + btree_gc_min_nodes(b->c))) {
gc->nodes_pre = gc->nodes;
ret = -EAGAIN;
break;
@@ -1846,8 +1846,8 @@ static void bch_btree_gc(struct cache_set *c)
cond_resched();
if (ret == -EAGAIN)
schedule_timeout_interruptible(msecs_to_jiffies
(GC_SLEEP_MS));
schedule_timeout_interruptible(
msecs_to_jiffies(btree_gc_sleep_ms(c)));
else if (ret)
pr_warn("gc failed!\n");
} while (ret && !test_bit(CACHE_SET_IO_DISABLE, &c->flags));
@@ -2822,7 +2822,8 @@ void bch_btree_exit(void)
int __init bch_btree_init(void)
{
btree_io_wq = alloc_workqueue("bch_btree_io", WQ_MEM_RECLAIM, 0);
btree_io_wq = alloc_workqueue("bch_btree_io",
WQ_MEM_RECLAIM | WQ_PERCPU, 0);
if (!btree_io_wq)
return -ENOMEM;

93
drivers/md/bcache/journal.c
View File

@@ -275,8 +275,7 @@ bsearch:
* ja->cur_idx
*/
ja->cur_idx = i;
ja->last_idx = ja->discard_idx = (i + 1) %
ca->sb.njournal_buckets;
ja->last_idx = (i + 1) % ca->sb.njournal_buckets;
}
@@ -336,16 +335,6 @@ void bch_journal_mark(struct cache_set *c, struct list_head *list)
}
}
static bool is_discard_enabled(struct cache_set *s)
{
struct cache *ca = s->cache;
if (ca->discard)
return true;
return false;
}
int bch_journal_replay(struct cache_set *s, struct list_head *list)
{
int ret = 0, keys = 0, entries = 0;
@@ -360,15 +349,10 @@ int bch_journal_replay(struct cache_set *s, struct list_head *list)
BUG_ON(i->pin && atomic_read(i->pin) != 1);
if (n != i->j.seq) {
if (n == start && is_discard_enabled(s))
pr_info("journal entries %llu-%llu may be discarded! (replaying %llu-%llu)\n",
n, i->j.seq - 1, start, end);
else {
pr_err("journal entries %llu-%llu missing! (replaying %llu-%llu)\n",
n, i->j.seq - 1, start, end);
ret = -EIO;
goto err;
}
pr_err("journal entries %llu-%llu missing! (replaying %llu-%llu)\n",
n, i->j.seq - 1, start, end);
ret = -EIO;
goto err;
}
for (k = i->j.start;
@@ -568,65 +552,6 @@ out:
#define last_seq(j) ((j)->seq - fifo_used(&(j)->pin) + 1)
static void journal_discard_endio(struct bio *bio)
{
struct journal_device *ja =
container_of(bio, struct journal_device, discard_bio);
struct cache *ca = container_of(ja, struct cache, journal);
atomic_set(&ja->discard_in_flight, DISCARD_DONE);
closure_wake_up(&ca->set->journal.wait);
closure_put(&ca->set->cl);
}
static void journal_discard_work(struct work_struct *work)
{
struct journal_device *ja =
container_of(work, struct journal_device, discard_work);
submit_bio(&ja->discard_bio);
}
static void do_journal_discard(struct cache *ca)
{
struct journal_device *ja = &ca->journal;
struct bio *bio = &ja->discard_bio;
if (!ca->discard) {
ja->discard_idx = ja->last_idx;
return;
}
switch (atomic_read(&ja->discard_in_flight)) {
case DISCARD_IN_FLIGHT:
return;
case DISCARD_DONE:
ja->discard_idx = (ja->discard_idx + 1) %
ca->sb.njournal_buckets;
atomic_set(&ja->discard_in_flight, DISCARD_READY);
fallthrough;
case DISCARD_READY:
if (ja->discard_idx == ja->last_idx)
return;
atomic_set(&ja->discard_in_flight, DISCARD_IN_FLIGHT);
bio_init_inline(bio, ca->bdev, 1, REQ_OP_DISCARD);
bio->bi_iter.bi_sector = bucket_to_sector(ca->set,
ca->sb.d[ja->discard_idx]);
bio->bi_iter.bi_size = bucket_bytes(ca);
bio->bi_end_io = journal_discard_endio;
closure_get(&ca->set->cl);
INIT_WORK(&ja->discard_work, journal_discard_work);
queue_work(bch_journal_wq, &ja->discard_work);
}
}
static unsigned int free_journal_buckets(struct cache_set *c)
{
struct journal *j = &c->journal;
@@ -635,10 +560,10 @@ static unsigned int free_journal_buckets(struct cache_set *c)
unsigned int n;
/* In case njournal_buckets is not power of 2 */
if (ja->cur_idx >= ja->discard_idx)
n = ca->sb.njournal_buckets + ja->discard_idx - ja->cur_idx;
if (ja->cur_idx >= ja->last_idx)
n = ca->sb.njournal_buckets + ja->last_idx - ja->cur_idx;
else
n = ja->discard_idx - ja->cur_idx;
n = ja->last_idx - ja->cur_idx;
if (n > (1 + j->do_reserve))
return n - (1 + j->do_reserve);
@@ -668,8 +593,6 @@ static void journal_reclaim(struct cache_set *c)
ja->last_idx = (ja->last_idx + 1) %
ca->sb.njournal_buckets;
do_journal_discard(ca);
if (c->journal.blocks_free)
goto out;

13
drivers/md/bcache/journal.h
View File

@@ -139,19 +139,6 @@ struct journal_device {
/* Last journal bucket that still contains an open journal entry */
unsigned int last_idx;
/* Next journal bucket to be discarded */
unsigned int discard_idx;
#define DISCARD_READY 0
#define DISCARD_IN_FLIGHT 1
#define DISCARD_DONE 2
/* 1 - discard in flight, -1 - discard completed */
atomic_t discard_in_flight;
struct work_struct discard_work;
struct bio discard_bio;
struct bio_vec discard_bv;
/* Bio for journal reads/writes to this device */
struct bio bio;
struct bio_vec bv[8];

33
drivers/md/bcache/super.c
View File

@@ -1388,7 +1388,7 @@ static CLOSURE_CALLBACK(cached_dev_flush)
bch_cache_accounting_destroy(&dc->accounting);
kobject_del(&d->kobj);
continue_at(cl, cached_dev_free, system_wq);
continue_at(cl, cached_dev_free, system_percpu_wq);
}
static int cached_dev_init(struct cached_dev *dc, unsigned int block_size)
@@ -1400,7 +1400,7 @@ static int cached_dev_init(struct cached_dev *dc, unsigned int block_size)
__module_get(THIS_MODULE);
INIT_LIST_HEAD(&dc->list);
closure_init(&dc->disk.cl, NULL);
set_closure_fn(&dc->disk.cl, cached_dev_flush, system_wq);
set_closure_fn(&dc->disk.cl, cached_dev_flush, system_percpu_wq);
kobject_init(&dc->disk.kobj, &bch_cached_dev_ktype);
INIT_WORK(&dc->detach, cached_dev_detach_finish);
sema_init(&dc->sb_write_mutex, 1);
@@ -1513,7 +1513,7 @@ static CLOSURE_CALLBACK(flash_dev_flush)
bcache_device_unlink(d);
mutex_unlock(&bch_register_lock);
kobject_del(&d->kobj);
continue_at(cl, flash_dev_free, system_wq);
continue_at(cl, flash_dev_free, system_percpu_wq);
}
static int flash_dev_run(struct cache_set *c, struct uuid_entry *u)
@@ -1525,7 +1525,7 @@ static int flash_dev_run(struct cache_set *c, struct uuid_entry *u)
goto err_ret;
closure_init(&d->cl, NULL);
set_closure_fn(&d->cl, flash_dev_flush, system_wq);
set_closure_fn(&d->cl, flash_dev_flush, system_percpu_wq);
kobject_init(&d->kobj, &bch_flash_dev_ktype);
@@ -1833,7 +1833,7 @@ static CLOSURE_CALLBACK(__cache_set_unregister)
mutex_unlock(&bch_register_lock);
continue_at(cl, cache_set_flush, system_wq);
continue_at(cl, cache_set_flush, system_percpu_wq);
}
void bch_cache_set_stop(struct cache_set *c)
@@ -1863,10 +1863,10 @@ struct cache_set *bch_cache_set_alloc(struct cache_sb *sb)
__module_get(THIS_MODULE);
closure_init(&c->cl, NULL);
set_closure_fn(&c->cl, cache_set_free, system_wq);
set_closure_fn(&c->cl, cache_set_free, system_percpu_wq);
closure_init(&c->caching, &c->cl);
set_closure_fn(&c->caching, __cache_set_unregister, system_wq);
set_closure_fn(&c->caching, __cache_set_unregister, system_percpu_wq);
/* Maybe create continue_at_noreturn() and use it here? */
closure_set_stopped(&c->cl);
@@ -1939,7 +1939,8 @@ struct cache_set *bch_cache_set_alloc(struct cache_sb *sb)
if (!c->uuids)
goto err;
c->moving_gc_wq = alloc_workqueue("bcache_gc", WQ_MEM_RECLAIM, 0);
c->moving_gc_wq = alloc_workqueue("bcache_gc",
WQ_MEM_RECLAIM | WQ_PERCPU, 0);
if (!c->moving_gc_wq)
goto err;
@@ -2382,9 +2383,6 @@ static int register_cache(struct cache_sb *sb, struct cache_sb_disk *sb_disk,
ca->bdev = file_bdev(bdev_file);
ca->sb_disk = sb_disk;
if (bdev_max_discard_sectors(file_bdev(bdev_file)))
ca->discard = CACHE_DISCARD(&ca->sb);
ret = cache_alloc(ca);
if (ret != 0) {
if (ret == -ENOMEM)
@@ -2531,7 +2529,7 @@ static void register_device_async(struct async_reg_args *args)
INIT_DELAYED_WORK(&args->reg_work, register_cache_worker);
/* 10 jiffies is enough for a delay */
queue_delayed_work(system_wq, &args->reg_work, 10);
queue_delayed_work(system_percpu_wq, &args->reg_work, 10);
}
static void *alloc_holder_object(struct cache_sb *sb)
@@ -2905,24 +2903,25 @@ static int __init bcache_init(void)
if (bch_btree_init())
goto err;
bcache_wq = alloc_workqueue("bcache", WQ_MEM_RECLAIM, 0);
bcache_wq = alloc_workqueue("bcache", WQ_MEM_RECLAIM | WQ_PERCPU, 0);
if (!bcache_wq)
goto err;
/*
* Let's not make this `WQ_MEM_RECLAIM` for the following reasons:
*
* 1. It used `system_wq` before which also does no memory reclaim.
* 1. It used `system_percpu_wq` before which also does no memory reclaim.
* 2. With `WQ_MEM_RECLAIM` desktop stalls, increased boot times, and
* reduced throughput can be observed.
*
* We still want to user our own queue to not congest the `system_wq`.
* We still want to user our own queue to not congest the `system_percpu_wq`.
*/
bch_flush_wq = alloc_workqueue("bch_flush", 0, 0);
bch_flush_wq = alloc_workqueue("bch_flush", WQ_PERCPU, 0);
if (!bch_flush_wq)
goto err;
bch_journal_wq = alloc_workqueue("bch_journal", WQ_MEM_RECLAIM, 0);
bch_journal_wq = alloc_workqueue("bch_journal",
WQ_MEM_RECLAIM | WQ_PERCPU, 0);
if (!bch_journal_wq)
goto err;

15
drivers/md/bcache/sysfs.c
View File

@@ -134,7 +134,6 @@ read_attribute(partial_stripes_expensive);
rw_attribute(synchronous);
rw_attribute(journal_delay_ms);
rw_attribute(io_disable);
rw_attribute(discard);
rw_attribute(running);
rw_attribute(label);
rw_attribute(errors);
@@ -1036,7 +1035,6 @@ SHOW(__bch_cache)
sysfs_hprint(bucket_size, bucket_bytes(ca));
sysfs_hprint(block_size, block_bytes(ca));
sysfs_print(nbuckets, ca->sb.nbuckets);
sysfs_print(discard, ca->discard);
sysfs_hprint(written, atomic_long_read(&ca->sectors_written) << 9);
sysfs_hprint(btree_written,
atomic_long_read(&ca->btree_sectors_written) << 9);
@@ -1142,18 +1140,6 @@ STORE(__bch_cache)
if (bcache_is_reboot)
return -EBUSY;
if (attr == &sysfs_discard) {
bool v = strtoul_or_return(buf);
if (bdev_max_discard_sectors(ca->bdev))
ca->discard = v;
if (v != CACHE_DISCARD(&ca->sb)) {
SET_CACHE_DISCARD(&ca->sb, v);
bcache_write_super(ca->set);
}
}
if (attr == &sysfs_cache_replacement_policy) {
v = __sysfs_match_string(cache_replacement_policies, -1, buf);
if (v < 0)
@@ -1185,7 +1171,6 @@ static struct attribute *bch_cache_attrs[] = {
&sysfs_block_size,
&sysfs_nbuckets,
&sysfs_priority_stats,
&sysfs_discard,
&sysfs_written,
&sysfs_btree_written,
&sysfs_metadata_written,

5
drivers/md/bcache/writeback.c
View File

@@ -805,8 +805,7 @@ static int bch_writeback_thread(void *arg)
* may set BCH_ENABLE_AUTO_GC via sysfs, then when
* BCH_DO_AUTO_GC is set, garbage collection thread
* will be wake up here. After moving gc, the shrunk
* btree and discarded free buckets SSD space may be
* helpful for following write requests.
* btree may be helpful for following write requests.
*/
if (c->gc_after_writeback ==
(BCH_ENABLE_AUTO_GC|BCH_DO_AUTO_GC)) {
@@ -1076,7 +1075,7 @@ void bch_cached_dev_writeback_init(struct cached_dev *dc)
int bch_cached_dev_writeback_start(struct cached_dev *dc)
{
dc->writeback_write_wq = alloc_workqueue("bcache_writeback_wq",
WQ_MEM_RECLAIM, 0);
WQ_MEM_RECLAIM | WQ_PERCPU, 0);
if (!dc->writeback_write_wq)
return -ENOMEM;

63
drivers/md/dm-zone.c
View File

@@ -17,33 +17,26 @@
* For internal zone reports bypassing the top BIO submission path.
*/
static int dm_blk_do_report_zones(struct mapped_device *md, struct dm_table *t,
sector_t sector, unsigned int nr_zones,
report_zones_cb cb, void *data)
unsigned int nr_zones,
struct dm_report_zones_args *args)
{
struct gendisk *disk = md->disk;
int ret;
struct dm_report_zones_args args = {
.next_sector = sector,
.orig_data = data,
.orig_cb = cb,
};
do {
struct dm_target *tgt;
int ret;
tgt = dm_table_find_target(t, args.next_sector);
tgt = dm_table_find_target(t, args->next_sector);
if (WARN_ON_ONCE(!tgt->type->report_zones))
return -EIO;
args.tgt = tgt;
ret = tgt->type->report_zones(tgt, &args,
nr_zones - args.zone_idx);
args->tgt = tgt;
ret = tgt->type->report_zones(tgt, args,
nr_zones - args->zone_idx);
if (ret < 0)
return ret;
} while (args.zone_idx < nr_zones &&
args.next_sector < get_capacity(disk));
} while (args->zone_idx < nr_zones &&
args->next_sector < get_capacity(md->disk));
return args.zone_idx;
return args->zone_idx;
}
/*
@@ -52,7 +45,8 @@ static int dm_blk_do_report_zones(struct mapped_device *md, struct dm_table *t,
* generally implemented by targets using dm_report_zones().
*/
int dm_blk_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones,
struct blk_report_zones_args *args)
{
struct mapped_device *md = disk->private_data;
struct dm_table *map;
@@ -76,9 +70,14 @@ int dm_blk_report_zones(struct gendisk *disk, sector_t sector,
map = zone_revalidate_map;
}
if (map)
ret = dm_blk_do_report_zones(md, map, sector, nr_zones, cb,
data);
if (map) {
struct dm_report_zones_args dm_args = {
.disk = md->disk,
.next_sector = sector,
.rep_args = args,
};
ret = dm_blk_do_report_zones(md, map, nr_zones, &dm_args);
}
if (put_table)
dm_put_live_table(md, srcu_idx);
@@ -113,7 +112,18 @@ static int dm_report_zones_cb(struct blk_zone *zone, unsigned int idx,
}
args->next_sector = zone->start + zone->len;
return args->orig_cb(zone, args->zone_idx++, args->orig_data);
/* If we have an internal callback, call it first. */
if (args->cb) {
int ret;
ret = args->cb(zone, args->zone_idx, args->data);
if (ret)
return ret;
}
return disk_report_zone(args->disk, zone, args->zone_idx++,
args->rep_args);
}
/*
@@ -492,10 +502,15 @@ int dm_zone_get_reset_bitmap(struct mapped_device *md, struct dm_table *t,
sector_t sector, unsigned int nr_zones,
unsigned long *need_reset)
{
struct dm_report_zones_args args = {
.disk = md->disk,
.next_sector = sector,
.cb = dm_zone_need_reset_cb,
.data = need_reset,
};
int ret;
ret = dm_blk_do_report_zones(md, t, sector, nr_zones,
dm_zone_need_reset_cb, need_reset);
ret = dm_blk_do_report_zones(md, t, nr_zones, &args);
if (ret != nr_zones) {
DMERR("Get %s zone reset bitmap failed\n",
md->disk->disk_name);

3
drivers/md/dm.h
View File

@@ -109,7 +109,8 @@ void dm_finalize_zone_settings(struct dm_table *t, struct queue_limits *lim);
void dm_zone_endio(struct dm_io *io, struct bio *clone);
#ifdef CONFIG_BLK_DEV_ZONED
int dm_blk_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data);
unsigned int nr_zones,
struct blk_report_zones_args *args);
bool dm_is_zone_write(struct mapped_device *md, struct bio *bio);
int dm_zone_get_reset_bitmap(struct mapped_device *md, struct dm_table *t,
sector_t sector, unsigned int nr_zones,

2
drivers/md/md-linear.c
View File

@@ -72,9 +72,11 @@ static int linear_set_limits(struct mddev *mddev)
md_init_stacking_limits(&lim);
lim.max_hw_sectors = mddev->chunk_sectors;
lim.logical_block_size = mddev->logical_block_size;
lim.max_write_zeroes_sectors = mddev->chunk_sectors;
lim.max_hw_wzeroes_unmap_sectors = mddev->chunk_sectors;
lim.io_min = mddev->chunk_sectors << 9;
lim.features |= BLK_FEAT_ATOMIC_WRITES;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)
return err;

2
drivers/md/md-llbitmap.c
View File

@@ -378,7 +378,7 @@ static void llbitmap_infect_dirty_bits(struct llbitmap *llbitmap,
case BitClean:
pctl->state[pos] = BitDirty;
break;
};
}
}
}

259
drivers/md/md.c
View File

@@ -99,7 +99,7 @@ static int remove_and_add_spares(struct mddev *mddev,
struct md_rdev *this);
static void mddev_detach(struct mddev *mddev);
static void export_rdev(struct md_rdev *rdev, struct mddev *mddev);
static void md_wakeup_thread_directly(struct md_thread __rcu *thread);
static void md_wakeup_thread_directly(struct md_thread __rcu **thread);
/*
* Default number of read corrections we'll attempt on an rdev
@@ -339,6 +339,7 @@ static int start_readonly;
*/
static bool create_on_open = true;
static bool legacy_async_del_gendisk = true;
static bool check_new_feature = true;
/*
* We have a system wide 'event count' that is incremented
@@ -730,6 +731,8 @@ static void mddev_clear_bitmap_ops(struct mddev *mddev)
int mddev_init(struct mddev *mddev)
{
int err = 0;
if (!IS_ENABLED(CONFIG_MD_BITMAP))
mddev->bitmap_id = ID_BITMAP_NONE;
else
@@ -741,10 +744,23 @@ int mddev_init(struct mddev *mddev)
if (percpu_ref_init(&mddev->writes_pending, no_op,
PERCPU_REF_ALLOW_REINIT, GFP_KERNEL)) {
percpu_ref_exit(&mddev->active_io);
return -ENOMEM;
err = -ENOMEM;
goto exit_acitve_io;
}
err = bioset_init(&mddev->bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (err)
goto exit_writes_pending;
err = bioset_init(&mddev->sync_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (err)
goto exit_bio_set;
err = bioset_init(&mddev->io_clone_set, BIO_POOL_SIZE,
offsetof(struct md_io_clone, bio_clone), 0);
if (err)
goto exit_sync_set;
/* We want to start with the refcount at zero */
percpu_ref_put(&mddev->writes_pending);
@@ -773,11 +789,24 @@ int mddev_init(struct mddev *mddev)
INIT_WORK(&mddev->del_work, mddev_delayed_delete);
return 0;
exit_sync_set:
bioset_exit(&mddev->sync_set);
exit_bio_set:
bioset_exit(&mddev->bio_set);
exit_writes_pending:
percpu_ref_exit(&mddev->writes_pending);
exit_acitve_io:
percpu_ref_exit(&mddev->active_io);
return err;
}
EXPORT_SYMBOL_GPL(mddev_init);
void mddev_destroy(struct mddev *mddev)
{
bioset_exit(&mddev->bio_set);
bioset_exit(&mddev->sync_set);
bioset_exit(&mddev->io_clone_set);
percpu_ref_exit(&mddev->active_io);
percpu_ref_exit(&mddev->writes_pending);
}
@@ -941,8 +970,11 @@ void mddev_unlock(struct mddev *mddev)
* do_md_stop. dm raid only uses md_stop to stop. So dm raid
* doesn't need to check MD_DELETED when getting reconfig lock
*/
if (test_bit(MD_DELETED, &mddev->flags))
if (test_bit(MD_DELETED, &mddev->flags) &&
!test_and_set_bit(MD_DO_DELETE, &mddev->flags)) {
kobject_del(&mddev->kobj);
del_gendisk(mddev->gendisk);
}
}
}
EXPORT_SYMBOL_GPL(mddev_unlock);
@@ -1820,9 +1852,13 @@ static int super_1_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor_
}
if (sb->pad0 ||
sb->pad3[0] ||
memcmp(sb->pad3, sb->pad3+1, sizeof(sb->pad3) - sizeof(sb->pad3[1])))
/* Some padding is non-zero, might be a new feature */
return -EINVAL;
memcmp(sb->pad3, sb->pad3+1, sizeof(sb->pad3) - sizeof(sb->pad3[1]))) {
pr_warn("Some padding is non-zero on %pg, might be a new feature\n",
rdev->bdev);
if (check_new_feature)
return -EINVAL;
pr_warn("check_new_feature is disabled, data corruption possible\n");
}
rdev->preferred_minor = 0xffff;
rdev->data_offset = le64_to_cpu(sb->data_offset);
@@ -1963,6 +1999,7 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *freshest, struc
mddev->layout = le32_to_cpu(sb->layout);
mddev->raid_disks = le32_to_cpu(sb->raid_disks);
mddev->dev_sectors = le64_to_cpu(sb->size);
mddev->logical_block_size = le32_to_cpu(sb->logical_block_size);
mddev->events = ev1;
mddev->bitmap_info.offset = 0;
mddev->bitmap_info.space = 0;
@@ -2172,6 +2209,7 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
sb->chunksize = cpu_to_le32(mddev->chunk_sectors);
sb->level = cpu_to_le32(mddev->level);
sb->layout = cpu_to_le32(mddev->layout);
sb->logical_block_size = cpu_to_le32(mddev->logical_block_size);
if (test_bit(FailFast, &rdev->flags))
sb->devflags |= FailFast1;
else
@@ -2750,6 +2788,7 @@ void md_update_sb(struct mddev *mddev, int force_change)
if (!md_is_rdwr(mddev)) {
if (force_change)
set_bit(MD_SB_CHANGE_DEVS, &mddev->sb_flags);
pr_err("%s: can't update sb for read-only array %s\n", __func__, mdname(mddev));
return;
}
@@ -5134,7 +5173,7 @@ static void stop_sync_thread(struct mddev *mddev, bool locked)
* Thread might be blocked waiting for metadata update which will now
* never happen
*/
md_wakeup_thread_directly(mddev->sync_thread);
md_wakeup_thread_directly(&mddev->sync_thread);
if (work_pending(&mddev->sync_work))
flush_work(&mddev->sync_work);
@@ -5900,6 +5939,68 @@ static struct md_sysfs_entry md_serialize_policy =
__ATTR(serialize_policy, S_IRUGO | S_IWUSR, serialize_policy_show,
serialize_policy_store);
static int mddev_set_logical_block_size(struct mddev *mddev,
unsigned int lbs)
{
int err = 0;
struct queue_limits lim;
if (queue_logical_block_size(mddev->gendisk->queue) >= lbs) {
pr_err("%s: Cannot set LBS smaller than mddev LBS %u\n",
mdname(mddev), lbs);
return -EINVAL;
}
lim = queue_limits_start_update(mddev->gendisk->queue);
lim.logical_block_size = lbs;
pr_info("%s: logical_block_size is changed, data may be lost\n",
mdname(mddev));
err = queue_limits_commit_update(mddev->gendisk->queue, &lim);
if (err)
return err;
mddev->logical_block_size = lbs;
/* New lbs will be written to superblock after array is running */
set_bit(MD_SB_CHANGE_DEVS, &mddev->sb_flags);
return 0;
}
static ssize_t
lbs_show(struct mddev *mddev, char *page)
{
return sprintf(page, "%u\n", mddev->logical_block_size);
}
static ssize_t
lbs_store(struct mddev *mddev, const char *buf, size_t len)
{
unsigned int lbs;
int err = -EBUSY;
/* Only 1.x meta supports configurable LBS */
if (mddev->major_version == 0)
return -EINVAL;
if (mddev->pers)
return -EBUSY;
err = kstrtouint(buf, 10, &lbs);
if (err < 0)
return -EINVAL;
err = mddev_lock(mddev);
if (err)
goto unlock;
err = mddev_set_logical_block_size(mddev, lbs);
unlock:
mddev_unlock(mddev);
return err ?: len;
}
static struct md_sysfs_entry md_logical_block_size =
__ATTR(logical_block_size, 0644, lbs_show, lbs_store);
static struct attribute *md_default_attrs[] = {
&md_level.attr,
@@ -5922,6 +6023,7 @@ static struct attribute *md_default_attrs[] = {
&md_consistency_policy.attr,
&md_fail_last_dev.attr,
&md_serialize_policy.attr,
&md_logical_block_size.attr,
NULL,
};
@@ -6052,6 +6154,17 @@ int mddev_stack_rdev_limits(struct mddev *mddev, struct queue_limits *lim,
return -EINVAL;
}
/*
* Before RAID adding folio support, the logical_block_size
* should be smaller than the page size.
*/
if (lim->logical_block_size > PAGE_SIZE) {
pr_err("%s: logical_block_size must not larger than PAGE_SIZE\n",
mdname(mddev));
return -EINVAL;
}
mddev->logical_block_size = lim->logical_block_size;
return 0;
}
EXPORT_SYMBOL_GPL(mddev_stack_rdev_limits);
@@ -6064,6 +6177,13 @@ int mddev_stack_new_rdev(struct mddev *mddev, struct md_rdev *rdev)
if (mddev_is_dm(mddev))
return 0;
if (queue_logical_block_size(rdev->bdev->bd_disk->queue) >
queue_logical_block_size(mddev->gendisk->queue)) {
pr_err("%s: incompatible logical_block_size, can not add\n",
mdname(mddev));
return -EINVAL;
}
lim = queue_limits_start_update(mddev->gendisk->queue);
queue_limits_stack_bdev(&lim, rdev->bdev, rdev->data_offset,
mddev->gendisk->disk_name);
@@ -6384,29 +6504,9 @@ int md_run(struct mddev *mddev)
nowait = nowait && bdev_nowait(rdev->bdev);
}
if (!bioset_initialized(&mddev->bio_set)) {
err = bioset_init(&mddev->bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (err)
return err;
}
if (!bioset_initialized(&mddev->sync_set)) {
err = bioset_init(&mddev->sync_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
if (err)
goto exit_bio_set;
}
if (!bioset_initialized(&mddev->io_clone_set)) {
err = bioset_init(&mddev->io_clone_set, BIO_POOL_SIZE,
offsetof(struct md_io_clone, bio_clone), 0);
if (err)
goto exit_sync_set;
}
pers = get_pers(mddev->level, mddev->clevel);
if (!pers) {
err = -EINVAL;
goto abort;
}
if (!pers)
return -EINVAL;
if (mddev->level != pers->head.id) {
mddev->level = pers->head.id;
mddev->new_level = pers->head.id;
@@ -6417,8 +6517,7 @@ int md_run(struct mddev *mddev)
pers->start_reshape == NULL) {
/* This personality cannot handle reshaping... */
put_pers(pers);
err = -EINVAL;
goto abort;
return -EINVAL;
}
if (pers->sync_request) {
@@ -6545,12 +6644,6 @@ bitmap_abort:
mddev->private = NULL;
put_pers(pers);
md_bitmap_destroy(mddev);
abort:
bioset_exit(&mddev->io_clone_set);
exit_sync_set:
bioset_exit(&mddev->sync_set);
exit_bio_set:
bioset_exit(&mddev->bio_set);
return err;
}
EXPORT_SYMBOL_GPL(md_run);
@@ -6683,6 +6776,7 @@ static void md_clean(struct mddev *mddev)
mddev->chunk_sectors = 0;
mddev->ctime = mddev->utime = 0;
mddev->layout = 0;
mddev->logical_block_size = 0;
mddev->max_disks = 0;
mddev->events = 0;
mddev->can_decrease_events = 0;
@@ -6775,10 +6869,6 @@ static void __md_stop(struct mddev *mddev)
mddev->private = NULL;
put_pers(pers);
clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
bioset_exit(&mddev->bio_set);
bioset_exit(&mddev->sync_set);
bioset_exit(&mddev->io_clone_set);
}
void md_stop(struct mddev *mddev)
@@ -6869,6 +6959,10 @@ static int do_md_stop(struct mddev *mddev, int mode)
if (!md_is_rdwr(mddev))
set_disk_ro(disk, 0);
if (mode == 2 && mddev->pers->sync_request &&
mddev->to_remove == NULL)
mddev->to_remove = &md_redundancy_group;
__md_stop_writes(mddev);
__md_stop(mddev);
@@ -8373,22 +8467,21 @@ static int md_thread(void *arg)
return 0;
}
static void md_wakeup_thread_directly(struct md_thread __rcu *thread)
static void md_wakeup_thread_directly(struct md_thread __rcu **thread)
{
struct md_thread *t;
rcu_read_lock();
t = rcu_dereference(thread);
t = rcu_dereference(*thread);
if (t)
wake_up_process(t->tsk);
rcu_read_unlock();
}
void md_wakeup_thread(struct md_thread __rcu *thread)
void __md_wakeup_thread(struct md_thread __rcu *thread)
{
struct md_thread *t;
rcu_read_lock();
t = rcu_dereference(thread);
if (t) {
pr_debug("md: waking up MD thread %s.\n", t->tsk->comm);
@@ -8396,9 +8489,8 @@ void md_wakeup_thread(struct md_thread __rcu *thread)
if (wq_has_sleeper(&t->wqueue))
wake_up(&t->wqueue);
}
rcu_read_unlock();
}
EXPORT_SYMBOL(md_wakeup_thread);
EXPORT_SYMBOL(__md_wakeup_thread);
struct md_thread *md_register_thread(void (*run) (struct md_thread *),
struct mddev *mddev, const char *name)
@@ -9978,6 +10070,52 @@ static void unregister_sync_thread(struct mddev *mddev)
md_reap_sync_thread(mddev);
}
static bool md_should_do_recovery(struct mddev *mddev)
{
/*
* As long as one of the following flags is set,
* recovery needs to do or cleanup.
*/
if (test_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
test_bit(MD_RECOVERY_DONE, &mddev->recovery))
return true;
/*
* If no flags are set and it is in read-only status,
* there is nothing to do.
*/
if (!md_is_rdwr(mddev))
return false;
/*
* MD_SB_CHANGE_PENDING indicates that the array is switching from clean to
* active, and no action is needed for now.
* All other MD_SB_* flags require to update the superblock.
*/
if (mddev->sb_flags & ~ (1<<MD_SB_CHANGE_PENDING))
return true;
/*
* If the array is not using external metadata and there has been no data
* written for some time, then the array's status needs to be set to
* in_sync.
*/
if (mddev->external == 0 && mddev->safemode == 1)
return true;
/*
* When the system is about to restart or the process receives an signal,
* the array needs to be synchronized as soon as possible.
* Once the data synchronization is completed, need to change the array
* status to in_sync.
*/
if (mddev->safemode == 2 && !mddev->in_sync &&
mddev->resync_offset == MaxSector)
return true;
return false;
}
/*
* This routine is regularly called by all per-raid-array threads to
* deal with generic issues like resync and super-block update.
@@ -10014,18 +10152,7 @@ void md_check_recovery(struct mddev *mddev)
flush_signals(current);
}
if (!md_is_rdwr(mddev) &&
!test_bit(MD_RECOVERY_NEEDED, &mddev->recovery) &&
!test_bit(MD_RECOVERY_DONE, &mddev->recovery))
return;
if ( ! (
(mddev->sb_flags & ~ (1<<MD_SB_CHANGE_PENDING)) ||
test_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
test_bit(MD_RECOVERY_DONE, &mddev->recovery) ||
(mddev->external == 0 && mddev->safemode == 1) ||
(mddev->safemode == 2
&& !mddev->in_sync && mddev->resync_offset == MaxSector)
))
if (!md_should_do_recovery(mddev))
return;
if (mddev_trylock(mddev)) {
@@ -10281,7 +10408,6 @@ static int md_notify_reboot(struct notifier_block *this,
unsigned long code, void *x)
{
struct mddev *mddev;
int need_delay = 0;
spin_lock(&all_mddevs_lock);
list_for_each_entry(mddev, &all_mddevs, all_mddevs) {
@@ -10295,21 +10421,11 @@ static int md_notify_reboot(struct notifier_block *this,
mddev->safemode = 2;
mddev_unlock(mddev);
}
need_delay = 1;
spin_lock(&all_mddevs_lock);
mddev_put_locked(mddev);
}
spin_unlock(&all_mddevs_lock);
/*
* certain more exotic SCSI devices are known to be
* volatile wrt too early system reboots. While the
* right place to handle this issue is the given
* driver, we do want to have a safe RAID driver ...
*/
if (need_delay)
msleep(1000);
return NOTIFY_DONE;
}
@@ -10697,6 +10813,7 @@ module_param(start_dirty_degraded, int, S_IRUGO|S_IWUSR);
module_param_call(new_array, add_named_array, NULL, NULL, S_IWUSR);
module_param(create_on_open, bool, S_IRUSR|S_IWUSR);
module_param(legacy_async_del_gendisk, bool, 0600);
module_param(check_new_feature, bool, 0600);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("MD RAID framework");

10
drivers/md/md.h
View File

@@ -354,6 +354,7 @@ enum mddev_flags {
MD_HAS_MULTIPLE_PPLS,
MD_NOT_READY,
MD_BROKEN,
MD_DO_DELETE,
MD_DELETED,
};
@@ -432,6 +433,7 @@ struct mddev {
sector_t array_sectors; /* exported array size */
int external_size; /* size managed
* externally */
unsigned int logical_block_size;
__u64 events;
/* If the last 'event' was simply a clean->dirty transition, and
* we didn't write it to the spares, then it is safe and simple
@@ -882,6 +884,12 @@ struct md_io_clone {
#define THREAD_WAKEUP 0
#define md_wakeup_thread(thread) do { \
rcu_read_lock(); \
__md_wakeup_thread(thread); \
rcu_read_unlock(); \
} while (0)
static inline void safe_put_page(struct page *p)
{
if (p) put_page(p);
@@ -895,7 +903,7 @@ extern struct md_thread *md_register_thread(
struct mddev *mddev,
const char *name);
extern void md_unregister_thread(struct mddev *mddev, struct md_thread __rcu **threadp);
extern void md_wakeup_thread(struct md_thread __rcu *thread);
extern void __md_wakeup_thread(struct md_thread __rcu *thread);
extern void md_check_recovery(struct mddev *mddev);
extern void md_reap_sync_thread(struct mddev *mddev);
extern enum sync_action md_sync_action(struct mddev *mddev);

20
drivers/md/raid0.c
View File

@@ -68,7 +68,10 @@ static int create_strip_zones(struct mddev *mddev, struct r0conf **private_conf)
struct strip_zone *zone;
int cnt;
struct r0conf *conf = kzalloc(sizeof(*conf), GFP_KERNEL);
unsigned blksize = 512;
unsigned int blksize = 512;
if (!mddev_is_dm(mddev))
blksize = queue_logical_block_size(mddev->gendisk->queue);
*private_conf = ERR_PTR(-ENOMEM);
if (!conf)
@@ -84,7 +87,8 @@ static int create_strip_zones(struct mddev *mddev, struct r0conf **private_conf)
sector_div(sectors, mddev->chunk_sectors);
rdev1->sectors = sectors * mddev->chunk_sectors;
blksize = max(blksize, queue_logical_block_size(
if (mddev_is_dm(mddev))
blksize = max(blksize, queue_logical_block_size(
rdev1->bdev->bd_disk->queue));
rdev_for_each(rdev2, mddev) {
@@ -383,6 +387,7 @@ static int raid0_set_limits(struct mddev *mddev)
lim.max_hw_sectors = mddev->chunk_sectors;
lim.max_write_zeroes_sectors = mddev->chunk_sectors;
lim.max_hw_wzeroes_unmap_sectors = mddev->chunk_sectors;
lim.logical_block_size = mddev->logical_block_size;
lim.io_min = mddev->chunk_sectors << 9;
lim.io_opt = lim.io_min * mddev->raid_disks;
lim.chunk_sectors = mddev->chunk_sectors;
@@ -405,6 +410,12 @@ static int raid0_run(struct mddev *mddev)
if (md_check_no_bitmap(mddev))
return -EINVAL;
if (!mddev_is_dm(mddev)) {
ret = raid0_set_limits(mddev);
if (ret)
return ret;
}
/* if private is not null, we are here after takeover */
if (mddev->private == NULL) {
ret = create_strip_zones(mddev, &conf);
@@ -413,11 +424,6 @@ static int raid0_run(struct mddev *mddev)
mddev->private = conf;
}
conf = mddev->private;
if (!mddev_is_dm(mddev)) {
ret = raid0_set_limits(mddev);
if (ret)
return ret;
}
/* calculate array device size */
md_set_array_sectors(mddev, raid0_size(mddev, 0, 0));

1
drivers/md/raid1.c
View File

@@ -3213,6 +3213,7 @@ static int raid1_set_limits(struct mddev *mddev)
md_init_stacking_limits(&lim);
lim.max_write_zeroes_sectors = 0;
lim.max_hw_wzeroes_unmap_sectors = 0;
lim.logical_block_size = mddev->logical_block_size;
lim.features |= BLK_FEAT_ATOMIC_WRITES;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)

1
drivers/md/raid10.c
View File

@@ -4000,6 +4000,7 @@ static int raid10_set_queue_limits(struct mddev *mddev)
md_init_stacking_limits(&lim);
lim.max_write_zeroes_sectors = 0;
lim.max_hw_wzeroes_unmap_sectors = 0;
lim.logical_block_size = mddev->logical_block_size;
lim.io_min = mddev->chunk_sectors << 9;
lim.chunk_sectors = mddev->chunk_sectors;
lim.io_opt = lim.io_min * raid10_nr_stripes(conf);

2
drivers/md/raid5-cache.c
View File

@@ -3104,7 +3104,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
goto out_mempool;
spin_lock_init(&log->tree_lock);
INIT_RADIX_TREE(&log->big_stripe_tree, GFP_NOWAIT | __GFP_NOWARN);
INIT_RADIX_TREE(&log->big_stripe_tree, GFP_NOWAIT);
thread = md_register_thread(r5l_reclaim_thread, log->rdev->mddev,
"reclaim");

7
drivers/md/raid5.c
View File

@@ -4956,7 +4956,8 @@ static void handle_stripe(struct stripe_head *sh)
goto finish;
if (s.handle_bad_blocks ||
test_bit(MD_SB_CHANGE_PENDING, &conf->mddev->sb_flags)) {
(md_is_rdwr(conf->mddev) &&
test_bit(MD_SB_CHANGE_PENDING, &conf->mddev->sb_flags))) {
set_bit(STRIPE_HANDLE, &sh->state);
goto finish;
}
@@ -6768,7 +6769,8 @@ static void raid5d(struct md_thread *thread)
int batch_size, released;
unsigned int offset;
if (test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags))
if (md_is_rdwr(mddev) &&
test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags))
break;
released = release_stripe_list(conf, conf->temp_inactive_list);
@@ -7745,6 +7747,7 @@ static int raid5_set_limits(struct mddev *mddev)
stripe = roundup_pow_of_two(data_disks * (mddev->chunk_sectors << 9));
md_init_stacking_limits(&lim);
lim.logical_block_size = mddev->logical_block_size;
lim.io_min = mddev->chunk_sectors << 9;
lim.io_opt = lim.io_min * (conf->raid_disks - conf->max_degraded);
lim.features |= BLK_FEAT_RAID_PARTIAL_STRIPES_EXPENSIVE;

1
drivers/nvme/host/apple.c
View File

@@ -1283,6 +1283,7 @@ static const struct nvme_ctrl_ops nvme_ctrl_ops = {
.reg_read64 = apple_nvme_reg_read64,
.free_ctrl = apple_nvme_free_ctrl,
.get_address = apple_nvme_get_address,
.get_virt_boundary = nvme_get_virt_boundary,
};
static void apple_nvme_async_probe(void *data, async_cookie_t cookie)

15
drivers/nvme/host/core.c
View File

@@ -2069,13 +2069,13 @@ static u32 nvme_max_drv_segments(struct nvme_ctrl *ctrl)
}
static void nvme_set_ctrl_limits(struct nvme_ctrl *ctrl,
struct queue_limits *lim)
struct queue_limits *lim, bool is_admin)
{
lim->max_hw_sectors = ctrl->max_hw_sectors;
lim->max_segments = min_t(u32, USHRT_MAX,
min_not_zero(nvme_max_drv_segments(ctrl), ctrl->max_segments));
lim->max_integrity_segments = ctrl->max_integrity_segments;
lim->virt_boundary_mask = NVME_CTRL_PAGE_SIZE - 1;
lim->virt_boundary_mask = ctrl->ops->get_virt_boundary(ctrl, is_admin);
lim->max_segment_size = UINT_MAX;
lim->dma_alignment = 3;
}
@@ -2177,7 +2177,7 @@ static int nvme_update_ns_info_generic(struct nvme_ns *ns,
int ret;
lim = queue_limits_start_update(ns->disk->queue);
nvme_set_ctrl_limits(ns->ctrl, &lim);
nvme_set_ctrl_limits(ns->ctrl, &lim, false);
memflags = blk_mq_freeze_queue(ns->disk->queue);
ret = queue_limits_commit_update(ns->disk->queue, &lim);
@@ -2381,7 +2381,7 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
ns->head->lba_shift = id->lbaf[lbaf].ds;
ns->head->nuse = le64_to_cpu(id->nuse);
capacity = nvme_lba_to_sect(ns->head, le64_to_cpu(id->nsze));
nvme_set_ctrl_limits(ns->ctrl, &lim);
nvme_set_ctrl_limits(ns->ctrl, &lim, false);
nvme_configure_metadata(ns->ctrl, ns->head, id, nvm, info);
nvme_set_chunk_sectors(ns, id, &lim);
if (!nvme_update_disk_info(ns, id, &lim))
@@ -2599,10 +2599,9 @@ static void nvme_configure_opal(struct nvme_ctrl *ctrl, bool was_suspended)
#ifdef CONFIG_BLK_DEV_ZONED
static int nvme_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones, struct blk_report_zones_args *args)
{
return nvme_ns_report_zones(disk->private_data, sector, nr_zones, cb,
data);
return nvme_ns_report_zones(disk->private_data, sector, nr_zones, args);
}
#else
#define nvme_report_zones NULL
@@ -3589,7 +3588,7 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl)
min_not_zero(ctrl->max_hw_sectors, max_hw_sectors);
lim = queue_limits_start_update(ctrl->admin_q);
nvme_set_ctrl_limits(ctrl, &lim);
nvme_set_ctrl_limits(ctrl, &lim, true);
ret = queue_limits_commit_update(ctrl->admin_q, &lim);
if (ret)
goto out_free;

6
drivers/nvme/host/fabrics.h
View File

@@ -217,6 +217,12 @@ static inline unsigned int nvmf_nr_io_queues(struct nvmf_ctrl_options *opts)
min(opts->nr_poll_queues, num_online_cpus());
}
static inline unsigned long nvmf_get_virt_boundary(struct nvme_ctrl *ctrl,
bool is_admin)
{
return 0;
}
int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val);
int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val);
int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val);

1
drivers/nvme/host/fc.c
View File

@@ -3361,6 +3361,7 @@ static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
.submit_async_event = nvme_fc_submit_async_event,
.delete_ctrl = nvme_fc_delete_ctrl,
.get_address = nvmf_get_address,
.get_virt_boundary = nvmf_get_virt_boundary,
};
static void

4
drivers/nvme/host/multipath.c
View File

@@ -576,7 +576,7 @@ static int nvme_ns_head_get_unique_id(struct gendisk *disk, u8 id[16],
#ifdef CONFIG_BLK_DEV_ZONED
static int nvme_ns_head_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones, struct blk_report_zones_args *args)
{
struct nvme_ns_head *head = disk->private_data;
struct nvme_ns *ns;
@@ -585,7 +585,7 @@ static int nvme_ns_head_report_zones(struct gendisk *disk, sector_t sector,
srcu_idx = srcu_read_lock(&head->srcu);
ns = nvme_find_path(head);
if (ns)
ret = nvme_ns_report_zones(ns, sector, nr_zones, cb, data);
ret = nvme_ns_report_zones(ns, sector, nr_zones, args);
srcu_read_unlock(&head->srcu, srcu_idx);
return ret;
}

9
drivers/nvme/host/nvme.h
View File

@@ -558,6 +558,12 @@ static inline bool nvme_ns_has_pi(struct nvme_ns_head *head)
return head->pi_type && head->ms == head->pi_size;
}
static inline unsigned long nvme_get_virt_boundary(struct nvme_ctrl *ctrl,
bool is_admin)
{
return NVME_CTRL_PAGE_SIZE - 1;
}
struct nvme_ctrl_ops {
const char *name;
struct module *module;
@@ -578,6 +584,7 @@ struct nvme_ctrl_ops {
int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
void (*print_device_info)(struct nvme_ctrl *ctrl);
bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);
unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl, bool is_admin);
};
/*
@@ -1108,7 +1115,7 @@ struct nvme_zone_info {
};
int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data);
unsigned int nr_zones, struct blk_report_zones_args *args);
int nvme_query_zone_info(struct nvme_ns *ns, unsigned lbaf,
struct nvme_zone_info *zi);
void nvme_update_zone_info(struct nvme_ns *ns, struct queue_limits *lim,

118
drivers/nvme/host/pci.c
View File

@@ -260,8 +260,20 @@ enum nvme_iod_flags {
/* single segment dma mapping */
IOD_SINGLE_SEGMENT = 1U << 2,
/* Data payload contains p2p memory */
IOD_DATA_P2P = 1U << 3,
/* Metadata contains p2p memory */
IOD_META_P2P = 1U << 4,
/* Data payload contains MMIO memory */
IOD_DATA_MMIO = 1U << 5,
/* Metadata contains MMIO memory */
IOD_META_MMIO = 1U << 6,
/* Metadata using non-coalesced MPTR */
IOD_SINGLE_META_SEGMENT = 1U << 5,
IOD_SINGLE_META_SEGMENT = 1U << 7,
};
struct nvme_dma_vec {
@@ -613,9 +625,22 @@ static inline enum nvme_use_sgl nvme_pci_use_sgls(struct nvme_dev *dev,
struct nvme_queue *nvmeq = req->mq_hctx->driver_data;
if (nvmeq->qid && nvme_ctrl_sgl_supported(&dev->ctrl)) {
if (nvme_req(req)->flags & NVME_REQ_USERCMD)
return SGL_FORCED;
if (req->nr_integrity_segments > 1)
/*
* When the controller is capable of using SGL, there are
* several conditions that we force to use it:
*
* 1. A request containing page gaps within the controller's
* mask can not use the PRP format.
*
* 2. User commands use SGL because that lets the device
* validate the requested transfer lengths.
*
* 3. Multiple integrity segments must use SGL as that's the
* only way to describe such a command in NVMe.
*/
if (req_phys_gap_mask(req) & (NVME_CTRL_PAGE_SIZE - 1) ||
nvme_req(req)->flags & NVME_REQ_USERCMD ||
req->nr_integrity_segments > 1)
return SGL_FORCED;
return SGL_SUPPORTED;
}
@@ -685,20 +710,20 @@ static void nvme_free_descriptors(struct request *req)
}
}
static void nvme_free_prps(struct request *req)
static void nvme_free_prps(struct request *req, unsigned int attrs)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
struct nvme_queue *nvmeq = req->mq_hctx->driver_data;
unsigned int i;
for (i = 0; i < iod->nr_dma_vecs; i++)
dma_unmap_page(nvmeq->dev->dev, iod->dma_vecs[i].addr,
iod->dma_vecs[i].len, rq_dma_dir(req));
dma_unmap_phys(nvmeq->dev->dev, iod->dma_vecs[i].addr,
iod->dma_vecs[i].len, rq_dma_dir(req), attrs);
mempool_free(iod->dma_vecs, nvmeq->dev->dmavec_mempool);
}
static void nvme_free_sgls(struct request *req, struct nvme_sgl_desc *sge,
struct nvme_sgl_desc *sg_list)
struct nvme_sgl_desc *sg_list, unsigned int attrs)
{
struct nvme_queue *nvmeq = req->mq_hctx->driver_data;
enum dma_data_direction dir = rq_dma_dir(req);
@@ -707,22 +732,25 @@ static void nvme_free_sgls(struct request *req, struct nvme_sgl_desc *sge,
unsigned int i;
if (sge->type == (NVME_SGL_FMT_DATA_DESC << 4)) {
dma_unmap_page(dma_dev, le64_to_cpu(sge->addr), len, dir);
dma_unmap_phys(dma_dev, le64_to_cpu(sge->addr), len, dir,
attrs);
return;
}
for (i = 0; i < len / sizeof(*sg_list); i++)
dma_unmap_page(dma_dev, le64_to_cpu(sg_list[i].addr),
le32_to_cpu(sg_list[i].length), dir);
dma_unmap_phys(dma_dev, le64_to_cpu(sg_list[i].addr),
le32_to_cpu(sg_list[i].length), dir, attrs);
}
static void nvme_unmap_metadata(struct request *req)
{
struct nvme_queue *nvmeq = req->mq_hctx->driver_data;
enum pci_p2pdma_map_type map = PCI_P2PDMA_MAP_NONE;
enum dma_data_direction dir = rq_dma_dir(req);
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
struct device *dma_dev = nvmeq->dev->dev;
struct nvme_sgl_desc *sge = iod->meta_descriptor;
unsigned int attrs = 0;
if (iod->flags & IOD_SINGLE_META_SEGMENT) {
dma_unmap_page(dma_dev, iod->meta_dma,
@@ -731,13 +759,20 @@ static void nvme_unmap_metadata(struct request *req)
return;
}
if (!blk_rq_integrity_dma_unmap(req, dma_dev, &iod->meta_dma_state,
iod->meta_total_len)) {
if (iod->flags & IOD_META_P2P)
map = PCI_P2PDMA_MAP_BUS_ADDR;
else if (iod->flags & IOD_META_MMIO) {
map = PCI_P2PDMA_MAP_THRU_HOST_BRIDGE;
attrs |= DMA_ATTR_MMIO;
}
if (!blk_rq_dma_unmap(req, dma_dev, &iod->meta_dma_state,
iod->meta_total_len, map)) {
if (nvme_pci_cmd_use_meta_sgl(&iod->cmd))
nvme_free_sgls(req, sge, &sge[1]);
nvme_free_sgls(req, sge, &sge[1], attrs);
else
dma_unmap_page(dma_dev, iod->meta_dma,
iod->meta_total_len, dir);
dma_unmap_phys(dma_dev, iod->meta_dma,
iod->meta_total_len, dir, attrs);
}
if (iod->meta_descriptor)
@@ -747,9 +782,11 @@ static void nvme_unmap_metadata(struct request *req)
static void nvme_unmap_data(struct request *req)
{
enum pci_p2pdma_map_type map = PCI_P2PDMA_MAP_NONE;
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
struct nvme_queue *nvmeq = req->mq_hctx->driver_data;
struct device *dma_dev = nvmeq->dev->dev;
unsigned int attrs = 0;
if (iod->flags & IOD_SINGLE_SEGMENT) {
static_assert(offsetof(union nvme_data_ptr, prp1) ==
@@ -759,12 +796,20 @@ static void nvme_unmap_data(struct request *req)
return;
}
if (!blk_rq_dma_unmap(req, dma_dev, &iod->dma_state, iod->total_len)) {
if (iod->flags & IOD_DATA_P2P)
map = PCI_P2PDMA_MAP_BUS_ADDR;
else if (iod->flags & IOD_DATA_MMIO) {
map = PCI_P2PDMA_MAP_THRU_HOST_BRIDGE;
attrs |= DMA_ATTR_MMIO;
}
if (!blk_rq_dma_unmap(req, dma_dev, &iod->dma_state, iod->total_len,
map)) {
if (nvme_pci_cmd_use_sgl(&iod->cmd))
nvme_free_sgls(req, iod->descriptors[0],
&iod->cmd.common.dptr.sgl);
&iod->cmd.common.dptr.sgl, attrs);
else
nvme_free_prps(req);
nvme_free_prps(req, attrs);
}
if (iod->nr_descriptors)
@@ -1035,6 +1080,19 @@ static blk_status_t nvme_map_data(struct request *req)
if (!blk_rq_dma_map_iter_start(req, dev->dev, &iod->dma_state, &iter))
return iter.status;
switch (iter.p2pdma.map) {
case PCI_P2PDMA_MAP_BUS_ADDR:
iod->flags |= IOD_DATA_P2P;
break;
case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
iod->flags |= IOD_DATA_MMIO;
break;
case PCI_P2PDMA_MAP_NONE:
break;
default:
return BLK_STS_RESOURCE;
}
if (use_sgl == SGL_FORCED ||
(use_sgl == SGL_SUPPORTED &&
(sgl_threshold && nvme_pci_avg_seg_size(req) >= sgl_threshold)))
@@ -1057,6 +1115,19 @@ static blk_status_t nvme_pci_setup_meta_iter(struct request *req)
&iod->meta_dma_state, &iter))
return iter.status;
switch (iter.p2pdma.map) {
case PCI_P2PDMA_MAP_BUS_ADDR:
iod->flags |= IOD_META_P2P;
break;
case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
iod->flags |= IOD_META_MMIO;
break;
case PCI_P2PDMA_MAP_NONE:
break;
default:
return BLK_STS_RESOURCE;
}
if (blk_rq_dma_map_coalesce(&iod->meta_dma_state))
entries = 1;
@@ -3250,6 +3321,14 @@ static bool nvme_pci_supports_pci_p2pdma(struct nvme_ctrl *ctrl)
return dma_pci_p2pdma_supported(dev->dev);
}
static unsigned long nvme_pci_get_virt_boundary(struct nvme_ctrl *ctrl,
bool is_admin)
{
if (!nvme_ctrl_sgl_supported(ctrl) || is_admin)
return NVME_CTRL_PAGE_SIZE - 1;
return 0;
}
static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.name = "pcie",
.module = THIS_MODULE,
@@ -3264,6 +3343,7 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.get_address = nvme_pci_get_address,
.print_device_info = nvme_pci_print_device_info,
.supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma,
.get_virt_boundary = nvme_pci_get_virt_boundary,
};
static int nvme_dev_map(struct nvme_dev *dev)

1
drivers/nvme/host/rdma.c
View File

@@ -2202,6 +2202,7 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
.delete_ctrl = nvme_rdma_delete_ctrl,
.get_address = nvmf_get_address,
.stop_ctrl = nvme_rdma_stop_ctrl,
.get_virt_boundary = nvme_get_virt_boundary,
};
/*

1
drivers/nvme/host/tcp.c
View File

@@ -2865,6 +2865,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ctrl_ops = {
.delete_ctrl = nvme_tcp_delete_ctrl,
.get_address = nvme_tcp_get_address,
.stop_ctrl = nvme_tcp_stop_ctrl,
.get_virt_boundary = nvmf_get_virt_boundary,
};
static bool

10
drivers/nvme/host/zns.c
View File

@@ -148,8 +148,8 @@ static void *nvme_zns_alloc_report_buffer(struct nvme_ns *ns,
static int nvme_zone_parse_entry(struct nvme_ns *ns,
struct nvme_zone_descriptor *entry,
unsigned int idx, report_zones_cb cb,
void *data)
unsigned int idx,
struct blk_report_zones_args *args)
{
struct nvme_ns_head *head = ns->head;
struct blk_zone zone = { };
@@ -169,11 +169,11 @@ static int nvme_zone_parse_entry(struct nvme_ns *ns,
else
zone.wp = nvme_lba_to_sect(head, le64_to_cpu(entry->wp));
return cb(&zone, idx, data);
return disk_report_zone(ns->disk, &zone, idx, args);
}
int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones, struct blk_report_zones_args *args)
{
struct nvme_zone_report *report;
struct nvme_command c = { };
@@ -213,7 +213,7 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector,
for (i = 0; i < nz && zone_idx < nr_zones; i++) {
ret = nvme_zone_parse_entry(ns, &report->entries[i],
zone_idx, cb, data);
zone_idx, args);
if (ret)
goto out_free;
zone_idx++;

1
drivers/nvme/target/loop.c
View File

@@ -511,6 +511,7 @@ static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
.submit_async_event = nvme_loop_submit_async_event,
.delete_ctrl = nvme_loop_delete_ctrl_host,
.get_address = nvmf_get_address,
.get_virt_boundary = nvme_get_virt_boundary,
};
static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl)

64
drivers/s390/block/dasd.c
View File

@@ -207,19 +207,6 @@ static int dasd_state_known_to_new(struct dasd_device *device)
return 0;
}
static struct dentry *dasd_debugfs_setup(const char *name,
struct dentry *base_dentry)
{
struct dentry *pde;
if (!base_dentry)
return NULL;
pde = debugfs_create_dir(name, base_dentry);
if (!pde || IS_ERR(pde))
return NULL;
return pde;
}
/*
* Request the irq line for the device.
*/
@@ -234,14 +221,14 @@ static int dasd_state_known_to_basic(struct dasd_device *device)
if (rc)
return rc;
block->debugfs_dentry =
dasd_debugfs_setup(block->gdp->disk_name,
debugfs_create_dir(block->gdp->disk_name,
dasd_debugfs_root_entry);
dasd_profile_init(&block->profile, block->debugfs_dentry);
if (dasd_global_profile_level == DASD_PROFILE_ON)
dasd_profile_on(&device->block->profile);
}
device->debugfs_dentry =
dasd_debugfs_setup(dev_name(&device->cdev->dev),
debugfs_create_dir(dev_name(&device->cdev->dev),
dasd_debugfs_root_entry);
dasd_profile_init(&device->profile, device->debugfs_dentry);
dasd_hosts_init(device->debugfs_dentry, device);
@@ -1057,19 +1044,9 @@ static const struct file_operations dasd_stats_raw_fops = {
static void dasd_profile_init(struct dasd_profile *profile,
struct dentry *base_dentry)
{
umode_t mode;
struct dentry *pde;
if (!base_dentry)
return;
profile->dentry = NULL;
profile->data = NULL;
mode = (S_IRUSR | S_IWUSR | S_IFREG);
pde = debugfs_create_file("statistics", mode, base_dentry,
profile, &dasd_stats_raw_fops);
if (pde && !IS_ERR(pde))
profile->dentry = pde;
return;
profile->dentry = debugfs_create_file("statistics", 0600, base_dentry,
profile, &dasd_stats_raw_fops);
}
static void dasd_profile_exit(struct dasd_profile *profile)
@@ -1089,25 +1066,9 @@ static void dasd_statistics_removeroot(void)
static void dasd_statistics_createroot(void)
{
struct dentry *pde;
dasd_debugfs_root_entry = NULL;
pde = debugfs_create_dir("dasd", NULL);
if (!pde || IS_ERR(pde))
goto error;
dasd_debugfs_root_entry = pde;
pde = debugfs_create_dir("global", dasd_debugfs_root_entry);
if (!pde || IS_ERR(pde))
goto error;
dasd_debugfs_global_entry = pde;
dasd_debugfs_root_entry = debugfs_create_dir("dasd", NULL);
dasd_debugfs_global_entry = debugfs_create_dir("global", dasd_debugfs_root_entry);
dasd_profile_init(&dasd_global_profile, dasd_debugfs_global_entry);
return;
error:
DBF_EVENT(DBF_ERR, "%s",
"Creation of the dasd debugfs interface failed");
dasd_statistics_removeroot();
return;
}
#else
@@ -1168,17 +1129,8 @@ static void dasd_hosts_exit(struct dasd_device *device)
static void dasd_hosts_init(struct dentry *base_dentry,
struct dasd_device *device)
{
struct dentry *pde;
umode_t mode;
if (!base_dentry)
return;
mode = S_IRUSR | S_IFREG;
pde = debugfs_create_file("host_access_list", mode, base_dentry,
device, &dasd_hosts_fops);
if (pde && !IS_ERR(pde))
device->hosts_dentry = pde;
device->hosts_dentry = debugfs_create_file("host_access_list", 0400, base_dentry,
device, &dasd_hosts_fops);
}
struct dasd_ccw_req *dasd_smalloc_request(int magic, int cplength, int datasize,

3
drivers/s390/block/dasd_devmap.c
View File

@@ -355,7 +355,8 @@ static int __init dasd_parse_range(const char *range)
/* each device in dasd= parameter should be set initially online */
features |= DASD_FEATURE_INITIAL_ONLINE;
while (from <= to) {
sprintf(bus_id, "%01x.%01x.%04x", from_id0, from_id1, from++);
scnprintf(bus_id, sizeof(bus_id),
"%01x.%01x.%04x", from_id0, from_id1, from++);
devmap = dasd_add_busid(bus_id, features);
if (IS_ERR(devmap)) {
rc = PTR_ERR(devmap);

8
drivers/s390/block/dasd_eckd.c
View File

@@ -6139,6 +6139,7 @@ static int dasd_eckd_copy_pair_swap(struct dasd_device *device, char *prim_busid
struct dasd_copy_relation *copy;
struct dasd_block *block;
struct gendisk *gdp;
int rc;
copy = device->copy;
if (!copy)
@@ -6173,6 +6174,13 @@ static int dasd_eckd_copy_pair_swap(struct dasd_device *device, char *prim_busid
/* swap blocklayer device link */
gdp = block->gdp;
dasd_add_link_to_gendisk(gdp, secondary);
rc = device_move(disk_to_dev(gdp), &secondary->cdev->dev, DPM_ORDER_NONE);
if (rc) {
dev_err(&primary->cdev->dev,
"copy_pair_swap: moving blockdevice parent %s->%s failed (%d)\n",
dev_name(&primary->cdev->dev),
dev_name(&secondary->cdev->dev), rc);
}
/* re-enable device */
dasd_device_remove_stop_bits(primary, DASD_STOPPED_PPRC);

80
drivers/s390/block/dasd_genhd.c
View File

@@ -22,6 +22,7 @@
static unsigned int queue_depth = 32;
static unsigned int nr_hw_queues = 4;
static void dasd_gd_free(struct gendisk *gdp);
module_param(queue_depth, uint, 0444);
MODULE_PARM_DESC(queue_depth, "Default queue depth for new DASD devices");
@@ -29,6 +30,37 @@ MODULE_PARM_DESC(queue_depth, "Default queue depth for new DASD devices");
module_param(nr_hw_queues, uint, 0444);
MODULE_PARM_DESC(nr_hw_queues, "Default number of hardware queues for new DASD devices");
/*
* Set device name.
* dasda - dasdz : 26 devices
* dasdaa - dasdzz : 676 devices, added up = 702
* dasdaaa - dasdzzz : 17576 devices, added up = 18278
* dasdaaaa - dasdzzzz : 456976 devices, added up = 475252
*/
static int dasd_name_format(char *prefix, int index, char *buf, int buflen)
{
const int base = 'z' - 'a' + 1;
char *begin = buf + strlen(prefix);
char *end = buf + buflen;
char *p;
int unit;
p = end - 1;
*p = '\0';
unit = base;
do {
if (p == begin)
return -EINVAL;
*--p = 'a' + (index % unit);
index = (index / unit) - 1;
} while (index >= 0);
memmove(begin, p, end - p);
memcpy(buf, prefix, strlen(prefix));
return 0;
}
/*
* Allocate and register gendisk structure for device.
*/
@@ -45,11 +77,13 @@ int dasd_gendisk_alloc(struct dasd_block *block)
};
struct gendisk *gdp;
struct dasd_device *base;
int len, rc;
unsigned int devindex;
int rc;
/* Make sure the minor for this device exists. */
base = block->base;
if (base->devindex >= DASD_PER_MAJOR)
devindex = base->devindex;
if (devindex >= DASD_PER_MAJOR)
return -EBUSY;
block->tag_set.ops = &dasd_mq_ops;
@@ -69,31 +103,17 @@ int dasd_gendisk_alloc(struct dasd_block *block)
/* Initialize gendisk structure. */
gdp->major = DASD_MAJOR;
gdp->first_minor = base->devindex << DASD_PARTN_BITS;
gdp->first_minor = devindex << DASD_PARTN_BITS;
gdp->minors = 1 << DASD_PARTN_BITS;
gdp->fops = &dasd_device_operations;
/*
* Set device name.
* dasda - dasdz : 26 devices
* dasdaa - dasdzz : 676 devices, added up = 702
* dasdaaa - dasdzzz : 17576 devices, added up = 18278
* dasdaaaa - dasdzzzz : 456976 devices, added up = 475252
*/
len = sprintf(gdp->disk_name, "dasd");
if (base->devindex > 25) {
if (base->devindex > 701) {
if (base->devindex > 18277)
len += sprintf(gdp->disk_name + len, "%c",
'a'+(((base->devindex-18278)
/17576)%26));
len += sprintf(gdp->disk_name + len, "%c",
'a'+(((base->devindex-702)/676)%26));
}
len += sprintf(gdp->disk_name + len, "%c",
'a'+(((base->devindex-26)/26)%26));
rc = dasd_name_format("dasd", devindex, gdp->disk_name, sizeof(gdp->disk_name));
if (rc) {
DBF_DEV_EVENT(DBF_ERR, block->base,
"setting disk name failed, rc %d", rc);
dasd_gd_free(gdp);
return rc;
}
len += sprintf(gdp->disk_name + len, "%c", 'a'+(base->devindex%26));
if (base->features & DASD_FEATURE_READONLY ||
test_bit(DASD_FLAG_DEVICE_RO, &base->flags))
@@ -111,15 +131,23 @@ int dasd_gendisk_alloc(struct dasd_block *block)
return 0;
}
/*
* Free gendisk structure
*/
static void dasd_gd_free(struct gendisk *gd)
{
del_gendisk(gd);
gd->private_data = NULL;
put_disk(gd);
}
/*
* Unregister and free gendisk structure for device.
*/
void dasd_gendisk_free(struct dasd_block *block)
{
if (block->gdp) {
del_gendisk(block->gdp);
block->gdp->private_data = NULL;
put_disk(block->gdp);
dasd_gd_free(block->gdp);
block->gdp = NULL;
blk_mq_free_tag_set(&block->tag_set);
}

2
drivers/scsi/sd.h
View File

@@ -240,7 +240,7 @@ blk_status_t sd_zbc_setup_zone_mgmt_cmnd(struct scsi_cmnd *cmd,
unsigned int sd_zbc_complete(struct scsi_cmnd *cmd, unsigned int good_bytes,
struct scsi_sense_hdr *sshdr);
int sd_zbc_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data);
unsigned int nr_zones, struct blk_report_zones_args *args);
#else /* CONFIG_BLK_DEV_ZONED */

20
drivers/scsi/sd_zbc.c
View File

@@ -35,8 +35,7 @@ static bool sd_zbc_is_gap_zone(const u8 buf[64])
* @buf: SCSI zone descriptor.
* @idx: Index of the zone relative to the first zone reported by the current
* sd_zbc_report_zones() call.
* @cb: Callback function pointer.
* @data: Second argument passed to @cb.
* @args: report zones arguments (callback, etc)
*
* Return: Value returned by @cb.
*
@@ -44,12 +43,11 @@ static bool sd_zbc_is_gap_zone(const u8 buf[64])
* call @cb(blk_zone, @data).
*/
static int sd_zbc_parse_report(struct scsi_disk *sdkp, const u8 buf[64],
unsigned int idx, report_zones_cb cb, void *data)
unsigned int idx, struct blk_report_zones_args *args)
{
struct scsi_device *sdp = sdkp->device;
struct blk_zone zone = { 0 };
sector_t start_lba, gran;
int ret;
if (WARN_ON_ONCE(sd_zbc_is_gap_zone(buf)))
return -EINVAL;
@@ -87,11 +85,7 @@ static int sd_zbc_parse_report(struct scsi_disk *sdkp, const u8 buf[64],
else
zone.wp = logical_to_sectors(sdp, get_unaligned_be64(&buf[24]));
ret = cb(&zone, idx, data);
if (ret)
return ret;
return 0;
return disk_report_zone(sdkp->disk, &zone, idx, args);
}
/**
@@ -217,14 +211,14 @@ static inline sector_t sd_zbc_zone_sectors(struct scsi_disk *sdkp)
* @disk: Disk to report zones for.
* @sector: Start sector.
* @nr_zones: Maximum number of zones to report.
* @cb: Callback function called to report zone information.
* @data: Second argument passed to @cb.
* @args: Callback arguments.
*
* Called by the block layer to iterate over zone information. See also the
* disk->fops->report_zones() calls in block/blk-zoned.c.
*/
int sd_zbc_report_zones(struct gendisk *disk, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data)
unsigned int nr_zones,
struct blk_report_zones_args *args)
{
struct scsi_disk *sdkp = scsi_disk(disk);
sector_t lba = sectors_to_logical(sdkp->device, sector);
@@ -283,7 +277,7 @@ int sd_zbc_report_zones(struct gendisk *disk, sector_t sector,
}
ret = sd_zbc_parse_report(sdkp, buf + offset, zone_idx,
cb, data);
args);
if (ret)
goto out;

11
fs/btrfs/zoned.c
View File

@@ -264,8 +264,8 @@ static int btrfs_get_dev_zones(struct btrfs_device *device, u64 pos,
}
}
ret = blkdev_report_zones(device->bdev, pos >> SECTOR_SHIFT, *nr_zones,
copy_zone_info_cb, zones);
ret = blkdev_report_zones_cached(device->bdev, pos >> SECTOR_SHIFT,
*nr_zones, copy_zone_info_cb, zones);
if (ret < 0) {
btrfs_err(device->fs_info,
"zoned: failed to read zone %llu on %s (devid %llu)",
@@ -494,6 +494,7 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache)
case BLK_ZONE_COND_IMP_OPEN:
case BLK_ZONE_COND_EXP_OPEN:
case BLK_ZONE_COND_CLOSED:
case BLK_ZONE_COND_ACTIVE:
__set_bit(nreported, zone_info->active_zones);
nactive++;
break;
@@ -896,9 +897,9 @@ int btrfs_sb_log_location_bdev(struct block_device *bdev, int mirror, int rw,
if (sb_zone + 1 >= nr_zones)
return -ENOENT;
ret = blkdev_report_zones(bdev, zone_start_sector(sb_zone, bdev),
BTRFS_NR_SB_LOG_ZONES, copy_zone_info_cb,
zones);
ret = blkdev_report_zones_cached(bdev, zone_start_sector(sb_zone, bdev),
BTRFS_NR_SB_LOG_ZONES,
copy_zone_info_cb, zones);
if (ret < 0)
return ret;
if (unlikely(ret != BTRFS_NR_SB_LOG_ZONES))

1
fs/xfs/libxfs/xfs_zones.c
View File

@@ -95,6 +95,7 @@ xfs_zone_validate_seq(
case BLK_ZONE_COND_IMP_OPEN:
case BLK_ZONE_COND_EXP_OPEN:
case BLK_ZONE_COND_CLOSED:
case BLK_ZONE_COND_ACTIVE:
return xfs_zone_validate_wp(zone, rtg, write_pointer);
case BLK_ZONE_COND_FULL:
return xfs_zone_validate_full(zone, rtg, write_pointer);

2
fs/xfs/xfs_zone_alloc.c
View File

@@ -1263,7 +1263,7 @@ xfs_mount_zones(
PAGE_SHIFT;
if (bdev_is_zoned(bt->bt_bdev)) {
error = blkdev_report_zones(bt->bt_bdev,
error = blkdev_report_zones_cached(bt->bt_bdev,
XFS_FSB_TO_BB(mp, mp->m_sb.sb_rtstart),
mp->m_sb.sb_rgcount, xfs_get_zone_info_cb, &iz);
if (error < 0)

4
include/linux/backing-dev-defs.h
View File

@@ -170,7 +170,9 @@ struct backing_dev_info {
u64 id;
struct rb_node rb_node; /* keyed by ->id */
struct list_head bdi_list;
unsigned long ra_pages; /* max readahead in PAGE_SIZE units */
/* max readahead in PAGE_SIZE units */
unsigned long __data_racy ra_pages;
unsigned long io_pages; /* max allowed IO size */
struct kref refcnt; /* Reference counter for the structure */

7
include/linux/bio-integrity.h
View File

@@ -13,7 +13,8 @@ enum bip_flags {
BIP_CHECK_GUARD = 1 << 5, /* guard check */
BIP_CHECK_REFTAG = 1 << 6, /* reftag check */
BIP_CHECK_APPTAG = 1 << 7, /* apptag check */
BIP_P2P_DMA = 1 << 8, /* using P2P address */
BIP_MEMPOOL = 1 << 15, /* buffer backed by mempool */
};
struct bio_integrity_payload {
@@ -140,4 +141,8 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page,
return 0;
}
#endif /* CONFIG_BLK_DEV_INTEGRITY */
void bio_integrity_alloc_buf(struct bio *bio, bool zero_buffer);
void bio_integrity_free_buf(struct bio_integrity_payload *bip);
#endif /* _LINUX_BIO_INTEGRITY_H */

2
include/linux/bio.h
View File

@@ -324,6 +324,8 @@ extern struct bio *bio_split(struct bio *bio, int sectors,
gfp_t gfp, struct bio_set *bs);
int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
unsigned *segs, unsigned max_bytes, unsigned len_align);
u8 bio_seg_gap(struct request_queue *q, struct bio *prev, struct bio *next,
u8 gaps_bit);
/**
* bio_next_split - get next @sectors from a bio, splitting if necessary

19
include/linux/blk-integrity.h
View File

@@ -8,6 +8,11 @@
struct request;
/*
* Maximum contiguous integrity buffer allocation.
*/
#define BLK_INTEGRITY_MAX_SIZE SZ_2M
enum blk_integrity_flags {
BLK_INTEGRITY_NOVERIFY = 1 << 0,
BLK_INTEGRITY_NOGENERATE = 1 << 1,
@@ -28,14 +33,6 @@ static inline bool queue_limits_stack_integrity_bdev(struct queue_limits *t,
#ifdef CONFIG_BLK_DEV_INTEGRITY
int blk_rq_map_integrity_sg(struct request *, struct scatterlist *);
static inline bool blk_rq_integrity_dma_unmap(struct request *req,
struct device *dma_dev, struct dma_iova_state *state,
size_t mapped_len)
{
return blk_dma_unmap(req, dma_dev, state, mapped_len,
bio_integrity(req->bio)->bip_flags & BIP_P2P_DMA);
}
int blk_rq_count_integrity_sg(struct request_queue *, struct bio *);
int blk_rq_integrity_map_user(struct request *rq, void __user *ubuf,
ssize_t bytes);
@@ -124,12 +121,6 @@ static inline int blk_rq_map_integrity_sg(struct request *q,
{
return 0;
}
static inline bool blk_rq_integrity_dma_unmap(struct request *req,
struct device *dma_dev, struct dma_iova_state *state,
size_t mapped_len)
{
return false;
}
static inline int blk_rq_integrity_map_user(struct request *rq,
void __user *ubuf,
ssize_t bytes)

28
include/linux/blk-mq-dma.h
View File

@@ -16,13 +16,13 @@ struct blk_dma_iter {
/* Output address range for this iteration */
dma_addr_t addr;
u32 len;
struct pci_p2pdma_map_state p2pdma;
/* Status code. Only valid when blk_rq_dma_map_iter_* returned false */
blk_status_t status;
/* Internal to blk_rq_dma_map_iter_* */
struct blk_map_iter iter;
struct pci_p2pdma_map_state p2pdma;
};
bool blk_rq_dma_map_iter_start(struct request *req, struct device *dma_dev,
@@ -43,36 +43,34 @@ static inline bool blk_rq_dma_map_coalesce(struct dma_iova_state *state)
}
/**
* blk_dma_unmap - try to DMA unmap a request
* blk_rq_dma_unmap - try to DMA unmap a request
* @req: request to unmap
* @dma_dev: device to unmap from
* @state: DMA IOVA state
* @mapped_len: number of bytes to unmap
* @is_p2p: true if mapped with PCI_P2PDMA_MAP_BUS_ADDR
* @map: peer-to-peer mapping type
*
* Returns %false if the callers need to manually unmap every DMA segment
* mapped using @iter or %true if no work is left to be done.
*/
static inline bool blk_dma_unmap(struct request *req, struct device *dma_dev,
struct dma_iova_state *state, size_t mapped_len, bool is_p2p)
static inline bool blk_rq_dma_unmap(struct request *req, struct device *dma_dev,
struct dma_iova_state *state, size_t mapped_len,
enum pci_p2pdma_map_type map)
{
if (is_p2p)
if (map == PCI_P2PDMA_MAP_BUS_ADDR)
return true;
if (dma_use_iova(state)) {
unsigned int attrs = 0;
if (map == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE)
attrs |= DMA_ATTR_MMIO;
dma_iova_destroy(dma_dev, state, mapped_len, rq_dma_dir(req),
0);
attrs);
return true;
}
return !dma_need_unmap(dma_dev);
}
static inline bool blk_rq_dma_unmap(struct request *req, struct device *dma_dev,
struct dma_iova_state *state, size_t mapped_len)
{
return blk_dma_unmap(req, dma_dev, state, mapped_len,
req->cmd_flags & REQ_P2PDMA);
}
#endif /* BLK_MQ_DMA_H */

30
include/linux/blk-mq.h
View File

@@ -152,6 +152,14 @@ struct request {
unsigned short nr_phys_segments;
unsigned short nr_integrity_segments;
/*
* The lowest set bit for address gaps between physical segments. This
* provides information necessary for dma optimization opprotunities,
* like for testing if the segments can be coalesced against the
* device's iommu granule.
*/
unsigned char phys_gap_bit;
#ifdef CONFIG_BLK_INLINE_ENCRYPTION
struct bio_crypt_ctx *crypt_ctx;
struct blk_crypto_keyslot *crypt_keyslot;
@@ -208,6 +216,14 @@ struct request {
void *end_io_data;
};
/*
* Returns a mask with all bits starting at req->phys_gap_bit set to 1.
*/
static inline unsigned long req_phys_gap_mask(const struct request *req)
{
return ~(((1 << req->phys_gap_bit) >> 1) - 1);
}
static inline enum req_op req_op(const struct request *req)
{
return req->cmd_flags & REQ_OP_MASK;
@@ -999,8 +1015,20 @@ static inline void *blk_mq_rq_to_pdu(struct request *rq)
return rq + 1;
}
static inline struct blk_mq_hw_ctx *queue_hctx(struct request_queue *q, int id)
{
struct blk_mq_hw_ctx *hctx;
rcu_read_lock();
hctx = rcu_dereference(q->queue_hw_ctx)[id];
rcu_read_unlock();
return hctx;
}
#define queue_for_each_hw_ctx(q, hctx, i) \
xa_for_each(&(q)->hctx_table, (i), (hctx))
for ((i) = 0; (i) < (q)->nr_hw_queues && \
({ hctx = queue_hctx((q), i); 1; }); (i)++)
#define hctx_for_each_ctx(hctx, ctx, i) \
for ((i) = 0; (i) < (hctx)->nr_ctx && \

14
include/linux/blk_types.h
View File

@@ -218,6 +218,18 @@ struct bio {
enum rw_hint bi_write_hint;
u8 bi_write_stream;
blk_status_t bi_status;
/*
* The bvec gap bit indicates the lowest set bit in any address offset
* between all bi_io_vecs. This field is initialized only after the bio
* is split to the hardware limits (see bio_split_io_at()). The value
* may be used to consider DMA optimization when performing that
* mapping. The value is compared to a power of two mask where the
* result depends on any bit set within the mask, so saving the lowest
* bit is sufficient to know if any segment gap collides with the mask.
*/
u8 bi_bvec_gap_bit;
atomic_t __bi_remaining;
struct bvec_iter bi_iter;
@@ -381,7 +393,6 @@ enum req_flag_bits {
__REQ_DRV, /* for driver use */
__REQ_FS_PRIVATE, /* for file system (submitter) use */
__REQ_ATOMIC, /* for atomic write operations */
__REQ_P2PDMA, /* contains P2P DMA pages */
/*
* Command specific flags, keep last:
*/
@@ -414,7 +425,6 @@ enum req_flag_bits {
#define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV)
#define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE)
#define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC)
#define REQ_P2PDMA (__force blk_opf_t)(1ULL << __REQ_P2PDMA)
#define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP)

62
include/linux/blkdev.h
View File

@@ -38,6 +38,7 @@ struct blk_flush_queue;
struct kiocb;
struct pr_ops;
struct rq_qos;
struct blk_report_zones_args;
struct blk_queue_stats;
struct blk_stat_callback;
struct blk_crypto_profile;
@@ -172,6 +173,7 @@ struct gendisk {
#define GD_ADDED 4
#define GD_SUPPRESS_PART_SCAN 5
#define GD_OWNS_QUEUE 6
#define GD_ZONE_APPEND_USED 7
struct mutex open_mutex; /* open/close mutex */
unsigned open_partitions; /* number of open partitions */
@@ -195,7 +197,7 @@ struct gendisk {
unsigned int nr_zones;
unsigned int zone_capacity;
unsigned int last_zone_capacity;
unsigned long __rcu *conv_zones_bitmap;
u8 __rcu *zones_cond;
unsigned int zone_wplugs_hash_bits;
atomic_t nr_zone_wplugs;
spinlock_t zone_wplugs_lock;
@@ -378,7 +380,7 @@ struct queue_limits {
unsigned int max_sectors;
unsigned int max_user_sectors;
unsigned int max_segment_size;
unsigned int min_segment_size;
unsigned int max_fast_segment_size;
unsigned int physical_block_size;
unsigned int logical_block_size;
unsigned int alignment_offset;
@@ -432,9 +434,17 @@ struct queue_limits {
typedef int (*report_zones_cb)(struct blk_zone *zone, unsigned int idx,
void *data);
int disk_report_zone(struct gendisk *disk, struct blk_zone *zone,
unsigned int idx, struct blk_report_zones_args *args);
int blkdev_get_zone_info(struct block_device *bdev, sector_t sector,
struct blk_zone *zone);
#define BLK_ALL_ZONES ((unsigned int)-1)
int blkdev_report_zones(struct block_device *bdev, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data);
int blkdev_report_zones_cached(struct block_device *bdev, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data);
int blkdev_zone_mgmt(struct block_device *bdev, enum req_op op,
sector_t sectors, sector_t nr_sectors);
int blk_revalidate_disk_zones(struct gendisk *disk);
@@ -485,7 +495,7 @@ struct request_queue {
*/
unsigned long queue_flags;
unsigned int rq_timeout;
unsigned int __data_racy rq_timeout;
unsigned int queue_depth;
@@ -493,7 +503,7 @@ struct request_queue {
/* hw dispatch queues */
unsigned int nr_hw_queues;
struct xarray hctx_table;
struct blk_mq_hw_ctx * __rcu *queue_hw_ctx;
struct percpu_ref q_usage_counter;
struct lock_class_key io_lock_cls_key;
@@ -921,12 +931,20 @@ static inline unsigned int bdev_zone_capacity(struct block_device *bdev,
{
return disk_zone_capacity(bdev->bd_disk, pos);
}
bool bdev_zone_is_seq(struct block_device *bdev, sector_t sector);
#else /* CONFIG_BLK_DEV_ZONED */
static inline unsigned int disk_nr_zones(struct gendisk *disk)
{
return 0;
}
static inline bool bdev_zone_is_seq(struct block_device *bdev, sector_t sector)
{
return false;
}
static inline bool bio_needs_zone_write_plugging(struct bio *bio)
{
return false;
@@ -1504,6 +1522,12 @@ static inline sector_t bdev_zone_sectors(struct block_device *bdev)
return q->limits.chunk_sectors;
}
static inline sector_t bdev_zone_start(struct block_device *bdev,
sector_t sector)
{
return sector & ~(bdev_zone_sectors(bdev) - 1);
}
static inline sector_t bdev_offset_from_zone_start(struct block_device *bdev,
sector_t sector)
{
@@ -1529,33 +1553,6 @@ static inline bool bdev_is_zone_aligned(struct block_device *bdev,
return bdev_is_zone_start(bdev, sector);
}
/**
* bdev_zone_is_seq - check if a sector belongs to a sequential write zone
* @bdev: block device to check
* @sector: sector number
*
* Check if @sector on @bdev is contained in a sequential write required zone.
*/
static inline bool bdev_zone_is_seq(struct block_device *bdev, sector_t sector)
{
bool is_seq = false;
#if IS_ENABLED(CONFIG_BLK_DEV_ZONED)
if (bdev_is_zoned(bdev)) {
struct gendisk *disk = bdev->bd_disk;
unsigned long *bitmap;
rcu_read_lock();
bitmap = rcu_dereference(disk->conv_zones_bitmap);
is_seq = !bitmap ||
!test_bit(disk_zone_no(disk, sector), bitmap);
rcu_read_unlock();
}
#endif
return is_seq;
}
int blk_zone_issue_zeroout(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp_mask);
@@ -1662,7 +1659,8 @@ struct block_device_operations {
/* this callback is with swap_lock and sometimes page table lock held */
void (*swap_slot_free_notify) (struct block_device *, unsigned long);
int (*report_zones)(struct gendisk *, sector_t sector,
unsigned int nr_zones, report_zones_cb cb, void *data);
unsigned int nr_zones,
struct blk_report_zones_args *args);
char *(*devnode)(struct gendisk *disk, umode_t *mode);
/* returns the length of the identifier or a negative errno: */
int (*get_unique_id)(struct gendisk *disk, u8 id[16],

3
include/linux/blktrace_api.h
View File

@@ -14,11 +14,12 @@
#include <linux/sysfs.h>
struct blk_trace {
int version;
int trace_state;
struct rchan *rchan;
unsigned long __percpu *sequence;
unsigned char __percpu *msg_data;
u16 act_mask;
u64 act_mask;
u64 start_lba;
u64 end_lba;
u32 pid;

10
include/linux/device-mapper.h
View File

@@ -538,12 +538,18 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone);
#ifdef CONFIG_BLK_DEV_ZONED
struct dm_report_zones_args {
struct dm_target *tgt;
struct gendisk *disk;
sector_t next_sector;
void *orig_data;
report_zones_cb orig_cb;
unsigned int zone_idx;
/* for block layer ->report_zones */
struct blk_report_zones_args *rep_args;
/* for internal users */
report_zones_cb cb;
void *data;
/* must be filled by ->report_zones before calling dm_report_zones_cb */
sector_t start;
};

34
include/linux/kfifo.h
View File

@@ -369,6 +369,30 @@ __kfifo_int_must_check_helper( \
}) \
)
/**
* kfifo_alloc_node - dynamically allocates a new fifo buffer on a NUMA node
* @fifo: pointer to the fifo
* @size: the number of elements in the fifo, this must be a power of 2
* @gfp_mask: get_free_pages mask, passed to kmalloc()
* @node: NUMA node to allocate memory on
*
* This macro dynamically allocates a new fifo buffer with NUMA node awareness.
*
* The number of elements will be rounded-up to a power of 2.
* The fifo will be release with kfifo_free().
* Return 0 if no error, otherwise an error code.
*/
#define kfifo_alloc_node(fifo, size, gfp_mask, node) \
__kfifo_int_must_check_helper( \
({ \
typeof((fifo) + 1) __tmp = (fifo); \
struct __kfifo *__kfifo = &__tmp->kfifo; \
__is_kfifo_ptr(__tmp) ? \
__kfifo_alloc_node(__kfifo, size, sizeof(*__tmp->type), gfp_mask, node) : \
-EINVAL; \
}) \
)
/**
* kfifo_free - frees the fifo
* @fifo: the fifo to be freed
@@ -899,8 +923,14 @@ __kfifo_uint_must_check_helper( \
)
extern int __kfifo_alloc(struct __kfifo *fifo, unsigned int size,
size_t esize, gfp_t gfp_mask);
extern int __kfifo_alloc_node(struct __kfifo *fifo, unsigned int size,
size_t esize, gfp_t gfp_mask, int node);
static inline int __kfifo_alloc(struct __kfifo *fifo, unsigned int size,
size_t esize, gfp_t gfp_mask)
{
return __kfifo_alloc_node(fifo, size, esize, gfp_mask, NUMA_NO_NODE);
}
extern void __kfifo_free(struct __kfifo *fifo);

6
include/linux/sbitmap.h
View File

@@ -75,7 +75,7 @@ struct sbitmap {
*/
struct sbitmap_word *map;
/*
/**
* @alloc_hint: Cache of last successfully allocated or freed bit.
*
* This is per-cpu, which allows multiple users to stick to different
@@ -128,7 +128,7 @@ struct sbitmap_queue {
*/
struct sbq_wait_state *ws;
/*
/**
* @ws_active: count of currently active ws waitqueues
*/
atomic_t ws_active;
@@ -547,6 +547,8 @@ static inline void sbq_index_atomic_inc(atomic_t *index)
* sbitmap_queue.
* @sbq: Bitmap queue to wait on.
* @wait_index: A counter per "user" of @sbq.
*
* Return: Next wait queue to be used
*/
static inline struct sbq_wait_state *sbq_wait_ptr(struct sbitmap_queue *sbq,
atomic_t *wait_index)

55
include/uapi/linux/blktrace_api.h
View File

@@ -26,11 +26,22 @@ enum blktrace_cat {
BLK_TC_DRV_DATA = 1 << 14, /* binary per-driver data */
BLK_TC_FUA = 1 << 15, /* fua requests */
BLK_TC_END = 1 << 15, /* we've run out of bits! */
BLK_TC_END_V1 = 1 << 15, /* we've run out of bits! */
BLK_TC_ZONE_APPEND = 1ull << 16, /* zone append */
BLK_TC_ZONE_RESET = 1ull << 17, /* zone reset */
BLK_TC_ZONE_RESET_ALL = 1ull << 18, /* zone reset all */
BLK_TC_ZONE_FINISH = 1ull << 19, /* zone finish */
BLK_TC_ZONE_OPEN = 1ull << 20, /* zone open */
BLK_TC_ZONE_CLOSE = 1ull << 21, /* zone close */
BLK_TC_WRITE_ZEROES = 1ull << 22, /* write-zeroes */
BLK_TC_END_V2 = 1ull << 22,
};
#define BLK_TC_SHIFT (16)
#define BLK_TC_ACT(act) ((act) << BLK_TC_SHIFT)
#define BLK_TC_ACT(act) ((u64)(act) << BLK_TC_SHIFT)
/*
* Basic trace actions
@@ -53,6 +64,8 @@ enum blktrace_act {
__BLK_TA_REMAP, /* bio was remapped */
__BLK_TA_ABORT, /* request aborted */
__BLK_TA_DRV_DATA, /* driver-specific binary data */
__BLK_TA_ZONE_PLUG, /* zone write plug was plugged */
__BLK_TA_ZONE_UNPLUG, /* zone write plug was unplugged */
__BLK_TA_CGROUP = 1 << 8, /* from a cgroup*/
};
@@ -88,12 +101,19 @@ enum blktrace_notify {
#define BLK_TA_ABORT (__BLK_TA_ABORT | BLK_TC_ACT(BLK_TC_QUEUE))
#define BLK_TA_DRV_DATA (__BLK_TA_DRV_DATA | BLK_TC_ACT(BLK_TC_DRV_DATA))
#define BLK_TA_ZONE_APPEND (__BLK_TA_COMPLETE |\
BLK_TC_ACT(BLK_TC_ZONE_APPEND))
#define BLK_TA_ZONE_PLUG (__BLK_TA_ZONE_PLUG | BLK_TC_ACT(BLK_TC_QUEUE))
#define BLK_TA_ZONE_UNPLUG (__BLK_TA_ZONE_UNPLUG |\
BLK_TC_ACT(BLK_TC_QUEUE))
#define BLK_TN_PROCESS (__BLK_TN_PROCESS | BLK_TC_ACT(BLK_TC_NOTIFY))
#define BLK_TN_TIMESTAMP (__BLK_TN_TIMESTAMP | BLK_TC_ACT(BLK_TC_NOTIFY))
#define BLK_TN_MESSAGE (__BLK_TN_MESSAGE | BLK_TC_ACT(BLK_TC_NOTIFY))
#define BLK_IO_TRACE_MAGIC 0x65617400
#define BLK_IO_TRACE_VERSION 0x07
#define BLK_IO_TRACE2_VERSION 0x08
/*
* The trace itself
@@ -113,6 +133,21 @@ struct blk_io_trace {
/* cgroup id will be stored here if exists */
};
struct blk_io_trace2 {
__u32 magic; /* MAGIC << 8 | BLK_IO_TRACE2_VERSION */
__u32 sequence; /* event number */
__u64 time; /* in nanoseconds */
__u64 sector; /* disk offset */
__u32 bytes; /* transfer length */
__u32 pid; /* who did it */
__u64 action; /* what happened */
__u32 device; /* device number */
__u32 cpu; /* on what cpu did it happen */
__u16 error; /* completion error */
__u16 pdu_len; /* length of data after this trace */
__u8 pad[12];
/* cgroup id will be stored here if it exists */
};
/*
* The remap event
*/
@@ -129,6 +164,7 @@ enum {
};
#define BLKTRACE_BDEV_SIZE 32
#define BLKTRACE_BDEV_SIZE2 64
/*
* User setup structure passed with BLKTRACESETUP
@@ -143,4 +179,19 @@ struct blk_user_trace_setup {
__u32 pid;
};
/*
* User setup structure passed with BLKTRACESETUP2
*/
struct blk_user_trace_setup2 {
char name[BLKTRACE_BDEV_SIZE2]; /* output */
__u64 act_mask; /* input */
__u32 buf_size; /* input */
__u32 buf_nr; /* input */
__u64 start_lba;
__u64 end_lba;
__u32 pid;
__u32 flags; /* currently unused */
__u64 reserved[11];
};
#endif /* _UAPIBLKTRACE_H */

46
include/uapi/linux/blkzoned.h
View File

@@ -48,6 +48,8 @@ enum blk_zone_type {
* FINISH ZONE command.
* @BLK_ZONE_COND_READONLY: The zone is read-only.
* @BLK_ZONE_COND_OFFLINE: The zone is offline (sectors cannot be read/written).
* @BLK_ZONE_COND_ACTIVE: The zone is either implicitly open, explicitly open,
* or closed.
*
* The Zone Condition state machine in the ZBC/ZAC standards maps the above
* deinitions as:
@@ -61,6 +63,13 @@ enum blk_zone_type {
*
* Conditions 0x5 to 0xC are reserved by the current ZBC/ZAC spec and should
* be considered invalid.
*
* The condition BLK_ZONE_COND_ACTIVE is used only with cached zone reports.
* It is used to report any of the BLK_ZONE_COND_IMP_OPEN,
* BLK_ZONE_COND_EXP_OPEN and BLK_ZONE_COND_CLOSED conditions. Conversely, a
* regular zone report will never report a zone condition using
* BLK_ZONE_COND_ACTIVE and instead use the conditions BLK_ZONE_COND_IMP_OPEN,
* BLK_ZONE_COND_EXP_OPEN or BLK_ZONE_COND_CLOSED as reported by the device.
*/
enum blk_zone_cond {
BLK_ZONE_COND_NOT_WP = 0x0,
@@ -71,15 +80,27 @@ enum blk_zone_cond {
BLK_ZONE_COND_READONLY = 0xD,
BLK_ZONE_COND_FULL = 0xE,
BLK_ZONE_COND_OFFLINE = 0xF,
BLK_ZONE_COND_ACTIVE = 0xFF,
};
/**
* enum blk_zone_report_flags - Feature flags of reported zone descriptors.
*
* @BLK_ZONE_REP_CAPACITY: Zone descriptor has capacity field.
* @BLK_ZONE_REP_CAPACITY: Output only. Indicates that zone descriptors in a
* zone report have a valid capacity field.
* @BLK_ZONE_REP_CACHED: Input only. Indicates that the zone report should be
* generated using cached zone information. In this case,
* the implicit open, explicit open and closed zone
* conditions are all reported with the
* BLK_ZONE_COND_ACTIVE condition.
*/
enum blk_zone_report_flags {
BLK_ZONE_REP_CAPACITY = (1 << 0),
/* Output flags */
BLK_ZONE_REP_CAPACITY = (1U << 0),
/* Input flags */
BLK_ZONE_REP_CACHED = (1U << 31),
};
/**
@@ -122,6 +143,10 @@ struct blk_zone {
* @sector: starting sector of report
* @nr_zones: IN maximum / OUT actual
* @flags: one or more flags as defined by enum blk_zone_report_flags.
* @flags: one or more flags as defined by enum blk_zone_report_flags.
* With BLKREPORTZONE, this field is ignored as an input and is valid
* only as an output. Using BLKREPORTZONEV2, this field is used as both
* input and output.
* @zones: Space to hold @nr_zones @zones entries on reply.
*
* The array of at most @nr_zones must follow this structure in memory.
@@ -148,9 +173,19 @@ struct blk_zone_range {
/**
* Zoned block device ioctl's:
*
* @BLKREPORTZONE: Get zone information. Takes a zone report as argument.
* The zone report will start from the zone containing the
* sector specified in the report request structure.
* @BLKREPORTZONE: Get zone information from a zoned device. Takes a zone report
* as argument. The zone report will start from the zone
* containing the sector specified in struct blk_zone_report.
* The flags field of struct blk_zone_report is used as an
* output only and ignored as an input.
* DEPRECATED, use BLKREPORTZONEV2 instead.
* @BLKREPORTZONEV2: Same as @BLKREPORTZONE but uses the flags field of
* struct blk_zone_report as an input, allowing to get a zone
* report using cached zone information if the flag
* BLK_ZONE_REP_CACHED is set. In such case, the zone report
* may include zones with the condition @BLK_ZONE_COND_ACTIVE
* (c.f. the description of this condition above for more
* details).
* @BLKRESETZONE: Reset the write pointer of the zones in the specified
* sector range. The sector range must be zone aligned.
* @BLKGETZONESZ: Get the device zone size in number of 512 B sectors.
@@ -169,5 +204,6 @@ struct blk_zone_range {
#define BLKOPENZONE _IOW(0x12, 134, struct blk_zone_range)
#define BLKCLOSEZONE _IOW(0x12, 135, struct blk_zone_range)
#define BLKFINISHZONE _IOW(0x12, 136, struct blk_zone_range)
#define BLKREPORTZONEV2 _IOWR(0x12, 142, struct blk_zone_report)
#endif /* _UAPI_BLKZONED_H */

3
include/uapi/linux/fs.h
View File

@@ -298,8 +298,9 @@ struct file_attr {
#define BLKROTATIONAL _IO(0x12,126)
#define BLKZEROOUT _IO(0x12,127)
#define BLKGETDISKSEQ _IOR(0x12,128,__u64)
/* 130-136 are used by zoned block device ioctls (uapi/linux/blkzoned.h) */
/* 130-136 and 142 are used by zoned block device ioctls (uapi/linux/blkzoned.h) */
/* 137-141 are used by blk-crypto ioctls (uapi/linux/blk-crypto.h) */
#define BLKTRACESETUP2 _IOWR(0x12, 142, struct blk_user_trace_setup2)
#define BMAP_IOCTL 1 /* obsolete - kept for compatibility */
#define FIBMAP _IO(0x00,1) /* bmap access */

Some files were not shown because too many files have changed in this diff Show More