== sanity test 398c: run fio to test AIO ================= 14:13:07 (1782929587) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.201003 s, 209 MB/s osc.lustre-OST0000-osc-ffff9081c062f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154518: Wed Jul 1 14:13:56 2026 write: IOPS=56, BW=227KiB/s (233kB/s)(10.0MiB/45084msec); 0 zone resets slat (usec): min=40, max=4801, avg=217.81, stdev=226.11 clat (msec): min=28, max=567, avg=281.38, stdev=66.04 lat (msec): min=29, max=567, avg=281.60, stdev=66.04 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 182], 10.00th=[ 205], 20.00th=[ 228], | 30.00th=[ 247], 40.00th=[ 264], 50.00th=[ 279], 60.00th=[ 296], | 70.00th=[ 313], 80.00th=[ 334], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 464], 99.50th=[ 481], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 567] bw ( KiB/s): min= 136, max= 421, per=25.06%, avg=226.33, stdev=49.39, samples=89 iops : min= 34, max= 105, avg=56.37, stdev=12.37, samples=89 lat (msec) : 50=0.12%, 100=0.51%, 250=31.41%, 500=67.66%, 750=0.31% cpu : usr=0.11%, sys=0.94%, ctx=2961, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154519: Wed Jul 1 14:13:56 2026 write: IOPS=56, BW=226KiB/s (232kB/s)(10.0MiB/45291msec); 0 zone resets slat (usec): min=35, max=4770, avg=205.34, stdev=237.71 clat (msec): min=73, max=555, avg=282.54, stdev=67.25 lat (msec): min=73, max=555, avg=282.75, stdev=67.26 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 178], 10.00th=[ 205], 20.00th=[ 226], | 30.00th=[ 247], 40.00th=[ 266], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 313], 80.00th=[ 334], 90.00th=[ 363], 95.00th=[ 397], | 99.00th=[ 468], 99.50th=[ 514], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 127, max= 408, per=24.88%, avg=224.71, stdev=48.29, samples=90 iops : min= 31, max= 102, avg=55.97, stdev=12.11, samples=90 lat (msec) : 100=0.62%, 250=31.76%, 500=67.11%, 750=0.51% cpu : usr=0.10%, sys=0.87%, ctx=2859, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154520: Wed Jul 1 14:13:56 2026 write: IOPS=56, BW=226KiB/s (232kB/s)(10.0MiB/45290msec); 0 zone resets slat (usec): min=45, max=3734, avg=204.88, stdev=180.20 clat (msec): min=53, max=548, avg=282.58, stdev=64.26 lat (msec): min=53, max=548, avg=282.79, stdev=64.26 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 186], 10.00th=[ 207], 20.00th=[ 228], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 279], 60.00th=[ 296], | 70.00th=[ 313], 80.00th=[ 334], 90.00th=[ 368], 95.00th=[ 384], | 99.00th=[ 456], 99.50th=[ 472], 99.90th=[ 531], 99.95th=[ 542], | 99.99th=[ 550] bw ( KiB/s): min= 135, max= 409, per=24.89%, avg=224.72, stdev=48.12, samples=90 iops : min= 33, max= 102, avg=55.97, stdev=12.06, samples=90 lat (msec) : 100=0.23%, 250=31.80%, 500=67.70%, 750=0.27% cpu : usr=0.08%, sys=0.91%, ctx=3022, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154521: Wed Jul 1 14:13:56 2026 write: IOPS=56, BW=226KiB/s (231kB/s)(10.0MiB/45332msec); 0 zone resets slat (usec): min=39, max=4091, avg=220.84, stdev=241.40 clat (msec): min=109, max=567, avg=282.83, stdev=63.44 lat (msec): min=109, max=567, avg=283.05, stdev=63.44 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 188], 10.00th=[ 205], 20.00th=[ 230], | 30.00th=[ 247], 40.00th=[ 266], 50.00th=[ 284], 60.00th=[ 296], | 70.00th=[ 313], 80.00th=[ 334], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 464], 99.50th=[ 481], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 128, max= 368, per=24.92%, avg=224.99, stdev=48.55, samples=90 iops : min= 32, max= 92, avg=56.03, stdev=12.18, samples=90 lat (msec) : 250=32.38%, 500=67.34%, 750=0.27% cpu : usr=0.12%, sys=0.92%, ctx=3017, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=904KiB/s (925kB/s), 226KiB/s-227KiB/s (231kB/s-233kB/s), io=40.0MiB (41.9MB), run=45084-45332msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154537: Wed Jul 1 14:14:37 2026 read: IOPS=34, BW=136KiB/s (140kB/s)(5048KiB/37029msec) slat (usec): min=1204, max=19989, avg=4820.12, stdev=2347.98 clat (msec): min=5, max=498, avg=146.69, stdev=39.83 lat (msec): min=10, max=504, avg=151.51, stdev=40.25 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 99], 10.00th=[ 108], 20.00th=[ 117], | 30.00th=[ 127], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 255], 99.50th=[ 338], 99.90th=[ 451], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 48, max= 256, per=25.27%, avg=136.71, stdev=41.45, samples=73 iops : min= 12, max= 64, avg=33.93, stdev=10.42, samples=73 write: IOPS=35, BW=140KiB/s (144kB/s)(5192KiB/37029msec); 0 zone resets slat (usec): min=56, max=3449, avg=244.14, stdev=272.12 clat (msec): min=72, max=596, avg=307.83, stdev=82.20 lat (msec): min=72, max=597, avg=308.07, stdev=82.20 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 188], 10.00th=[ 209], 20.00th=[ 236], | 30.00th=[ 257], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 326], | 70.00th=[ 351], 80.00th=[ 372], 90.00th=[ 418], 95.00th=[ 451], | 99.00th=[ 518], 99.50th=[ 542], 99.90th=[ 584], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 87, max= 208, per=24.94%, avg=140.40, stdev=29.61, samples=73 iops : min= 21, max= 52, avg=34.86, stdev= 7.45, samples=73 lat (msec) : 10=0.16%, 20=0.04%, 50=0.35%, 100=2.73%, 250=58.67% lat (msec) : 500=37.11%, 750=0.94% cpu : usr=0.17%, sys=1.48%, ctx=5089, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154538: Wed Jul 1 14:14:37 2026 read: IOPS=33, BW=134KiB/s (138kB/s)(4976KiB/37052msec) slat (usec): min=1368, max=22255, avg=4799.32, stdev=2344.02 clat (msec): min=4, max=273, avg=146.03, stdev=36.13 lat (msec): min=6, max=280, avg=150.84, stdev=36.59 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 116], | 30.00th=[ 128], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 239], 99.50th=[ 245], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 48, max= 224, per=24.75%, avg=133.90, stdev=41.57, samples=73 iops : min= 12, max= 56, avg=33.23, stdev=10.44, samples=73 write: IOPS=35, BW=142KiB/s (145kB/s)(5264KiB/37052msec); 0 zone resets slat (usec): min=38, max=3200, avg=233.50, stdev=228.80 clat (msec): min=69, max=572, avg=306.76, stdev=84.17 lat (msec): min=69, max=573, avg=306.99, stdev=84.18 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 176], 10.00th=[ 209], 20.00th=[ 232], | 30.00th=[ 255], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 326], | 70.00th=[ 351], 80.00th=[ 376], 90.00th=[ 418], 95.00th=[ 464], | 99.00th=[ 518], 99.50th=[ 542], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 63, max= 216, per=25.15%, avg=141.62, stdev=32.41, samples=73 iops : min= 15, max= 54, avg=35.16, stdev= 8.11, samples=73 lat (msec) : 10=0.16%, 20=0.12%, 50=0.23%, 100=3.20%, 250=58.91% lat (msec) : 500=36.48%, 750=0.90% cpu : usr=0.16%, sys=1.51%, ctx=5042, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154539: Wed Jul 1 14:14:37 2026 read: IOPS=33, BW=135KiB/s (138kB/s)(4984KiB/37031msec) slat (usec): min=65, max=21064, avg=4899.50, stdev=2386.21 clat (msec): min=6, max=494, avg=145.00, stdev=35.66 lat (msec): min=10, max=498, avg=149.90, stdev=36.05 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 96], 10.00th=[ 106], 20.00th=[ 116], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 234], 99.50th=[ 239], 99.90th=[ 401], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 39, max= 256, per=25.03%, avg=135.40, stdev=43.57, samples=73 iops : min= 9, max= 64, avg=33.60, stdev=10.94, samples=73 write: IOPS=35, BW=142KiB/s (145kB/s)(5256KiB/37031msec); 0 zone resets slat (usec): min=42, max=5897, avg=241.03, stdev=323.68 clat (msec): min=56, max=586, avg=308.05, stdev=86.27 lat (msec): min=56, max=586, avg=308.30, stdev=86.27 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 178], 10.00th=[ 205], 20.00th=[ 234], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 326], | 70.00th=[ 351], 80.00th=[ 376], 90.00th=[ 426], 95.00th=[ 468], | 99.00th=[ 531], 99.50th=[ 542], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 80, max= 216, per=25.15%, avg=141.59, stdev=28.98, samples=73 iops : min= 20, max= 54, avg=35.16, stdev= 7.30, samples=73 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=3.71%, 250=58.28% lat (msec) : 500=36.64%, 750=1.21% cpu : usr=0.17%, sys=1.46%, ctx=5026, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154540: Wed Jul 1 14:14:37 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(5060KiB/37032msec) slat (usec): min=1275, max=65396, avg=4943.74, stdev=2892.73 clat (msec): min=5, max=274, avg=146.10, stdev=34.54 lat (msec): min=13, max=280, avg=151.05, stdev=34.97 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 121], | 30.00th=[ 130], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 192], 95.00th=[ 211], | 99.00th=[ 236], 99.50th=[ 247], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 56, max= 224, per=25.30%, avg=136.89, stdev=36.50, samples=73 iops : min= 14, max= 56, avg=33.99, stdev= 9.17, samples=73 write: IOPS=34, BW=140KiB/s (143kB/s)(5180KiB/37032msec); 0 zone resets slat (usec): min=36, max=3702, avg=236.96, stdev=259.68 clat (msec): min=43, max=577, avg=308.83, stdev=82.18 lat (msec): min=43, max=577, avg=309.07, stdev=82.18 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 190], 10.00th=[ 211], 20.00th=[ 232], | 30.00th=[ 257], 40.00th=[ 284], 50.00th=[ 305], 60.00th=[ 330], | 70.00th=[ 351], 80.00th=[ 376], 90.00th=[ 418], 95.00th=[ 460], | 99.00th=[ 518], 99.50th=[ 527], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 87, max= 224, per=24.72%, avg=139.16, stdev=31.13, samples=73 iops : min= 21, max= 56, avg=34.55, stdev= 7.83, samples=73 lat (msec) : 10=0.04%, 20=0.12%, 50=0.31%, 100=2.73%, 250=59.84% lat (msec) : 500=36.25%, 750=0.70% cpu : usr=0.16%, sys=1.49%, ctx=5008, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=542KiB/s (555kB/s), 134KiB/s-137KiB/s (138kB/s-140kB/s), io=19.6MiB (20.5MB), run=37029-37052msec WRITE: bw=564KiB/s (577kB/s), 140KiB/s-142KiB/s (143kB/s-145kB/s), io=20.4MiB (21.4MB), run=37029-37052msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154550: Wed Jul 1 14:14:42 2026 read: IOPS=5, BW=211MiB/s (221MB/s)(40.0MiB/190msec) slat (nsec): min=13338k, max=13338k, avg=13337738.00, stdev= 0.00 clat (nsec): min=168933k, max=168933k, avg=168932604.00, stdev= 0.00 lat (nsec): min=182275k, max=182275k, avg=182274648.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.82%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=40.0MiB (41.9MB), run=190-190msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout