== sanity test 398c: run fio to test AIO ================= 10:15:44 (1790777744) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.394625 s, 106 MB/s osc.lustre-OST0000-osc-ffff91d150a0d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181698: Wed Sep 30 10:16:26 2026 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30481msec); 0 zone resets slat (usec): min=28, max=14259, avg=231.82, stdev=663.49 clat (msec): min=12, max=357, avg=189.88, stdev=39.56 lat (msec): min=13, max=357, avg=190.12, stdev=39.55 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 207], 80.00th=[ 220], 90.00th=[ 239], 95.00th=[ 257], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 254, max= 518, per=25.02%, avg=336.07, stdev=59.37, samples=60 iops : min= 63, max= 129, avg=83.62, stdev=14.81, samples=60 lat (msec) : 20=0.08%, 50=0.16%, 100=0.43%, 250=93.12%, 500=6.21% cpu : usr=0.15%, sys=1.00%, ctx=2854, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181699: Wed Sep 30 10:16:26 2026 write: IOPS=84, BW=337KiB/s (345kB/s)(10.0MiB/30377msec); 0 zone resets slat (usec): min=28, max=20282, avg=258.16, stdev=1010.14 clat (msec): min=23, max=346, avg=189.41, stdev=40.31 lat (msec): min=24, max=346, avg=189.67, stdev=40.30 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 123], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 220], 90.00th=[ 241], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 347] bw ( KiB/s): min= 190, max= 495, per=25.07%, avg=336.73, stdev=65.18, samples=60 iops : min= 47, max= 123, avg=83.77, stdev=16.32, samples=60 lat (msec) : 50=0.16%, 100=1.09%, 250=91.68%, 500=7.07% cpu : usr=0.20%, sys=0.98%, ctx=2142, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181700: Wed Sep 30 10:16:26 2026 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30482msec); 0 zone resets slat (usec): min=27, max=20372, avg=235.70, stdev=1045.12 clat (msec): min=42, max=303, avg=189.35, stdev=38.08 lat (msec): min=44, max=303, avg=189.59, stdev=38.13 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 209], 80.00th=[ 220], 90.00th=[ 236], 95.00th=[ 251], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 246, max= 495, per=24.89%, avg=334.32, stdev=60.63, samples=60 iops : min= 61, max= 123, avg=83.17, stdev=15.17, samples=60 lat (msec) : 50=0.04%, 100=0.66%, 250=93.59%, 500=5.70% cpu : usr=0.19%, sys=0.83%, ctx=1965, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181701: Wed Sep 30 10:16:26 2026 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30477msec); 0 zone resets slat (usec): min=29, max=16728, avg=218.16, stdev=827.23 clat (msec): min=25, max=308, avg=189.28, stdev=38.63 lat (msec): min=25, max=309, avg=189.50, stdev=38.63 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 220], 90.00th=[ 241], 95.00th=[ 255], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 309] bw ( KiB/s): min= 227, max= 510, per=24.89%, avg=334.30, stdev=69.01, samples=60 iops : min= 56, max= 127, avg=83.17, stdev=17.23, samples=60 lat (msec) : 50=0.12%, 100=0.08%, 250=93.48%, 500=6.33% cpu : usr=0.17%, sys=0.92%, ctx=2068, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1344KiB/s (1376kB/s), 336KiB/s-337KiB/s (344kB/s-345kB/s), io=40.0MiB (41.9MB), run=30377-30482msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181715: Wed Sep 30 10:17:08 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5048KiB/35863msec) slat (usec): min=49, max=261912, avg=15662.41, stdev=11250.76 clat (msec): min=31, max=497, avg=211.81, stdev=58.98 lat (msec): min=47, max=533, avg=227.49, stdev=61.63 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 393], 99.90th=[ 489], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 55, max= 276, per=25.00%, avg=139.77, stdev=38.94, samples=71 iops : min= 13, max= 69, avg=34.41, stdev= 9.77, samples=71 write: IOPS=36, BW=145KiB/s (148kB/s)(5192KiB/35863msec); 0 zone resets slat (usec): min=31, max=21810, avg=253.51, stdev=1004.70 clat (msec): min=30, max=495, avg=218.01, stdev=61.54 lat (msec): min=31, max=495, avg=218.28, stdev=61.55 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 131], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 230], | 70.00th=[ 249], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 330], | 99.00th=[ 388], 99.50th=[ 405], 99.90th=[ 477], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 63, max= 286, per=24.77%, avg=144.14, stdev=44.15, samples=71 iops : min= 15, max= 71, avg=35.49, stdev=11.11, samples=71 lat (msec) : 50=0.12%, 100=1.45%, 250=71.56%, 500=26.88% cpu : usr=0.33%, sys=1.68%, ctx=3774, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181716: Wed Sep 30 10:17:08 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(4976KiB/35695msec) slat (usec): min=28, max=57787, avg=15564.86, stdev=8383.76 clat (msec): min=21, max=421, avg=210.74, stdev=56.37 lat (msec): min=32, max=447, avg=226.32, stdev=59.06 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 397], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 71, max= 221, per=24.65%, avg=137.80, stdev=37.60, samples=70 iops : min= 17, max= 55, avg=33.91, stdev= 9.45, samples=70 write: IOPS=36, BW=147KiB/s (151kB/s)(5264KiB/35695msec); 0 zone resets slat (usec): min=28, max=15793, avg=258.58, stdev=862.77 clat (msec): min=58, max=405, avg=219.25, stdev=55.44 lat (msec): min=59, max=405, avg=219.51, stdev=55.46 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 249], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 355], 99.50th=[ 380], 99.90th=[ 397], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 63, max= 284, per=25.15%, avg=146.37, stdev=40.51, samples=70 iops : min= 15, max= 71, avg=36.06, stdev=10.20, samples=70 lat (msec) : 50=0.12%, 100=1.17%, 250=72.07%, 500=26.64% cpu : usr=0.20%, sys=1.49%, ctx=3786, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181717: Wed Sep 30 10:17:08 2026 read: IOPS=34, BW=140KiB/s (143kB/s)(4984KiB/35696msec) slat (usec): min=2451, max=86756, avg=15387.20, stdev=8916.44 clat (msec): min=58, max=405, avg=210.68, stdev=56.09 lat (msec): min=67, max=418, avg=226.07, stdev=58.86 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 163], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 397], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 62, max= 276, per=24.81%, avg=138.71, stdev=42.07, samples=70 iops : min= 15, max= 69, avg=34.11, stdev=10.60, samples=70 write: IOPS=36, BW=147KiB/s (151kB/s)(5256KiB/35696msec); 0 zone resets slat (usec): min=27, max=19303, avg=262.40, stdev=948.51 clat (msec): min=31, max=397, avg=217.92, stdev=55.94 lat (msec): min=31, max=397, avg=218.18, stdev=55.92 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 245], 80.00th=[ 266], 90.00th=[ 296], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 380], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 55, max= 246, per=24.98%, avg=145.36, stdev=41.03, samples=70 iops : min= 13, max= 61, avg=35.80, stdev=10.30, samples=70 lat (msec) : 50=0.04%, 100=0.82%, 250=73.91%, 500=25.23% cpu : usr=0.23%, sys=1.68%, ctx=4241, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181718: Wed Sep 30 10:17:08 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(5060KiB/35693msec) slat (usec): min=3029, max=78465, avg=15339.02, stdev=9179.04 clat (msec): min=13, max=436, avg=211.41, stdev=57.26 lat (msec): min=37, max=468, avg=226.76, stdev=59.71 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 401], 99.90th=[ 435], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 63, max= 260, per=25.11%, avg=140.36, stdev=36.61, samples=70 iops : min= 15, max= 65, avg=34.56, stdev= 9.23, samples=70 write: IOPS=36, BW=145KiB/s (149kB/s)(5180KiB/35693msec); 0 zone resets slat (usec): min=30, max=11874, avg=263.26, stdev=860.33 clat (msec): min=94, max=421, avg=216.83, stdev=56.09 lat (msec): min=94, max=424, avg=217.09, stdev=56.14 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 130], 10.00th=[ 153], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 245], 80.00th=[ 264], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 384], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 80, max= 231, per=24.66%, avg=143.53, stdev=37.52, samples=70 iops : min= 20, max= 57, avg=35.34, stdev= 9.43, samples=70 lat (msec) : 20=0.04%, 100=0.70%, 250=73.59%, 500=25.66% cpu : usr=0.23%, sys=1.45%, ctx=4029, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=560KiB/s (573kB/s), 139KiB/s-142KiB/s (143kB/s-145kB/s), io=19.6MiB (20.5MB), run=35693-35863msec WRITE: bw=583KiB/s (597kB/s), 145KiB/s-147KiB/s (148kB/s-151kB/s), io=20.4MiB (21.4MB), run=35693-35863msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181727: Wed Sep 30 10:17:15 2026 read: IOPS=2, BW=107MiB/s (112MB/s)(40.0MiB/374msec) slat (nsec): min=37212k, max=37212k, avg=37212466.00, stdev= 0.00 clat (nsec): min=299297k, max=299297k, avg=299296659.00, stdev= 0.00 lat (nsec): min=336513k, max=336513k, avg=336513242.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.02%, ctx=10, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=40.0MiB (41.9MB), run=374-374msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout