== sanity test 398c: run fio to test AIO ================= 20:31:59 (1781224319) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.61517 s, 68.2 MB/s osc.lustre-OST0000-osc-ffff97e3f0b96800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181564: Thu Jun 11 20:32:46 2026 write: IOPS=75, BW=300KiB/s (308kB/s)(10.0MiB/34078msec); 0 zone resets slat (usec): min=53, max=13713, avg=367.83, stdev=922.90 clat (msec): min=28, max=422, avg=211.88, stdev=47.00 lat (msec): min=28, max=425, avg=212.25, stdev=46.97 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 363], 99.90th=[ 393], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 158, max= 431, per=25.03%, avg=299.88, stdev=56.40, samples=67 iops : min= 39, max= 107, avg=74.57, stdev=14.14, samples=67 lat (msec) : 50=0.16%, 100=0.55%, 250=79.06%, 500=20.23% cpu : usr=0.18%, sys=1.23%, ctx=2956, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181565: Thu Jun 11 20:32:46 2026 write: IOPS=75, BW=301KiB/s (308kB/s)(10.0MiB/34057msec); 0 zone resets slat (usec): min=52, max=20867, avg=326.12, stdev=977.32 clat (msec): min=19, max=426, avg=212.20, stdev=47.39 lat (msec): min=19, max=426, avg=212.53, stdev=47.37 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 368], 99.50th=[ 405], 99.90th=[ 418], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 167, max= 407, per=25.08%, avg=300.40, stdev=50.87, samples=67 iops : min= 41, max= 101, avg=74.72, stdev=12.69, samples=67 lat (msec) : 20=0.04%, 100=0.78%, 250=80.74%, 500=18.44% cpu : usr=0.15%, sys=1.03%, ctx=2481, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181566: Thu Jun 11 20:32:46 2026 write: IOPS=74, BW=300KiB/s (307kB/s)(10.0MiB/34181msec); 0 zone resets slat (usec): min=50, max=22979, avg=339.08, stdev=1089.88 clat (msec): min=90, max=416, avg=212.16, stdev=45.80 lat (msec): min=91, max=417, avg=212.50, stdev=45.82 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 176], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 359], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 418] bw ( KiB/s): min= 200, max= 415, per=24.89%, avg=298.19, stdev=48.96, samples=67 iops : min= 50, max= 103, avg=74.15, stdev=12.22, samples=67 lat (msec) : 100=0.47%, 250=81.99%, 500=17.54% cpu : usr=0.18%, sys=1.01%, ctx=2647, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181567: Thu Jun 11 20:32:46 2026 write: IOPS=74, BW=300KiB/s (307kB/s)(10.0MiB/34174msec); 0 zone resets slat (usec): min=43, max=13075, avg=318.64, stdev=862.44 clat (msec): min=93, max=417, avg=212.43, stdev=45.17 lat (msec): min=93, max=417, avg=212.75, stdev=45.16 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 157], 10.00th=[ 165], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 355], 99.50th=[ 393], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 143, max= 384, per=24.85%, avg=297.67, stdev=53.34, samples=67 iops : min= 35, max= 96, avg=74.03, stdev=13.32, samples=67 lat (msec) : 100=0.12%, 250=81.41%, 500=18.48% cpu : usr=0.17%, sys=0.87%, ctx=2452, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1198KiB/s (1227kB/s), 300KiB/s-301KiB/s (307kB/s-308kB/s), io=40.0MiB (41.9MB), run=34057-34181msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181582: Thu Jun 11 20:33:41 2026 read: IOPS=28, BW=113KiB/s (115kB/s)(5048KiB/44870msec) slat (usec): min=73, max=94475, avg=19038.30, stdev=10860.47 clat (msec): min=96, max=648, avg=264.88, stdev=72.09 lat (msec): min=106, max=692, avg=283.93, stdev=75.20 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 171], 10.00th=[ 188], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 239], 50.00th=[ 255], 60.00th=[ 271], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 355], 95.00th=[ 401], | 99.00th=[ 489], 99.50th=[ 527], 99.90th=[ 642], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 40, max= 188, per=25.06%, avg=111.53, stdev=33.05, samples=89 iops : min= 10, max= 47, avg=27.34, stdev= 8.28, samples=89 write: IOPS=28, BW=116KiB/s (118kB/s)(5192KiB/44870msec); 0 zone resets slat (usec): min=54, max=21086, avg=430.12, stdev=1374.87 clat (msec): min=130, max=638, avg=274.40, stdev=65.21 lat (msec): min=130, max=638, avg=274.84, stdev=65.21 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 180], 10.00th=[ 199], 20.00th=[ 220], | 30.00th=[ 239], 40.00th=[ 253], 50.00th=[ 268], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 355], 95.00th=[ 388], | 99.00th=[ 468], 99.50th=[ 514], 99.90th=[ 592], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 40, max= 205, per=24.78%, avg=114.97, stdev=33.88, samples=89 iops : min= 10, max= 51, avg=28.20, stdev= 8.51, samples=89 lat (msec) : 100=0.04%, 250=42.62%, 500=56.60%, 750=0.74% cpu : usr=0.23%, sys=1.58%, ctx=4473, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181583: Thu Jun 11 20:33:41 2026 read: IOPS=27, BW=111KiB/s (113kB/s)(4976KiB/45001msec) slat (usec): min=3142, max=96202, avg=19480.69, stdev=11004.34 clat (msec): min=69, max=549, avg=262.94, stdev=65.76 lat (msec): min=113, max=572, avg=282.42, stdev=68.45 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 171], 10.00th=[ 188], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 239], 50.00th=[ 255], 60.00th=[ 271], | 70.00th=[ 292], 80.00th=[ 313], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 447], 99.50th=[ 464], 99.90th=[ 506], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 15, max= 181, per=24.67%, avg=109.76, stdev=34.22, samples=89 iops : min= 3, max= 45, avg=26.91, stdev= 8.62, samples=89 write: IOPS=29, BW=117KiB/s (120kB/s)(5264KiB/45001msec); 0 zone resets slat (usec): min=56, max=24473, avg=411.98, stdev=1441.39 clat (msec): min=71, max=595, avg=277.17, stdev=68.87 lat (msec): min=72, max=595, avg=277.58, stdev=68.87 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 182], 10.00th=[ 197], 20.00th=[ 220], | 30.00th=[ 239], 40.00th=[ 255], 50.00th=[ 271], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 368], 95.00th=[ 405], | 99.00th=[ 489], 99.50th=[ 514], 99.90th=[ 584], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 55, max= 231, per=25.16%, avg=116.74, stdev=32.12, samples=89 iops : min= 13, max= 57, avg=28.63, stdev= 8.03, samples=89 lat (msec) : 100=0.08%, 250=41.48%, 500=57.93%, 750=0.51% cpu : usr=0.18%, sys=1.59%, ctx=4214, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181584: Thu Jun 11 20:33:41 2026 read: IOPS=28, BW=113KiB/s (116kB/s)(4984KiB/44126msec) slat (usec): min=62, max=83303, avg=18607.01, stdev=10950.29 clat (msec): min=21, max=565, avg=262.24, stdev=70.77 lat (msec): min=32, max=571, avg=280.86, stdev=73.25 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 165], 10.00th=[ 184], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 241], 50.00th=[ 255], 60.00th=[ 271], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 355], 95.00th=[ 388], | 99.00th=[ 456], 99.50th=[ 498], 99.90th=[ 558], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 47, max= 175, per=25.38%, avg=112.95, stdev=29.81, samples=87 iops : min= 11, max= 43, avg=27.68, stdev= 7.48, samples=87 write: IOPS=29, BW=119KiB/s (122kB/s)(5256KiB/44126msec); 0 zone resets slat (usec): min=53, max=20539, avg=324.99, stdev=904.49 clat (msec): min=28, max=545, avg=269.93, stdev=69.53 lat (msec): min=29, max=545, avg=270.25, stdev=69.53 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 169], 10.00th=[ 190], 20.00th=[ 215], | 30.00th=[ 234], 40.00th=[ 247], 50.00th=[ 262], 60.00th=[ 279], | 70.00th=[ 296], 80.00th=[ 321], 90.00th=[ 368], 95.00th=[ 393], | 99.00th=[ 468], 99.50th=[ 502], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 39, max= 215, per=25.56%, avg=118.62, stdev=37.80, samples=87 iops : min= 9, max= 53, avg=29.10, stdev= 9.50, samples=87 lat (msec) : 50=0.27%, 100=0.74%, 250=43.36%, 500=55.12%, 750=0.51% cpu : usr=0.18%, sys=1.61%, ctx=4242, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181585: Thu Jun 11 20:33:41 2026 read: IOPS=28, BW=112KiB/s (115kB/s)(5060KiB/44998msec) slat (msec): min=3, max=102, avg=19.08, stdev=11.36 clat (msec): min=74, max=578, avg=263.49, stdev=66.55 lat (msec): min=87, max=600, avg=282.58, stdev=69.34 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 171], 10.00th=[ 186], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 241], 50.00th=[ 257], 60.00th=[ 275], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 456], 99.50th=[ 481], 99.90th=[ 558], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 7, max= 184, per=25.15%, avg=111.93, stdev=34.37, samples=89 iops : min= 1, max= 46, avg=27.44, stdev= 8.63, samples=89 write: IOPS=28, BW=115KiB/s (118kB/s)(5180KiB/44998msec); 0 zone resets slat (usec): min=52, max=19969, avg=362.71, stdev=1015.51 clat (msec): min=47, max=628, avg=276.26, stdev=68.43 lat (msec): min=48, max=628, avg=276.62, stdev=68.44 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 178], 10.00th=[ 197], 20.00th=[ 220], | 30.00th=[ 239], 40.00th=[ 251], 50.00th=[ 266], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 368], 95.00th=[ 401], | 99.00th=[ 468], 99.50th=[ 502], 99.90th=[ 584], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 32, max= 207, per=24.64%, avg=114.33, stdev=33.38, samples=89 iops : min= 8, max= 51, avg=28.06, stdev= 8.39, samples=89 lat (msec) : 50=0.04%, 100=0.04%, 250=42.66%, 500=56.80%, 750=0.47% cpu : usr=0.15%, sys=1.60%, ctx=4530, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=446KiB/s (457kB/s), 111KiB/s-113KiB/s (113kB/s-116kB/s), io=19.6MiB (20.5MB), run=44126-45001msec WRITE: bw=464KiB/s (475kB/s), 115KiB/s-119KiB/s (118kB/s-122kB/s), io=20.4MiB (21.4MB), run=44126-45001msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181596: Thu Jun 11 20:33:53 2026 read: IOPS=1, BW=57.1MiB/s (59.9MB/s)(40.0MiB/700msec) slat (nsec): min=38283k, max=38283k, avg=38282767.00, stdev= 0.00 clat (nsec): min=639380k, max=639380k, avg=639380182.00, stdev= 0.00 lat (nsec): min=677668k, max=677668k, avg=677668211.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.86%, ctx=7, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=57.1MiB/s (59.9MB/s), 57.1MiB/s-57.1MiB/s (59.9MB/s-59.9MB/s), io=40.0MiB (41.9MB), run=700-700msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout