== sanity test 398c: run fio to test AIO ================= 12:29:36 (1777566576) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.179391 s, 234 MB/s osc.lustre-OST0000-osc-ffff9d59c8b94800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=194024: Thu Apr 30 12:29:52 2026 write: IOPS=227, BW=912KiB/s (934kB/s)(10.0MiB/11230msec); 0 zone resets slat (usec): min=51, max=7266, avg=205.37, stdev=284.76 clat (msec): min=36, max=130, avg=69.62, stdev=13.50 lat (msec): min=36, max=131, avg=69.82, stdev=13.51 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 87], 95.00th=[ 95], | 99.00th=[ 111], 99.50th=[ 116], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 758, max= 1157, per=24.98%, avg=911.05, stdev=101.43, samples=22 iops : min= 189, max= 289, avg=227.68, stdev=25.33, samples=22 lat (msec) : 50=4.30%, 100=93.12%, 250=2.58% cpu : usr=0.43%, sys=3.05%, ctx=2802, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194025: Thu Apr 30 12:29:52 2026 write: IOPS=230, BW=923KiB/s (945kB/s)(10.0MiB/11100msec); 0 zone resets slat (usec): min=38, max=13467, avg=195.80, stdev=332.02 clat (msec): min=14, max=154, avg=69.08, stdev=14.44 lat (msec): min=14, max=154, avg=69.28, stdev=14.45 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 94], | 99.00th=[ 112], 99.50th=[ 117], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 155] bw ( KiB/s): min= 704, max= 1274, per=25.19%, avg=918.81, stdev=127.55, samples=21 iops : min= 176, max= 318, avg=229.57, stdev=31.80, samples=21 lat (msec) : 20=0.08%, 50=5.39%, 100=91.99%, 250=2.54% cpu : usr=0.45%, sys=2.95%, ctx=2678, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194026: Thu Apr 30 12:29:52 2026 write: IOPS=228, BW=913KiB/s (935kB/s)(10.0MiB/11211msec); 0 zone resets slat (usec): min=48, max=3944, avg=211.52, stdev=251.01 clat (msec): min=11, max=164, avg=69.54, stdev=13.94 lat (msec): min=11, max=164, avg=69.76, stdev=13.94 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 94], | 99.00th=[ 108], 99.50th=[ 120], 99.90th=[ 140], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 744, max= 1155, per=25.00%, avg=911.68, stdev=109.69, samples=22 iops : min= 186, max= 288, avg=227.82, stdev=27.35, samples=22 lat (msec) : 20=0.08%, 50=4.06%, 100=93.36%, 250=2.50% cpu : usr=0.36%, sys=3.18%, ctx=2840, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=194027: Thu Apr 30 12:29:52 2026 write: IOPS=227, BW=912KiB/s (934kB/s)(10.0MiB/11229msec); 0 zone resets slat (usec): min=51, max=4037, avg=191.67, stdev=248.36 clat (msec): min=12, max=136, avg=69.67, stdev=13.48 lat (msec): min=12, max=136, avg=69.87, stdev=13.49 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 51], 10.00th=[ 54], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 88], 95.00th=[ 92], | 99.00th=[ 110], 99.50th=[ 114], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 136] bw ( KiB/s): min= 758, max= 1151, per=24.96%, avg=910.36, stdev=104.53, samples=22 iops : min= 189, max= 287, avg=227.50, stdev=26.07, samples=22 lat (msec) : 20=0.08%, 50=4.06%, 100=93.55%, 250=2.30% cpu : usr=0.23%, sys=2.87%, ctx=2669, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3647KiB/s (3735kB/s), 912KiB/s-923KiB/s (934kB/s-945kB/s), io=40.0MiB (41.9MB), run=11100-11230msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=194038: Thu Apr 30 12:30:05 2026 read: IOPS=130, BW=521KiB/s (533kB/s)(5048KiB/9692msec) slat (usec): min=1245, max=18521, avg=3960.66, stdev=1512.67 clat (msec): min=13, max=111, avg=57.43, stdev=11.67 lat (msec): min=16, max=114, avg=61.39, stdev=12.04 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 47], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 93], 99.90th=[ 109], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 384, max= 632, per=25.22%, avg=521.53, stdev=77.82, samples=19 iops : min= 96, max= 158, avg=130.26, stdev=19.42, samples=19 write: IOPS=133, BW=536KiB/s (549kB/s)(5192KiB/9692msec); 0 zone resets slat (usec): min=37, max=3599, avg=226.60, stdev=231.69 clat (msec): min=30, max=105, avg=58.38, stdev=10.29 lat (msec): min=30, max=105, avg=58.61, stdev=10.29 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 86], 99.50th=[ 96], 99.90th=[ 102], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 416, max= 760, per=25.00%, avg=538.32, stdev=80.78, samples=19 iops : min= 104, max= 190, avg=134.47, stdev=20.18, samples=19 lat (msec) : 20=0.04%, 50=24.77%, 100=74.96%, 250=0.23% cpu : usr=0.47%, sys=4.51%, ctx=4623, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194039: Thu Apr 30 12:30:05 2026 read: IOPS=129, BW=518KiB/s (531kB/s)(4976KiB/9597msec) slat (usec): min=77, max=10277, avg=3884.59, stdev=1450.54 clat (msec): min=8, max=134, avg=57.20, stdev=12.36 lat (msec): min=10, max=138, avg=61.09, stdev=12.68 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 90], 99.50th=[ 105], 99.90th=[ 131], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 414, max= 637, per=25.30%, avg=523.11, stdev=62.25, samples=18 iops : min= 103, max= 159, avg=130.61, stdev=15.64, samples=18 write: IOPS=137, BW=549KiB/s (562kB/s)(5264KiB/9597msec); 0 zone resets slat (usec): min=39, max=2727, avg=215.76, stdev=209.39 clat (msec): min=32, max=109, avg=58.46, stdev=10.68 lat (msec): min=32, max=109, avg=58.67, stdev=10.70 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 77], | 99.00th=[ 87], 99.50th=[ 96], 99.90th=[ 107], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 448, max= 760, per=25.73%, avg=554.00, stdev=73.21, samples=18 iops : min= 112, max= 190, avg=138.39, stdev=18.26, samples=18 lat (msec) : 10=0.08%, 20=0.08%, 50=25.43%, 100=73.91%, 250=0.51% cpu : usr=0.38%, sys=4.53%, ctx=4540, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194040: Thu Apr 30 12:30:05 2026 read: IOPS=128, BW=514KiB/s (527kB/s)(4984KiB/9690msec) slat (usec): min=1214, max=16662, avg=3962.26, stdev=1491.32 clat (msec): min=29, max=123, avg=57.75, stdev=11.72 lat (msec): min=33, max=127, avg=61.71, stdev=12.02 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 90], 99.50th=[ 100], 99.90th=[ 120], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 385, max= 656, per=25.00%, avg=517.05, stdev=74.20, samples=19 iops : min= 96, max= 164, avg=129.11, stdev=18.61, samples=19 write: IOPS=135, BW=542KiB/s (555kB/s)(5256KiB/9690msec); 0 zone resets slat (usec): min=56, max=4300, avg=222.77, stdev=243.73 clat (msec): min=29, max=118, avg=58.39, stdev=10.42 lat (msec): min=30, max=118, avg=58.61, stdev=10.42 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 71], 95.00th=[ 75], | 99.00th=[ 85], 99.50th=[ 91], 99.90th=[ 113], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 408, max= 710, per=25.17%, avg=541.89, stdev=80.50, samples=19 iops : min= 102, max= 177, avg=135.32, stdev=20.09, samples=19 lat (msec) : 50=24.14%, 100=75.59%, 250=0.27% cpu : usr=0.34%, sys=4.55%, ctx=4579, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194041: Thu Apr 30 12:30:05 2026 read: IOPS=130, BW=522KiB/s (534kB/s)(5060KiB/9702msec) slat (usec): min=1318, max=17321, avg=3960.11, stdev=1548.68 clat (msec): min=11, max=109, avg=57.28, stdev=11.11 lat (msec): min=16, max=113, avg=61.25, stdev=11.47 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 77], | 99.00th=[ 88], 99.50th=[ 93], 99.90th=[ 101], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 390, max= 664, per=25.23%, avg=521.84, stdev=74.37, samples=19 iops : min= 97, max= 166, avg=130.32, stdev=18.64, samples=19 write: IOPS=133, BW=534KiB/s (547kB/s)(5180KiB/9702msec); 0 zone resets slat (usec): min=41, max=3201, avg=220.40, stdev=234.97 clat (msec): min=34, max=104, avg=58.91, stdev=10.42 lat (msec): min=34, max=104, avg=59.13, stdev=10.42 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 77], | 99.00th=[ 87], 99.50th=[ 92], 99.90th=[ 102], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 440, max= 653, per=24.74%, avg=532.58, stdev=64.73, samples=19 iops : min= 110, max= 163, avg=133.00, stdev=16.20, samples=19 lat (msec) : 20=0.04%, 50=23.44%, 100=76.37%, 250=0.16% cpu : usr=0.49%, sys=4.40%, ctx=4598, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2068KiB/s (2118kB/s), 514KiB/s-522KiB/s (527kB/s-534kB/s), io=19.6MiB (20.5MB), run=9597-9702msec WRITE: bw=2153KiB/s (2205kB/s), 534KiB/s-549KiB/s (547kB/s-562kB/s), io=20.4MiB (21.4MB), run=9597-9702msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=194047: Thu Apr 30 12:30:09 2026 read: IOPS=5, BW=212MiB/s (222MB/s)(40.0MiB/189msec) slat (nsec): min=13301k, max=13301k, avg=13301320.00, stdev= 0.00 clat (nsec): min=171224k, max=171224k, avg=171223825.00, stdev= 0.00 lat (nsec): min=184530k, max=184530k, avg=184530284.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.85%, ctx=12, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=40.0MiB (41.9MB), run=189-189msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout