== sanity test 398c: run fio to test AIO ================= 16:53:52 (1788468832) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.619334 s, 67.7 MB/s osc.lustre-OST0000-osc-ffff8f3ad1ddb000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146324: Thu Sep 3 16:57:10 2026 write: IOPS=13, BW=54.3KiB/s (55.6kB/s)(10.0MiB/188680msec); 0 zone resets slat (usec): min=28, max=20586, avg=283.26, stdev=1080.66 clat (msec): min=302, max=2642, avg=1178.11, stdev=316.02 lat (msec): min=305, max=2642, avg=1178.40, stdev=316.05 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 743], 10.00th=[ 818], 20.00th=[ 927], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1586], 95.00th=[ 1703], | 99.00th=[ 2089], 99.50th=[ 2265], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 128, per=25.60%, avg=55.55, stdev=29.32, samples=362 iops : min= 1, max= 32, avg=13.42, stdev= 7.37, samples=362 lat (msec) : 500=1.88%, 750=3.95%, 1000=21.41%, 2000=71.25%, >=2000=1.52% cpu : usr=0.04%, sys=0.23%, ctx=1902, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146325: Thu Sep 3 16:57:10 2026 write: IOPS=13, BW=54.3KiB/s (55.6kB/s)(10.0MiB/188684msec); 0 zone resets slat (usec): min=29, max=18069, avg=232.31, stdev=736.40 clat (msec): min=178, max=2561, avg=1178.68, stdev=314.86 lat (msec): min=185, max=2561, avg=1178.92, stdev=314.87 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 726], 10.00th=[ 827], 20.00th=[ 936], | 30.00th=[ 1028], 40.00th=[ 1099], 50.00th=[ 1167], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1435], 90.00th=[ 1552], 95.00th=[ 1687], | 99.00th=[ 2072], 99.50th=[ 2232], 99.90th=[ 2534], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 214, per=26.08%, avg=56.60, stdev=34.31, samples=355 iops : min= 1, max= 53, avg=13.68, stdev= 8.60, samples=355 lat (msec) : 250=0.59%, 500=1.33%, 750=3.98%, 1000=20.43%, 2000=72.34% lat (msec) : >=2000=1.33% cpu : usr=0.04%, sys=0.21%, ctx=1772, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146326: Thu Sep 3 16:57:10 2026 write: IOPS=13, BW=54.3KiB/s (55.6kB/s)(10.0MiB/188682msec); 0 zone resets slat (usec): min=29, max=19738, avg=268.11, stdev=1037.33 clat (msec): min=304, max=2544, avg=1178.15, stdev=311.50 lat (msec): min=304, max=2544, avg=1178.42, stdev=311.55 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 743], 10.00th=[ 827], 20.00th=[ 936], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1418], 90.00th=[ 1552], 95.00th=[ 1703], | 99.00th=[ 2072], 99.50th=[ 2265], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 144, per=25.03%, avg=54.32, stdev=28.71, samples=370 iops : min= 1, max= 36, avg=13.11, stdev= 7.21, samples=370 lat (msec) : 500=1.95%, 750=3.83%, 1000=21.68%, 2000=71.33%, >=2000=1.21% cpu : usr=0.05%, sys=0.22%, ctx=1772, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146327: Thu Sep 3 16:57:10 2026 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(10.0MiB/188355msec); 0 zone resets slat (usec): min=28, max=31510, avg=288.72, stdev=1205.06 clat (msec): min=181, max=2616, avg=1176.12, stdev=319.54 lat (msec): min=181, max=2616, avg=1176.41, stdev=319.51 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 726], 10.00th=[ 818], 20.00th=[ 927], | 30.00th=[ 1011], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1401], 90.00th=[ 1569], 95.00th=[ 1720], | 99.00th=[ 2123], 99.50th=[ 2265], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 176, per=25.17%, avg=54.62, stdev=25.54, samples=368 iops : min= 1, max= 44, avg=13.18, stdev= 6.42, samples=368 lat (msec) : 250=0.20%, 500=1.95%, 750=4.02%, 1000=22.34%, 2000=70.00% lat (msec) : >=2000=1.48% cpu : usr=0.05%, sys=0.23%, ctx=1997, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=217KiB/s (222kB/s), 54.3KiB/s-54.4KiB/s (55.6kB/s-55.7kB/s), io=40.0MiB (41.9MB), run=188355-188684msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146366: Thu Sep 3 16:59:56 2026 read: IOPS=8, BW=32.8KiB/s (33.6kB/s)(5048KiB/153791msec) slat (usec): min=3582, max=84899, avg=19710.01, stdev=12430.12 clat (msec): min=17, max=1523, avg=605.11, stdev=223.75 lat (msec): min=29, max=1591, avg=624.84, stdev=227.45 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 321], 10.00th=[ 368], 20.00th=[ 426], | 30.00th=[ 468], 40.00th=[ 518], 50.00th=[ 558], 60.00th=[ 609], | 70.00th=[ 667], 80.00th=[ 785], 90.00th=[ 927], 95.00th=[ 1003], | 99.00th=[ 1267], 99.50th=[ 1385], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 104, per=25.93%, avg=33.45, stdev=18.67, samples=293 iops : min= 1, max= 26, avg= 7.78, stdev= 4.67, samples=293 write: IOPS=8, BW=33.8KiB/s (34.6kB/s)(5192KiB/153791msec); 0 zone resets slat (usec): min=33, max=24588, avg=407.84, stdev=1473.68 clat (msec): min=332, max=2959, avg=1284.20, stdev=460.23 lat (msec): min=332, max=2959, avg=1284.61, stdev=460.22 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 625], 10.00th=[ 701], 20.00th=[ 869], | 30.00th=[ 986], 40.00th=[ 1099], 50.00th=[ 1267], 60.00th=[ 1401], | 70.00th=[ 1536], 80.00th=[ 1670], 90.00th=[ 1871], 95.00th=[ 2089], | 99.00th=[ 2467], 99.50th=[ 2668], 99.90th=[ 2903], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 104, per=25.54%, avg=34.22, stdev=18.21, samples=295 iops : min= 1, max= 26, avg= 7.98, stdev= 4.60, samples=295 lat (msec) : 20=0.04%, 50=0.04%, 100=0.23%, 250=0.39%, 500=17.30% lat (msec) : 750=26.29%, 1000=18.01%, 2000=34.26%, >=2000=3.44% cpu : usr=0.10%, sys=0.58%, ctx=4434, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146367: Thu Sep 3 16:59:56 2026 read: IOPS=8, BW=32.1KiB/s (32.9kB/s)(4976KiB/155103msec) slat (msec): min=3, max=540, avg=21.56, stdev=25.93 clat (msec): min=14, max=2064, avg=606.07, stdev=230.86 lat (msec): min=28, max=2094, avg=627.64, stdev=235.53 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 313], 10.00th=[ 355], 20.00th=[ 426], | 30.00th=[ 472], 40.00th=[ 523], 50.00th=[ 575], 60.00th=[ 625], | 70.00th=[ 693], 80.00th=[ 768], 90.00th=[ 927], 95.00th=[ 1020], | 99.00th=[ 1250], 99.50th=[ 1385], 99.90th=[ 1687], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 95, per=25.21%, avg=32.52, stdev=19.14, samples=297 iops : min= 1, max= 23, avg= 7.54, stdev= 4.79, samples=297 write: IOPS=8, BW=33.9KiB/s (34.8kB/s)(5264KiB/155103msec); 0 zone resets slat (usec): min=31, max=41106, avg=437.33, stdev=1867.57 clat (msec): min=220, max=3146, avg=1289.23, stdev=482.03 lat (msec): min=220, max=3146, avg=1289.67, stdev=482.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 869], | 30.00th=[ 995], 40.00th=[ 1083], 50.00th=[ 1250], 60.00th=[ 1385], | 70.00th=[ 1502], 80.00th=[ 1687], 90.00th=[ 1921], 95.00th=[ 2165], | 99.00th=[ 2601], 99.50th=[ 2903], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 128, per=25.00%, avg=33.50, stdev=20.28, samples=305 iops : min= 1, max= 32, avg= 7.80, stdev= 5.10, samples=305 lat (msec) : 20=0.08%, 50=0.16%, 100=0.31%, 250=0.74%, 500=16.56% lat (msec) : 750=26.41%, 1000=17.46%, 2000=34.57%, >=2000=3.71% cpu : usr=0.11%, sys=0.64%, ctx=4646, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146368: Thu Sep 3 16:59:56 2026 read: IOPS=8, BW=32.3KiB/s (33.1kB/s)(4984KiB/154304msec) slat (usec): min=26, max=101744, avg=19767.73, stdev=12581.23 clat (msec): min=45, max=1583, avg=598.37, stdev=223.78 lat (msec): min=55, max=1647, avg=618.15, stdev=227.66 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 326], 10.00th=[ 368], 20.00th=[ 422], | 30.00th=[ 468], 40.00th=[ 514], 50.00th=[ 550], 60.00th=[ 609], | 70.00th=[ 684], 80.00th=[ 768], 90.00th=[ 919], 95.00th=[ 1011], | 99.00th=[ 1267], 99.50th=[ 1351], 99.90th=[ 1569], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 104, per=26.14%, avg=33.72, stdev=19.51, samples=288 iops : min= 1, max= 26, avg= 7.85, stdev= 4.89, samples=288 write: IOPS=8, BW=34.1KiB/s (34.9kB/s)(5256KiB/154304msec); 0 zone resets slat (usec): min=28, max=41007, avg=440.28, stdev=1759.62 clat (msec): min=176, max=2857, avg=1291.24, stdev=462.46 lat (msec): min=176, max=2857, avg=1291.68, stdev=462.53 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 609], 10.00th=[ 718], 20.00th=[ 894], | 30.00th=[ 995], 40.00th=[ 1116], 50.00th=[ 1284], 60.00th=[ 1401], | 70.00th=[ 1519], 80.00th=[ 1687], 90.00th=[ 1888], 95.00th=[ 2089], | 99.00th=[ 2467], 99.50th=[ 2601], 99.90th=[ 2836], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 103, per=24.97%, avg=33.46, stdev=18.96, samples=304 iops : min= 1, max= 25, avg= 7.79, stdev= 4.78, samples=304 lat (msec) : 50=0.08%, 100=0.43%, 250=0.90%, 500=17.50%, 750=25.27% lat (msec) : 1000=17.66%, 2000=35.12%, >=2000=3.05% cpu : usr=0.12%, sys=0.60%, ctx=4661, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146369: Thu Sep 3 16:59:56 2026 read: IOPS=8, BW=32.8KiB/s (33.6kB/s)(5060KiB/154167msec) slat (usec): min=3241, max=89652, avg=20218.70, stdev=12232.18 clat (msec): min=43, max=1621, avg=608.11, stdev=228.92 lat (msec): min=70, max=1642, avg=628.33, stdev=232.32 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 330], 10.00th=[ 372], 20.00th=[ 426], | 30.00th=[ 468], 40.00th=[ 518], 50.00th=[ 558], 60.00th=[ 617], | 70.00th=[ 684], 80.00th=[ 768], 90.00th=[ 919], 95.00th=[ 1045], | 99.00th=[ 1368], 99.50th=[ 1552], 99.90th=[ 1603], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 87, per=26.44%, avg=34.11, stdev=18.69, samples=288 iops : min= 1, max= 21, avg= 7.94, stdev= 4.70, samples=288 write: IOPS=8, BW=33.6KiB/s (34.4kB/s)(5180KiB/154167msec); 0 zone resets slat (usec): min=29, max=24972, avg=389.22, stdev=1483.39 clat (msec): min=142, max=2816, avg=1287.23, stdev=450.75 lat (msec): min=143, max=2816, avg=1287.62, stdev=450.72 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 625], 10.00th=[ 735], 20.00th=[ 877], | 30.00th=[ 1003], 40.00th=[ 1116], 50.00th=[ 1267], 60.00th=[ 1401], | 70.00th=[ 1502], 80.00th=[ 1670], 90.00th=[ 1871], 95.00th=[ 2089], | 99.00th=[ 2433], 99.50th=[ 2500], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 120, per=25.29%, avg=33.89, stdev=18.88, samples=297 iops : min= 1, max= 30, avg= 7.90, stdev= 4.74, samples=297 lat (msec) : 50=0.04%, 100=0.20%, 250=0.47%, 500=17.50%, 750=25.90% lat (msec) : 1000=17.54%, 2000=35.08%, >=2000=3.28% cpu : usr=0.10%, sys=0.59%, ctx=4443, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129KiB/s (132kB/s), 32.1KiB/s-32.8KiB/s (32.9kB/s-33.6kB/s), io=19.6MiB (20.5MB), run=153791-155103msec WRITE: bw=135KiB/s (138kB/s), 33.6KiB/s-34.1KiB/s (34.4kB/s-34.9kB/s), io=20.4MiB (21.4MB), run=153791-155103msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146395: Thu Sep 3 17:00:02 2026 read: IOPS=1, BW=52.9MiB/s (55.5MB/s)(40.0MiB/756msec) slat (nsec): min=21590k, max=21590k, avg=21590338.00, stdev= 0.00 clat (nsec): min=720478k, max=720478k, avg=720477723.00, stdev= 0.00 lat (nsec): min=742072k, max=742072k, avg=742072365.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 718], 5.00th=[ 718], 10.00th=[ 718], 20.00th=[ 718], | 30.00th=[ 718], 40.00th=[ 718], 50.00th=[ 718], 60.00th=[ 718], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.72%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=52.9MiB/s (55.5MB/s), 52.9MiB/s-52.9MiB/s (55.5MB/s-55.5MB/s), io=40.0MiB (41.9MB), run=756-756msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout