== sanity test 398c: run fio to test AIO ================= 00:06:32 (1787889992) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.565719 s, 74.1 MB/s osc.lustre-OST0000-osc-ffff911678063000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179439: Fri Aug 28 00:07:15 2026 write: IOPS=81, BW=326KiB/s (334kB/s)(10.0MiB/31367msec); 0 zone resets slat (usec): min=26, max=16147, avg=300.42, stdev=987.14 clat (msec): min=30, max=433, avg=195.53, stdev=48.50 lat (msec): min=30, max=433, avg=195.84, stdev=48.49 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 338], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 435] bw ( KiB/s): min= 135, max= 482, per=25.14%, avg=325.50, stdev=70.58, samples=62 iops : min= 33, max= 120, avg=81.02, stdev=17.64, samples=62 lat (msec) : 50=0.08%, 100=1.25%, 250=86.09%, 500=12.58% cpu : usr=0.21%, sys=0.91%, ctx=2167, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179440: Fri Aug 28 00:07:15 2026 write: IOPS=80, BW=324KiB/s (332kB/s)(10.0MiB/31613msec); 0 zone resets slat (usec): min=29, max=16986, avg=306.08, stdev=991.63 clat (msec): min=19, max=390, avg=196.01, stdev=48.77 lat (msec): min=19, max=390, avg=196.32, stdev=48.77 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 255], 95.00th=[ 284], | 99.00th=[ 338], 99.50th=[ 372], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 167, max= 468, per=24.87%, avg=322.06, stdev=56.50, samples=62 iops : min= 41, max= 117, avg=80.13, stdev=14.16, samples=62 lat (msec) : 20=0.04%, 50=0.08%, 100=1.33%, 250=86.48%, 500=12.07% cpu : usr=0.19%, sys=1.03%, ctx=2505, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179441: Fri Aug 28 00:07:15 2026 write: IOPS=80, BW=324KiB/s (332kB/s)(10.0MiB/31611msec); 0 zone resets slat (usec): min=28, max=14155, avg=238.78, stdev=791.07 clat (msec): min=39, max=398, avg=196.24, stdev=47.78 lat (msec): min=39, max=398, avg=196.48, stdev=47.77 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 136], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 159, max= 477, per=24.89%, avg=322.35, stdev=61.69, samples=62 iops : min= 39, max= 119, avg=80.21, stdev=15.39, samples=62 lat (msec) : 50=0.04%, 100=0.66%, 250=86.95%, 500=12.34% cpu : usr=0.20%, sys=0.92%, ctx=1925, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179442: Fri Aug 28 00:07:15 2026 write: IOPS=81, BW=325KiB/s (332kB/s)(10.0MiB/31549msec); 0 zone resets slat (usec): min=27, max=30749, avg=315.27, stdev=1141.33 clat (msec): min=28, max=410, avg=196.07, stdev=47.18 lat (msec): min=29, max=410, avg=196.39, stdev=47.13 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 133], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 167, max= 428, per=24.94%, avg=322.94, stdev=61.83, samples=62 iops : min= 41, max= 107, avg=80.37, stdev=15.50, samples=62 lat (msec) : 50=0.20%, 100=0.66%, 250=86.80%, 500=12.34% cpu : usr=0.20%, sys=0.93%, ctx=2261, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1296KiB/s (1327kB/s), 324KiB/s-326KiB/s (332kB/s-334kB/s), io=40.0MiB (41.9MB), run=31367-31613msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179457: Fri Aug 28 00:08:02 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(5048KiB/36755msec) slat (msec): min=2, max=244, avg=16.02, stdev=10.64 clat (msec): min=92, max=435, avg=217.33, stdev=51.73 lat (msec): min=99, max=496, avg=233.36, stdev=53.61 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 305], | 99.00th=[ 380], 99.50th=[ 397], 99.90th=[ 409], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 39, max= 240, per=24.99%, avg=136.18, stdev=31.93, samples=73 iops : min= 9, max= 60, avg=33.53, stdev= 8.05, samples=73 write: IOPS=35, BW=141KiB/s (145kB/s)(5192KiB/36755msec); 0 zone resets slat (usec): min=34, max=11146, avg=271.32, stdev=810.60 clat (msec): min=91, max=462, avg=223.42, stdev=52.59 lat (msec): min=92, max=462, avg=223.70, stdev=52.61 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 155], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 397], 99.50th=[ 414], 99.90th=[ 443], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 45, max= 223, per=24.72%, avg=140.42, stdev=40.07, samples=73 iops : min= 11, max= 55, avg=34.62, stdev=10.08, samples=73 lat (msec) : 100=0.20%, 250=73.87%, 500=25.94% cpu : usr=0.25%, sys=1.58%, ctx=4370, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179458: Fri Aug 28 00:08:02 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4976KiB/36393msec) slat (usec): min=88, max=79542, avg=15661.82, stdev=8714.82 clat (msec): min=31, max=486, avg=217.14, stdev=54.29 lat (msec): min=31, max=515, avg=232.81, stdev=56.44 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 363], 99.50th=[ 372], 99.90th=[ 397], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 62, max= 230, per=24.95%, avg=136.00, stdev=32.79, samples=72 iops : min= 15, max= 57, avg=33.49, stdev= 8.26, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5264KiB/36393msec); 0 zone resets slat (usec): min=34, max=14980, avg=261.01, stdev=829.40 clat (msec): min=69, max=427, avg=221.46, stdev=48.13 lat (msec): min=69, max=427, avg=221.72, stdev=48.13 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 157], 10.00th=[ 167], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 384], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 55, max= 224, per=25.43%, avg=144.42, stdev=38.58, samples=72 iops : min= 13, max= 56, avg=35.63, stdev= 9.71, samples=72 lat (msec) : 50=0.04%, 100=0.16%, 250=74.92%, 500=24.88% cpu : usr=0.25%, sys=1.61%, ctx=4108, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179459: Fri Aug 28 00:08:02 2026 read: IOPS=33, BW=136KiB/s (139kB/s)(4984KiB/36767msec) slat (usec): min=3207, max=65628, avg=15813.99, stdev=9012.10 clat (msec): min=89, max=435, avg=217.24, stdev=51.11 lat (msec): min=99, max=451, avg=233.06, stdev=53.21 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 426], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 39, max= 247, per=24.72%, avg=134.70, stdev=37.77, samples=73 iops : min= 9, max= 61, avg=33.16, stdev= 9.52, samples=73 write: IOPS=35, BW=143KiB/s (146kB/s)(5256KiB/36767msec); 0 zone resets slat (usec): min=31, max=22064, avg=257.70, stdev=965.26 clat (msec): min=89, max=472, avg=224.44, stdev=51.39 lat (msec): min=89, max=472, avg=224.70, stdev=51.41 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 155], 10.00th=[ 167], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 249], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 368], 99.50th=[ 384], 99.90th=[ 460], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 55, max= 230, per=24.99%, avg=141.95, stdev=35.29, samples=73 iops : min= 13, max= 57, avg=34.99, stdev= 8.87, samples=73 lat (msec) : 100=0.20%, 250=73.67%, 500=26.13% cpu : usr=0.22%, sys=1.49%, ctx=4192, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179460: Fri Aug 28 00:08:02 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(5060KiB/36713msec) slat (usec): min=2792, max=80316, avg=15915.98, stdev=8681.66 clat (msec): min=30, max=440, avg=215.45, stdev=50.73 lat (msec): min=38, max=499, avg=231.38, stdev=52.98 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 368], 99.50th=[ 397], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 60, max= 231, per=25.24%, avg=137.58, stdev=37.15, samples=72 iops : min= 15, max= 57, avg=33.89, stdev= 9.30, samples=72 write: IOPS=35, BW=141KiB/s (144kB/s)(5180KiB/36713msec); 0 zone resets slat (usec): min=29, max=14190, avg=274.98, stdev=804.18 clat (msec): min=101, max=453, avg=224.98, stdev=49.67 lat (msec): min=102, max=453, avg=225.26, stdev=49.69 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 157], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 451], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 47, max= 216, per=24.75%, avg=140.60, stdev=33.57, samples=72 iops : min= 11, max= 54, avg=34.62, stdev= 8.52, samples=72 lat (msec) : 50=0.04%, 250=75.35%, 500=24.61% cpu : usr=0.28%, sys=1.62%, ctx=4139, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=546KiB/s (559kB/s), 136KiB/s-138KiB/s (139kB/s-141kB/s), io=19.6MiB (20.5MB), run=36393-36767msec WRITE: bw=568KiB/s (582kB/s), 141KiB/s-145KiB/s (144kB/s-148kB/s), io=20.4MiB (21.4MB), run=36393-36767msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179470: Fri Aug 28 00:08:13 2026 read: IOPS=1, BW=66.9MiB/s (70.1MB/s)(40.0MiB/598msec) slat (nsec): min=66488k, max=66488k, avg=66487909.00, stdev= 0.00 clat (nsec): min=503835k, max=503835k, avg=503834837.00, stdev= 0.00 lat (nsec): min=570328k, max=570328k, avg=570328004.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.35%, ctx=7, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.9MiB/s (70.1MB/s), 66.9MiB/s-66.9MiB/s (70.1MB/s-70.1MB/s), io=40.0MiB (41.9MB), run=598-598msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout