== sanity test 398c: run fio to test AIO ================= 23:11:44 (1763352704) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.286387 s, 146 MB/s osc.lustre-OST0000-osc-ffff96834533c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184001: Sun Nov 16 23:12:05 2025 write: IOPS=155, BW=623KiB/s (638kB/s)(10.0MiB/16426msec); 0 zone resets slat (usec): min=29, max=3741, avg=138.02, stdev=240.55 clat (msec): min=14, max=214, avg=102.18, stdev=26.38 lat (msec): min=15, max=214, avg=102.32, stdev=26.40 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 87], 40.00th=[ 93], 50.00th=[ 99], 60.00th=[ 107], | 70.00th=[ 115], 80.00th=[ 124], 90.00th=[ 136], 95.00th=[ 148], | 99.00th=[ 184], 99.50th=[ 192], 99.90th=[ 209], 99.95th=[ 211], | 99.99th=[ 215] bw ( KiB/s): min= 415, max= 830, per=24.98%, avg=622.50, stdev=123.31, samples=32 iops : min= 103, max= 207, avg=155.38, stdev=30.83, samples=32 lat (msec) : 20=0.08%, 50=0.66%, 100=52.85%, 250=46.41% cpu : usr=0.26%, sys=1.55%, ctx=2654, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184002: Sun Nov 16 23:12:05 2025 write: IOPS=156, BW=625KiB/s (640kB/s)(10.0MiB/16395msec); 0 zone resets slat (usec): min=25, max=3597, avg=125.14, stdev=190.54 clat (msec): min=25, max=246, avg=102.20, stdev=26.52 lat (msec): min=25, max=246, avg=102.32, stdev=26.53 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 88], 40.00th=[ 93], 50.00th=[ 99], 60.00th=[ 107], | 70.00th=[ 114], 80.00th=[ 124], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 178], 99.50th=[ 188], 99.90th=[ 215], 99.95th=[ 228], | 99.99th=[ 247] bw ( KiB/s): min= 384, max= 891, per=25.06%, avg=624.50, stdev=130.07, samples=32 iops : min= 96, max= 222, avg=155.88, stdev=32.53, samples=32 lat (msec) : 50=1.29%, 100=50.35%, 250=48.36% cpu : usr=0.18%, sys=1.51%, ctx=2759, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184003: Sun Nov 16 23:12:05 2025 write: IOPS=156, BW=625KiB/s (640kB/s)(10.0MiB/16395msec); 0 zone resets slat (usec): min=27, max=3383, avg=135.38, stdev=213.62 clat (msec): min=8, max=246, avg=102.04, stdev=27.95 lat (msec): min=8, max=246, avg=102.17, stdev=27.95 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 100], 60.00th=[ 106], | 70.00th=[ 112], 80.00th=[ 123], 90.00th=[ 136], 95.00th=[ 148], | 99.00th=[ 199], 99.50th=[ 207], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 392, max= 894, per=25.12%, avg=626.00, stdev=129.76, samples=32 iops : min= 98, max= 223, avg=156.28, stdev=32.48, samples=32 lat (msec) : 10=0.04%, 20=0.08%, 50=1.37%, 100=49.69%, 250=48.83% cpu : usr=0.22%, sys=1.53%, ctx=2956, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184004: Sun Nov 16 23:12:05 2025 write: IOPS=155, BW=623KiB/s (638kB/s)(10.0MiB/16431msec); 0 zone resets slat (usec): min=30, max=3816, avg=149.75, stdev=254.04 clat (msec): min=13, max=217, avg=102.17, stdev=26.15 lat (msec): min=13, max=218, avg=102.32, stdev=26.16 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 68], 10.00th=[ 71], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 92], 50.00th=[ 100], 60.00th=[ 107], | 70.00th=[ 114], 80.00th=[ 123], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 180], 99.50th=[ 190], 99.90th=[ 201], 99.95th=[ 207], | 99.99th=[ 218] bw ( KiB/s): min= 400, max= 894, per=24.91%, avg=620.84, stdev=123.18, samples=32 iops : min= 100, max= 223, avg=154.97, stdev=30.82, samples=32 lat (msec) : 20=0.04%, 50=0.78%, 100=49.88%, 250=49.30% cpu : usr=0.26%, sys=1.67%, ctx=3001, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2493KiB/s (2553kB/s), 623KiB/s-625KiB/s (638kB/s-640kB/s), io=40.0MiB (41.9MB), run=16395-16431msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184016: Sun Nov 16 23:12:28 2025 read: IOPS=71, BW=286KiB/s (292kB/s)(5048KiB/17679msec) slat (usec): min=39, max=33395, avg=7572.65, stdev=3448.66 clat (msec): min=9, max=208, avg=104.62, stdev=27.47 lat (msec): min=9, max=220, avg=112.20, stdev=28.78 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 82], | 30.00th=[ 88], 40.00th=[ 95], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 128], 90.00th=[ 140], 95.00th=[ 155], | 99.00th=[ 180], 99.50th=[ 188], 99.90th=[ 197], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 184, max= 422, per=25.12%, avg=284.37, stdev=62.99, samples=35 iops : min= 46, max= 105, avg=70.66, stdev=15.78, samples=35 write: IOPS=73, BW=294KiB/s (301kB/s)(5192KiB/17679msec); 0 zone resets slat (usec): min=36, max=7716, avg=168.27, stdev=370.64 clat (msec): min=44, max=193, avg=108.27, stdev=24.76 lat (msec): min=50, max=193, avg=108.44, stdev=24.76 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 79], 20.00th=[ 88], | 30.00th=[ 94], 40.00th=[ 101], 50.00th=[ 106], 60.00th=[ 112], | 70.00th=[ 120], 80.00th=[ 128], 90.00th=[ 144], 95.00th=[ 155], | 99.00th=[ 174], 99.50th=[ 178], 99.90th=[ 192], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 174, max= 400, per=24.88%, avg=293.03, stdev=59.99, samples=35 iops : min= 43, max= 100, avg=72.80, stdev=15.06, samples=35 lat (msec) : 10=0.04%, 20=0.16%, 50=0.20%, 100=43.01%, 250=56.60% cpu : usr=0.33%, sys=2.49%, ctx=4750, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184017: Sun Nov 16 23:12:28 2025 read: IOPS=70, BW=281KiB/s (287kB/s)(4976KiB/17724msec) slat (usec): min=2281, max=25238, avg=7587.03, stdev=3164.64 clat (msec): min=24, max=193, avg=104.40, stdev=25.36 lat (msec): min=36, max=212, avg=111.99, stdev=26.45 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 89], 40.00th=[ 95], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 127], 90.00th=[ 140], 95.00th=[ 148], | 99.00th=[ 171], 99.50th=[ 178], 99.90th=[ 192], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 144, max= 424, per=24.70%, avg=279.63, stdev=69.66, samples=35 iops : min= 36, max= 106, avg=69.49, stdev=17.42, samples=35 write: IOPS=74, BW=297KiB/s (304kB/s)(5264KiB/17724msec); 0 zone resets slat (usec): min=34, max=7868, avg=160.48, stdev=322.94 clat (msec): min=55, max=203, avg=108.43, stdev=25.41 lat (msec): min=55, max=203, avg=108.59, stdev=25.42 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 79], 20.00th=[ 87], | 30.00th=[ 92], 40.00th=[ 100], 50.00th=[ 106], 60.00th=[ 113], | 70.00th=[ 121], 80.00th=[ 131], 90.00th=[ 142], 95.00th=[ 155], | 99.00th=[ 178], 99.50th=[ 186], 99.90th=[ 199], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 191, max= 472, per=25.26%, avg=297.51, stdev=66.37, samples=35 iops : min= 47, max= 118, avg=73.94, stdev=16.60, samples=35 lat (msec) : 50=0.16%, 100=44.57%, 250=55.27% cpu : usr=0.34%, sys=2.61%, ctx=4716, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184018: Sun Nov 16 23:12:28 2025 read: IOPS=70, BW=281KiB/s (288kB/s)(4984KiB/17725msec) slat (usec): min=2123, max=28775, avg=7665.64, stdev=3232.04 clat (msec): min=19, max=202, avg=104.44, stdev=26.08 lat (msec): min=28, max=221, avg=112.11, stdev=27.22 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 83], | 30.00th=[ 89], 40.00th=[ 95], 50.00th=[ 102], 60.00th=[ 109], | 70.00th=[ 117], 80.00th=[ 128], 90.00th=[ 140], 95.00th=[ 153], | 99.00th=[ 176], 99.50th=[ 182], 99.90th=[ 197], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 150, max= 432, per=24.84%, avg=281.14, stdev=62.18, samples=35 iops : min= 37, max= 108, avg=69.86, stdev=15.64, samples=35 write: IOPS=74, BW=297KiB/s (304kB/s)(5256KiB/17725msec); 0 zone resets slat (usec): min=31, max=7362, avg=158.76, stdev=320.13 clat (msec): min=24, max=196, avg=107.75, stdev=24.84 lat (msec): min=25, max=197, avg=107.91, stdev=24.86 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 78], 20.00th=[ 88], | 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 105], 60.00th=[ 112], | 70.00th=[ 120], 80.00th=[ 129], 90.00th=[ 142], 95.00th=[ 155], | 99.00th=[ 171], 99.50th=[ 180], 99.90th=[ 190], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 136, max= 448, per=25.11%, avg=295.77, stdev=72.44, samples=35 iops : min= 34, max= 112, avg=73.49, stdev=18.07, samples=35 lat (msec) : 20=0.04%, 50=0.20%, 100=45.04%, 250=54.73% cpu : usr=0.34%, sys=2.43%, ctx=4821, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184019: Sun Nov 16 23:12:28 2025 read: IOPS=71, BW=285KiB/s (292kB/s)(5060KiB/17727msec) slat (usec): min=2105, max=25654, avg=7484.56, stdev=3252.59 clat (msec): min=46, max=230, avg=104.35, stdev=26.25 lat (msec): min=55, max=237, avg=111.85, stdev=27.41 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 81], | 30.00th=[ 89], 40.00th=[ 95], 50.00th=[ 102], 60.00th=[ 108], | 70.00th=[ 116], 80.00th=[ 127], 90.00th=[ 142], 95.00th=[ 153], | 99.00th=[ 174], 99.50th=[ 178], 99.90th=[ 228], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 171, max= 432, per=25.21%, avg=285.43, stdev=64.35, samples=35 iops : min= 42, max= 108, avg=70.91, stdev=16.13, samples=35 write: IOPS=73, BW=292KiB/s (299kB/s)(5180KiB/17727msec); 0 zone resets slat (usec): min=36, max=5289, avg=160.47, stdev=280.12 clat (msec): min=58, max=191, avg=108.14, stdev=24.99 lat (msec): min=59, max=191, avg=108.30, stdev=24.99 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 72], 10.00th=[ 78], 20.00th=[ 86], | 30.00th=[ 92], 40.00th=[ 100], 50.00th=[ 106], 60.00th=[ 112], | 70.00th=[ 121], 80.00th=[ 129], 90.00th=[ 142], 95.00th=[ 157], | 99.00th=[ 174], 99.50th=[ 178], 99.90th=[ 186], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 160, max= 464, per=24.80%, avg=292.20, stdev=66.21, samples=35 iops : min= 40, max= 116, avg=72.60, stdev=16.57, samples=35 lat (msec) : 50=0.04%, 100=44.10%, 250=55.86% cpu : usr=0.41%, sys=2.54%, ctx=4751, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1132KiB/s (1159kB/s), 281KiB/s-286KiB/s (287kB/s-292kB/s), io=19.6MiB (20.5MB), run=17679-17727msec WRITE: bw=1179KiB/s (1207kB/s), 292KiB/s-297KiB/s (299kB/s-304kB/s), io=20.4MiB (21.4MB), run=17679-17727msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184025: Sun Nov 16 23:12:33 2025 read: IOPS=3, BW=143MiB/s (150MB/s)(40.0MiB/279msec) slat (nsec): min=14311k, max=14311k, avg=14310929.00, stdev= 0.00 clat (nsec): min=249530k, max=249530k, avg=249529654.00, stdev= 0.00 lat (nsec): min=263846k, max=263846k, avg=263845670.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.68%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=40.0MiB (41.9MB), run=279-279msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout