== sanity test 398c: run fio to test AIO ================= 10:34:01 (1749479641) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0773248 s, 542 MB/s osc.lustre-OST0000-osc-ffff8800b3e6b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=19327: Mon Jun 9 10:34:32 2025 write: IOPS=85, BW=342KiB/s (351kB/s)(10.0MiB/29907msec) slat (usec): min=13, max=393, avg=82.85, stdev=50.82 clat (msec): min=56, max=331, avg=186.58, stdev=35.99 lat (msec): min=56, max=332, avg=186.66, stdev=35.99 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 132], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 334] bw ( KiB/s): min= 208, max= 456, per=24.94%, avg=341.17, stdev=45.76, samples=59 iops : min= 52, max= 114, avg=85.20, stdev=11.43, samples=59 lat (msec) : 100=0.23%, 250=94.96%, 500=4.80% cpu : usr=0.16%, sys=0.78%, ctx=2466, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19328: Mon Jun 9 10:34:32 2025 write: IOPS=85, BW=343KiB/s (352kB/s)(10.0MiB/29811msec) slat (usec): min=13, max=989, avg=82.28, stdev=55.87 clat (msec): min=33, max=336, avg=186.03, stdev=37.50 lat (msec): min=33, max=336, avg=186.11, stdev=37.50 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 251], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 338] bw ( KiB/s): min= 200, max= 488, per=25.07%, avg=342.93, stdev=50.00, samples=59 iops : min= 50, max= 122, avg=85.64, stdev=12.50, samples=59 lat (msec) : 50=0.12%, 100=1.09%, 250=93.87%, 500=4.92% cpu : usr=0.16%, sys=0.75%, ctx=2385, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19329: Mon Jun 9 10:34:32 2025 write: IOPS=85, BW=342KiB/s (350kB/s)(10.0MiB/29925msec) slat (usec): min=14, max=391, avg=79.76, stdev=48.92 clat (msec): min=56, max=335, avg=186.70, stdev=35.56 lat (msec): min=56, max=336, avg=186.79, stdev=35.56 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 133], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 251], | 99.00th=[ 284], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 338] bw ( KiB/s): min= 224, max= 456, per=24.89%, avg=340.49, stdev=44.63, samples=59 iops : min= 56, max= 114, avg=85.03, stdev=11.15, samples=59 lat (msec) : 100=0.16%, 250=94.73%, 500=5.12% cpu : usr=0.18%, sys=0.73%, ctx=2380, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19330: Mon Jun 9 10:34:32 2025 write: IOPS=85, BW=342KiB/s (350kB/s)(10.0MiB/29934msec) slat (usec): min=13, max=536, avg=81.40, stdev=53.57 clat (msec): min=56, max=332, avg=186.82, stdev=35.96 lat (msec): min=56, max=332, avg=186.90, stdev=35.96 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 136], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 232, max= 455, per=24.90%, avg=340.66, stdev=44.90, samples=59 iops : min= 58, max= 113, avg=85.10, stdev=11.22, samples=59 lat (msec) : 100=0.27%, 250=94.61%, 500=5.12% cpu : usr=0.15%, sys=0.74%, ctx=2433, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1368KiB/s (1401kB/s), 342KiB/s-343KiB/s (350kB/s-352kB/s), io=40.0MiB (41.9MB), run=29811-29934msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19364: Mon Jun 9 10:34:51 2025 read: IOPS=70, BW=280KiB/s (287kB/s)(5048KiB/18001msec) slat (usec): min=760, max=3451, avg=1295.09, stdev=299.67 clat (usec): min=1266, max=153572, avg=71123.31, stdev=12956.41 lat (msec): min=2, max=154, avg=72.42, stdev=12.96 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 74], | 70.00th=[ 79], 80.00th=[ 82], 90.00th=[ 87], 95.00th=[ 90], | 99.00th=[ 100], 99.50th=[ 102], 99.90th=[ 140], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 192, max= 400, per=25.17%, avg=280.37, stdev=50.47, samples=35 iops : min= 48, max= 100, avg=70.03, stdev=12.64, samples=35 write: IOPS=72, BW=288KiB/s (295kB/s)(5192KiB/18001msec) slat (usec): min=14, max=259, avg=65.06, stdev=36.26 clat (msec): min=51, max=239, avg=151.09, stdev=31.17 lat (msec): min=51, max=239, avg=151.15, stdev=31.17 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 103], 10.00th=[ 114], 20.00th=[ 126], | 30.00th=[ 133], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 192], 95.00th=[ 203], | 99.00th=[ 230], 99.50th=[ 234], 99.90th=[ 239], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 200, max= 368, per=24.80%, avg=287.46, stdev=35.95, samples=35 iops : min= 50, max= 92, avg=71.80, stdev= 9.01, samples=35 lat (msec) : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.23%, 50=1.33% lat (msec) : 100=49.26%, 250=49.02% cpu : usr=0.21%, sys=1.59%, ctx=3263, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19365: Mon Jun 9 10:34:51 2025 read: IOPS=69, BW=276KiB/s (283kB/s)(4976KiB/18011msec) slat (usec): min=718, max=3847, avg=1294.47, stdev=302.05 clat (msec): min=2, max=104, avg=71.03, stdev=12.62 lat (msec): min=3, max=105, avg=72.33, stdev=12.63 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 74], | 70.00th=[ 79], 80.00th=[ 82], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 99], 99.50th=[ 100], 99.90th=[ 103], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 144, max= 400, per=24.67%, avg=274.81, stdev=57.03, samples=36 iops : min= 36, max= 100, avg=68.64, stdev=14.24, samples=36 write: IOPS=73, BW=292KiB/s (299kB/s)(5264KiB/18011msec) slat (usec): min=14, max=705, avg=67.97, stdev=42.34 clat (msec): min=22, max=253, avg=150.35, stdev=33.19 lat (msec): min=23, max=253, avg=150.42, stdev=33.20 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 99], 10.00th=[ 111], 20.00th=[ 125], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 194], 95.00th=[ 205], | 99.00th=[ 228], 99.50th=[ 236], 99.90th=[ 245], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 240, max= 376, per=25.07%, avg=290.58, stdev=30.79, samples=36 iops : min= 60, max= 94, avg=72.58, stdev= 7.75, samples=36 lat (msec) : 4=0.04%, 10=0.12%, 20=0.16%, 50=1.64%, 100=49.38% lat (msec) : 250=48.63%, 500=0.04% cpu : usr=0.24%, sys=1.57%, ctx=3153, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19366: Mon Jun 9 10:34:51 2025 read: IOPS=69, BW=278KiB/s (284kB/s)(4984KiB/17957msec) slat (usec): min=13, max=3130, avg=1279.81, stdev=306.27 clat (usec): min=1358, max=153933, avg=70819.25, stdev=13704.41 lat (usec): min=1601, max=155491, avg=72101.18, stdev=13730.66 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 79], 80.00th=[ 82], 90.00th=[ 87], 95.00th=[ 92], | 99.00th=[ 100], 99.50th=[ 101], 99.90th=[ 110], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 184, max= 408, per=25.07%, avg=279.29, stdev=56.09, samples=35 iops : min= 46, max= 102, avg=69.80, stdev=14.03, samples=35 write: IOPS=73, BW=293KiB/s (300kB/s)(5256KiB/17957msec) slat (usec): min=12, max=333, avg=65.78, stdev=37.63 clat (msec): min=17, max=255, avg=150.07, stdev=32.43 lat (msec): min=17, max=255, avg=150.14, stdev=32.43 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 165], 80.00th=[ 178], 90.00th=[ 192], 95.00th=[ 205], | 99.00th=[ 222], 99.50th=[ 230], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 216, max= 368, per=25.12%, avg=291.17, stdev=37.60, samples=35 iops : min= 54, max= 92, avg=72.77, stdev= 9.41, samples=35 lat (msec) : 2=0.16%, 4=0.16%, 10=0.04%, 20=0.16%, 50=1.91% lat (msec) : 100=48.75%, 250=48.75%, 500=0.08% cpu : usr=0.28%, sys=1.54%, ctx=3082, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19367: Mon Jun 9 10:34:51 2025 read: IOPS=70, BW=281KiB/s (288kB/s)(5060KiB/18001msec) slat (usec): min=744, max=2844, avg=1289.58, stdev=283.90 clat (usec): min=1746, max=115906, avg=71106.91, stdev=13122.85 lat (msec): min=3, max=117, avg=72.40, stdev=13.13 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 82], 90.00th=[ 87], 95.00th=[ 91], | 99.00th=[ 99], 99.50th=[ 105], 99.90th=[ 114], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 168, max= 392, per=25.40%, avg=282.91, stdev=56.43, samples=35 iops : min= 42, max= 98, avg=70.69, stdev=14.07, samples=35 write: IOPS=71, BW=288KiB/s (295kB/s)(5180KiB/18001msec) slat (usec): min=14, max=1508, avg=68.46, stdev=57.05 clat (msec): min=51, max=260, avg=151.27, stdev=31.97 lat (msec): min=51, max=260, avg=151.34, stdev=31.97 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 178], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 236], 99.50th=[ 239], 99.90th=[ 259], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 224, max= 376, per=24.75%, avg=286.80, stdev=32.87, samples=35 iops : min= 56, max= 94, avg=71.66, stdev= 8.26, samples=35 lat (msec) : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.31%, 50=1.29% lat (msec) : 100=49.65%, 250=48.48%, 500=0.12% cpu : usr=0.19%, sys=1.67%, ctx=3246, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1114KiB/s (1141kB/s), 276KiB/s-281KiB/s (283kB/s-288kB/s), io=19.6MiB (20.5MB), run=17957-18011msec WRITE: bw=1160KiB/s (1188kB/s), 288KiB/s-293KiB/s (295kB/s-300kB/s), io=20.4MiB (21.4MB), run=17957-18011msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=19385: Mon Jun 9 10:34:52 2025 read: IOPS=10, BW=404MiB/s (424MB/s)(40.0MiB/99msec) slat (nsec): min=8296.9k, max=8296.9k, avg=8296902.00, stdev= 0.00 clat (nsec): min=86861k, max=86861k, avg=86861005.00, stdev= 0.00 lat (nsec): min=95175k, max=95175k, avg=95174977.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[86508], 5.00th=[86508], 10.00th=[86508], 20.00th=[86508], | 30.00th=[86508], 40.00th=[86508], 50.00th=[86508], 60.00th=[86508], | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.16%, ctx=17, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=40.0MiB (41.9MB), run=99-99msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout