== sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:56:35 (1785459395) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats=0 llite.lustre-ffff9dcdd2208000.statahead_max=32 llite.lustre-ffff9dcdd2208000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats=0 total: 100 open/close in 3.13 seconds: 31.96 ops/second llite.lustre-ffff9dcdd2208000.statahead_max=0 101 real 0m2.333s user 0m0.001s sys 0m0.309s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9dcdd2208000.statahead_max=32 llite.lustre-ffff9dcdd2208000.statahead_max=32 101 real 0m1.587s user 0m0.005s sys 0m0.161s 'ls -l' 100 files with statahead: 1 sec statahead total: 167 statahead wrong: 2 agl total: 160 list_total: 30 fname_total: 114 hit_total: 3843 miss_total: 1654 swrong=2 ewrong=2 - open/close 490 (time 1785459424.41 total 10.00 last 48.99) total: 900 open/close in 15.49 seconds: 58.12 ops/second llite.lustre-ffff9dcdd2208000.statahead_max=0 1001 real 0m30.343s user 0m0.042s sys 0m2.903s 'ls -l' 1000 files without statahead: 31 sec llite.lustre-ffff9dcdd2208000.statahead_max=32 llite.lustre-ffff9dcdd2208000.statahead_max=32 1001 real 0m17.046s user 0m0.023s sys 0m1.534s 'ls -l' 1000 files with statahead: 17 sec statahead total: 168 statahead wrong: 2 agl total: 161 list_total: 31 fname_total: 114 hit_total: 4841 miss_total: 1656 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 36 seconds llite.lustre-ffff9dcdd2208000.statahead_stats= statahead total: 168 statahead wrong: 2 agl total: 161 list_total: 31 fname_total: 114 hit_total: 4841 miss_total: 1656 mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats= snapshot_time: 1785459534.118122109 secs.nsecs start_time: 1785459400.112949705 secs.nsecs elapsed_time: 134.005172404 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 7 7 16: 2 5 12 32: 34 87 100 - open/close 549 (time 1785459547.22 total 10.03 last 54.75) - open/close 1129 (time 1785459557.23 total 20.04 last 57.94) - open/close 1776 (time 1785459567.25 total 30.06 last 64.58) - open/close 2358 (time 1785459577.26 total 40.06 last 58.15) - open/close 2972 (time 1785459587.27 total 50.08 last 61.32) - open/close 3498 (time 1785459597.28 total 60.09 last 52.54) - open/close 4070 (time 1785459607.29 total 70.10 last 57.14) - open/close 4617 (time 1785459617.30 total 80.11 last 54.67) - open/close 5231 (time 1785459627.31 total 90.12 last 61.33) - open/close 5830 (time 1785459637.31 total 100.12 last 59.87) - open/close 6398 (time 1785459647.32 total 110.13 last 56.76) - open/close 7041 (time 1785459657.34 total 120.15 last 64.16) - open/close 7668 (time 1785459667.35 total 130.16 last 62.63) - open/close 8216 (time 1785459677.37 total 140.18 last 54.69) - open/close 8655 (time 1785459687.39 total 150.19 last 43.84) - open/close 9158 (time 1785459697.39 total 160.20 last 50.26) - open/close 9611 (time 1785459707.40 total 170.21 last 45.26) total: 10000 open/close in 177.65 seconds: 56.29 ops/second llite.lustre-ffff9dcdd2208000.statahead_batch_max=0 llite.lustre-ffff9dcdd2208000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m33.043s user 0m0.243s sys 0m14.651s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9dcdd2208000.statahead_batch_max=32 llite.lustre-ffff9dcdd2208000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m26.878s user 0m0.254s sys 0m13.454s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats= snapshot_time: 1785459990.606084270 secs.nsecs start_time: 1785459820.268365392 secs.nsecs elapsed_time: 170.337718878 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats=0 llite.lustre-ffff9dcdd2208000.statahead_max=2048 llite.lustre-ffff9dcdd2208000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats=0 total: 100 open/close in 3.62 seconds: 27.60 ops/second llite.lustre-ffff9dcdd2208000.statahead_max=0 101 real 0m3.369s user 0m0.003s sys 0m0.304s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff9dcdd2208000.statahead_max=2048 llite.lustre-ffff9dcdd2208000.statahead_max=2048 101 real 0m1.287s user 0m0.002s sys 0m0.139s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 610 (time 1785460390.21 total 10.00 last 60.98) total: 900 open/close in 16.99 seconds: 52.96 ops/second llite.lustre-ffff9dcdd2208000.statahead_max=0 1001 real 0m37.890s user 0m0.034s sys 0m3.071s 'ls -l' 1000 files without statahead: 38 sec llite.lustre-ffff9dcdd2208000.statahead_max=2048 llite.lustre-ffff9dcdd2208000.statahead_max=2048 1001 real 0m15.588s user 0m0.023s sys 0m1.317s 'ls -l' 1000 files with statahead: 15 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 26 seconds llite.lustre-ffff9dcdd2208000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats= snapshot_time: 1785460499.387911597 secs.nsecs start_time: 1785459993.921425162 secs.nsecs elapsed_time: 505.466486435 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 16 16 16: 2 16 33 32: 3 25 58 64: 1 8 66 128: 0 0 66 256: 4 33 100 - open/close 605 (time 1785460512.87 total 10.01 last 60.47) - open/close 1280 (time 1785460522.88 total 20.01 last 67.44) - open/close 1897 (time 1785460532.90 total 30.03 last 61.62) - open/close 2424 (time 1785460542.91 total 40.05 last 52.60) - open/close 2941 (time 1785460552.92 total 50.05 last 51.68) - open/close 3504 (time 1785460562.92 total 60.05 last 56.29) - open/close 4290 (time 1785460572.93 total 70.06 last 78.55) - open/close 4853 (time 1785460582.94 total 80.07 last 56.24) - open/close 5421 (time 1785460592.95 total 90.09 last 56.70) - open/close 5933 (time 1785460602.98 total 100.11 last 51.08) - open/close 6441 (time 1785460612.99 total 110.12 last 50.72) - open/close 7011 (time 1785460623.00 total 120.14 last 56.93) - open/close 7572 (time 1785460633.02 total 130.16 last 55.99) - open/close 8201 (time 1785460643.04 total 140.17 last 62.82) - open/close 8809 (time 1785460653.05 total 150.18 last 60.72) - open/close 9362 (time 1785460663.05 total 160.18 last 55.30) - open/close 9881 (time 1785460673.09 total 170.22 last 51.73) total: 10000 open/close in 172.77 seconds: 57.88 ops/second llite.lustre-ffff9dcdd2208000.statahead_batch_max=0 llite.lustre-ffff9dcdd2208000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m15.722s user 0m0.254s sys 0m13.226s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9dcdd2208000.statahead_batch_max=256 llite.lustre-ffff9dcdd2208000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.738s user 0m0.235s sys 0m13.470s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9dcdd2208000.batch_stats= snapshot_time: 1785460855.470578021 secs.nsecs start_time: 1785460766.335939887 secs.nsecs elapsed_time: 89.134638134 secs.nsecs subreqs per batch batches %% cum %% 1: 139 40 40 2: 70 20 60 4: 40 11 72 8: 20 5 78 16: 21 6 84 32: 16 4 89 64: 3 0 90 128: 0 0 90 256: 34 9 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9dcdd2208000.statahead_batch_max=64 llite.lustre-ffff9dcdd2208000.statahead_max=128