== sanity test 123ad: Verify batching statahead works correctly ========================================================== 21:04:21 (1786755861) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats=0 llite.lustre-ffff974911d0b000.statahead_max=32 llite.lustre-ffff974911d0b000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.87 seconds: 34.78 ops/second llite.lustre-ffff974911d0b000.statahead_max=0 101 real 0m2.425s user 0m0.002s sys 0m0.327s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff974911d0b000.statahead_max=32 llite.lustre-ffff974911d0b000.statahead_max=32 101 real 0m1.130s user 0m0.003s sys 0m0.197s 'ls -l' 100 files with statahead: 1 sec statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 31 fname_total: 115 hit_total: 3976 miss_total: 1338 swrong=2 ewrong=2 - open/close 833 (time 1786755889.84 total 10.00 last 83.27) total: 900 open/close in 10.93 seconds: 82.33 ops/second llite.lustre-ffff974911d0b000.statahead_max=0 1001 real 0m22.633s user 0m0.040s sys 0m3.263s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff974911d0b000.statahead_max=32 llite.lustre-ffff974911d0b000.statahead_max=32 1001 real 0m9.404s user 0m0.036s sys 0m1.678s 'ls -l' 1000 files with statahead: 10 sec statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 32 fname_total: 115 hit_total: 4974 miss_total: 1340 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff974911d0b000.statahead_stats= statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 32 fname_total: 115 hit_total: 4974 miss_total: 1340 mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786755956.226668594 secs.nsecs start_time: 1786755866.622334831 secs.nsecs elapsed_time: 89.604333763 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 1 2 10 4: 2 5 16 8: 0 0 16 16: 26 70 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786755956.226731952 secs.nsecs start_time: 1786755866.622746146 secs.nsecs elapsed_time: 89.603985806 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 2 5 7 16: 9 23 31 32: 26 68 100 - open/close 775 (time 1786755968.66 total 10.00 last 77.48) - open/close 1575 (time 1786755978.67 total 20.01 last 79.91) - open/close 2332 (time 1786755988.68 total 30.02 last 75.62) - open/close 3130 (time 1786755998.68 total 40.03 last 79.76) - open/close 3951 (time 1786756008.68 total 50.03 last 82.09) - open/close 4702 (time 1786756018.70 total 60.05 last 74.99) - open/close 5478 (time 1786756028.70 total 70.05 last 77.60) - open/close 6276 (time 1786756038.71 total 80.05 last 79.74) - open/close 7127 (time 1786756048.71 total 90.06 last 85.09) - open/close 7954 (time 1786756058.71 total 100.06 last 82.69) - open/close 8734 (time 1786756068.72 total 110.06 last 77.97) - open/close 9544 (time 1786756078.72 total 120.07 last 80.96) total: 10000 open/close in 125.63 seconds: 79.60 ops/second llite.lustre-ffff974911d0b000.statahead_batch_max=0 llite.lustre-ffff974911d0b000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff974911d0b000.stats=clear mdc.lustre-MDT0001-mdc-ffff974911d0b000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.505s user 0m0.317s sys 0m18.930s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff974911d0b000.statahead_batch_max=32 llite.lustre-ffff974911d0b000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff974911d0b000.stats=clear mdc.lustre-MDT0001-mdc-ffff974911d0b000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.375s user 0m0.322s sys 0m16.761s batched statahead RPC= hit count=9988 mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786756277.536678781 secs.nsecs start_time: 1786756163.650829106 secs.nsecs elapsed_time: 113.885849675 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786756277.536753966 secs.nsecs start_time: 1786756163.650921199 secs.nsecs elapsed_time: 113.885832767 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats=0 llite.lustre-ffff974911d0b000.statahead_max=2048 llite.lustre-ffff974911d0b000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.61 seconds: 38.31 ops/second llite.lustre-ffff974911d0b000.statahead_max=0 101 real 0m2.902s user 0m0.003s sys 0m0.346s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff974911d0b000.statahead_max=2048 llite.lustre-ffff974911d0b000.statahead_max=2048 101 real 0m0.717s user 0m0.007s sys 0m0.186s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10087 miss_total: 13 swrong=0 ewrong=0 - open/close 713 (time 1786756606.64 total 10.01 last 71.24) total: 900 open/close in 12.77 seconds: 70.49 ops/second llite.lustre-ffff974911d0b000.statahead_max=0 1001 real 0m26.604s user 0m0.033s sys 0m3.224s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff974911d0b000.statahead_max=2048 llite.lustre-ffff974911d0b000.statahead_max=2048 1001 real 0m7.548s user 0m0.024s sys 0m1.621s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11086 miss_total: 14 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff974911d0b000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11086 miss_total: 14 mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786756679.354878440 secs.nsecs start_time: 1786756280.005901350 secs.nsecs elapsed_time: 399.348977090 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 4 50 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786756679.356631691 secs.nsecs start_time: 1786756280.006070687 secs.nsecs elapsed_time: 399.350561004 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 2 25 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 733 (time 1786756692.06 total 10.00 last 73.29) - open/close 1375 (time 1786756702.06 total 20.00 last 64.19) - open/close 2142 (time 1786756712.06 total 30.01 last 76.67) - open/close 2878 (time 1786756722.07 total 40.02 last 73.53) - open/close 3632 (time 1786756732.08 total 50.02 last 75.35) - open/close 4406 (time 1786756742.08 total 60.03 last 77.36) - open/close 5184 (time 1786756752.09 total 70.03 last 77.74) - open/close 5938 (time 1786756762.10 total 80.04 last 75.35) - open/close 6702 (time 1786756772.11 total 90.05 last 76.30) - open/close 7429 (time 1786756782.11 total 100.05 last 72.70) - open/close 8187 (time 1786756792.12 total 110.07 last 75.71) - open/close 8944 (time 1786756802.12 total 120.07 last 75.67) - open/close 9677 (time 1786756812.14 total 130.08 last 73.20) total: 10000 open/close in 135.12 seconds: 74.01 ops/second llite.lustre-ffff974911d0b000.statahead_batch_max=0 llite.lustre-ffff974911d0b000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff974911d0b000.stats=clear mdc.lustre-MDT0001-mdc-ffff974911d0b000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.220s user 0m0.346s sys 0m17.841s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff974911d0b000.statahead_batch_max=256 llite.lustre-ffff974911d0b000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff974911d0b000.stats=clear mdc.lustre-MDT0001-mdc-ffff974911d0b000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.328s user 0m0.332s sys 0m18.597s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786756953.211298288 secs.nsecs start_time: 1786756884.777551924 secs.nsecs elapsed_time: 68.433746364 secs.nsecs subreqs per batch batches %% cum %% 1: 254 57 57 2: 51 11 68 4: 55 12 81 8: 16 3 84 16: 10 2 86 32: 20 4 91 64: 3 0 92 128: 1 0 92 256: 34 7 100 mdc.lustre-MDT0001-mdc-ffff974911d0b000.batch_stats= snapshot_time: 1786756953.211353490 secs.nsecs start_time: 1786756884.777730796 secs.nsecs elapsed_time: 68.433622694 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff974911d0b000.statahead_batch_max=64 llite.lustre-ffff974911d0b000.statahead_max=128