== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:06:54 (1786637214) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats=0 llite.lustre-ffff9db6085d0800.statahead_max=32 llite.lustre-ffff9db6085d0800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.73 seconds: 57.75 ops/second llite.lustre-ffff9db6085d0800.statahead_max=0 101 real 0m1.685s user 0m0.004s sys 0m0.335s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9db6085d0800.statahead_max=32 llite.lustre-ffff9db6085d0800.statahead_max=32 101 real 0m0.941s user 0m0.005s sys 0m0.173s 'ls -l' 100 files with statahead: 1 sec statahead total: 142 statahead wrong: 2 agl total: 136 list_total: 31 fname_total: 108 hit_total: 4084 miss_total: 1575 swrong=2 ewrong=2 total: 900 open/close in 9.07 seconds: 99.24 ops/second llite.lustre-ffff9db6085d0800.statahead_max=0 1001 real 0m18.197s user 0m0.028s sys 0m3.055s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff9db6085d0800.statahead_max=32 llite.lustre-ffff9db6085d0800.statahead_max=32 1001 real 0m9.770s user 0m0.031s sys 0m1.547s 'ls -l' 1000 files with statahead: 10 sec statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 108 hit_total: 5082 miss_total: 1577 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff9db6085d0800.statahead_stats= statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 108 hit_total: 5082 miss_total: 1577 mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786637292.450284551 secs.nsecs start_time: 1786637217.896298719 secs.nsecs elapsed_time: 74.553985832 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 2 5 15 16: 17 44 60 32: 15 39 100 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786637292.450653550 secs.nsecs start_time: 1786637217.896413368 secs.nsecs elapsed_time: 74.554240182 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 4 10 15 16: 26 65 80 32: 8 20 100 - open/close 871 (time 1786637304.75 total 10.02 last 86.90) - open/close 1733 (time 1786637314.76 total 20.03 last 86.17) - open/close 2739 (time 1786637324.76 total 30.04 last 100.51) - open/close 3805 (time 1786637334.78 total 40.05 last 106.45) - open/close 4897 (time 1786637344.79 total 50.06 last 109.12) - open/close 5921 (time 1786637354.79 total 60.06 last 102.35) - open/close 6699 (time 1786637364.79 total 70.06 last 77.80) - open/close 7761 (time 1786637374.80 total 80.07 last 106.09) - open/close 8726 (time 1786637384.80 total 90.08 last 96.48) - open/close 9758 (time 1786637394.81 total 100.08 last 103.19) total: 10000 open/close in 102.13 seconds: 97.91 ops/second llite.lustre-ffff9db6085d0800.statahead_batch_max=0 llite.lustre-ffff9db6085d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db6085d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db6085d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.691s user 0m0.310s sys 0m15.371s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9db6085d0800.statahead_batch_max=32 llite.lustre-ffff9db6085d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9db6085d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db6085d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m38.679s user 0m0.303s sys 0m14.636s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786637575.020250436 secs.nsecs start_time: 1786637460.405133411 secs.nsecs elapsed_time: 114.615117025 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786637575.020482054 secs.nsecs start_time: 1786637460.405370693 secs.nsecs elapsed_time: 114.615111361 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats=0 llite.lustre-ffff9db6085d0800.statahead_max=2048 llite.lustre-ffff9db6085d0800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 1.95 seconds: 51.15 ops/second llite.lustre-ffff9db6085d0800.statahead_max=0 101 real 0m1.564s user 0m0.005s sys 0m0.335s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9db6085d0800.statahead_max=2048 llite.lustre-ffff9db6085d0800.statahead_max=2048 101 real 0m0.543s user 0m0.003s sys 0m0.162s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 total: 900 open/close in 7.91 seconds: 113.74 ops/second llite.lustre-ffff9db6085d0800.statahead_max=0 1001 real 0m16.981s user 0m0.032s sys 0m3.025s 'ls -l' 1000 files without statahead: 17 sec llite.lustre-ffff9db6085d0800.statahead_max=2048 llite.lustre-ffff9db6085d0800.statahead_max=2048 1001 real 0m6.124s user 0m0.020s sys 0m1.544s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 14 seconds llite.lustre-ffff9db6085d0800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786637868.588583689 secs.nsecs start_time: 1786637577.259063070 secs.nsecs elapsed_time: 291.329520619 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 1 10 20 8: 3 30 50 16: 3 30 80 32: 0 0 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786637868.588662994 secs.nsecs start_time: 1786637577.259450813 secs.nsecs elapsed_time: 291.329212181 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 3 30 50 16: 2 20 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 1060 (time 1786637880.68 total 10.00 last 105.96) - open/close 2121 (time 1786637890.70 total 20.02 last 105.90) - open/close 3135 (time 1786637900.70 total 30.02 last 101.38) - open/close 4125 (time 1786637910.71 total 40.03 last 98.98) - open/close 5181 (time 1786637920.71 total 50.03 last 105.58) - open/close 6217 (time 1786637930.71 total 60.03 last 103.55) - open/close 7225 (time 1786637940.73 total 70.05 last 100.66) - open/close 8325 (time 1786637950.73 total 80.05 last 109.99) - open/close 9412 (time 1786637960.73 total 90.06 last 108.62) total: 10000 open/close in 95.55 seconds: 104.66 ops/second llite.lustre-ffff9db6085d0800.statahead_batch_max=0 llite.lustre-ffff9db6085d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db6085d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db6085d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.653s user 0m0.305s sys 0m16.403s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9db6085d0800.statahead_batch_max=256 llite.lustre-ffff9db6085d0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9db6085d0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db6085d0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.829s user 0m0.287s sys 0m16.620s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786638113.858894807 secs.nsecs start_time: 1786638044.162685151 secs.nsecs elapsed_time: 69.696209656 secs.nsecs subreqs per batch batches %% cum %% 1: 165 49 49 2: 41 12 61 4: 41 12 73 8: 17 5 78 16: 16 4 83 32: 16 4 88 64: 5 1 89 128: 0 0 89 256: 35 10 100 mdc.lustre-MDT0001-mdc-ffff9db6085d0800.batch_stats= snapshot_time: 1786638113.859318768 secs.nsecs start_time: 1786638044.162848932 secs.nsecs elapsed_time: 69.696469836 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9db6085d0800.statahead_batch_max=64 llite.lustre-ffff9db6085d0800.statahead_max=128