== sanity test 123ad: Verify batching statahead works correctly ========================================================== 11:40:02 (1787845202) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats=0 llite.lustre-ffff98d1057b0800.statahead_max=32 llite.lustre-ffff98d1057b0800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.51 seconds: 39.78 ops/second llite.lustre-ffff98d1057b0800.statahead_max=0 101 real 0m2.575s user 0m0.009s sys 0m0.342s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff98d1057b0800.statahead_max=32 llite.lustre-ffff98d1057b0800.statahead_max=32 101 real 0m0.837s user 0m0.004s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 141 statahead wrong: 2 agl total: 135 list_total: 35 fname_total: 96 hit_total: 4123 miss_total: 1352 swrong=2 ewrong=2 - open/close 799 (time 1787845228.81 total 10.00 last 79.89) total: 900 open/close in 11.34 seconds: 79.37 ops/second llite.lustre-ffff98d1057b0800.statahead_max=0 1001 real 0m23.291s user 0m0.037s sys 0m3.054s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff98d1057b0800.statahead_max=32 llite.lustre-ffff98d1057b0800.statahead_max=32 1001 real 0m7.285s user 0m0.028s sys 0m1.477s 'ls -l' 1000 files with statahead: 8 sec statahead total: 142 statahead wrong: 2 agl total: 136 list_total: 36 fname_total: 96 hit_total: 5119 miss_total: 1356 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff98d1057b0800.statahead_stats= statahead total: 142 statahead wrong: 2 agl total: 136 list_total: 36 fname_total: 96 hit_total: 5119 miss_total: 1356 mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787845292.058346803 secs.nsecs start_time: 1787845207.032683385 secs.nsecs elapsed_time: 85.025663418 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 3 8 10 4: 2 5 16 8: 2 5 21 16: 24 64 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787845292.058477795 secs.nsecs start_time: 1787845207.032899336 secs.nsecs elapsed_time: 85.025578459 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 3 7 12 16: 8 20 33 32: 26 66 100 - open/close 680 (time 1787845304.73 total 10.01 last 67.94) - open/close 1373 (time 1787845314.73 total 20.02 last 69.24) - open/close 2093 (time 1787845324.75 total 30.03 last 71.90) - open/close 2863 (time 1787845334.75 total 40.03 last 77.00) - open/close 3652 (time 1787845344.75 total 50.03 last 78.88) - open/close 4417 (time 1787845354.76 total 60.05 last 76.40) - open/close 5195 (time 1787845364.77 total 70.05 last 77.78) - open/close 5802 (time 1787845374.77 total 80.05 last 60.69) - open/close 6275 (time 1787845384.78 total 90.07 last 47.23) - open/close 6785 (time 1787845394.78 total 100.07 last 51.00) - open/close 7418 (time 1787845404.80 total 110.09 last 63.17) - open/close 8076 (time 1787845414.81 total 120.09 last 65.77) - open/close 8791 (time 1787845424.81 total 130.09 last 71.49) - open/close 9579 (time 1787845434.82 total 140.10 last 78.75) total: 10000 open/close in 145.80 seconds: 68.58 ops/second llite.lustre-ffff98d1057b0800.statahead_batch_max=0 llite.lustre-ffff98d1057b0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98d1057b0800.stats=clear mdc.lustre-MDT0001-mdc-ffff98d1057b0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.988s user 0m0.338s sys 0m18.689s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98d1057b0800.statahead_batch_max=32 llite.lustre-ffff98d1057b0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98d1057b0800.stats=clear mdc.lustre-MDT0001-mdc-ffff98d1057b0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m34.977s user 0m0.323s sys 0m14.801s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787845608.496137330 secs.nsecs start_time: 1787845504.867089732 secs.nsecs elapsed_time: 103.629047598 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787845608.496476357 secs.nsecs start_time: 1787845504.867340443 secs.nsecs elapsed_time: 103.629135914 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats=0 llite.lustre-ffff98d1057b0800.statahead_max=2048 llite.lustre-ffff98d1057b0800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.93 seconds: 34.12 ops/second llite.lustre-ffff98d1057b0800.statahead_max=0 101 real 0m2.631s user 0m0.006s sys 0m0.315s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff98d1057b0800.statahead_max=2048 llite.lustre-ffff98d1057b0800.statahead_max=2048 101 real 0m1.205s user 0m0.004s sys 0m0.188s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 618 (time 1787845932.10 total 10.01 last 61.75) total: 900 open/close in 14.68 seconds: 61.30 ops/second llite.lustre-ffff98d1057b0800.statahead_max=0 1001 real 0m23.352s user 0m0.039s sys 0m3.140s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff98d1057b0800.statahead_max=2048 llite.lustre-ffff98d1057b0800.statahead_max=2048 1001 real 0m7.712s user 0m0.038s sys 0m1.464s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff98d1057b0800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787846001.510869003 secs.nsecs start_time: 1787845610.991683346 secs.nsecs elapsed_time: 390.519185657 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 1 16 33 4: 2 33 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787846001.510948408 secs.nsecs start_time: 1787845610.991858952 secs.nsecs elapsed_time: 390.519089456 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 3 37 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 711 (time 1787846014.12 total 10.01 last 71.05) - open/close 1425 (time 1787846024.12 total 20.01 last 71.39) - open/close 2119 (time 1787846034.13 total 30.01 last 69.36) - open/close 2869 (time 1787846044.13 total 40.02 last 74.97) - open/close 3532 (time 1787846054.14 total 50.02 last 66.28) - open/close 4280 (time 1787846064.15 total 60.03 last 74.71) - open/close 5020 (time 1787846074.16 total 70.04 last 73.95) - open/close 5767 (time 1787846084.16 total 80.05 last 74.66) - open/close 6490 (time 1787846094.17 total 90.05 last 72.26) - open/close 7243 (time 1787846104.17 total 100.06 last 75.25) - open/close 7858 (time 1787846114.20 total 110.08 last 61.35) - open/close 8467 (time 1787846124.20 total 120.09 last 60.86) - open/close 9047 (time 1787846134.21 total 130.09 last 57.96) - open/close 9733 (time 1787846144.21 total 140.10 last 68.59) total: 10000 open/close in 143.81 seconds: 69.53 ops/second llite.lustre-ffff98d1057b0800.statahead_batch_max=0 llite.lustre-ffff98d1057b0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98d1057b0800.stats=clear mdc.lustre-MDT0001-mdc-ffff98d1057b0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.747s user 0m0.302s sys 0m15.450s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98d1057b0800.statahead_batch_max=256 llite.lustre-ffff98d1057b0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98d1057b0800.stats=clear mdc.lustre-MDT0001-mdc-ffff98d1057b0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.864s user 0m0.307s sys 0m15.580s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787846295.087255554 secs.nsecs start_time: 1787846226.529074040 secs.nsecs elapsed_time: 68.558181514 secs.nsecs subreqs per batch batches %% cum %% 1: 109 45 45 2: 29 12 57 4: 19 7 65 8: 16 6 72 16: 10 4 76 32: 15 6 83 64: 4 1 84 128: 1 0 85 256: 35 14 100 mdc.lustre-MDT0001-mdc-ffff98d1057b0800.batch_stats= snapshot_time: 1787846295.093944279 secs.nsecs start_time: 1787846226.529224308 secs.nsecs elapsed_time: 68.564719971 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff98d1057b0800.statahead_batch_max=64 llite.lustre-ffff98d1057b0800.statahead_max=128