== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:28:05 (1786206485) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats=0 llite.lustre-ffff8e351c292800.statahead_max=32 llite.lustre-ffff8e351c292800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.68 seconds: 59.52 ops/second llite.lustre-ffff8e351c292800.statahead_max=0 101 real 0m1.580s user 0m0.006s sys 0m0.313s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff8e351c292800.statahead_max=32 llite.lustre-ffff8e351c292800.statahead_max=32 101 real 0m0.556s user 0m0.003s sys 0m0.183s 'ls -l' 100 files with statahead: 0 sec statahead total: 72 statahead wrong: 2 agl total: 67 list_total: 29 fname_total: 42 hit_total: 13704 miss_total: 513 swrong=2 ewrong=2 total: 900 open/close in 6.56 seconds: 137.12 ops/second llite.lustre-ffff8e351c292800.statahead_max=0 1001 real 0m13.773s user 0m0.032s sys 0m2.942s 'ls -l' 1000 files without statahead: 14 sec llite.lustre-ffff8e351c292800.statahead_max=32 llite.lustre-ffff8e351c292800.statahead_max=32 1001 real 0m5.044s user 0m0.026s sys 0m1.588s 'ls -l' 1000 files with statahead: 5 sec statahead total: 73 statahead wrong: 2 agl total: 68 list_total: 30 fname_total: 42 hit_total: 14702 miss_total: 515 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffff8e351c292800.statahead_stats= statahead total: 73 statahead wrong: 2 agl total: 68 list_total: 30 fname_total: 42 hit_total: 14702 miss_total: 515 mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786206538.372340143 secs.nsecs start_time: 1786206488.398213116 secs.nsecs elapsed_time: 49.974127027 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 3 7 15 4: 1 2 18 8: 2 5 23 16: 25 65 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786206538.372408806 secs.nsecs start_time: 1786206488.398338054 secs.nsecs elapsed_time: 49.974070752 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 4 10 15 16: 11 27 42 32: 23 57 100 - open/close 1247 (time 1786206550.06 total 10.01 last 124.62) - open/close 2636 (time 1786206560.06 total 20.01 last 138.83) - open/close 4046 (time 1786206570.06 total 30.01 last 140.96) - open/close 5368 (time 1786206580.07 total 40.02 last 132.13) - open/close 6520 (time 1786206590.08 total 50.03 last 115.12) - open/close 7162 (time 1786206600.09 total 60.04 last 64.13) - open/close 7853 (time 1786206610.09 total 70.04 last 69.06) - open/close 8380 (time 1786206620.10 total 80.05 last 52.65) - open/close 8720 (time 1786206630.11 total 90.06 last 33.99) - open/close 9071 (time 1786206640.14 total 100.09 last 34.98) - open/close 9403 (time 1786206650.16 total 110.11 last 33.12) - open/close 9884 (time 1786206660.17 total 120.12 last 48.06) total: 10000 open/close in 121.90 seconds: 82.03 ops/second llite.lustre-ffff8e351c292800.statahead_batch_max=0 llite.lustre-ffff8e351c292800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e351c292800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e351c292800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.335s user 0m0.301s sys 0m15.959s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e351c292800.statahead_batch_max=32 llite.lustre-ffff8e351c292800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e351c292800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e351c292800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m47.355s user 0m0.299s sys 0m15.238s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786206862.840224842 secs.nsecs start_time: 1786206740.574412349 secs.nsecs elapsed_time: 122.265812493 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786206862.840300929 secs.nsecs start_time: 1786206740.574604597 secs.nsecs elapsed_time: 122.265696332 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats=0 llite.lustre-ffff8e351c292800.statahead_max=2048 llite.lustre-ffff8e351c292800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.82 seconds: 54.81 ops/second llite.lustre-ffff8e351c292800.statahead_max=0 101 real 0m2.085s user 0m0.011s sys 0m0.341s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8e351c292800.statahead_max=2048 llite.lustre-ffff8e351c292800.statahead_max=2048 101 real 0m0.785s user 0m0.004s sys 0m0.185s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 883 (time 1786207154.29 total 10.01 last 88.25) total: 900 open/close in 10.18 seconds: 88.42 ops/second llite.lustre-ffff8e351c292800.statahead_max=0 1001 real 0m19.152s user 0m0.035s sys 0m3.047s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8e351c292800.statahead_max=2048 llite.lustre-ffff8e351c292800.statahead_max=2048 1001 real 0m6.806s user 0m0.029s sys 0m1.542s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff8e351c292800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786207208.804976911 secs.nsecs start_time: 1786206865.481244280 secs.nsecs elapsed_time: 343.323732631 secs.nsecs subreqs per batch batches %% cum %% 1: 3 37 37 2: 2 25 62 4: 1 12 75 8: 0 0 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786207208.805041729 secs.nsecs start_time: 1786206865.485810909 secs.nsecs elapsed_time: 343.319230820 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 44 44 16: 1 11 55 32: 0 0 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 913 (time 1786207221.53 total 10.02 last 91.11) - open/close 1806 (time 1786207231.53 total 20.02 last 89.29) - open/close 2730 (time 1786207241.54 total 30.03 last 92.33) - open/close 3758 (time 1786207251.55 total 40.04 last 102.70) - open/close 4567 (time 1786207261.55 total 50.04 last 80.88) - open/close 5420 (time 1786207271.56 total 60.05 last 85.27) - open/close 6346 (time 1786207281.57 total 70.06 last 92.49) - open/close 7238 (time 1786207291.58 total 80.07 last 89.14) - open/close 8176 (time 1786207301.58 total 90.07 last 93.80) - open/close 9122 (time 1786207311.58 total 100.07 last 94.53) total: 10000 open/close in 109.09 seconds: 91.67 ops/second llite.lustre-ffff8e351c292800.statahead_batch_max=0 llite.lustre-ffff8e351c292800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e351c292800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e351c292800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.572s user 0m0.302s sys 0m17.262s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e351c292800.statahead_batch_max=256 llite.lustre-ffff8e351c292800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e351c292800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e351c292800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.282s user 0m0.296s sys 0m16.243s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786207473.079922319 secs.nsecs start_time: 1786207395.524695133 secs.nsecs elapsed_time: 77.555227186 secs.nsecs subreqs per batch batches %% cum %% 1: 212 53 53 2: 65 16 69 4: 31 7 77 8: 27 6 83 16: 11 2 86 32: 12 3 89 64: 7 1 91 128: 0 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff8e351c292800.batch_stats= snapshot_time: 1786207473.079997044 secs.nsecs start_time: 1786207395.524879885 secs.nsecs elapsed_time: 77.555117159 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8e351c292800.statahead_batch_max=64 llite.lustre-ffff8e351c292800.statahead_max=128