== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:29:14 (1786404554) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats=0 llite.lustre-ffff97cc111ff000.statahead_max=32 llite.lustre-ffff97cc111ff000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.36 seconds: 42.39 ops/second llite.lustre-ffff97cc111ff000.statahead_max=0 101 real 0m1.755s user 0m0.005s sys 0m0.305s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff97cc111ff000.statahead_max=32 llite.lustre-ffff97cc111ff000.statahead_max=32 101 real 0m1.083s user 0m0.005s sys 0m0.171s 'ls -l' 100 files with statahead: 1 sec statahead total: 159 statahead wrong: 2 agl total: 152 list_total: 31 fname_total: 122 hit_total: 3867 miss_total: 1602 swrong=2 ewrong=2 total: 900 open/close in 9.06 seconds: 99.39 ops/second llite.lustre-ffff97cc111ff000.statahead_max=0 1001 real 0m18.359s user 0m0.025s sys 0m3.008s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff97cc111ff000.statahead_max=32 llite.lustre-ffff97cc111ff000.statahead_max=32 1001 real 0m11.202s user 0m0.026s sys 0m1.408s 'ls -l' 1000 files with statahead: 11 sec statahead total: 160 statahead wrong: 2 agl total: 153 list_total: 32 fname_total: 122 hit_total: 4866 miss_total: 1603 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 26 seconds llite.lustre-ffff97cc111ff000.statahead_stats= statahead total: 160 statahead wrong: 2 agl total: 153 list_total: 32 fname_total: 122 hit_total: 4866 miss_total: 1603 mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786404648.734218363 secs.nsecs start_time: 1786404559.353838830 secs.nsecs elapsed_time: 89.380379533 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 0 0 8 4: 2 5 13 8: 3 8 21 16: 23 62 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786404648.734290338 secs.nsecs start_time: 1786404559.354222275 secs.nsecs elapsed_time: 89.380068063 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 2 5 15 16: 13 32 47 32: 21 52 100 - open/close 939 (time 1786404661.68 total 10.00 last 93.89) - open/close 1812 (time 1786404671.68 total 20.00 last 87.27) - open/close 2736 (time 1786404681.69 total 30.01 last 92.35) - open/close 3702 (time 1786404691.70 total 40.02 last 96.48) - open/close 4575 (time 1786404701.71 total 50.03 last 87.27) - open/close 5273 (time 1786404711.71 total 60.03 last 69.77) - open/close 6171 (time 1786404721.72 total 70.04 last 89.69) - open/close 7098 (time 1786404731.72 total 80.04 last 92.68) - open/close 8021 (time 1786404741.73 total 90.05 last 92.21) - open/close 8945 (time 1786404751.74 total 100.06 last 92.33) - open/close 9881 (time 1786404761.74 total 110.06 last 93.59) total: 10000 open/close in 111.30 seconds: 89.85 ops/second llite.lustre-ffff97cc111ff000.statahead_batch_max=0 llite.lustre-ffff97cc111ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cc111ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff97cc111ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.688s user 0m0.317s sys 0m17.077s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff97cc111ff000.statahead_batch_max=32 llite.lustre-ffff97cc111ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff97cc111ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff97cc111ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m47.505s user 0m0.296s sys 0m15.512s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786404996.048325526 secs.nsecs start_time: 1786404876.513508228 secs.nsecs elapsed_time: 119.534817298 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786404996.048394087 secs.nsecs start_time: 1786404876.513748789 secs.nsecs elapsed_time: 119.534645298 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats=0 llite.lustre-ffff97cc111ff000.statahead_max=2048 llite.lustre-ffff97cc111ff000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.85 seconds: 54.20 ops/second llite.lustre-ffff97cc111ff000.statahead_max=0 101 real 0m1.691s user 0m0.009s sys 0m0.302s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff97cc111ff000.statahead_max=2048 llite.lustre-ffff97cc111ff000.statahead_max=2048 101 real 0m0.924s user 0m0.004s sys 0m0.207s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 total: 900 open/close in 9.83 seconds: 91.56 ops/second llite.lustre-ffff97cc111ff000.statahead_max=0 1001 real 0m18.830s user 0m0.030s sys 0m3.083s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff97cc111ff000.statahead_max=2048 llite.lustre-ffff97cc111ff000.statahead_max=2048 1001 real 0m8.964s user 0m0.027s sys 0m1.586s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 26 seconds llite.lustre-ffff97cc111ff000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786405357.976391437 secs.nsecs start_time: 1786404998.754476879 secs.nsecs elapsed_time: 359.221914558 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 2 28 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786405357.976469906 secs.nsecs start_time: 1786404998.754649982 secs.nsecs elapsed_time: 359.221819924 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 2 25 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 840 (time 1786405370.45 total 10.00 last 83.98) - open/close 1752 (time 1786405380.49 total 20.04 last 90.83) - open/close 2714 (time 1786405390.49 total 30.05 last 96.17) - open/close 3649 (time 1786405400.50 total 40.05 last 93.44) - open/close 4561 (time 1786405410.51 total 50.06 last 91.14) - open/close 5435 (time 1786405420.54 total 60.09 last 87.12) - open/close 6324 (time 1786405430.54 total 70.10 last 88.86) - open/close 7216 (time 1786405440.55 total 80.10 last 89.18) - open/close 8036 (time 1786405450.55 total 90.10 last 81.98) - open/close 8931 (time 1786405460.55 total 100.10 last 89.46) - open/close 9851 (time 1786405470.55 total 110.11 last 91.99) total: 10000 open/close in 111.64 seconds: 89.57 ops/second llite.lustre-ffff97cc111ff000.statahead_batch_max=0 llite.lustre-ffff97cc111ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cc111ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff97cc111ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m52.774s user 0m0.294s sys 0m17.015s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff97cc111ff000.statahead_batch_max=256 llite.lustre-ffff97cc111ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff97cc111ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff97cc111ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.154s user 0m0.272s sys 0m16.456s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786405665.327022212 secs.nsecs start_time: 1786405592.419748363 secs.nsecs elapsed_time: 72.907273849 secs.nsecs subreqs per batch batches %% cum %% 1: 68 36 36 2: 24 12 49 4: 13 6 56 8: 13 6 63 16: 15 8 71 32: 11 5 77 64: 3 1 79 128: 4 2 81 256: 35 18 100 mdc.lustre-MDT0001-mdc-ffff97cc111ff000.batch_stats= snapshot_time: 1786405665.327080792 secs.nsecs start_time: 1786405592.419876988 secs.nsecs elapsed_time: 72.907203804 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff97cc111ff000.statahead_batch_max=64 llite.lustre-ffff97cc111ff000.statahead_max=128