== sanity test 123ad: Verify batching statahead works correctly ========================================================== 06:06:58 (1787566018) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats=0 llite.lustre-ffff9ca943f61800.statahead_max=32 llite.lustre-ffff9ca943f61800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.57 seconds: 38.97 ops/second llite.lustre-ffff9ca943f61800.statahead_max=0 101 real 0m2.152s user 0m0.007s sys 0m0.286s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ca943f61800.statahead_max=32 llite.lustre-ffff9ca943f61800.statahead_max=32 101 real 0m0.948s user 0m0.004s sys 0m0.143s 'ls -l' 100 files with statahead: 1 sec statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 30 fname_total: 75 hit_total: 3999 miss_total: 1305 swrong=2 ewrong=2 - open/close 798 (time 1787566046.35 total 10.00 last 79.77) total: 900 open/close in 11.19 seconds: 80.44 ops/second llite.lustre-ffff9ca943f61800.statahead_max=0 1001 real 0m21.722s user 0m0.030s sys 0m2.812s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9ca943f61800.statahead_max=32 llite.lustre-ffff9ca943f61800.statahead_max=32 1001 real 0m8.135s user 0m0.023s sys 0m1.367s 'ls -l' 1000 files with statahead: 8 sec statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 31 fname_total: 75 hit_total: 4997 miss_total: 1307 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9ca943f61800.statahead_stats= statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 31 fname_total: 75 hit_total: 4997 miss_total: 1307 mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566107.664984344 secs.nsecs start_time: 1787566023.614026107 secs.nsecs elapsed_time: 84.050958237 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 3 8 13 8: 1 2 16 16: 25 69 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566107.665282705 secs.nsecs start_time: 1787566023.614238158 secs.nsecs elapsed_time: 84.051044547 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 0 0 5 8: 3 7 12 16: 10 25 38 32: 24 61 100 - open/close 807 (time 1787566120.46 total 10.00 last 80.67) - open/close 1453 (time 1787566130.47 total 20.01 last 64.56) - open/close 2241 (time 1787566140.48 total 30.03 last 78.68) - open/close 3165 (time 1787566150.50 total 40.04 last 92.24) - open/close 4109 (time 1787566160.50 total 50.05 last 94.36) - open/close 4993 (time 1787566170.51 total 60.05 last 88.34) - open/close 5943 (time 1787566180.52 total 70.07 last 94.86) - open/close 6863 (time 1787566190.53 total 80.08 last 91.93) - open/close 7721 (time 1787566200.54 total 90.08 last 85.74) - open/close 8669 (time 1787566210.54 total 100.08 last 94.79) - open/close 9620 (time 1787566220.55 total 110.09 last 95.04) total: 10000 open/close in 114.24 seconds: 87.53 ops/second llite.lustre-ffff9ca943f61800.statahead_batch_max=0 llite.lustre-ffff9ca943f61800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ca943f61800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ca943f61800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.596s user 0m0.293s sys 0m15.505s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ca943f61800.statahead_batch_max=32 llite.lustre-ffff9ca943f61800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ca943f61800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ca943f61800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m31.235s user 0m0.288s sys 0m13.703s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566386.675001623 secs.nsecs start_time: 1787566285.792061745 secs.nsecs elapsed_time: 100.882939878 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566386.675067499 secs.nsecs start_time: 1787566285.792300323 secs.nsecs elapsed_time: 100.882767176 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats=0 llite.lustre-ffff9ca943f61800.statahead_max=2048 llite.lustre-ffff9ca943f61800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 1.91 seconds: 52.23 ops/second llite.lustre-ffff9ca943f61800.statahead_max=0 101 real 0m2.007s user 0m0.007s sys 0m0.285s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ca943f61800.statahead_max=2048 llite.lustre-ffff9ca943f61800.statahead_max=2048 101 real 0m0.734s user 0m0.004s sys 0m0.185s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 total: 900 open/close in 9.74 seconds: 92.37 ops/second llite.lustre-ffff9ca943f61800.statahead_max=0 1001 real 0m19.022s user 0m0.032s sys 0m2.760s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff9ca943f61800.statahead_max=2048 llite.lustre-ffff9ca943f61800.statahead_max=2048 1001 real 0m6.742s user 0m0.030s sys 0m1.385s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9ca943f61800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566715.309961076 secs.nsecs start_time: 1787566389.050855128 secs.nsecs elapsed_time: 326.259105948 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 3 33 33 4: 0 0 33 8: 2 22 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566715.310028522 secs.nsecs start_time: 1787566389.050994126 secs.nsecs elapsed_time: 326.259034396 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 0 0 11 8: 4 44 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 886 (time 1787566728.22 total 10.02 last 88.46) - open/close 1801 (time 1787566738.22 total 20.02 last 91.50) - open/close 2773 (time 1787566748.22 total 30.02 last 97.18) - open/close 3738 (time 1787566758.23 total 40.03 last 96.42) - open/close 4774 (time 1787566768.25 total 50.04 last 103.41) - open/close 5696 (time 1787566778.25 total 60.05 last 92.16) - open/close 6576 (time 1787566788.26 total 70.05 last 87.97) - open/close 7442 (time 1787566798.27 total 80.07 last 86.46) - open/close 8180 (time 1787566808.28 total 90.08 last 73.75) - open/close 9170 (time 1787566818.29 total 100.08 last 98.93) total: 10000 open/close in 108.74 seconds: 91.96 ops/second llite.lustre-ffff9ca943f61800.statahead_batch_max=0 llite.lustre-ffff9ca943f61800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ca943f61800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ca943f61800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.617s user 0m0.296s sys 0m15.223s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ca943f61800.statahead_batch_max=256 llite.lustre-ffff9ca943f61800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ca943f61800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ca943f61800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.316s user 0m0.265s sys 0m14.414s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566951.979937989 secs.nsecs start_time: 1787566892.529949411 secs.nsecs elapsed_time: 59.449988578 secs.nsecs subreqs per batch batches %% cum %% 1: 77 39 39 2: 20 10 49 4: 18 9 58 8: 11 5 63 16: 11 5 69 32: 15 7 77 64: 7 3 80 128: 4 2 82 256: 34 17 100 mdc.lustre-MDT0001-mdc-ffff9ca943f61800.batch_stats= snapshot_time: 1787566951.980006819 secs.nsecs start_time: 1787566892.530075551 secs.nsecs elapsed_time: 59.449931268 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ca943f61800.statahead_batch_max=64 llite.lustre-ffff9ca943f61800.statahead_max=128