== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:20:01 (1787869201) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats=0 llite.lustre-ffff8d4ba30ec000.statahead_max=32 llite.lustre-ffff8d4ba30ec000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.26 seconds: 30.71 ops/second llite.lustre-ffff8d4ba30ec000.statahead_max=0 101 real 0m2.638s user 0m0.004s sys 0m0.312s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8d4ba30ec000.statahead_max=32 llite.lustre-ffff8d4ba30ec000.statahead_max=32 101 real 0m1.054s user 0m0.004s sys 0m0.160s 'ls -l' 100 files with statahead: 1 sec statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 31 fname_total: 88 hit_total: 4613 miss_total: 1281 swrong=2 ewrong=2 total: 900 open/close in 9.65 seconds: 93.25 ops/second llite.lustre-ffff8d4ba30ec000.statahead_max=0 1001 real 0m21.442s user 0m0.033s sys 0m2.747s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8d4ba30ec000.statahead_max=32 llite.lustre-ffff8d4ba30ec000.statahead_max=32 1001 real 0m9.384s user 0m0.029s sys 0m1.413s 'ls -l' 1000 files with statahead: 9 sec statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 32 fname_total: 88 hit_total: 5612 miss_total: 1282 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8d4ba30ec000.statahead_stats= statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 32 fname_total: 88 hit_total: 5612 miss_total: 1282 mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787869293.940163520 secs.nsecs start_time: 1787869206.485252974 secs.nsecs elapsed_time: 87.454910546 secs.nsecs subreqs per batch batches %% cum %% 1: 5 12 12 2: 0 0 12 4: 3 7 20 8: 1 2 23 16: 25 64 87 32: 5 12 100 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787869293.940230259 secs.nsecs start_time: 1787869206.485331295 secs.nsecs elapsed_time: 87.454898964 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 0 0 7 4: 2 4 12 8: 2 4 17 16: 10 24 41 32: 24 58 100 - open/close 872 (time 1787869307.20 total 10.03 last 86.94) - open/close 1722 (time 1787869317.21 total 20.04 last 84.96) - open/close 2606 (time 1787869327.21 total 30.04 last 88.36) - open/close 3511 (time 1787869337.21 total 40.04 last 90.46) - open/close 4346 (time 1787869347.22 total 50.05 last 83.47) - open/close 5300 (time 1787869357.22 total 60.05 last 95.35) - open/close 6357 (time 1787869367.22 total 70.05 last 105.70) - open/close 7151 (time 1787869377.24 total 80.07 last 79.29) - open/close 8037 (time 1787869387.24 total 90.07 last 88.53) - open/close 8973 (time 1787869397.25 total 100.08 last 93.56) - open/close 9956 (time 1787869407.26 total 110.09 last 98.21) total: 10000 open/close in 110.47 seconds: 90.52 ops/second llite.lustre-ffff8d4ba30ec000.statahead_batch_max=0 llite.lustre-ffff8d4ba30ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.263s user 0m0.296s sys 0m13.398s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d4ba30ec000.statahead_batch_max=32 llite.lustre-ffff8d4ba30ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.573s user 0m0.300s sys 0m13.221s batched statahead RPC= hit count=9975 mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787869590.672344968 secs.nsecs start_time: 1787869478.477082826 secs.nsecs elapsed_time: 112.195262142 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787869590.672415578 secs.nsecs start_time: 1787869478.477160455 secs.nsecs elapsed_time: 112.195255123 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats=0 llite.lustre-ffff8d4ba30ec000.statahead_max=2048 llite.lustre-ffff8d4ba30ec000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.22 seconds: 45.11 ops/second llite.lustre-ffff8d4ba30ec000.statahead_max=0 101 real 0m2.488s user 0m0.003s sys 0m0.307s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8d4ba30ec000.statahead_max=2048 llite.lustre-ffff8d4ba30ec000.statahead_max=2048 101 real 0m0.745s user 0m0.006s sys 0m0.134s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 total: 900 open/close in 9.60 seconds: 93.77 ops/second llite.lustre-ffff8d4ba30ec000.statahead_max=0 1001 real 0m18.448s user 0m0.035s sys 0m2.722s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8d4ba30ec000.statahead_max=2048 llite.lustre-ffff8d4ba30ec000.statahead_max=2048 1001 real 0m7.161s user 0m0.036s sys 0m1.364s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8d4ba30ec000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787869910.152275157 secs.nsecs start_time: 1787869593.470728086 secs.nsecs elapsed_time: 316.681547071 secs.nsecs subreqs per batch batches %% cum %% 1: 3 37 37 2: 1 12 50 4: 1 12 62 8: 1 12 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787869910.152357600 secs.nsecs start_time: 1787869593.470901262 secs.nsecs elapsed_time: 316.681456338 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 2 20 40 16: 1 10 50 32: 1 10 60 64: 2 20 80 128: 0 0 80 256: 2 20 100 - open/close 863 (time 1787869922.64 total 10.00 last 86.28) - open/close 1864 (time 1787869932.65 total 20.01 last 100.06) - open/close 2933 (time 1787869942.65 total 30.01 last 106.89) - open/close 3909 (time 1787869952.65 total 40.01 last 97.53) - open/close 4925 (time 1787869962.66 total 50.02 last 101.57) - open/close 5885 (time 1787869972.66 total 60.02 last 95.97) - open/close 6816 (time 1787869982.67 total 70.02 last 93.06) - open/close 7754 (time 1787869992.67 total 80.03 last 93.77) - open/close 8656 (time 1787870002.67 total 90.03 last 90.17) - open/close 9626 (time 1787870012.68 total 100.04 last 96.94) total: 10000 open/close in 103.74 seconds: 96.39 ops/second llite.lustre-ffff8d4ba30ec000.statahead_batch_max=0 llite.lustre-ffff8d4ba30ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.932s user 0m0.270s sys 0m13.509s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d4ba30ec000.statahead_batch_max=256 llite.lustre-ffff8d4ba30ec000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m49.731s user 0m0.277s sys 0m12.746s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787870152.435705340 secs.nsecs start_time: 1787870092.599501510 secs.nsecs elapsed_time: 59.836203830 secs.nsecs subreqs per batch batches %% cum %% 1: 77 35 35 2: 27 12 48 4: 30 13 62 8: 16 7 69 16: 4 1 71 32: 17 7 79 64: 9 4 83 128: 1 0 84 256: 34 15 100 mdc.lustre-MDT0001-mdc-ffff8d4ba30ec000.batch_stats= snapshot_time: 1787870152.435784165 secs.nsecs start_time: 1787870092.599649313 secs.nsecs elapsed_time: 59.836134852 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8d4ba30ec000.statahead_batch_max=64 llite.lustre-ffff8d4ba30ec000.statahead_max=128