== sanity test 123ad: Verify batching statahead works correctly ========================================================== 11:59:24 (1786982364) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats=0 llite.lustre-ffff8baa43fe0800.statahead_max=32 llite.lustre-ffff8baa43fe0800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 4.29 seconds: 23.29 ops/second llite.lustre-ffff8baa43fe0800.statahead_max=0 101 real 0m3.460s user 0m0.006s sys 0m0.385s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8baa43fe0800.statahead_max=32 llite.lustre-ffff8baa43fe0800.statahead_max=32 101 real 0m1.303s user 0m0.005s sys 0m0.232s 'ls -l' 100 files with statahead: 1 sec statahead total: 117 statahead wrong: 2 agl total: 111 list_total: 30 fname_total: 87 hit_total: 3571 miss_total: 1646 swrong=2 ewrong=2 - open/close 658 (time 1786982399.69 total 10.00 last 65.78) total: 900 open/close in 14.56 seconds: 61.81 ops/second llite.lustre-ffff8baa43fe0800.statahead_max=0 1001 real 0m27.161s user 0m0.037s sys 0m3.450s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff8baa43fe0800.statahead_max=32 llite.lustre-ffff8baa43fe0800.statahead_max=32 1001 real 0m10.581s user 0m0.030s sys 0m1.747s 'ls -l' 1000 files with statahead: 11 sec statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 31 fname_total: 87 hit_total: 4570 miss_total: 1647 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff8baa43fe0800.statahead_stats= statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 31 fname_total: 87 hit_total: 4570 miss_total: 1647 mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786982479.709873461 secs.nsecs start_time: 1786982371.601190406 secs.nsecs elapsed_time: 108.108683055 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 2 5 10 4: 1 2 13 8: 3 8 21 16: 24 64 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786982479.710012671 secs.nsecs start_time: 1786982371.601322512 secs.nsecs elapsed_time: 108.108690159 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 12 30 43 32: 22 56 100 - open/close 667 (time 1786982492.79 total 10.00 last 66.70) - open/close 1319 (time 1786982502.80 total 20.01 last 65.13) - open/close 1962 (time 1786982512.81 total 30.02 last 64.23) - open/close 2645 (time 1786982522.83 total 40.04 last 68.16) - open/close 3234 (time 1786982532.85 total 50.06 last 58.82) - open/close 3860 (time 1786982542.86 total 60.07 last 62.54) - open/close 4518 (time 1786982552.86 total 70.07 last 65.76) - open/close 5125 (time 1786982562.86 total 80.08 last 60.69) - open/close 5876 (time 1786982572.88 total 90.09 last 75.01) - open/close 6484 (time 1786982582.89 total 100.10 last 60.72) - open/close 7085 (time 1786982592.90 total 110.11 last 60.04) - open/close 7647 (time 1786982602.91 total 120.12 last 56.14) - open/close 8339 (time 1786982612.94 total 130.15 last 69.02) - open/close 9003 (time 1786982622.94 total 140.15 last 66.40) - open/close 9681 (time 1786982632.95 total 150.16 last 67.73) total: 10000 open/close in 155.23 seconds: 64.42 ops/second llite.lustre-ffff8baa43fe0800.statahead_batch_max=0 llite.lustre-ffff8baa43fe0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.299s user 0m0.425s sys 0m20.299s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8baa43fe0800.statahead_batch_max=32 llite.lustre-ffff8baa43fe0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m5.190s user 0m0.391s sys 0m17.547s batched statahead RPC= hit count=9994 mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786982855.097878984 secs.nsecs start_time: 1786982715.334899059 secs.nsecs elapsed_time: 139.762979925 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 312 99 100 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786982855.097943108 secs.nsecs start_time: 1786982715.335037491 secs.nsecs elapsed_time: 139.762905617 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats=0 llite.lustre-ffff8baa43fe0800.statahead_max=2048 llite.lustre-ffff8baa43fe0800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.99 seconds: 33.47 ops/second llite.lustre-ffff8baa43fe0800.statahead_max=0 101 real 0m2.518s user 0m0.002s sys 0m0.354s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8baa43fe0800.statahead_max=2048 llite.lustre-ffff8baa43fe0800.statahead_max=2048 101 real 0m1.156s user 0m0.006s sys 0m0.235s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10093 miss_total: 7 swrong=0 ewrong=0 - open/close 742 (time 1786983223.11 total 10.00 last 74.18) total: 900 open/close in 12.46 seconds: 72.21 ops/second llite.lustre-ffff8baa43fe0800.statahead_max=0 1001 real 0m25.824s user 0m0.039s sys 0m3.494s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff8baa43fe0800.statahead_max=2048 llite.lustre-ffff8baa43fe0800.statahead_max=2048 1001 real 0m8.321s user 0m0.029s sys 0m1.658s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11092 miss_total: 8 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8baa43fe0800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11092 miss_total: 8 mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786983294.836859153 secs.nsecs start_time: 1786982858.512557169 secs.nsecs elapsed_time: 436.324301984 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 2 28 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786983294.836940630 secs.nsecs start_time: 1786982858.512746454 secs.nsecs elapsed_time: 436.324194176 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 2 22 33 16: 2 22 55 32: 0 0 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 676 (time 1786983307.34 total 10.00 last 67.59) - open/close 1391 (time 1786983317.36 total 20.02 last 71.38) - open/close 2167 (time 1786983327.37 total 30.03 last 77.53) - open/close 2865 (time 1786983337.37 total 40.03 last 69.78) - open/close 3543 (time 1786983347.38 total 50.04 last 67.75) - open/close 4252 (time 1786983357.38 total 60.04 last 70.89) - open/close 4929 (time 1786983367.38 total 70.04 last 67.70) - open/close 5608 (time 1786983377.39 total 80.05 last 67.84) - open/close 6280 (time 1786983387.40 total 90.06 last 67.13) - open/close 7015 (time 1786983397.43 total 100.08 last 73.33) - open/close 7669 (time 1786983407.43 total 110.08 last 65.39) - open/close 8380 (time 1786983417.44 total 120.09 last 71.04) - open/close 9089 (time 1786983427.44 total 130.10 last 70.85) - open/close 9807 (time 1786983437.45 total 140.11 last 71.74) total: 10000 open/close in 143.11 seconds: 69.88 ops/second llite.lustre-ffff8baa43fe0800.statahead_batch_max=0 llite.lustre-ffff8baa43fe0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.361s user 0m0.355s sys 0m18.604s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8baa43fe0800.statahead_batch_max=256 llite.lustre-ffff8baa43fe0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.stats=clear mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.019s user 0m0.389s sys 0m18.838s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786983599.665987866 secs.nsecs start_time: 1786983515.964442395 secs.nsecs elapsed_time: 83.701545471 secs.nsecs subreqs per batch batches %% cum %% 1: 224 54 54 2: 62 15 69 4: 43 10 79 8: 17 4 83 16: 11 2 86 32: 15 3 90 64: 7 1 91 128: 0 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff8baa43fe0800.batch_stats= snapshot_time: 1786983599.666132970 secs.nsecs start_time: 1786983515.964602457 secs.nsecs elapsed_time: 83.701530513 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8baa43fe0800.statahead_batch_max=64 llite.lustre-ffff8baa43fe0800.statahead_max=128