== sanity test 123ad: Verify batching statahead works correctly ========================================================== 14:14:01 (1785953641) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats=0 llite.lustre-ffff986158618000.statahead_max=32 llite.lustre-ffff986158618000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.17 seconds: 31.51 ops/second llite.lustre-ffff986158618000.statahead_max=0 101 real 0m3.282s user 0m0.006s sys 0m0.334s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff986158618000.statahead_max=32 llite.lustre-ffff986158618000.statahead_max=32 101 real 0m1.265s user 0m0.001s sys 0m0.174s 'ls -l' 100 files with statahead: 2 sec statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 36 fname_total: 110 hit_total: 4059 miss_total: 1272 swrong=2 ewrong=2 - open/close 457 (time 1785953673.50 total 10.03 last 45.58) total: 900 open/close in 15.53 seconds: 57.96 ops/second llite.lustre-ffff986158618000.statahead_max=0 1001 real 0m24.282s user 0m0.038s sys 0m3.119s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff986158618000.statahead_max=32 llite.lustre-ffff986158618000.statahead_max=32 1001 real 0m10.168s user 0m0.027s sys 0m1.449s 'ls -l' 1000 files with statahead: 10 sec statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 37 fname_total: 110 hit_total: 5053 miss_total: 1278 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff986158618000.statahead_stats= statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 37 fname_total: 110 hit_total: 5053 miss_total: 1278 mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats= snapshot_time: 1785953752.861232463 secs.nsecs start_time: 1785953647.460696007 secs.nsecs elapsed_time: 105.400536456 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 4 10 17 16: 18 46 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats= snapshot_time: 1785953752.861669728 secs.nsecs start_time: 1785953647.463288156 secs.nsecs elapsed_time: 105.398381572 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 2 5 15 16: 23 57 72 32: 11 27 100 - open/close 641 (time 1785953766.23 total 10.00 last 64.09) - open/close 1276 (time 1785953776.24 total 20.00 last 63.49) - open/close 2188 (time 1785953786.24 total 30.01 last 91.19) - open/close 2967 (time 1785953796.26 total 40.02 last 77.76) - open/close 3798 (time 1785953806.26 total 50.02 last 83.09) - open/close 4649 (time 1785953816.26 total 60.03 last 85.07) - open/close 5170 (time 1785953826.27 total 70.03 last 52.07) - open/close 5778 (time 1785953836.27 total 80.03 last 60.80) - open/close 6405 (time 1785953846.27 total 90.04 last 62.69) - open/close 6921 (time 1785953856.27 total 100.04 last 51.60) - open/close 7414 (time 1785953866.28 total 110.04 last 49.27) - open/close 8050 (time 1785953876.28 total 120.05 last 63.57) - open/close 8802 (time 1785953886.29 total 130.05 last 75.16) - open/close 9551 (time 1785953896.29 total 140.06 last 74.86) total: 10000 open/close in 147.40 seconds: 67.84 ops/second llite.lustre-ffff986158618000.statahead_batch_max=0 llite.lustre-ffff986158618000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff986158618000.stats=clear mdc.lustre-MDT0001-mdc-ffff986158618000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m20.469s user 0m0.297s sys 0m15.713s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff986158618000.statahead_batch_max=32 llite.lustre-ffff986158618000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff986158618000.stats=clear mdc.lustre-MDT0001-mdc-ffff986158618000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m12.232s user 0m0.277s sys 0m14.919s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats= snapshot_time: 1785954152.838790303 secs.nsecs start_time: 1785954002.067832599 secs.nsecs elapsed_time: 150.770957704 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats= snapshot_time: 1785954152.838848975 secs.nsecs start_time: 1785954002.067977051 secs.nsecs elapsed_time: 150.770871924 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats=0 llite.lustre-ffff986158618000.statahead_max=2048 llite.lustre-ffff986158618000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.32 seconds: 43.14 ops/second llite.lustre-ffff986158618000.statahead_max=0 101 real 0m2.681s user 0m0.002s sys 0m0.337s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff986158618000.statahead_max=2048 llite.lustre-ffff986158618000.statahead_max=2048 101 real 0m0.941s user 0m0.001s sys 0m0.174s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 - open/close 580 (time 1785954502.97 total 10.00 last 58.00) total: 900 open/close in 15.41 seconds: 58.41 ops/second llite.lustre-ffff986158618000.statahead_max=0 1001 real 0m30.923s user 0m0.034s sys 0m3.172s 'ls -l' 1000 files without statahead: 31 sec llite.lustre-ffff986158618000.statahead_max=2048 llite.lustre-ffff986158618000.statahead_max=2048 1001 real 0m8.995s user 0m0.029s sys 0m1.374s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 31 seconds llite.lustre-ffff986158618000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats= snapshot_time: 1785954596.121864405 secs.nsecs start_time: 1785954156.708009551 secs.nsecs elapsed_time: 439.413854854 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 2 25 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats= snapshot_time: 1785954596.121937717 secs.nsecs start_time: 1785954156.709876741 secs.nsecs elapsed_time: 439.412060976 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 2 25 50 16: 1 12 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 - open/close 626 (time 1785954611.16 total 10.00 last 62.59) - open/close 1010 (time 1785954621.17 total 20.01 last 38.37) - open/close 1530 (time 1785954631.18 total 30.02 last 51.95) - open/close 2015 (time 1785954641.21 total 40.05 last 48.35) - open/close 2775 (time 1785954651.22 total 50.06 last 75.95) - open/close 3267 (time 1785954661.23 total 60.07 last 49.12) - open/close 3977 (time 1785954671.25 total 70.08 last 70.92) - open/close 4710 (time 1785954681.26 total 80.10 last 73.20) - open/close 5470 (time 1785954691.27 total 90.11 last 75.93) - open/close 6083 (time 1785954701.30 total 100.13 last 61.13) - open/close 6579 (time 1785954711.33 total 110.16 last 49.45) - open/close 7181 (time 1785954721.34 total 120.17 last 60.14) - open/close 7903 (time 1785954731.36 total 130.20 last 72.02) - open/close 8513 (time 1785954741.37 total 140.21 last 60.93) - open/close 9198 (time 1785954751.38 total 150.21 last 68.48) - open/close 9768 (time 1785954761.38 total 160.22 last 56.98) total: 10000 open/close in 163.49 seconds: 61.16 ops/second llite.lustre-ffff986158618000.statahead_batch_max=0 llite.lustre-ffff986158618000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff986158618000.stats=clear mdc.lustre-MDT0001-mdc-ffff986158618000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.994s user 0m0.277s sys 0m15.421s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff986158618000.statahead_batch_max=256 llite.lustre-ffff986158618000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff986158618000.stats=clear mdc.lustre-MDT0001-mdc-ffff986158618000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.440s user 0m0.269s sys 0m14.847s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff986158618000.batch_stats= snapshot_time: 1785954946.100439321 secs.nsecs start_time: 1785954860.264519779 secs.nsecs elapsed_time: 85.835919542 secs.nsecs subreqs per batch batches %% cum %% 1: 157 47 47 2: 56 16 63 4: 32 9 73 8: 20 5 79 16: 11 3 82 32: 16 4 87 64: 8 2 89 128: 0 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff986158618000.batch_stats= snapshot_time: 1785954946.100515351 secs.nsecs start_time: 1785954860.264652919 secs.nsecs elapsed_time: 85.835862432 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff986158618000.statahead_batch_max=64 llite.lustre-ffff986158618000.statahead_max=128