== sanity test 123ad: Verify batching statahead works correctly ========================================================== 00:01:50 (1787371310) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats=0 llite.lustre-ffff8cfdc35c4000.statahead_max=32 llite.lustre-ffff8cfdc35c4000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.82 seconds: 35.42 ops/second llite.lustre-ffff8cfdc35c4000.statahead_max=0 101 real 0m2.571s user 0m0.005s sys 0m0.360s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8cfdc35c4000.statahead_max=32 llite.lustre-ffff8cfdc35c4000.statahead_max=32 101 real 0m1.017s user 0m0.004s sys 0m0.225s 'ls -l' 100 files with statahead: 2 sec statahead total: 109 statahead wrong: 2 agl total: 103 list_total: 35 fname_total: 71 hit_total: 3832 miss_total: 1489 swrong=2 ewrong=2 - open/close 654 (time 1787371338.98 total 10.00 last 65.37) total: 900 open/close in 13.87 seconds: 64.90 ops/second llite.lustre-ffff8cfdc35c4000.statahead_max=0 1001 real 0m24.937s user 0m0.038s sys 0m3.216s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8cfdc35c4000.statahead_max=32 llite.lustre-ffff8cfdc35c4000.statahead_max=32 1001 real 0m8.640s user 0m0.025s sys 0m1.387s 'ls -l' 1000 files with statahead: 8 sec statahead total: 110 statahead wrong: 2 agl total: 104 list_total: 36 fname_total: 71 hit_total: 4828 miss_total: 1493 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8cfdc35c4000.statahead_stats= statahead total: 110 statahead wrong: 2 agl total: 104 list_total: 36 fname_total: 71 hit_total: 4828 miss_total: 1493 mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787371410.521476504 secs.nsecs start_time: 1787371314.765855588 secs.nsecs elapsed_time: 95.755620916 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 3 7 12 8: 2 5 17 16: 18 46 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787371410.521553189 secs.nsecs start_time: 1787371314.765964396 secs.nsecs elapsed_time: 95.755588793 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 0 0 5 8: 4 10 15 16: 20 50 65 32: 14 35 100 - open/close 785 (time 1787371422.91 total 10.01 last 78.42) - open/close 1536 (time 1787371432.92 total 20.02 last 75.02) - open/close 2257 (time 1787371442.92 total 30.02 last 72.08) - open/close 2984 (time 1787371452.92 total 40.03 last 72.68) - open/close 3753 (time 1787371462.93 total 50.03 last 76.87) - open/close 4484 (time 1787371472.93 total 60.03 last 73.08) - open/close 5204 (time 1787371482.93 total 70.03 last 72.00) - open/close 5947 (time 1787371492.93 total 80.03 last 74.29) - open/close 6754 (time 1787371502.94 total 90.04 last 80.66) - open/close 7574 (time 1787371512.94 total 100.05 last 81.95) - open/close 8280 (time 1787371522.96 total 110.06 last 70.51) - open/close 9029 (time 1787371532.96 total 120.07 last 74.85) - open/close 9740 (time 1787371542.97 total 130.08 last 71.04) total: 10000 open/close in 133.65 seconds: 74.82 ops/second llite.lustre-ffff8cfdc35c4000.statahead_batch_max=0 llite.lustre-ffff8cfdc35c4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.827s user 0m0.317s sys 0m15.634s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8cfdc35c4000.statahead_batch_max=32 llite.lustre-ffff8cfdc35c4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.494s user 0m0.297s sys 0m14.846s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787371730.102434216 secs.nsecs start_time: 1787371618.412779393 secs.nsecs elapsed_time: 111.689654823 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787371730.102895050 secs.nsecs start_time: 1787371618.413017171 secs.nsecs elapsed_time: 111.689877879 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats=0 llite.lustre-ffff8cfdc35c4000.statahead_max=2048 llite.lustre-ffff8cfdc35c4000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.76 seconds: 36.18 ops/second llite.lustre-ffff8cfdc35c4000.statahead_max=0 101 real 0m2.577s user 0m0.009s sys 0m0.350s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8cfdc35c4000.statahead_max=2048 llite.lustre-ffff8cfdc35c4000.statahead_max=2048 101 real 0m0.944s user 0m0.006s sys 0m0.199s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 - open/close 667 (time 1787372071.19 total 10.01 last 66.64) total: 900 open/close in 13.91 seconds: 64.72 ops/second llite.lustre-ffff8cfdc35c4000.statahead_max=0 1001 real 0m24.566s user 0m0.040s sys 0m3.217s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8cfdc35c4000.statahead_max=2048 llite.lustre-ffff8cfdc35c4000.statahead_max=2048 1001 real 0m7.300s user 0m0.031s sys 0m1.252s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8cfdc35c4000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787372141.236142697 secs.nsecs start_time: 1787371732.660015449 secs.nsecs elapsed_time: 408.576127248 secs.nsecs subreqs per batch batches %% cum %% 1: 1 12 12 2: 1 12 25 4: 2 25 50 8: 1 12 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787372141.236254832 secs.nsecs start_time: 1787371732.662382504 secs.nsecs elapsed_time: 408.573872328 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 50 50 16: 1 12 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 - open/close 661 (time 1787372154.00 total 10.01 last 66.04) - open/close 1395 (time 1787372164.00 total 20.01 last 73.38) - open/close 2190 (time 1787372174.01 total 30.02 last 79.44) - open/close 2880 (time 1787372184.01 total 40.03 last 68.94) - open/close 3522 (time 1787372194.01 total 50.03 last 64.20) - open/close 4216 (time 1787372204.02 total 60.04 last 69.32) - open/close 4890 (time 1787372214.04 total 70.05 last 67.33) - open/close 5631 (time 1787372224.04 total 80.05 last 74.06) - open/close 6318 (time 1787372234.05 total 90.06 last 68.65) - open/close 7032 (time 1787372244.07 total 100.08 last 71.27) - open/close 7738 (time 1787372254.08 total 110.09 last 70.51) - open/close 8440 (time 1787372264.08 total 120.10 last 70.17) - open/close 9172 (time 1787372274.10 total 130.11 last 73.08) - open/close 9860 (time 1787372284.11 total 140.12 last 68.76) total: 10000 open/close in 142.14 seconds: 70.35 ops/second llite.lustre-ffff8cfdc35c4000.statahead_batch_max=0 llite.lustre-ffff8cfdc35c4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.597s user 0m0.350s sys 0m16.888s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8cfdc35c4000.statahead_batch_max=256 llite.lustre-ffff8cfdc35c4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.605s user 0m0.310s sys 0m16.278s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787372426.128818609 secs.nsecs start_time: 1787372361.262518905 secs.nsecs elapsed_time: 64.866299704 secs.nsecs subreqs per batch batches %% cum %% 1: 82 41 41 2: 27 13 55 4: 17 8 63 8: 7 3 67 16: 5 2 70 32: 15 7 77 64: 9 4 82 128: 0 0 82 256: 35 17 100 mdc.lustre-MDT0001-mdc-ffff8cfdc35c4000.batch_stats= snapshot_time: 1787372426.130622804 secs.nsecs start_time: 1787372361.262614631 secs.nsecs elapsed_time: 64.868008173 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8cfdc35c4000.statahead_batch_max=64 llite.lustre-ffff8cfdc35c4000.statahead_max=128