== sanity test 123ad: Verify batching statahead works correctly ========================================================== 07:47:55 (1787831275) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats=0 llite.lustre-ffff9b91ff55a000.statahead_max=32 llite.lustre-ffff9b91ff55a000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.21 seconds: 31.17 ops/second llite.lustre-ffff9b91ff55a000.statahead_max=0 101 real 0m3.303s user 0m0.006s sys 0m0.337s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9b91ff55a000.statahead_max=32 llite.lustre-ffff9b91ff55a000.statahead_max=32 101 real 0m0.939s user 0m0.004s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 105 statahead wrong: 2 agl total: 100 list_total: 32 fname_total: 71 hit_total: 4129 miss_total: 1614 swrong=2 ewrong=2 - open/close 650 (time 1787831307.89 total 10.02 last 64.85) total: 900 open/close in 13.70 seconds: 65.71 ops/second llite.lustre-ffff9b91ff55a000.statahead_max=0 1001 real 0m22.423s user 0m0.036s sys 0m2.961s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9b91ff55a000.statahead_max=32 llite.lustre-ffff9b91ff55a000.statahead_max=32 1001 real 0m8.287s user 0m0.031s sys 0m1.430s 'ls -l' 1000 files with statahead: 8 sec statahead total: 106 statahead wrong: 2 agl total: 101 list_total: 33 fname_total: 71 hit_total: 5125 miss_total: 1618 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9b91ff55a000.statahead_stats= statahead total: 106 statahead wrong: 2 agl total: 101 list_total: 33 fname_total: 71 hit_total: 5125 miss_total: 1618 mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787831378.976424214 secs.nsecs start_time: 1787831282.334576460 secs.nsecs elapsed_time: 96.641847754 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 3 7 7 4: 2 5 12 8: 0 0 12 16: 18 46 58 32: 16 41 100 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787831378.976484254 secs.nsecs start_time: 1787831282.334662486 secs.nsecs elapsed_time: 96.641821768 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 5 12 15 16: 21 53 69 32: 12 30 100 - open/close 859 (time 1787831392.16 total 10.01 last 85.81) - open/close 1665 (time 1787831402.18 total 20.02 last 80.50) - open/close 2553 (time 1787831412.18 total 30.03 last 88.77) - open/close 3353 (time 1787831422.19 total 40.04 last 79.91) - open/close 4190 (time 1787831432.20 total 50.04 last 83.66) - open/close 5097 (time 1787831442.20 total 60.05 last 90.64) - open/close 5989 (time 1787831452.20 total 70.05 last 89.20) - open/close 6796 (time 1787831462.22 total 80.06 last 80.58) - open/close 7626 (time 1787831472.22 total 90.07 last 82.98) - open/close 8484 (time 1787831482.22 total 100.07 last 85.76) - open/close 9315 (time 1787831492.23 total 110.08 last 83.06) total: 10000 open/close in 117.87 seconds: 84.84 ops/second llite.lustre-ffff9b91ff55a000.statahead_batch_max=0 llite.lustre-ffff9b91ff55a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.626s user 0m0.345s sys 0m15.338s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b91ff55a000.statahead_batch_max=32 llite.lustre-ffff9b91ff55a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m56.528s user 0m0.303s sys 0m14.096s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787831698.365617034 secs.nsecs start_time: 1787831569.609759204 secs.nsecs elapsed_time: 128.755857830 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787831698.365696417 secs.nsecs start_time: 1787831569.609846132 secs.nsecs elapsed_time: 128.755850285 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats=0 llite.lustre-ffff9b91ff55a000.statahead_max=2048 llite.lustre-ffff9b91ff55a000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.40 seconds: 29.41 ops/second llite.lustre-ffff9b91ff55a000.statahead_max=0 101 real 0m2.256s user 0m0.007s sys 0m0.325s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b91ff55a000.statahead_max=2048 llite.lustre-ffff9b91ff55a000.statahead_max=2048 101 real 0m1.120s user 0m0.008s sys 0m0.187s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 865 (time 1787832004.88 total 10.01 last 86.41) total: 900 open/close in 10.57 seconds: 85.14 ops/second llite.lustre-ffff9b91ff55a000.statahead_max=0 1001 real 0m21.755s user 0m0.036s sys 0m3.028s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9b91ff55a000.statahead_max=2048 llite.lustre-ffff9b91ff55a000.statahead_max=2048 1001 real 0m7.444s user 0m0.041s sys 0m1.438s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9b91ff55a000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787832068.375727889 secs.nsecs start_time: 1787831701.166718748 secs.nsecs elapsed_time: 367.209009141 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 1 16 33 4: 2 33 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787832068.375798116 secs.nsecs start_time: 1787831701.168641214 secs.nsecs elapsed_time: 367.207156902 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 50 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 758 (time 1787832081.10 total 10.01 last 75.72) - open/close 1346 (time 1787832091.11 total 20.02 last 58.75) - open/close 1978 (time 1787832101.11 total 30.02 last 63.20) - open/close 2588 (time 1787832111.12 total 40.03 last 60.93) - open/close 3238 (time 1787832121.13 total 50.04 last 64.96) - open/close 3999 (time 1787832131.13 total 60.04 last 76.10) - open/close 4728 (time 1787832141.13 total 70.04 last 72.85) - open/close 5422 (time 1787832151.14 total 80.05 last 69.34) - open/close 6000 (time 1787832161.16 total 90.07 last 57.68) - open/close 6525 (time 1787832171.18 total 100.09 last 52.43) - open/close 7163 (time 1787832181.18 total 110.09 last 63.80) - open/close 7755 (time 1787832191.18 total 120.09 last 59.15) - open/close 8373 (time 1787832201.18 total 130.09 last 61.80) - open/close 9075 (time 1787832211.19 total 140.10 last 70.20) - open/close 9697 (time 1787832221.20 total 150.11 last 62.11) total: 10000 open/close in 154.34 seconds: 64.79 ops/second llite.lustre-ffff9b91ff55a000.statahead_batch_max=0 llite.lustre-ffff9b91ff55a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.811s user 0m0.351s sys 0m16.418s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b91ff55a000.statahead_batch_max=256 llite.lustre-ffff9b91ff55a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.609s user 0m0.387s sys 0m16.213s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787832372.385574866 secs.nsecs start_time: 1787832303.252041754 secs.nsecs elapsed_time: 69.133533112 secs.nsecs subreqs per batch batches %% cum %% 1: 155 44 44 2: 47 13 58 4: 51 14 72 8: 21 6 78 16: 17 4 83 32: 16 4 88 64: 5 1 89 128: 2 0 90 256: 33 9 100 mdc.lustre-MDT0001-mdc-ffff9b91ff55a000.batch_stats= snapshot_time: 1787832372.385646508 secs.nsecs start_time: 1787832303.252245565 secs.nsecs elapsed_time: 69.133400943 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b91ff55a000.statahead_batch_max=64 llite.lustre-ffff9b91ff55a000.statahead_max=128