== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:50:17 (1788565817) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats=0 llite.lustre-ffff91e447cf0000.statahead_max=32 llite.lustre-ffff91e447cf0000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats=0 total: 100 open/close in 3.08 seconds: 32.48 ops/second llite.lustre-ffff91e447cf0000.statahead_max=0 101 real 0m3.040s user 0m0.002s sys 0m0.310s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff91e447cf0000.statahead_max=32 llite.lustre-ffff91e447cf0000.statahead_max=32 101 real 0m1.490s user 0m0.004s sys 0m0.177s 'ls -l' 100 files with statahead: 2 sec statahead total: 123 statahead wrong: 2 agl total: 117 list_total: 30 fname_total: 79 hit_total: 3627 miss_total: 1722 swrong=2 ewrong=2 - open/close 526 (time 1788565848.77 total 10.01 last 52.54) total: 900 open/close in 18.77 seconds: 47.95 ops/second llite.lustre-ffff91e447cf0000.statahead_max=0 1001 real 0m32.197s user 0m0.031s sys 0m2.981s 'ls -l' 1000 files without statahead: 32 sec llite.lustre-ffff91e447cf0000.statahead_max=32 llite.lustre-ffff91e447cf0000.statahead_max=32 1001 real 0m11.894s user 0m0.028s sys 0m1.461s 'ls -l' 1000 files with statahead: 12 sec statahead total: 124 statahead wrong: 2 agl total: 118 list_total: 31 fname_total: 79 hit_total: 4626 miss_total: 1723 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 29 seconds llite.lustre-ffff91e447cf0000.statahead_stats= statahead total: 124 statahead wrong: 2 agl total: 118 list_total: 31 fname_total: 79 hit_total: 4626 miss_total: 1723 mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats= snapshot_time: 1788565948.177708749 secs.nsecs start_time: 1788565823.006027107 secs.nsecs elapsed_time: 125.171681642 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 2 5 7 16: 2 5 12 32: 34 87 100 - open/close 558 (time 1788565962.12 total 10.00 last 55.78) - open/close 1119 (time 1788565972.13 total 20.02 last 56.04) - open/close 1662 (time 1788565982.17 total 30.05 last 54.09) - open/close 2189 (time 1788565992.19 total 40.07 last 52.60) - open/close 2785 (time 1788566002.20 total 50.08 last 59.55) - open/close 3370 (time 1788566012.20 total 60.09 last 58.46) - open/close 4002 (time 1788566022.21 total 70.10 last 63.14) - open/close 4653 (time 1788566032.22 total 80.10 last 65.07) - open/close 5258 (time 1788566042.22 total 90.10 last 60.49) - open/close 5888 (time 1788566052.22 total 100.10 last 63.00) - open/close 6426 (time 1788566062.22 total 110.11 last 53.79) - open/close 6901 (time 1788566072.24 total 120.13 last 47.40) - open/close 7407 (time 1788566082.24 total 130.13 last 50.58) - open/close 8021 (time 1788566092.25 total 140.13 last 61.38) - open/close 8624 (time 1788566102.25 total 150.14 last 60.28) - open/close 9240 (time 1788566112.26 total 160.15 last 61.52) - open/close 9822 (time 1788566122.27 total 170.16 last 58.16) total: 10000 open/close in 172.97 seconds: 57.81 ops/second llite.lustre-ffff91e447cf0000.statahead_batch_max=0 llite.lustre-ffff91e447cf0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91e447cf0000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m16.498s user 0m0.366s sys 0m15.031s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff91e447cf0000.statahead_batch_max=32 llite.lustre-ffff91e447cf0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff91e447cf0000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m53.547s user 0m0.301s sys 0m14.011s batched statahead RPC= hit count=9970 mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats= snapshot_time: 1788566333.592700381 secs.nsecs start_time: 1788566209.492089056 secs.nsecs elapsed_time: 124.100611325 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats=0 llite.lustre-ffff91e447cf0000.statahead_max=2048 llite.lustre-ffff91e447cf0000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats=0 total: 100 open/close in 3.22 seconds: 31.03 ops/second llite.lustre-ffff91e447cf0000.statahead_max=0 101 real 0m2.856s user 0m0.010s sys 0m0.329s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff91e447cf0000.statahead_max=2048 llite.lustre-ffff91e447cf0000.statahead_max=2048 101 real 0m0.952s user 0m0.004s sys 0m0.166s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10069 miss_total: 31 swrong=0 ewrong=0 - open/close 594 (time 1788566702.01 total 10.01 last 59.35) total: 900 open/close in 14.80 seconds: 60.82 ops/second llite.lustre-ffff91e447cf0000.statahead_max=0 1001 real 0m28.418s user 0m0.045s sys 0m2.948s 'ls -l' 1000 files without statahead: 28 sec llite.lustre-ffff91e447cf0000.statahead_max=2048 llite.lustre-ffff91e447cf0000.statahead_max=2048 1001 real 0m7.888s user 0m0.027s sys 0m1.325s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11068 miss_total: 32 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff91e447cf0000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11068 miss_total: 32 mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats= snapshot_time: 1788566782.264954305 secs.nsecs start_time: 1788566337.004086375 secs.nsecs elapsed_time: 445.260867930 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 18 18 16: 2 18 36 32: 1 9 45 64: 1 9 54 128: 1 9 63 256: 4 36 100 - open/close 640 (time 1788566794.63 total 10.00 last 63.99) - open/close 1140 (time 1788566804.65 total 20.02 last 49.89) - open/close 1764 (time 1788566814.66 total 30.03 last 62.37) - open/close 2324 (time 1788566824.67 total 40.04 last 55.94) - open/close 2955 (time 1788566834.68 total 50.05 last 63.03) - open/close 3537 (time 1788566844.69 total 60.07 last 58.11) - open/close 4101 (time 1788566854.72 total 70.09 last 56.26) - open/close 4594 (time 1788566864.73 total 80.10 last 49.26) - open/close 5239 (time 1788566874.73 total 90.10 last 64.47) - open/close 5758 (time 1788566884.74 total 100.11 last 51.85) - open/close 6346 (time 1788566894.75 total 110.12 last 58.73) - open/close 6905 (time 1788566904.76 total 120.14 last 55.83) - open/close 7537 (time 1788566914.77 total 130.14 last 63.17) - open/close 8112 (time 1788566924.77 total 140.14 last 57.48) - open/close 8666 (time 1788566934.78 total 150.15 last 55.37) - open/close 9215 (time 1788566944.78 total 160.15 last 54.88) - open/close 9779 (time 1788566954.80 total 170.17 last 56.33) total: 10000 open/close in 173.84 seconds: 57.52 ops/second llite.lustre-ffff91e447cf0000.statahead_batch_max=0 llite.lustre-ffff91e447cf0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91e447cf0000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.464s user 0m0.355s sys 0m14.844s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff91e447cf0000.statahead_batch_max=256 llite.lustre-ffff91e447cf0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff91e447cf0000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.410s user 0m0.326s sys 0m14.792s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff91e447cf0000.batch_stats= snapshot_time: 1788567099.232614771 secs.nsecs start_time: 1788567035.294448274 secs.nsecs elapsed_time: 63.938166497 secs.nsecs subreqs per batch batches %% cum %% 1: 76 38 38 2: 23 11 50 4: 15 7 57 8: 13 6 64 16: 15 7 71 32: 14 7 78 64: 7 3 82 128: 0 0 82 256: 35 17 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff91e447cf0000.statahead_batch_max=64 llite.lustre-ffff91e447cf0000.statahead_max=128