== sanity test 123ad: Verify batching statahead works correctly ========================================================== 02:22:02 (1787120522) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats=0 llite.lustre-ffff889f485f2000.statahead_max=32 llite.lustre-ffff889f485f2000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.23 seconds: 44.76 ops/second llite.lustre-ffff889f485f2000.statahead_max=0 101 real 0m1.992s user 0m0.004s sys 0m0.334s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff889f485f2000.statahead_max=32 llite.lustre-ffff889f485f2000.statahead_max=32 101 real 0m0.904s user 0m0.004s sys 0m0.198s 'ls -l' 100 files with statahead: 1 sec statahead total: 155 statahead wrong: 2 agl total: 149 list_total: 30 fname_total: 112 hit_total: 3772 miss_total: 1436 swrong=2 ewrong=2 - open/close 882 (time 1787120546.73 total 10.01 last 88.16) total: 900 open/close in 10.28 seconds: 87.58 ops/second llite.lustre-ffff889f485f2000.statahead_max=0 1001 real 0m21.656s user 0m0.033s sys 0m3.143s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff889f485f2000.statahead_max=32 llite.lustre-ffff889f485f2000.statahead_max=32 1001 real 0m9.032s user 0m0.035s sys 0m1.593s 'ls -l' 1000 files with statahead: 9 sec statahead total: 156 statahead wrong: 2 agl total: 150 list_total: 31 fname_total: 112 hit_total: 4771 miss_total: 1437 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff889f485f2000.statahead_stats= statahead total: 156 statahead wrong: 2 agl total: 150 list_total: 31 fname_total: 112 hit_total: 4771 miss_total: 1437 mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787120610.390075035 secs.nsecs start_time: 1787120526.248106491 secs.nsecs elapsed_time: 84.141968544 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 3 7 7 4: 2 5 12 8: 1 2 15 16: 20 51 66 32: 13 33 100 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787120610.390143055 secs.nsecs start_time: 1787120526.248301026 secs.nsecs elapsed_time: 84.141842029 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 5 12 17 16: 19 48 66 32: 13 33 100 - open/close 747 (time 1787120623.32 total 10.01 last 74.62) - open/close 1458 (time 1787120633.32 total 20.01 last 71.08) - open/close 2240 (time 1787120643.33 total 30.02 last 78.17) - open/close 2992 (time 1787120653.34 total 40.03 last 75.09) - open/close 3642 (time 1787120663.36 total 50.05 last 64.86) - open/close 4541 (time 1787120673.37 total 60.06 last 89.85) - open/close 5417 (time 1787120683.37 total 70.06 last 87.56) - open/close 6246 (time 1787120693.37 total 80.07 last 82.90) - open/close 7082 (time 1787120703.39 total 90.08 last 83.49) - open/close 7866 (time 1787120713.39 total 100.08 last 78.39) - open/close 8628 (time 1787120723.39 total 110.09 last 76.14) - open/close 9408 (time 1787120733.40 total 120.09 last 77.99) total: 10000 open/close in 127.51 seconds: 78.43 ops/second llite.lustre-ffff889f485f2000.statahead_batch_max=0 llite.lustre-ffff889f485f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff889f485f2000.stats=clear mdc.lustre-MDT0001-mdc-ffff889f485f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.000s user 0m0.307s sys 0m17.590s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff889f485f2000.statahead_batch_max=32 llite.lustre-ffff889f485f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff889f485f2000.stats=clear mdc.lustre-MDT0001-mdc-ffff889f485f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.119s user 0m0.261s sys 0m16.021s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787120929.921095919 secs.nsecs start_time: 1787120816.315747164 secs.nsecs elapsed_time: 113.605348755 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787120929.921160629 secs.nsecs start_time: 1787120816.315823256 secs.nsecs elapsed_time: 113.605337373 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats=0 llite.lustre-ffff889f485f2000.statahead_max=2048 llite.lustre-ffff889f485f2000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.23 seconds: 44.84 ops/second llite.lustre-ffff889f485f2000.statahead_max=0 101 real 0m2.116s user 0m0.003s sys 0m0.333s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff889f485f2000.statahead_max=2048 llite.lustre-ffff889f485f2000.statahead_max=2048 101 real 0m0.732s user 0m0.005s sys 0m0.204s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 - open/close 892 (time 1787121233.44 total 10.00 last 89.18) total: 900 open/close in 10.10 seconds: 89.12 ops/second llite.lustre-ffff889f485f2000.statahead_max=0 1001 real 0m20.733s user 0m0.040s sys 0m3.110s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff889f485f2000.statahead_max=2048 llite.lustre-ffff889f485f2000.statahead_max=2048 1001 real 0m6.555s user 0m0.027s sys 0m1.645s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff889f485f2000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787121292.215221032 secs.nsecs start_time: 1787120932.217269156 secs.nsecs elapsed_time: 359.997951876 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 3 30 50 16: 2 20 70 32: 0 0 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787121292.215288517 secs.nsecs start_time: 1787120932.217673838 secs.nsecs elapsed_time: 359.997614679 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 2 20 30 8: 2 20 50 16: 1 10 60 32: 2 20 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 867 (time 1787121304.33 total 10.00 last 86.70) - open/close 1717 (time 1787121314.34 total 20.00 last 84.99) - open/close 2494 (time 1787121324.34 total 30.01 last 77.63) - open/close 3271 (time 1787121334.36 total 40.02 last 77.59) - open/close 4150 (time 1787121344.36 total 50.03 last 87.88) - open/close 5058 (time 1787121354.37 total 60.04 last 90.71) - open/close 5930 (time 1787121364.37 total 70.04 last 87.18) - open/close 6796 (time 1787121374.37 total 80.04 last 86.59) - open/close 7642 (time 1787121384.39 total 90.05 last 84.48) - open/close 8475 (time 1787121394.39 total 100.06 last 83.25) - open/close 9282 (time 1787121404.41 total 110.08 last 80.53) total: 10000 open/close in 119.17 seconds: 83.92 ops/second llite.lustre-ffff889f485f2000.statahead_batch_max=0 llite.lustre-ffff889f485f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff889f485f2000.stats=clear mdc.lustre-MDT0001-mdc-ffff889f485f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.602s user 0m0.281s sys 0m17.628s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff889f485f2000.statahead_batch_max=256 llite.lustre-ffff889f485f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff889f485f2000.stats=clear mdc.lustre-MDT0001-mdc-ffff889f485f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.923s user 0m0.278s sys 0m17.484s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787121554.876044404 secs.nsecs start_time: 1787121486.712349366 secs.nsecs elapsed_time: 68.163695038 secs.nsecs subreqs per batch batches %% cum %% 1: 273 57 57 2: 62 12 70 4: 51 10 80 8: 26 5 86 16: 8 1 87 32: 19 3 91 64: 6 1 93 128: 0 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff889f485f2000.batch_stats= snapshot_time: 1787121554.876110898 secs.nsecs start_time: 1787121486.713853276 secs.nsecs elapsed_time: 68.162257622 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff889f485f2000.statahead_batch_max=64 llite.lustre-ffff889f485f2000.statahead_max=128