== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:12:06 (1787839926) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats=0 llite.lustre-ffff9a785234d800.statahead_max=32 llite.lustre-ffff9a785234d800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.80 seconds: 35.77 ops/second llite.lustre-ffff9a785234d800.statahead_max=0 101 real 0m2.365s user 0m0.008s sys 0m0.295s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9a785234d800.statahead_max=32 llite.lustre-ffff9a785234d800.statahead_max=32 101 real 0m1.074s user 0m0.005s sys 0m0.167s 'ls -l' 100 files with statahead: 1 sec statahead total: 117 statahead wrong: 2 agl total: 111 list_total: 31 fname_total: 71 hit_total: 3873 miss_total: 1545 swrong=2 ewrong=2 - open/close 817 (time 1787839954.52 total 10.01 last 81.63) total: 900 open/close in 10.96 seconds: 82.14 ops/second llite.lustre-ffff9a785234d800.statahead_max=0 1001 real 0m20.967s user 0m0.032s sys 0m2.925s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9a785234d800.statahead_max=32 llite.lustre-ffff9a785234d800.statahead_max=32 1001 real 0m9.549s user 0m0.035s sys 0m1.409s 'ls -l' 1000 files with statahead: 9 sec statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 32 fname_total: 71 hit_total: 4871 miss_total: 1547 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9a785234d800.statahead_stats= statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 32 fname_total: 71 hit_total: 4871 miss_total: 1547 mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840023.622010196 secs.nsecs start_time: 1787839931.698434613 secs.nsecs elapsed_time: 91.923575583 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 4 10 17 16: 20 51 69 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840023.622076220 secs.nsecs start_time: 1787839931.698564935 secs.nsecs elapsed_time: 91.923511285 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 2 5 10 8: 1 2 12 16: 23 58 71 32: 11 28 100 - open/close 807 (time 1787840037.21 total 10.01 last 80.65) - open/close 1646 (time 1787840047.22 total 20.01 last 83.83) - open/close 2438 (time 1787840057.23 total 30.03 last 79.07) - open/close 3248 (time 1787840067.24 total 40.04 last 80.96) - open/close 4052 (time 1787840077.24 total 50.04 last 80.39) - open/close 4854 (time 1787840087.24 total 60.04 last 80.15) - open/close 5658 (time 1787840097.25 total 70.05 last 80.35) - open/close 6521 (time 1787840107.25 total 80.05 last 86.28) - open/close 7264 (time 1787840117.26 total 90.06 last 74.25) - open/close 8009 (time 1787840127.26 total 100.06 last 74.47) - open/close 8747 (time 1787840137.26 total 110.06 last 73.79) - open/close 9512 (time 1787840147.28 total 120.07 last 76.41) total: 10000 open/close in 125.33 seconds: 79.79 ops/second llite.lustre-ffff9a785234d800.statahead_batch_max=0 llite.lustre-ffff9a785234d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a785234d800.stats=clear mdc.lustre-MDT0001-mdc-ffff9a785234d800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.511s user 0m0.301s sys 0m14.324s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9a785234d800.statahead_batch_max=32 llite.lustre-ffff9a785234d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9a785234d800.stats=clear mdc.lustre-MDT0001-mdc-ffff9a785234d800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m38.229s user 0m0.321s sys 0m13.623s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840328.869340037 secs.nsecs start_time: 1787840220.225808900 secs.nsecs elapsed_time: 108.643531137 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840328.869408165 secs.nsecs start_time: 1787840220.225907009 secs.nsecs elapsed_time: 108.643501156 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats=0 llite.lustre-ffff9a785234d800.statahead_max=2048 llite.lustre-ffff9a785234d800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.88 seconds: 34.77 ops/second llite.lustre-ffff9a785234d800.statahead_max=0 101 real 0m3.063s user 0m0.007s sys 0m0.366s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9a785234d800.statahead_max=2048 llite.lustre-ffff9a785234d800.statahead_max=2048 101 real 0m0.808s user 0m0.004s sys 0m0.170s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 - open/close 797 (time 1787840623.54 total 10.01 last 79.63) total: 900 open/close in 11.22 seconds: 80.21 ops/second llite.lustre-ffff9a785234d800.statahead_max=0 1001 real 0m22.007s user 0m0.036s sys 0m3.040s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9a785234d800.statahead_max=2048 llite.lustre-ffff9a785234d800.statahead_max=2048 1001 real 0m5.255s user 0m0.023s sys 0m1.370s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9a785234d800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840684.122407132 secs.nsecs start_time: 1787840331.135752583 secs.nsecs elapsed_time: 352.986654549 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840684.122483886 secs.nsecs start_time: 1787840331.135932625 secs.nsecs elapsed_time: 352.986551261 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 2 22 33 8: 1 11 44 16: 3 33 77 32: 0 0 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 840 (time 1787840696.52 total 10.00 last 83.98) - open/close 1668 (time 1787840706.53 total 20.01 last 82.73) - open/close 2412 (time 1787840716.55 total 30.03 last 74.24) - open/close 3167 (time 1787840726.56 total 40.04 last 75.43) - open/close 3936 (time 1787840736.56 total 50.05 last 76.86) - open/close 4572 (time 1787840746.57 total 60.05 last 63.57) - open/close 5238 (time 1787840756.57 total 70.06 last 66.58) - open/close 5912 (time 1787840766.58 total 80.06 last 67.35) - open/close 6751 (time 1787840776.58 total 90.07 last 83.86) - open/close 7601 (time 1787840786.59 total 100.08 last 84.93) - open/close 8442 (time 1787840796.60 total 110.08 last 84.05) - open/close 9042 (time 1787840806.62 total 120.10 last 59.89) - open/close 9502 (time 1787840816.63 total 130.12 last 45.94) total: 10000 open/close in 138.81 seconds: 72.04 ops/second llite.lustre-ffff9a785234d800.statahead_batch_max=0 llite.lustre-ffff9a785234d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a785234d800.stats=clear mdc.lustre-MDT0001-mdc-ffff9a785234d800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.869s user 0m0.311s sys 0m14.661s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9a785234d800.statahead_batch_max=256 llite.lustre-ffff9a785234d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9a785234d800.stats=clear mdc.lustre-MDT0001-mdc-ffff9a785234d800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.864s user 0m0.333s sys 0m15.242s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840966.901207059 secs.nsecs start_time: 1787840898.817555271 secs.nsecs elapsed_time: 68.083651788 secs.nsecs subreqs per batch batches %% cum %% 1: 123 45 45 2: 41 15 60 4: 24 8 69 8: 14 5 74 16: 12 4 78 32: 14 5 83 64: 9 3 87 128: 1 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff9a785234d800.batch_stats= snapshot_time: 1787840966.901278259 secs.nsecs start_time: 1787840898.817725881 secs.nsecs elapsed_time: 68.083552378 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9a785234d800.statahead_batch_max=64 llite.lustre-ffff9a785234d800.statahead_max=128