== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:38:11 (1779874691) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats=0 llite.lustre-ffff8b21cac07000.statahead_max=32 llite.lustre-ffff8b21cac07000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.02 seconds: 49.49 ops/second llite.lustre-ffff8b21cac07000.statahead_max=0 101 real 0m2.411s user 0m0.004s sys 0m0.324s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8b21cac07000.statahead_max=32 llite.lustre-ffff8b21cac07000.statahead_max=32 101 real 0m1.172s user 0m0.003s sys 0m0.167s 'ls -l' 100 files with statahead: 2 sec statahead total: 118 statahead wrong: 2 agl total: 113 list_total: 31 fname_total: 70 hit_total: 3842 miss_total: 1752 swrong=2 ewrong=2 - open/close 880 (time 1779874719.66 total 10.00 last 87.97) total: 900 open/close in 10.25 seconds: 87.77 ops/second llite.lustre-ffff8b21cac07000.statahead_max=0 1001 real 0m23.593s user 0m0.034s sys 0m3.139s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff8b21cac07000.statahead_max=32 llite.lustre-ffff8b21cac07000.statahead_max=32 1001 real 0m9.975s user 0m0.026s sys 0m1.413s 'ls -l' 1000 files with statahead: 10 sec statahead total: 119 statahead wrong: 2 agl total: 114 list_total: 32 fname_total: 70 hit_total: 4839 miss_total: 1755 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8b21cac07000.statahead_stats= statahead total: 119 statahead wrong: 2 agl total: 114 list_total: 32 fname_total: 70 hit_total: 4839 miss_total: 1755 mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779874783.654505712 secs.nsecs start_time: 1779874696.589144478 secs.nsecs elapsed_time: 87.065361234 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 1 2 10 16: 19 50 60 32: 15 39 100 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779874783.654572812 secs.nsecs start_time: 1779874696.589784282 secs.nsecs elapsed_time: 87.064788530 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 1 2 8 16: 21 56 64 32: 13 35 100 - open/close 852 (time 1779874796.14 total 10.01 last 85.09) - open/close 1677 (time 1779874806.15 total 20.02 last 82.46) - open/close 2735 (time 1779874816.15 total 30.02 last 105.73) - open/close 3646 (time 1779874826.16 total 40.03 last 91.08) - open/close 4703 (time 1779874836.16 total 50.03 last 105.64) - open/close 5644 (time 1779874846.17 total 60.04 last 94.04) - open/close 6524 (time 1779874856.18 total 70.05 last 87.88) - open/close 7405 (time 1779874866.19 total 80.06 last 88.06) - open/close 8326 (time 1779874876.20 total 90.07 last 91.95) - open/close 9159 (time 1779874886.20 total 100.07 last 83.29) total: 10000 open/close in 108.88 seconds: 91.85 ops/second llite.lustre-ffff8b21cac07000.statahead_batch_max=0 llite.lustre-ffff8b21cac07000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b21cac07000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b21cac07000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.850s user 0m0.297s sys 0m15.172s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b21cac07000.statahead_batch_max=32 llite.lustre-ffff8b21cac07000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b21cac07000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b21cac07000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m39.782s user 0m0.300s sys 0m14.873s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779875085.496064431 secs.nsecs start_time: 1779874973.414988602 secs.nsecs elapsed_time: 112.081075829 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779875085.496137641 secs.nsecs start_time: 1779874973.415057366 secs.nsecs elapsed_time: 112.081080275 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats=0 llite.lustre-ffff8b21cac07000.statahead_max=2048 llite.lustre-ffff8b21cac07000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.90 seconds: 52.67 ops/second llite.lustre-ffff8b21cac07000.statahead_max=0 101 real 0m1.973s user 0m0.005s sys 0m0.322s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8b21cac07000.statahead_max=2048 llite.lustre-ffff8b21cac07000.statahead_max=2048 101 real 0m0.747s user 0m0.005s sys 0m0.157s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 total: 900 open/close in 9.57 seconds: 94.08 ops/second llite.lustre-ffff8b21cac07000.statahead_max=0 1001 real 0m18.228s user 0m0.032s sys 0m2.954s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff8b21cac07000.statahead_max=2048 llite.lustre-ffff8b21cac07000.statahead_max=2048 1001 real 0m5.634s user 0m0.024s sys 0m1.378s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8b21cac07000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779875414.100490329 secs.nsecs start_time: 1779875088.292878886 secs.nsecs elapsed_time: 325.807611443 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 2 28 28 4: 2 28 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779875414.100546150 secs.nsecs start_time: 1779875088.293044639 secs.nsecs elapsed_time: 325.807501511 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 27 27 16: 3 27 54 32: 2 18 72 64: 1 9 81 128: 0 0 81 256: 2 18 100 - open/close 655 (time 1779875426.62 total 10.01 last 65.42) - open/close 1109 (time 1779875436.63 total 20.02 last 45.37) - open/close 2071 (time 1779875446.63 total 30.02 last 96.16) - open/close 2987 (time 1779875456.65 total 40.04 last 91.40) - open/close 3840 (time 1779875466.66 total 50.05 last 85.29) - open/close 4844 (time 1779875476.66 total 60.05 last 100.39) - open/close 5885 (time 1779875486.67 total 70.06 last 104.01) - open/close 6924 (time 1779875496.67 total 80.06 last 103.82) - open/close 7866 (time 1779875506.68 total 90.07 last 94.13) - open/close 8899 (time 1779875516.69 total 100.08 last 103.23) - open/close 9806 (time 1779875526.69 total 110.08 last 90.66) total: 10000 open/close in 112.13 seconds: 89.18 ops/second llite.lustre-ffff8b21cac07000.statahead_batch_max=0 llite.lustre-ffff8b21cac07000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b21cac07000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b21cac07000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.820s user 0m0.308s sys 0m15.006s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b21cac07000.statahead_batch_max=256 llite.lustre-ffff8b21cac07000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b21cac07000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b21cac07000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.767s user 0m0.305s sys 0m15.367s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779875669.512245031 secs.nsecs start_time: 1779875598.682493651 secs.nsecs elapsed_time: 70.829751380 secs.nsecs subreqs per batch batches % cum % 1: 197 51 51 2: 54 14 66 4: 33 8 74 8: 21 5 80 16: 20 5 85 32: 12 3 88 64: 7 1 90 128: 1 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff8b21cac07000.batch_stats= snapshot_time: 1779875669.512311524 secs.nsecs start_time: 1779875598.682575418 secs.nsecs elapsed_time: 70.829736106 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8b21cac07000.statahead_batch_max=64 llite.lustre-ffff8b21cac07000.statahead_max=128