== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:49:19 (1788648559) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats=0 llite.lustre-ffff9c67c9c3c000.statahead_max=32 llite.lustre-ffff9c67c9c3c000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.55 seconds: 39.20 ops/second llite.lustre-ffff9c67c9c3c000.statahead_max=0 101 real 0m2.177s user 0m0.006s sys 0m0.326s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9c67c9c3c000.statahead_max=32 llite.lustre-ffff9c67c9c3c000.statahead_max=32 101 real 0m1.020s user 0m0.004s sys 0m0.162s 'ls -l' 100 files with statahead: 1 sec statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 31 fname_total: 99 hit_total: 4462 miss_total: 1295 swrong=2 ewrong=2 - open/close 662 (time 1788648584.77 total 10.02 last 66.04) total: 900 open/close in 13.01 seconds: 69.20 ops/second llite.lustre-ffff9c67c9c3c000.statahead_max=0 1001 real 0m26.642s user 0m0.036s sys 0m3.002s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9c67c9c3c000.statahead_max=32 llite.lustre-ffff9c67c9c3c000.statahead_max=32 1001 real 0m9.608s user 0m0.036s sys 0m1.440s 'ls -l' 1000 files with statahead: 10 sec statahead total: 132 statahead wrong: 2 agl total: 126 list_total: 32 fname_total: 99 hit_total: 5458 miss_total: 1299 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9c67c9c3c000.statahead_stats= statahead total: 132 statahead wrong: 2 agl total: 126 list_total: 32 fname_total: 99 hit_total: 5458 miss_total: 1299 mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788648661.676937950 secs.nsecs start_time: 1788648563.317026980 secs.nsecs elapsed_time: 98.359910970 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 1 2 11 8: 3 8 19 16: 20 55 75 32: 9 25 100 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788648661.677012581 secs.nsecs start_time: 1788648563.317354506 secs.nsecs elapsed_time: 98.359658075 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 2 5 12 16: 11 28 41 32: 23 58 100 - open/close 624 (time 1788648674.50 total 10.01 last 62.37) - open/close 1230 (time 1788648684.52 total 20.02 last 60.52) - open/close 1868 (time 1788648694.52 total 30.02 last 63.79) - open/close 2624 (time 1788648704.52 total 40.03 last 75.56) - open/close 3329 (time 1788648714.54 total 50.04 last 70.42) - open/close 4025 (time 1788648724.56 total 60.06 last 69.44) - open/close 4676 (time 1788648734.57 total 70.07 last 65.01) - open/close 5425 (time 1788648744.58 total 80.08 last 74.89) - open/close 6212 (time 1788648754.58 total 90.08 last 78.63) - open/close 6935 (time 1788648764.59 total 100.09 last 72.25) - open/close 7554 (time 1788648774.60 total 110.10 last 61.87) - open/close 8108 (time 1788648784.61 total 120.11 last 55.35) - open/close 8607 (time 1788648794.61 total 130.11 last 49.89) - open/close 9210 (time 1788648804.63 total 140.13 last 60.17) - open/close 9948 (time 1788648814.64 total 150.14 last 73.71) total: 10000 open/close in 150.94 seconds: 66.25 ops/second llite.lustre-ffff9c67c9c3c000.statahead_batch_max=0 llite.lustre-ffff9c67c9c3c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.308s user 0m0.308s sys 0m15.628s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9c67c9c3c000.statahead_batch_max=32 llite.lustre-ffff9c67c9c3c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m30.745s user 0m0.346s sys 0m14.595s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788648987.062615733 secs.nsecs start_time: 1788648886.933834243 secs.nsecs elapsed_time: 100.128781490 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788648987.062682179 secs.nsecs start_time: 1788648886.933934040 secs.nsecs elapsed_time: 100.128748139 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats=0 llite.lustre-ffff9c67c9c3c000.statahead_max=2048 llite.lustre-ffff9c67c9c3c000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.96 seconds: 33.83 ops/second llite.lustre-ffff9c67c9c3c000.statahead_max=0 101 real 0m2.522s user 0m0.003s sys 0m0.355s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9c67c9c3c000.statahead_max=2048 llite.lustre-ffff9c67c9c3c000.statahead_max=2048 101 real 0m0.987s user 0m0.005s sys 0m0.208s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 - open/close 719 (time 1788649329.65 total 10.00 last 71.89) total: 900 open/close in 12.63 seconds: 71.27 ops/second llite.lustre-ffff9c67c9c3c000.statahead_max=0 1001 real 0m21.999s user 0m0.029s sys 0m2.857s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9c67c9c3c000.statahead_max=2048 llite.lustre-ffff9c67c9c3c000.statahead_max=2048 1001 real 0m5.281s user 0m0.024s sys 0m1.544s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9c67c9c3c000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788649387.237742995 secs.nsecs start_time: 1788648990.640371085 secs.nsecs elapsed_time: 396.597371910 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 2 28 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788649387.238028987 secs.nsecs start_time: 1788648990.640850896 secs.nsecs elapsed_time: 396.597178091 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 3 33 44 16: 1 11 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 658 (time 1788649400.05 total 10.01 last 65.71) - open/close 1410 (time 1788649410.06 total 20.02 last 75.16) - open/close 2116 (time 1788649420.08 total 30.04 last 70.45) - open/close 2912 (time 1788649430.08 total 40.04 last 79.57) - open/close 3714 (time 1788649440.09 total 50.05 last 80.15) - open/close 4523 (time 1788649450.10 total 60.06 last 80.81) - open/close 5315 (time 1788649460.11 total 70.08 last 79.07) - open/close 6059 (time 1788649470.12 total 80.08 last 74.36) - open/close 6839 (time 1788649480.13 total 90.09 last 77.94) - open/close 7619 (time 1788649490.13 total 100.10 last 77.95) - open/close 8392 (time 1788649500.13 total 110.10 last 77.30) - open/close 9132 (time 1788649510.15 total 120.12 last 73.87) - open/close 9941 (time 1788649520.15 total 130.12 last 80.88) total: 10000 open/close in 130.80 seconds: 76.45 ops/second llite.lustre-ffff9c67c9c3c000.statahead_batch_max=0 llite.lustre-ffff9c67c9c3c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.376s user 0m0.304s sys 0m15.931s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9c67c9c3c000.statahead_batch_max=256 llite.lustre-ffff9c67c9c3c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m46.503s user 0m0.312s sys 0m15.866s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788649653.235732275 secs.nsecs start_time: 1788649597.701574905 secs.nsecs elapsed_time: 55.534157370 secs.nsecs subreqs per batch batches %% cum %% 1: 145 48 48 2: 42 14 63 4: 38 12 76 8: 4 1 77 16: 11 3 81 32: 12 4 85 64: 8 2 87 128: 2 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff9c67c9c3c000.batch_stats= snapshot_time: 1788649653.235808065 secs.nsecs start_time: 1788649597.701721325 secs.nsecs elapsed_time: 55.534086740 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9c67c9c3c000.statahead_batch_max=64 llite.lustre-ffff9c67c9c3c000.statahead_max=128