== sanity test 123ad: Verify batching statahead works correctly ========================================================== 15:30:53 (1780342253) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats=0 llite.lustre-ffff9b9887dae800.statahead_max=32 llite.lustre-ffff9b9887dae800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.02 seconds: 49.50 ops/second llite.lustre-ffff9b9887dae800.statahead_max=0 101 real 0m1.526s user 0m0.002s sys 0m0.292s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9b9887dae800.statahead_max=32 llite.lustre-ffff9b9887dae800.statahead_max=32 101 real 0m0.646s user 0m0.004s sys 0m0.177s 'ls -l' 100 files with statahead: 1 sec statahead total: 117 statahead wrong: 2 agl total: 111 list_total: 30 fname_total: 84 hit_total: 3969 miss_total: 1619 swrong=2 ewrong=2 total: 900 open/close in 8.16 seconds: 110.28 ops/second llite.lustre-ffff9b9887dae800.statahead_max=0 1001 real 0m16.180s user 0m0.033s sys 0m3.096s 'ls -l' 1000 files without statahead: 16 sec llite.lustre-ffff9b9887dae800.statahead_max=32 llite.lustre-ffff9b9887dae800.statahead_max=32 1001 real 0m6.110s user 0m0.023s sys 0m1.487s 'ls -l' 1000 files with statahead: 6 sec statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 31 fname_total: 84 hit_total: 4966 miss_total: 1622 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 13 seconds llite.lustre-ffff9b9887dae800.statahead_stats= statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 31 fname_total: 84 hit_total: 4966 miss_total: 1622 mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342314.677592615 secs.nsecs start_time: 1780342256.860880742 secs.nsecs elapsed_time: 57.816711873 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 3 7 17 16: 17 42 60 32: 16 40 100 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342314.677693071 secs.nsecs start_time: 1780342256.861062252 secs.nsecs elapsed_time: 57.816630819 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 1 2 12 16: 20 51 64 32: 14 35 100 - open/close 1112 (time 1780342326.01 total 10.00 last 111.19) - open/close 2252 (time 1780342336.01 total 20.00 last 113.98) - open/close 3392 (time 1780342346.01 total 30.00 last 113.98) - open/close 4466 (time 1780342356.01 total 40.01 last 107.36) - open/close 5669 (time 1780342366.02 total 50.01 last 120.27) - open/close 6832 (time 1780342376.02 total 60.02 last 116.24) - open/close 7924 (time 1780342386.03 total 70.02 last 109.13) - open/close 9061 (time 1780342396.03 total 80.02 last 113.68) total: 10000 open/close in 88.16 seconds: 113.43 ops/second llite.lustre-ffff9b9887dae800.statahead_batch_max=0 llite.lustre-ffff9b9887dae800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9887dae800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9887dae800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m39.265s user 0m0.283s sys 0m19.869s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b9887dae800.statahead_batch_max=32 llite.lustre-ffff9b9887dae800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9887dae800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9887dae800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.107s user 0m0.267s sys 0m15.584s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342513.947911084 secs.nsecs start_time: 1780342448.572273453 secs.nsecs elapsed_time: 65.375637631 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342513.947971102 secs.nsecs start_time: 1780342448.572354058 secs.nsecs elapsed_time: 65.375617044 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats=0 llite.lustre-ffff9b9887dae800.statahead_max=2048 llite.lustre-ffff9b9887dae800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.46 seconds: 68.37 ops/second llite.lustre-ffff9b9887dae800.statahead_max=0 101 real 0m1.674s user 0m0.009s sys 0m0.298s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b9887dae800.statahead_max=2048 llite.lustre-ffff9b9887dae800.statahead_max=2048 101 real 0m0.481s user 0m0.003s sys 0m0.221s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 total: 900 open/close in 4.36 seconds: 206.36 ops/second llite.lustre-ffff9b9887dae800.statahead_max=0 1001 real 0m11.647s user 0m0.032s sys 0m3.047s 'ls -l' 1000 files without statahead: 12 sec llite.lustre-ffff9b9887dae800.statahead_max=2048 llite.lustre-ffff9b9887dae800.statahead_max=2048 1001 real 0m3.678s user 0m0.025s sys 0m1.525s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 11 seconds llite.lustre-ffff9b9887dae800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342733.045244822 secs.nsecs start_time: 1780342515.234575872 secs.nsecs elapsed_time: 217.810668950 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 2 33 33 8: 2 33 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342733.045295255 secs.nsecs start_time: 1780342515.234704678 secs.nsecs elapsed_time: 217.810590577 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 2 22 33 16: 2 22 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 1260 (time 1780342744.84 total 10.00 last 125.95) - open/close 2628 (time 1780342754.85 total 20.01 last 136.67) - open/close 4048 (time 1780342764.86 total 30.02 last 141.85) - open/close 5451 (time 1780342774.86 total 40.02 last 140.29) - open/close 7008 (time 1780342784.87 total 50.03 last 155.56) - open/close 8439 (time 1780342794.87 total 60.04 last 143.02) - open/close 9888 (time 1780342804.88 total 70.04 last 144.83) total: 10000 open/close in 70.80 seconds: 141.25 ops/second llite.lustre-ffff9b9887dae800.statahead_batch_max=0 llite.lustre-ffff9b9887dae800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9887dae800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9887dae800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.399s user 0m0.290s sys 0m21.693s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b9887dae800.statahead_batch_max=256 llite.lustre-ffff9b9887dae800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9887dae800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9887dae800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.252s user 0m0.295s sys 0m15.367s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342929.889100581 secs.nsecs start_time: 1780342858.283189298 secs.nsecs elapsed_time: 71.605911283 secs.nsecs subreqs per batch batches % cum % 1: 186 46 46 2: 73 18 64 4: 52 12 76 8: 20 4 81 16: 11 2 84 32: 19 4 89 64: 7 1 91 128: 4 0 92 256: 32 7 100 mdc.lustre-MDT0001-mdc-ffff9b9887dae800.batch_stats= snapshot_time: 1780342929.889169362 secs.nsecs start_time: 1780342858.283273270 secs.nsecs elapsed_time: 71.605896092 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b9887dae800.statahead_batch_max=64 llite.lustre-ffff9b9887dae800.statahead_max=128