== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:07:59 (1788624479) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats=0 llite.lustre-ffff9ccb42c9b800.statahead_max=32 llite.lustre-ffff9ccb42c9b800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.27 seconds: 44.07 ops/second llite.lustre-ffff9ccb42c9b800.statahead_max=0 101 real 0m2.164s user 0m0.004s sys 0m0.308s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ccb42c9b800.statahead_max=32 llite.lustre-ffff9ccb42c9b800.statahead_max=32 101 real 0m0.832s user 0m0.005s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 119 statahead wrong: 2 agl total: 113 list_total: 31 fname_total: 88 hit_total: 4107 miss_total: 777 swrong=2 ewrong=2 total: 900 open/close in 9.55 seconds: 94.26 ops/second llite.lustre-ffff9ccb42c9b800.statahead_max=0 1001 real 0m20.439s user 0m0.033s sys 0m2.878s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9ccb42c9b800.statahead_max=32 llite.lustre-ffff9ccb42c9b800.statahead_max=32 1001 real 0m8.548s user 0m0.034s sys 0m1.434s 'ls -l' 1000 files with statahead: 9 sec statahead total: 120 statahead wrong: 2 agl total: 114 list_total: 32 fname_total: 88 hit_total: 5106 miss_total: 778 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9ccb42c9b800.statahead_stats= statahead total: 120 statahead wrong: 2 agl total: 114 list_total: 32 fname_total: 88 hit_total: 5106 miss_total: 778 mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788624563.306332884 secs.nsecs start_time: 1788624483.809129325 secs.nsecs elapsed_time: 79.497203559 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 1 2 7 8: 3 7 15 16: 16 42 57 32: 16 42 100 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788624563.306416949 secs.nsecs start_time: 1788624483.809276164 secs.nsecs elapsed_time: 79.497140785 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 1 2 7 4: 0 0 7 8: 3 7 15 16: 25 62 77 32: 9 22 100 - open/close 805 (time 1788624576.02 total 10.01 last 80.44) - open/close 1660 (time 1788624586.02 total 20.01 last 85.45) - open/close 2430 (time 1788624596.03 total 30.02 last 76.92) - open/close 3248 (time 1788624606.05 total 40.04 last 81.66) - open/close 4015 (time 1788624616.05 total 50.04 last 76.70) - open/close 4882 (time 1788624626.05 total 60.04 last 86.67) - open/close 5702 (time 1788624636.06 total 70.05 last 81.94) - open/close 6502 (time 1788624646.08 total 80.07 last 79.84) - open/close 7324 (time 1788624656.09 total 90.08 last 82.15) - open/close 8144 (time 1788624666.10 total 100.10 last 81.85) - open/close 9022 (time 1788624676.13 total 110.12 last 87.57) - open/close 9823 (time 1788624686.14 total 120.13 last 79.99) total: 10000 open/close in 122.13 seconds: 81.88 ops/second llite.lustre-ffff9ccb42c9b800.statahead_batch_max=0 llite.lustre-ffff9ccb42c9b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.681s user 0m0.312s sys 0m15.529s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ccb42c9b800.statahead_batch_max=32 llite.lustre-ffff9ccb42c9b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m39.888s user 0m0.305s sys 0m13.684s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788624867.553367417 secs.nsecs start_time: 1788624756.835266644 secs.nsecs elapsed_time: 110.718100773 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788624867.553597969 secs.nsecs start_time: 1788624756.835367449 secs.nsecs elapsed_time: 110.718230520 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats=0 llite.lustre-ffff9ccb42c9b800.statahead_max=2048 llite.lustre-ffff9ccb42c9b800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.45 seconds: 40.86 ops/second llite.lustre-ffff9ccb42c9b800.statahead_max=0 101 real 0m1.697s user 0m0.006s sys 0m0.299s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ccb42c9b800.statahead_max=2048 llite.lustre-ffff9ccb42c9b800.statahead_max=2048 101 real 0m0.735s user 0m0.005s sys 0m0.166s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 total: 900 open/close in 9.61 seconds: 93.64 ops/second llite.lustre-ffff9ccb42c9b800.statahead_max=0 1001 real 0m23.557s user 0m0.034s sys 0m2.861s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9ccb42c9b800.statahead_max=2048 llite.lustre-ffff9ccb42c9b800.statahead_max=2048 1001 real 0m4.809s user 0m0.021s sys 0m1.507s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9ccb42c9b800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788625216.860023191 secs.nsecs start_time: 1788624870.444758334 secs.nsecs elapsed_time: 346.415264857 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 22 22 4: 1 11 33 8: 1 11 44 16: 3 33 77 32: 0 0 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788625216.860097673 secs.nsecs start_time: 1788624870.446241708 secs.nsecs elapsed_time: 346.413855965 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 0 0 11 8: 4 44 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 873 (time 1788625229.07 total 10.00 last 87.28) - open/close 1778 (time 1788625239.10 total 20.04 last 90.19) - open/close 2589 (time 1788625249.11 total 30.04 last 81.05) - open/close 3532 (time 1788625259.11 total 40.05 last 94.27) - open/close 4434 (time 1788625269.13 total 50.06 last 90.09) - open/close 5354 (time 1788625279.13 total 60.07 last 91.93) - open/close 6204 (time 1788625289.15 total 70.08 last 84.88) - open/close 7059 (time 1788625299.15 total 80.09 last 85.45) - open/close 7960 (time 1788625309.16 total 90.09 last 90.09) - open/close 8825 (time 1788625319.16 total 100.09 last 86.45) - open/close 9748 (time 1788625329.16 total 110.10 last 92.28) total: 10000 open/close in 112.66 seconds: 88.76 ops/second llite.lustre-ffff9ccb42c9b800.statahead_batch_max=0 llite.lustre-ffff9ccb42c9b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.599s user 0m0.269s sys 0m16.375s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ccb42c9b800.statahead_batch_max=256 llite.lustre-ffff9ccb42c9b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.240s user 0m0.288s sys 0m15.041s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788625450.558192760 secs.nsecs start_time: 1788625390.907157721 secs.nsecs elapsed_time: 59.651035039 secs.nsecs subreqs per batch batches %% cum %% 1: 98 43 43 2: 31 13 57 4: 17 7 64 8: 10 4 69 16: 9 4 73 32: 22 9 83 64: 3 1 84 128: 0 0 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffff9ccb42c9b800.batch_stats= snapshot_time: 1788625450.558262733 secs.nsecs start_time: 1788625390.907284641 secs.nsecs elapsed_time: 59.650978092 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ccb42c9b800.statahead_batch_max=64 llite.lustre-ffff9ccb42c9b800.statahead_max=128