== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:26:36 (1789133196) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats=0 llite.lustre-ffff9b79c2e0d000.statahead_max=32 llite.lustre-ffff9b79c2e0d000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.24 seconds: 44.61 ops/second llite.lustre-ffff9b79c2e0d000.statahead_max=0 101 real 0m2.421s user 0m0.004s sys 0m0.342s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b79c2e0d000.statahead_max=32 llite.lustre-ffff9b79c2e0d000.statahead_max=32 101 real 0m1.108s user 0m0.007s sys 0m0.184s 'ls -l' 100 files with statahead: 2 sec statahead total: 154 statahead wrong: 2 agl total: 148 list_total: 30 fname_total: 121 hit_total: 4132 miss_total: 752 swrong=2 ewrong=2 - open/close 723 (time 1789133224.38 total 10.01 last 72.26) total: 900 open/close in 12.35 seconds: 72.87 ops/second llite.lustre-ffff9b79c2e0d000.statahead_max=0 1001 real 0m23.223s user 0m0.029s sys 0m3.129s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9b79c2e0d000.statahead_max=32 llite.lustre-ffff9b79c2e0d000.statahead_max=32 1001 real 0m9.109s user 0m0.032s sys 0m1.483s 'ls -l' 1000 files with statahead: 9 sec statahead total: 155 statahead wrong: 2 agl total: 149 list_total: 31 fname_total: 121 hit_total: 5130 miss_total: 754 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9b79c2e0d000.statahead_stats= statahead total: 155 statahead wrong: 2 agl total: 149 list_total: 31 fname_total: 121 hit_total: 5130 miss_total: 754 mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789133291.183929981 secs.nsecs start_time: 1789133201.518672084 secs.nsecs elapsed_time: 89.665257897 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 1 2 12 16: 20 51 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789133291.184003106 secs.nsecs start_time: 1789133201.520679262 secs.nsecs elapsed_time: 89.663323844 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 2 5 10 16: 19 51 62 32: 14 37 100 - open/close 758 (time 1789133304.06 total 10.01 last 75.72) - open/close 1518 (time 1789133314.06 total 20.02 last 75.95) - open/close 2170 (time 1789133324.08 total 30.04 last 65.08) - open/close 2895 (time 1789133334.11 total 40.06 last 72.34) - open/close 3632 (time 1789133344.11 total 50.06 last 73.70) - open/close 4345 (time 1789133354.11 total 60.07 last 71.26) - open/close 5077 (time 1789133364.12 total 70.07 last 73.18) - open/close 5812 (time 1789133374.12 total 80.07 last 73.45) - open/close 6607 (time 1789133384.12 total 90.07 last 79.50) - open/close 7349 (time 1789133394.14 total 100.10 last 74.05) - open/close 7937 (time 1789133404.15 total 110.10 last 58.75) - open/close 8458 (time 1789133414.17 total 120.12 last 52.02) - open/close 9110 (time 1789133424.17 total 130.12 last 65.18) - open/close 9724 (time 1789133434.17 total 140.12 last 61.39) total: 10000 open/close in 145.14 seconds: 68.90 ops/second llite.lustre-ffff9b79c2e0d000.statahead_batch_max=0 llite.lustre-ffff9b79c2e0d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.877s user 0m0.382s sys 0m16.795s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b79c2e0d000.statahead_batch_max=32 llite.lustre-ffff9b79c2e0d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m52.340s user 0m0.320s sys 0m14.509s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789133642.055350279 secs.nsecs start_time: 1789133519.083714999 secs.nsecs elapsed_time: 122.971635280 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789133642.055418607 secs.nsecs start_time: 1789133519.083809429 secs.nsecs elapsed_time: 122.971609178 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats=0 llite.lustre-ffff9b79c2e0d000.statahead_max=2048 llite.lustre-ffff9b79c2e0d000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.19 seconds: 31.38 ops/second llite.lustre-ffff9b79c2e0d000.statahead_max=0 101 real 0m2.929s user 0m0.006s sys 0m0.368s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9b79c2e0d000.statahead_max=2048 llite.lustre-ffff9b79c2e0d000.statahead_max=2048 101 real 0m1.043s user 0m0.003s sys 0m0.192s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 777 (time 1789134004.39 total 10.01 last 77.65) total: 900 open/close in 11.95 seconds: 75.30 ops/second llite.lustre-ffff9b79c2e0d000.statahead_max=0 1001 real 0m25.648s user 0m0.041s sys 0m2.936s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9b79c2e0d000.statahead_max=2048 llite.lustre-ffff9b79c2e0d000.statahead_max=2048 1001 real 0m8.604s user 0m0.031s sys 0m1.566s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9b79c2e0d000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789134079.349498232 secs.nsecs start_time: 1789133645.160633533 secs.nsecs elapsed_time: 434.188864699 secs.nsecs subreqs per batch batches %% cum %% 1: 3 37 37 2: 0 0 37 4: 1 12 50 8: 2 25 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789134079.349575555 secs.nsecs start_time: 1789133645.160863825 secs.nsecs elapsed_time: 434.188711730 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 2 20 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 723 (time 1789134092.45 total 10.01 last 72.25) - open/close 1408 (time 1789134102.45 total 20.01 last 68.48) - open/close 2053 (time 1789134112.46 total 30.03 last 64.40) - open/close 2671 (time 1789134122.48 total 40.04 last 61.73) - open/close 3371 (time 1789134132.52 total 50.08 last 69.73) - open/close 4095 (time 1789134142.52 total 60.08 last 72.36) - open/close 4750 (time 1789134152.53 total 70.09 last 65.47) - open/close 5242 (time 1789134162.53 total 80.09 last 49.19) - open/close 5689 (time 1789134172.53 total 90.10 last 44.67) - open/close 6161 (time 1789134182.54 total 100.11 last 47.15) - open/close 6754 (time 1789134192.55 total 110.11 last 59.26) - open/close 7325 (time 1789134202.55 total 120.11 last 57.08) - open/close 8039 (time 1789134212.56 total 130.12 last 71.36) - open/close 8855 (time 1789134222.57 total 140.13 last 81.51) - open/close 9496 (time 1789134232.58 total 150.14 last 64.05) total: 10000 open/close in 157.09 seconds: 63.66 ops/second llite.lustre-ffff9b79c2e0d000.statahead_batch_max=0 llite.lustre-ffff9b79c2e0d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.809s user 0m0.323s sys 0m16.133s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b79c2e0d000.statahead_batch_max=256 llite.lustre-ffff9b79c2e0d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.946s user 0m0.293s sys 0m16.556s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789134379.759883544 secs.nsecs start_time: 1789134316.684072259 secs.nsecs elapsed_time: 63.075811285 secs.nsecs subreqs per batch batches %% cum %% 1: 125 47 47 2: 37 14 61 4: 29 10 72 8: 9 3 75 16: 6 2 78 32: 13 4 82 64: 9 3 86 128: 2 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff9b79c2e0d000.batch_stats= snapshot_time: 1789134379.759942454 secs.nsecs start_time: 1789134316.684541954 secs.nsecs elapsed_time: 63.075400500 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b79c2e0d000.statahead_batch_max=64 llite.lustre-ffff9b79c2e0d000.statahead_max=128