== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:17:58 (1785403078) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats=0 llite.lustre-ffff979311139000.statahead_max=32 llite.lustre-ffff979311139000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats=0 total: 100 open/close in 2.58 seconds: 38.82 ops/second llite.lustre-ffff979311139000.statahead_max=0 101 real 0m3.765s user 0m0.006s sys 0m0.310s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff979311139000.statahead_max=32 llite.lustre-ffff979311139000.statahead_max=32 101 real 0m1.246s user 0m0.001s sys 0m0.154s 'ls -l' 100 files with statahead: 1 sec statahead total: 148 statahead wrong: 2 agl total: 141 list_total: 30 fname_total: 114 hit_total: 3751 miss_total: 1669 swrong=2 ewrong=2 - open/close 741 (time 1785403112.16 total 10.00 last 74.10) total: 900 open/close in 12.67 seconds: 71.02 ops/second llite.lustre-ffff979311139000.statahead_max=0 1001 real 0m35.055s user 0m0.032s sys 0m3.061s 'ls -l' 1000 files without statahead: 36 sec llite.lustre-ffff979311139000.statahead_max=32 llite.lustre-ffff979311139000.statahead_max=32 1001 real 0m15.534s user 0m0.031s sys 0m1.478s 'ls -l' 1000 files with statahead: 16 sec statahead total: 149 statahead wrong: 2 agl total: 142 list_total: 31 fname_total: 114 hit_total: 4749 miss_total: 1671 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff979311139000.statahead_stats= statahead total: 149 statahead wrong: 2 agl total: 142 list_total: 31 fname_total: 114 hit_total: 4749 miss_total: 1671 mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats= snapshot_time: 1785403205.459115592 secs.nsecs start_time: 1785403085.456247453 secs.nsecs elapsed_time: 120.002868139 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 7 7 16: 2 5 12 32: 34 87 100 - open/close 461 (time 1785403218.64 total 10.01 last 46.04) - open/close 972 (time 1785403228.66 total 20.03 last 51.00) - open/close 1469 (time 1785403238.67 total 30.05 last 49.63) - open/close 2137 (time 1785403248.68 total 40.06 last 66.75) - open/close 2691 (time 1785403258.69 total 50.06 last 55.36) - open/close 3371 (time 1785403268.69 total 60.07 last 67.97) - open/close 3969 (time 1785403278.73 total 70.10 last 59.58) - open/close 4518 (time 1785403288.74 total 80.11 last 54.86) - open/close 5037 (time 1785403298.75 total 90.12 last 51.83) - open/close 5634 (time 1785403308.75 total 100.13 last 59.68) - open/close 6213 (time 1785403318.76 total 110.14 last 57.83) - open/close 6531 (time 1785403328.77 total 120.15 last 31.77) - open/close 7040 (time 1785403338.78 total 130.15 last 50.89) - open/close 7502 (time 1785403348.79 total 140.16 last 46.14) - open/close 7940 (time 1785403358.79 total 150.17 last 43.78) - open/close 8398 (time 1785403368.79 total 160.17 last 45.80) - open/close 8840 (time 1785403378.81 total 170.19 last 44.12) - open/close 9380 (time 1785403388.82 total 180.20 last 53.95) - open/close 9935 (time 1785403398.86 total 190.24 last 55.26) total: 10000 open/close in 191.41 seconds: 52.24 ops/second llite.lustre-ffff979311139000.statahead_batch_max=0 llite.lustre-ffff979311139000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff979311139000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m30.873s user 0m0.330s sys 0m15.048s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff979311139000.statahead_batch_max=32 llite.lustre-ffff979311139000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff979311139000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m22.013s user 0m0.294s sys 0m14.651s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats= snapshot_time: 1785403673.864821391 secs.nsecs start_time: 1785403510.505886877 secs.nsecs elapsed_time: 163.358934514 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats=0 llite.lustre-ffff979311139000.statahead_max=2048 llite.lustre-ffff979311139000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats=0 total: 100 open/close in 3.33 seconds: 29.99 ops/second llite.lustre-ffff979311139000.statahead_max=0 101 real 0m3.617s user 0m0.007s sys 0m0.324s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff979311139000.statahead_max=2048 llite.lustre-ffff979311139000.statahead_max=2048 101 real 0m1.543s user 0m0.004s sys 0m0.154s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 642 (time 1785404140.59 total 10.00 last 64.19) total: 900 open/close in 13.63 seconds: 66.01 ops/second llite.lustre-ffff979311139000.statahead_max=0 1001 real 0m28.846s user 0m0.031s sys 0m3.084s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffff979311139000.statahead_max=2048 llite.lustre-ffff979311139000.statahead_max=2048 1001 real 0m8.947s user 0m0.018s sys 0m1.449s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff979311139000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats= snapshot_time: 1785404224.957324377 secs.nsecs start_time: 1785403678.325002222 secs.nsecs elapsed_time: 546.632322155 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 20 20 16: 1 10 30 32: 2 20 50 64: 1 10 60 128: 0 0 60 256: 4 40 100 - open/close 619 (time 1785404238.20 total 10.01 last 61.84) - open/close 1258 (time 1785404248.21 total 20.03 last 63.80) - open/close 1835 (time 1785404258.23 total 30.04 last 57.62) - open/close 2333 (time 1785404268.23 total 40.04 last 49.79) - open/close 2971 (time 1785404278.24 total 50.05 last 63.72) - open/close 3371 (time 1785404288.25 total 60.07 last 39.95) - open/close 3832 (time 1785404298.26 total 70.07 last 46.07) - open/close 4626 (time 1785404308.27 total 80.08 last 79.31) - open/close 5198 (time 1785404318.28 total 90.10 last 57.14) - open/close 5675 (time 1785404328.28 total 100.10 last 47.69) - open/close 6159 (time 1785404338.30 total 110.11 last 48.32) - open/close 6529 (time 1785404348.31 total 120.12 last 36.98) - open/close 6988 (time 1785404358.31 total 130.13 last 45.87) - open/close 7413 (time 1785404368.33 total 140.14 last 42.45) - open/close 8084 (time 1785404378.33 total 150.15 last 67.05) - open/close 8600 (time 1785404388.38 total 160.19 last 51.36) - open/close 9178 (time 1785404398.39 total 170.20 last 57.76) - open/close 9521 (time 1785404408.40 total 180.21 last 34.26) - open/close 9886 (time 1785404418.41 total 190.22 last 36.47) total: 10000 open/close in 192.65 seconds: 51.91 ops/second llite.lustre-ffff979311139000.statahead_batch_max=0 llite.lustre-ffff979311139000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff979311139000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m30.399s user 0m0.316s sys 0m14.790s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff979311139000.statahead_batch_max=256 llite.lustre-ffff979311139000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff979311139000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m45.026s user 0m0.341s sys 0m17.199s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff979311139000.batch_stats= snapshot_time: 1785404659.544164220 secs.nsecs start_time: 1785404531.055294243 secs.nsecs elapsed_time: 128.488869977 secs.nsecs subreqs per batch batches %% cum %% 1: 153 44 44 2: 53 15 59 4: 41 11 71 8: 27 7 79 16: 18 5 84 32: 15 4 88 64: 4 1 89 128: 1 0 90 256: 34 9 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff979311139000.statahead_batch_max=64 llite.lustre-ffff979311139000.statahead_max=128