== sanity test 123ad: Verify batching statahead works correctly ========================================================== 02:31:31 (1787121091) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats=0 llite.lustre-ffff8fcea0a23000.statahead_max=32 llite.lustre-ffff8fcea0a23000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats=0 total: 100 open/close in 3.45 seconds: 29.02 ops/second llite.lustre-ffff8fcea0a23000.statahead_max=0 101 real 0m2.602s user 0m0.007s sys 0m0.342s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8fcea0a23000.statahead_max=32 llite.lustre-ffff8fcea0a23000.statahead_max=32 101 real 0m0.829s user 0m0.001s sys 0m0.177s 'ls -l' 100 files with statahead: 1 sec statahead total: 156 statahead wrong: 2 agl total: 150 list_total: 30 fname_total: 124 hit_total: 4113 miss_total: 1375 swrong=2 ewrong=2 - open/close 651 (time 1787121121.02 total 10.01 last 65.03) total: 900 open/close in 14.14 seconds: 63.67 ops/second llite.lustre-ffff8fcea0a23000.statahead_max=0 1001 real 0m25.719s user 0m0.028s sys 0m3.246s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8fcea0a23000.statahead_max=32 llite.lustre-ffff8fcea0a23000.statahead_max=32 1001 real 0m10.297s user 0m0.031s sys 0m1.668s 'ls -l' 1000 files with statahead: 10 sec statahead total: 157 statahead wrong: 2 agl total: 151 list_total: 31 fname_total: 124 hit_total: 5109 miss_total: 1379 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff8fcea0a23000.statahead_stats= statahead total: 157 statahead wrong: 2 agl total: 151 list_total: 31 fname_total: 124 hit_total: 5109 miss_total: 1379 mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats= snapshot_time: 1787121199.589624941 secs.nsecs start_time: 1787121096.704963911 secs.nsecs elapsed_time: 102.884661030 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 7 7 16: 2 5 12 32: 34 87 100 - open/close 658 (time 1787121212.39 total 10.01 last 65.74) - open/close 1302 (time 1787121222.39 total 20.01 last 64.38) - open/close 2006 (time 1787121232.42 total 30.04 last 70.17) - open/close 2713 (time 1787121242.45 total 40.07 last 70.52) - open/close 3454 (time 1787121252.46 total 50.09 last 73.98) - open/close 4116 (time 1787121262.46 total 60.09 last 66.20) - open/close 4779 (time 1787121272.48 total 70.10 last 66.19) - open/close 5527 (time 1787121282.49 total 80.11 last 74.73) - open/close 6251 (time 1787121292.49 total 90.12 last 72.37) - open/close 6982 (time 1787121302.51 total 100.13 last 72.98) - open/close 7704 (time 1787121312.51 total 110.13 last 72.19) - open/close 8500 (time 1787121322.51 total 120.13 last 79.60) - open/close 9190 (time 1787121332.52 total 130.14 last 68.96) - open/close 9903 (time 1787121342.52 total 140.15 last 71.27) total: 10000 open/close in 141.42 seconds: 70.71 ops/second llite.lustre-ffff8fcea0a23000.statahead_batch_max=0 llite.lustre-ffff8fcea0a23000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.238s user 0m0.323s sys 0m17.843s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fcea0a23000.statahead_batch_max=32 llite.lustre-ffff8fcea0a23000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.382s user 0m0.298s sys 0m15.651s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats= snapshot_time: 1787121532.861290227 secs.nsecs start_time: 1787121414.921958937 secs.nsecs elapsed_time: 117.939331290 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats=0 llite.lustre-ffff8fcea0a23000.statahead_max=2048 llite.lustre-ffff8fcea0a23000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats=0 total: 100 open/close in 2.67 seconds: 37.51 ops/second llite.lustre-ffff8fcea0a23000.statahead_max=0 101 real 0m2.609s user 0m0.006s sys 0m0.330s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8fcea0a23000.statahead_max=2048 llite.lustre-ffff8fcea0a23000.statahead_max=2048 101 real 0m0.804s user 0m0.004s sys 0m0.171s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 765 (time 1787121864.33 total 10.01 last 76.45) total: 900 open/close in 12.47 seconds: 72.16 ops/second llite.lustre-ffff8fcea0a23000.statahead_max=0 1001 real 0m21.775s user 0m0.043s sys 0m3.200s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8fcea0a23000.statahead_max=2048 llite.lustre-ffff8fcea0a23000.statahead_max=2048 1001 real 0m5.413s user 0m0.031s sys 0m1.791s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff8fcea0a23000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats= snapshot_time: 1787121933.069107699 secs.nsecs start_time: 1787121535.695473662 secs.nsecs elapsed_time: 397.373634037 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 18 18 16: 2 18 36 32: 2 18 54 64: 0 0 54 128: 1 9 63 256: 4 36 100 - open/close 660 (time 1787121945.51 total 10.01 last 65.90) - open/close 1300 (time 1787121955.51 total 20.02 last 63.99) - open/close 2036 (time 1787121965.52 total 30.02 last 73.55) - open/close 2744 (time 1787121975.54 total 40.04 last 70.69) - open/close 3446 (time 1787121985.54 total 50.04 last 70.18) - open/close 4144 (time 1787121995.54 total 60.04 last 69.80) - open/close 4780 (time 1787122005.56 total 70.06 last 63.47) - open/close 5520 (time 1787122015.57 total 80.07 last 73.92) - open/close 6313 (time 1787122025.57 total 90.07 last 79.29) - open/close 7043 (time 1787122035.59 total 100.09 last 72.88) - open/close 7792 (time 1787122045.59 total 110.09 last 74.86) - open/close 8468 (time 1787122055.60 total 120.10 last 67.58) - open/close 9123 (time 1787122065.62 total 130.12 last 65.35) - open/close 9638 (time 1787122075.63 total 140.13 last 51.47) total: 10000 open/close in 145.57 seconds: 68.70 ops/second llite.lustre-ffff8fcea0a23000.statahead_batch_max=0 llite.lustre-ffff8fcea0a23000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.370s user 0m0.358s sys 0m17.277s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fcea0a23000.statahead_batch_max=256 llite.lustre-ffff8fcea0a23000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.562s user 0m0.299s sys 0m16.854s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8fcea0a23000.batch_stats= snapshot_time: 1787122226.969605716 secs.nsecs start_time: 1787122154.282192702 secs.nsecs elapsed_time: 72.687413014 secs.nsecs subreqs per batch batches %% cum %% 1: 308 56 56 2: 84 15 71 4: 52 9 81 8: 33 6 87 16: 18 3 90 32: 12 2 93 64: 5 0 93 128: 0 0 93 256: 33 6 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8fcea0a23000.statahead_batch_max=64 llite.lustre-ffff8fcea0a23000.statahead_max=128