== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:18:21 (1786835901) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats=0 llite.lustre-ffff9ed277896800.statahead_max=32 llite.lustre-ffff9ed277896800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.13 seconds: 46.87 ops/second llite.lustre-ffff9ed277896800.statahead_max=0 101 real 0m1.931s user 0m0.005s sys 0m0.347s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ed277896800.statahead_max=32 llite.lustre-ffff9ed277896800.statahead_max=32 101 real 0m0.841s user 0m0.003s sys 0m0.214s 'ls -l' 100 files with statahead: 1 sec statahead total: 103 statahead wrong: 2 agl total: 97 list_total: 31 fname_total: 69 hit_total: 4614 miss_total: 777 swrong=2 ewrong=2 total: 900 open/close in 9.90 seconds: 90.90 ops/second llite.lustre-ffff9ed277896800.statahead_max=0 1001 real 0m17.893s user 0m0.028s sys 0m3.273s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff9ed277896800.statahead_max=32 llite.lustre-ffff9ed277896800.statahead_max=32 1001 real 0m5.508s user 0m0.029s sys 0m1.812s 'ls -l' 1000 files with statahead: 5 sec statahead total: 104 statahead wrong: 2 agl total: 98 list_total: 32 fname_total: 69 hit_total: 5610 miss_total: 781 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 13 seconds llite.lustre-ffff9ed277896800.statahead_stats= statahead total: 104 statahead wrong: 2 agl total: 98 list_total: 32 fname_total: 69 hit_total: 5610 miss_total: 781 mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786835968.593473111 secs.nsecs start_time: 1786835905.085474033 secs.nsecs elapsed_time: 63.507999078 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 1 2 8 8: 2 5 13 16: 27 75 88 32: 4 11 100 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786835968.593564908 secs.nsecs start_time: 1786835905.085677203 secs.nsecs elapsed_time: 63.507887705 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 7 4: 0 0 7 8: 2 5 12 16: 12 30 42 32: 23 57 100 - open/close 822 (time 1786835980.63 total 10.00 last 82.19) - open/close 1598 (time 1786835990.63 total 20.00 last 77.58) - open/close 2368 (time 1786836000.64 total 30.01 last 76.96) - open/close 3161 (time 1786836010.64 total 40.01 last 79.25) - open/close 3983 (time 1786836020.64 total 50.02 last 82.19) - open/close 4920 (time 1786836030.64 total 60.02 last 93.69) - open/close 5902 (time 1786836040.65 total 70.02 last 98.16) - open/close 6917 (time 1786836050.65 total 80.02 last 101.47) - open/close 7933 (time 1786836060.66 total 90.03 last 101.52) - open/close 8957 (time 1786836070.66 total 100.04 last 102.37) total: 10000 open/close in 108.94 seconds: 91.79 ops/second llite.lustre-ffff9ed277896800.statahead_batch_max=0 llite.lustre-ffff9ed277896800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ed277896800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ed277896800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m42.508s user 0m0.318s sys 0m24.013s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ed277896800.statahead_batch_max=32 llite.lustre-ffff9ed277896800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ed277896800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ed277896800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.884s user 0m0.286s sys 0m19.056s batched statahead RPC= hit count=9990 mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786836196.989889446 secs.nsecs start_time: 1786836127.417716763 secs.nsecs elapsed_time: 69.572172683 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786836196.989954024 secs.nsecs start_time: 1786836127.418283023 secs.nsecs elapsed_time: 69.571671001 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats=0 llite.lustre-ffff9ed277896800.statahead_max=2048 llite.lustre-ffff9ed277896800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.87 seconds: 53.47 ops/second llite.lustre-ffff9ed277896800.statahead_max=0 101 real 0m1.711s user 0m0.003s sys 0m0.350s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ed277896800.statahead_max=2048 llite.lustre-ffff9ed277896800.statahead_max=2048 101 real 0m0.555s user 0m0.007s sys 0m0.217s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10089 miss_total: 11 swrong=0 ewrong=0 total: 900 open/close in 8.29 seconds: 108.59 ops/second llite.lustre-ffff9ed277896800.statahead_max=0 1001 real 0m16.272s user 0m0.040s sys 0m3.157s 'ls -l' 1000 files without statahead: 16 sec llite.lustre-ffff9ed277896800.statahead_max=2048 llite.lustre-ffff9ed277896800.statahead_max=2048 1001 real 0m4.418s user 0m0.026s sys 0m1.854s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11088 miss_total: 12 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 14 seconds llite.lustre-ffff9ed277896800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11088 miss_total: 12 mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786836452.484101925 secs.nsecs start_time: 1786836198.611291363 secs.nsecs elapsed_time: 253.872810562 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 3 30 50 16: 0 0 50 32: 3 30 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786836452.484164952 secs.nsecs start_time: 1786836198.611445456 secs.nsecs elapsed_time: 253.872719496 secs.nsecs subreqs per batch batches %% cum %% 1: 1 10 10 2: 0 0 10 4: 2 20 30 8: 1 10 40 16: 3 30 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 1072 (time 1786836464.06 total 10.01 last 107.11) - open/close 2265 (time 1786836474.06 total 20.01 last 119.29) - open/close 3448 (time 1786836484.06 total 30.01 last 118.29) - open/close 4439 (time 1786836494.07 total 40.02 last 99.05) - open/close 5582 (time 1786836504.07 total 50.02 last 114.29) - open/close 6721 (time 1786836514.08 total 60.03 last 113.76) - open/close 7898 (time 1786836524.08 total 70.03 last 117.63) - open/close 8970 (time 1786836534.09 total 80.04 last 107.11) total: 10000 open/close in 88.70 seconds: 112.74 ops/second llite.lustre-ffff9ed277896800.statahead_batch_max=0 llite.lustre-ffff9ed277896800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ed277896800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ed277896800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m37.839s user 0m0.273s sys 0m26.821s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ed277896800.statahead_batch_max=256 llite.lustre-ffff9ed277896800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ed277896800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ed277896800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m31.913s user 0m0.209s sys 0m21.675s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786836624.717681867 secs.nsecs start_time: 1786836585.242513362 secs.nsecs elapsed_time: 39.475168505 secs.nsecs subreqs per batch batches %% cum %% 1: 308 64 64 2: 64 13 77 4: 28 5 83 8: 15 3 87 16: 6 1 88 32: 17 3 91 64: 5 1 92 128: 0 0 92 256: 34 7 100 mdc.lustre-MDT0001-mdc-ffff9ed277896800.batch_stats= snapshot_time: 1786836624.717752245 secs.nsecs start_time: 1786836585.242650228 secs.nsecs elapsed_time: 39.475102017 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ed277896800.statahead_batch_max=64 llite.lustre-ffff9ed277896800.statahead_max=128