== sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:32:37 (1787866357) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats=0 llite.lustre-ffff9d83ca64b800.statahead_max=32 llite.lustre-ffff9d83ca64b800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.30 seconds: 43.53 ops/second llite.lustre-ffff9d83ca64b800.statahead_max=0 101 real 0m2.570s user 0m0.002s sys 0m0.302s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9d83ca64b800.statahead_max=32 llite.lustre-ffff9d83ca64b800.statahead_max=32 101 real 0m1.027s user 0m0.002s sys 0m0.178s 'ls -l' 100 files with statahead: 1 sec statahead total: 152 statahead wrong: 2 agl total: 147 list_total: 31 fname_total: 112 hit_total: 4453 miss_total: 1292 swrong=2 ewrong=2 - open/close 817 (time 1787866384.84 total 10.01 last 81.62) total: 900 open/close in 11.08 seconds: 81.25 ops/second llite.lustre-ffff9d83ca64b800.statahead_max=0 1001 real 0m20.276s user 0m0.036s sys 0m2.721s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9d83ca64b800.statahead_max=32 llite.lustre-ffff9d83ca64b800.statahead_max=32 1001 real 0m8.982s user 0m0.028s sys 0m1.343s 'ls -l' 1000 files with statahead: 9 sec statahead total: 153 statahead wrong: 2 agl total: 148 list_total: 32 fname_total: 112 hit_total: 5449 miss_total: 1296 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9d83ca64b800.statahead_stats= statahead total: 153 statahead wrong: 2 agl total: 148 list_total: 32 fname_total: 112 hit_total: 5449 miss_total: 1296 mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787866446.706555254 secs.nsecs start_time: 1787866362.636366695 secs.nsecs elapsed_time: 84.070188559 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 1 2 8 4: 4 10 18 8: 3 8 27 16: 20 54 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787866446.706624452 secs.nsecs start_time: 1787866362.636468337 secs.nsecs elapsed_time: 84.070156115 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 4 10 15 16: 12 30 45 32: 22 55 100 - open/close 924 (time 1787866459.75 total 10.01 last 92.29) - open/close 1943 (time 1787866469.77 total 20.03 last 101.75) - open/close 2813 (time 1787866479.77 total 30.03 last 86.98) - open/close 3655 (time 1787866489.80 total 40.06 last 83.91) - open/close 4428 (time 1787866499.81 total 50.07 last 77.27) - open/close 5276 (time 1787866509.81 total 60.07 last 84.76) - open/close 6178 (time 1787866519.82 total 70.08 last 90.11) - open/close 7113 (time 1787866529.83 total 80.09 last 93.43) - open/close 8023 (time 1787866539.83 total 90.09 last 91.00) - open/close 9029 (time 1787866549.84 total 100.09 last 100.56) - open/close 9997 (time 1787866559.84 total 110.10 last 96.74) total: 10000 open/close in 110.12 seconds: 90.81 ops/second llite.lustre-ffff9d83ca64b800.statahead_batch_max=0 llite.lustre-ffff9d83ca64b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.961s user 0m0.242s sys 0m14.099s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d83ca64b800.statahead_batch_max=32 llite.lustre-ffff9d83ca64b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m35.802s user 0m0.264s sys 0m13.318s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787866733.967006849 secs.nsecs start_time: 1787866629.182487978 secs.nsecs elapsed_time: 104.784518871 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787866733.967067879 secs.nsecs start_time: 1787866629.182567340 secs.nsecs elapsed_time: 104.784500539 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats=0 llite.lustre-ffff9d83ca64b800.statahead_max=2048 llite.lustre-ffff9d83ca64b800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.41 seconds: 41.47 ops/second llite.lustre-ffff9d83ca64b800.statahead_max=0 101 real 0m2.037s user 0m0.003s sys 0m0.289s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9d83ca64b800.statahead_max=2048 llite.lustre-ffff9d83ca64b800.statahead_max=2048 101 real 0m0.924s user 0m0.003s sys 0m0.183s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 total: 900 open/close in 9.62 seconds: 93.57 ops/second llite.lustre-ffff9d83ca64b800.statahead_max=0 1001 real 0m21.794s user 0m0.030s sys 0m2.746s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9d83ca64b800.statahead_max=2048 llite.lustre-ffff9d83ca64b800.statahead_max=2048 1001 real 0m4.952s user 0m0.020s sys 0m1.364s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff9d83ca64b800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787867068.905737952 secs.nsecs start_time: 1787866736.595500209 secs.nsecs elapsed_time: 332.310237743 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 0 0 25 4: 4 50 75 8: 0 0 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787867068.905806300 secs.nsecs start_time: 1787866736.598750539 secs.nsecs elapsed_time: 332.307055761 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 3 33 44 16: 1 11 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 967 (time 1787867081.21 total 10.00 last 96.68) - open/close 1896 (time 1787867091.22 total 20.01 last 92.86) - open/close 2824 (time 1787867101.23 total 30.02 last 92.68) - open/close 3871 (time 1787867111.23 total 40.02 last 104.65) - open/close 4777 (time 1787867121.24 total 50.03 last 90.55) - open/close 5726 (time 1787867131.24 total 60.03 last 94.86) - open/close 6677 (time 1787867141.25 total 70.04 last 95.08) - open/close 7638 (time 1787867151.25 total 80.05 last 96.02) - open/close 8473 (time 1787867161.26 total 90.05 last 83.49) - open/close 9387 (time 1787867171.26 total 100.05 last 91.37) total: 10000 open/close in 106.35 seconds: 94.03 ops/second llite.lustre-ffff9d83ca64b800.statahead_batch_max=0 llite.lustre-ffff9d83ca64b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.120s user 0m0.258s sys 0m15.295s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d83ca64b800.statahead_batch_max=256 llite.lustre-ffff9d83ca64b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.424s user 0m0.249s sys 0m15.035s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787867302.381183579 secs.nsecs start_time: 1787867243.864781619 secs.nsecs elapsed_time: 58.516401960 secs.nsecs subreqs per batch batches %% cum %% 1: 145 52 52 2: 24 8 61 4: 25 9 70 8: 19 6 76 16: 6 2 79 32: 13 4 83 64: 9 3 87 128: 2 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff9d83ca64b800.batch_stats= snapshot_time: 1787867302.381255375 secs.nsecs start_time: 1787867243.864847494 secs.nsecs elapsed_time: 58.516407881 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9d83ca64b800.statahead_batch_max=64 llite.lustre-ffff9d83ca64b800.statahead_max=128