== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:18:01 (1781216281) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats=0 llite.lustre-ffff98bbc4af4000.statahead_max=32 llite.lustre-ffff98bbc4af4000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.41 seconds: 41.47 ops/second llite.lustre-ffff98bbc4af4000.statahead_max=0 101 real 0m2.665s user 0m0.002s sys 0m0.358s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff98bbc4af4000.statahead_max=32 llite.lustre-ffff98bbc4af4000.statahead_max=32 101 real 0m1.539s user 0m0.006s sys 0m0.189s 'ls -l' 100 files with statahead: 1 sec statahead total: 121 statahead wrong: 2 agl total: 115 list_total: 32 fname_total: 79 hit_total: 4108 miss_total: 1302 swrong=2 ewrong=2 total: 900 open/close in 9.92 seconds: 90.71 ops/second llite.lustre-ffff98bbc4af4000.statahead_max=0 1001 real 0m25.870s user 0m0.040s sys 0m3.347s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff98bbc4af4000.statahead_max=32 llite.lustre-ffff98bbc4af4000.statahead_max=32 1001 real 0m11.516s user 0m0.038s sys 0m1.571s 'ls -l' 1000 files with statahead: 11 sec statahead total: 122 statahead wrong: 2 agl total: 116 list_total: 33 fname_total: 79 hit_total: 5105 miss_total: 1305 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff98bbc4af4000.statahead_stats= statahead total: 122 statahead wrong: 2 agl total: 116 list_total: 33 fname_total: 79 hit_total: 5105 miss_total: 1305 mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781216387.510401887 secs.nsecs start_time: 1781216286.981915893 secs.nsecs elapsed_time: 100.528485994 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 3 8 10 4: 0 0 10 8: 4 10 21 16: 23 62 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781216387.510476849 secs.nsecs start_time: 1781216286.982008801 secs.nsecs elapsed_time: 100.528468048 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 2 5 10 16: 11 28 39 32: 23 60 100 - open/close 878 (time 1781216400.60 total 10.00 last 87.78) - open/close 1801 (time 1781216410.61 total 20.01 last 92.22) - open/close 2586 (time 1781216420.61 total 30.02 last 78.46) - open/close 3319 (time 1781216430.61 total 40.02 last 73.29) - open/close 4091 (time 1781216440.62 total 50.03 last 77.14) - open/close 4790 (time 1781216450.64 total 60.04 last 69.77) - open/close 5633 (time 1781216460.65 total 70.05 last 84.26) - open/close 6444 (time 1781216470.65 total 80.06 last 81.04) - open/close 7302 (time 1781216480.66 total 90.06 last 85.73) - open/close 7971 (time 1781216490.66 total 100.06 last 66.90) - open/close 8735 (time 1781216500.68 total 110.08 last 76.28) - open/close 9494 (time 1781216510.68 total 120.08 last 75.88) total: 10000 open/close in 126.15 seconds: 79.27 ops/second llite.lustre-ffff98bbc4af4000.statahead_batch_max=0 llite.lustre-ffff98bbc4af4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.745s user 0m0.361s sys 0m16.698s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98bbc4af4000.statahead_batch_max=32 llite.lustre-ffff98bbc4af4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m2.301s user 0m0.336s sys 0m15.644s batched statahead RPC= hit count=9976 mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781216745.070469501 secs.nsecs start_time: 1781216607.190750465 secs.nsecs elapsed_time: 137.879719036 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781216745.070534993 secs.nsecs start_time: 1781216607.190840133 secs.nsecs elapsed_time: 137.879694860 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats=0 llite.lustre-ffff98bbc4af4000.statahead_max=2048 llite.lustre-ffff98bbc4af4000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.55 seconds: 39.14 ops/second llite.lustre-ffff98bbc4af4000.statahead_max=0 101 real 0m2.613s user 0m0.007s sys 0m0.351s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff98bbc4af4000.statahead_max=2048 llite.lustre-ffff98bbc4af4000.statahead_max=2048 101 real 0m0.756s user 0m0.005s sys 0m0.161s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10075 miss_total: 25 swrong=0 ewrong=0 - open/close 865 (time 1781217060.16 total 10.00 last 86.47) total: 900 open/close in 10.67 seconds: 84.38 ops/second llite.lustre-ffff98bbc4af4000.statahead_max=0 1001 real 0m21.596s user 0m0.032s sys 0m3.156s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff98bbc4af4000.statahead_max=2048 llite.lustre-ffff98bbc4af4000.statahead_max=2048 1001 real 0m7.349s user 0m0.038s sys 0m1.468s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff98bbc4af4000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781217122.358994222 secs.nsecs start_time: 1781216748.475951409 secs.nsecs elapsed_time: 373.883042813 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 3 33 33 8: 2 22 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781217122.359097584 secs.nsecs start_time: 1781216748.476177239 secs.nsecs elapsed_time: 373.882920345 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 729 (time 1781217135.87 total 10.00 last 72.90) - open/close 1502 (time 1781217145.87 total 20.00 last 77.28) - open/close 2367 (time 1781217155.89 total 30.02 last 86.31) - open/close 3179 (time 1781217165.89 total 40.03 last 81.15) - open/close 4032 (time 1781217175.90 total 50.04 last 85.24) - open/close 4945 (time 1781217185.91 total 60.05 last 91.21) - open/close 5817 (time 1781217195.95 total 70.08 last 86.87) - open/close 6724 (time 1781217205.96 total 80.09 last 90.65) - open/close 7719 (time 1781217215.96 total 90.10 last 99.41) - open/close 8621 (time 1781217225.97 total 100.11 last 90.11) - open/close 9528 (time 1781217235.98 total 110.11 last 90.65) total: 10000 open/close in 115.11 seconds: 86.87 ops/second llite.lustre-ffff98bbc4af4000.statahead_batch_max=0 llite.lustre-ffff98bbc4af4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.897s user 0m0.336s sys 0m15.899s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98bbc4af4000.statahead_batch_max=256 llite.lustre-ffff98bbc4af4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.886s user 0m0.283s sys 0m15.767s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781217395.121763157 secs.nsecs start_time: 1781217315.695472949 secs.nsecs elapsed_time: 79.426290208 secs.nsecs subreqs per batch batches % cum % 1: 160 50 50 2: 50 15 66 4: 25 7 74 8: 16 5 79 16: 10 3 83 32: 13 4 87 64: 4 1 88 128: 0 0 88 256: 36 11 100 mdc.lustre-MDT0001-mdc-ffff98bbc4af4000.batch_stats= snapshot_time: 1781217395.121831386 secs.nsecs start_time: 1781217315.695543337 secs.nsecs elapsed_time: 79.426288049 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff98bbc4af4000.statahead_batch_max=64 llite.lustre-ffff98bbc4af4000.statahead_max=128