== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:31:02 (1788946262) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats=0 llite.lustre-ffff9089f5516000.statahead_max=32 llite.lustre-ffff9089f5516000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.45 seconds: 40.89 ops/second llite.lustre-ffff9089f5516000.statahead_max=0 101 real 0m2.027s user 0m0.004s sys 0m0.301s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9089f5516000.statahead_max=32 llite.lustre-ffff9089f5516000.statahead_max=32 101 real 0m1.252s user 0m0.005s sys 0m0.163s 'ls -l' 100 files with statahead: 2 sec statahead total: 125 statahead wrong: 1 agl total: 119 list_total: 31 fname_total: 73 hit_total: 3817 miss_total: 255 swrong=1 ewrong=1 total: 900 open/close in 9.98 seconds: 90.22 ops/second llite.lustre-ffff9089f5516000.statahead_max=0 1001 real 0m25.335s user 0m0.040s sys 0m2.922s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9089f5516000.statahead_max=32 llite.lustre-ffff9089f5516000.statahead_max=32 1001 real 0m8.583s user 0m0.021s sys 0m1.286s 'ls -l' 1000 files with statahead: 8 sec statahead total: 126 statahead wrong: 1 agl total: 120 list_total: 32 fname_total: 73 hit_total: 4812 miss_total: 260 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9089f5516000.statahead_stats= statahead total: 126 statahead wrong: 1 agl total: 120 list_total: 32 fname_total: 73 hit_total: 4812 miss_total: 260 mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788946356.429571601 secs.nsecs start_time: 1788946269.724001608 secs.nsecs elapsed_time: 86.705569993 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 3 8 10 4: 1 2 13 8: 3 8 21 16: 21 56 78 32: 8 21 100 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788946356.429638383 secs.nsecs start_time: 1788946269.724085840 secs.nsecs elapsed_time: 86.705552543 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 12 30 42 32: 23 57 100 - open/close 1027 (time 1788946368.97 total 10.00 last 102.67) - open/close 1939 (time 1788946378.98 total 20.01 last 91.14) - open/close 2845 (time 1788946389.00 total 30.04 last 90.35) - open/close 3792 (time 1788946399.01 total 40.04 last 94.68) - open/close 4391 (time 1788946409.02 total 50.05 last 59.82) - open/close 5056 (time 1788946419.02 total 60.05 last 66.49) - open/close 5912 (time 1788946429.03 total 70.07 last 85.48) - open/close 6875 (time 1788946439.04 total 80.07 last 96.25) - open/close 7832 (time 1788946449.04 total 90.08 last 95.68) - open/close 8720 (time 1788946459.04 total 100.08 last 88.77) - open/close 9673 (time 1788946469.05 total 110.09 last 95.24) total: 10000 open/close in 113.20 seconds: 88.34 ops/second llite.lustre-ffff9089f5516000.statahead_batch_max=0 llite.lustre-ffff9089f5516000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9089f5516000.stats=clear mdc.lustre-MDT0001-mdc-ffff9089f5516000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.870s user 0m0.290s sys 0m13.894s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9089f5516000.statahead_batch_max=32 llite.lustre-ffff9089f5516000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9089f5516000.stats=clear mdc.lustre-MDT0001-mdc-ffff9089f5516000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.276s user 0m0.306s sys 0m13.402s batched statahead RPC= hit count=9973 mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788946648.530137826 secs.nsecs start_time: 1788946538.259840356 secs.nsecs elapsed_time: 110.270297470 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788946648.530196435 secs.nsecs start_time: 1788946538.259948783 secs.nsecs elapsed_time: 110.270247652 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats=0 llite.lustre-ffff9089f5516000.statahead_max=2048 llite.lustre-ffff9089f5516000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 1.74 seconds: 57.40 ops/second llite.lustre-ffff9089f5516000.statahead_max=0 101 real 0m1.852s user 0m0.004s sys 0m0.300s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9089f5516000.statahead_max=2048 llite.lustre-ffff9089f5516000.statahead_max=2048 101 real 0m0.946s user 0m0.006s sys 0m0.205s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10072 miss_total: 28 swrong=0 ewrong=0 - open/close 781 (time 1788946906.75 total 10.01 last 78.00) total: 900 open/close in 11.30 seconds: 79.62 ops/second llite.lustre-ffff9089f5516000.statahead_max=0 1001 real 0m18.208s user 0m0.035s sys 0m2.806s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff9089f5516000.statahead_max=2048 llite.lustre-ffff9089f5516000.statahead_max=2048 1001 real 0m5.267s user 0m0.028s sys 0m1.340s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 14 seconds llite.lustre-ffff9089f5516000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788946956.949976227 secs.nsecs start_time: 1788946651.057734127 secs.nsecs elapsed_time: 305.892242100 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 0 0 11 4: 2 22 33 8: 1 11 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788946956.950038415 secs.nsecs start_time: 1788946651.057979376 secs.nsecs elapsed_time: 305.892059039 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 5 55 66 16: 0 0 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 772 (time 1788946970.03 total 10.00 last 77.18) - open/close 1513 (time 1788946980.03 total 20.01 last 74.07) - open/close 2189 (time 1788946990.03 total 30.01 last 67.59) - open/close 3058 (time 1788947000.04 total 40.01 last 86.86) - open/close 4195 (time 1788947010.04 total 50.02 last 113.66) - open/close 5005 (time 1788947020.04 total 60.02 last 80.99) - open/close 5973 (time 1788947030.04 total 70.02 last 96.79) - open/close 6939 (time 1788947040.05 total 80.02 last 96.55) - open/close 7668 (time 1788947050.06 total 90.03 last 72.83) - open/close 8582 (time 1788947060.06 total 100.04 last 91.37) - open/close 9617 (time 1788947070.06 total 110.04 last 103.47) total: 10000 open/close in 113.73 seconds: 87.93 ops/second llite.lustre-ffff9089f5516000.statahead_batch_max=0 llite.lustre-ffff9089f5516000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9089f5516000.stats=clear mdc.lustre-MDT0001-mdc-ffff9089f5516000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.274s user 0m0.285s sys 0m13.993s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9089f5516000.statahead_batch_max=256 llite.lustre-ffff9089f5516000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9089f5516000.stats=clear mdc.lustre-MDT0001-mdc-ffff9089f5516000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.237s user 0m0.318s sys 0m14.046s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788947200.924288685 secs.nsecs start_time: 1788947137.821644535 secs.nsecs elapsed_time: 63.102644150 secs.nsecs subreqs per batch batches %% cum %% 1: 77 37 37 2: 32 15 53 4: 24 11 65 8: 9 4 69 16: 2 0 70 32: 19 9 79 64: 6 2 82 128: 0 0 82 256: 35 17 100 mdc.lustre-MDT0001-mdc-ffff9089f5516000.batch_stats= snapshot_time: 1788947200.924358745 secs.nsecs start_time: 1788947137.821919065 secs.nsecs elapsed_time: 63.102439680 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9089f5516000.statahead_batch_max=64 llite.lustre-ffff9089f5516000.statahead_max=128