== sanity test 123ad: Verify batching statahead works correctly ========================================================== 13:28:35 (1784050115) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats=0 llite.lustre-ffff97cf90f7c000.statahead_max=32 llite.lustre-ffff97cf90f7c000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats=0 total: 100 open/close in 2.85 seconds: 35.10 ops/second llite.lustre-ffff97cf90f7c000.statahead_max=0 101 real 0m2.613s user 0m0.007s sys 0m0.367s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff97cf90f7c000.statahead_max=32 llite.lustre-ffff97cf90f7c000.statahead_max=32 101 real 0m1.472s user 0m0.007s sys 0m0.195s 'ls -l' 100 files with statahead: 2 sec statahead total: 127 statahead wrong: 2 agl total: 121 list_total: 31 fname_total: 95 hit_total: 4119 miss_total: 786 swrong=2 ewrong=2 - open/close 604 (time 1784050144.91 total 10.01 last 60.35) total: 900 open/close in 15.00 seconds: 60.02 ops/second llite.lustre-ffff97cf90f7c000.statahead_max=0 1001 real 0m26.723s user 0m0.043s sys 0m3.498s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff97cf90f7c000.statahead_max=32 llite.lustre-ffff97cf90f7c000.statahead_max=32 1001 real 0m11.420s user 0m0.033s sys 0m1.651s 'ls -l' 1000 files with statahead: 12 sec statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 32 fname_total: 95 hit_total: 5117 miss_total: 788 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff97cf90f7c000.statahead_stats= statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 32 fname_total: 95 hit_total: 5117 miss_total: 788 mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats= snapshot_time: 1784050225.071658023 secs.nsecs start_time: 1784050120.982377375 secs.nsecs elapsed_time: 104.089280648 secs.nsecs subreqs per batch batches %% cum %% 1: 2 4 4 2: 0 0 4 4: 1 2 7 8: 2 4 12 16: 2 4 17 32: 34 82 100 - open/close 532 (time 1784050237.63 total 10.01 last 53.16) - open/close 1113 (time 1784050247.63 total 20.01 last 58.10) - open/close 1748 (time 1784050257.65 total 30.03 last 63.37) - open/close 2386 (time 1784050267.65 total 40.03 last 63.77) - open/close 2952 (time 1784050277.67 total 50.05 last 56.52) - open/close 3452 (time 1784050287.67 total 60.05 last 49.96) - open/close 3997 (time 1784050297.69 total 70.07 last 54.41) - open/close 4608 (time 1784050307.70 total 80.08 last 61.05) - open/close 5167 (time 1784050317.70 total 90.08 last 55.87) - open/close 5751 (time 1784050327.72 total 100.10 last 58.33) - open/close 6325 (time 1784050337.72 total 110.10 last 57.40) - open/close 6928 (time 1784050347.73 total 120.11 last 60.21) - open/close 7520 (time 1784050357.75 total 130.12 last 59.11) - open/close 8080 (time 1784050367.75 total 140.13 last 55.97) - open/close 8672 (time 1784050377.76 total 150.14 last 59.13) - open/close 9260 (time 1784050387.77 total 160.15 last 58.76) - open/close 9820 (time 1784050397.77 total 170.15 last 55.98) total: 10000 open/close in 173.04 seconds: 57.79 ops/second llite.lustre-ffff97cf90f7c000.statahead_batch_max=0 llite.lustre-ffff97cf90f7c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m13.331s user 0m0.334s sys 0m17.532s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff97cf90f7c000.statahead_batch_max=32 llite.lustre-ffff97cf90f7c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m50.853s user 0m0.301s sys 0m16.200s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats= snapshot_time: 1784050611.043568756 secs.nsecs start_time: 1784050485.779824958 secs.nsecs elapsed_time: 125.263743798 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats=0 llite.lustre-ffff97cf90f7c000.statahead_max=2048 llite.lustre-ffff97cf90f7c000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats=0 total: 100 open/close in 2.98 seconds: 33.61 ops/second llite.lustre-ffff97cf90f7c000.statahead_max=0 101 real 0m2.563s user 0m0.005s sys 0m0.353s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff97cf90f7c000.statahead_max=2048 llite.lustre-ffff97cf90f7c000.statahead_max=2048 101 real 0m0.886s user 0m0.005s sys 0m0.160s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 666 (time 1784051006.74 total 10.02 last 66.45) total: 900 open/close in 13.66 seconds: 65.89 ops/second llite.lustre-ffff97cf90f7c000.statahead_max=0 1001 real 0m25.195s user 0m0.044s sys 0m3.173s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff97cf90f7c000.statahead_max=2048 llite.lustre-ffff97cf90f7c000.statahead_max=2048 1001 real 0m8.072s user 0m0.034s sys 0m1.566s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff97cf90f7c000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats= snapshot_time: 1784051082.721471969 secs.nsecs start_time: 1784050612.931458404 secs.nsecs elapsed_time: 469.790013565 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 16 16 16: 1 8 25 32: 3 25 50 64: 2 16 66 128: 0 0 66 256: 4 33 100 - open/close 646 (time 1784051095.00 total 10.03 last 64.41) - open/close 1180 (time 1784051105.00 total 20.03 last 53.39) - open/close 1807 (time 1784051115.00 total 30.03 last 62.68) - open/close 2421 (time 1784051125.01 total 40.04 last 61.38) - open/close 3014 (time 1784051135.01 total 50.04 last 59.27) - open/close 3669 (time 1784051145.02 total 60.05 last 65.46) - open/close 4330 (time 1784051155.02 total 70.05 last 66.09) - open/close 5036 (time 1784051165.02 total 80.05 last 70.60) - open/close 5710 (time 1784051175.02 total 90.05 last 67.39) - open/close 6396 (time 1784051185.04 total 100.07 last 68.51) - open/close 7113 (time 1784051195.04 total 110.07 last 71.65) - open/close 7776 (time 1784051205.05 total 120.08 last 66.24) - open/close 8380 (time 1784051215.06 total 130.09 last 60.35) - open/close 9043 (time 1784051225.07 total 140.10 last 66.24) - open/close 9696 (time 1784051235.07 total 150.10 last 65.30) total: 10000 open/close in 155.21 seconds: 64.43 ops/second llite.lustre-ffff97cf90f7c000.statahead_batch_max=0 llite.lustre-ffff97cf90f7c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.639s user 0m0.299s sys 0m17.142s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff97cf90f7c000.statahead_batch_max=256 llite.lustre-ffff97cf90f7c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.620s user 0m0.309s sys 0m16.918s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff97cf90f7c000.batch_stats= snapshot_time: 1784051392.546724938 secs.nsecs start_time: 1784051319.294053488 secs.nsecs elapsed_time: 73.252671450 secs.nsecs subreqs per batch batches %% cum %% 1: 215 52 52 2: 58 14 67 4: 42 10 77 8: 23 5 83 16: 19 4 87 32: 11 2 90 64: 3 0 91 128: 2 0 91 256: 34 8 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff97cf90f7c000.statahead_batch_max=64 llite.lustre-ffff97cf90f7c000.statahead_max=128