== sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:25:54 (1788395154) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats=0 llite.lustre-ffff98c0b7170800.statahead_max=32 llite.lustre-ffff98c0b7170800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.21 seconds: 45.32 ops/second llite.lustre-ffff98c0b7170800.statahead_max=0 101 real 0m2.563s user 0m0.006s sys 0m0.300s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff98c0b7170800.statahead_max=32 llite.lustre-ffff98c0b7170800.statahead_max=32 101 real 0m1.105s user 0m0.003s sys 0m0.151s 'ls -l' 100 files with statahead: 1 sec statahead total: 134 statahead wrong: 2 agl total: 129 list_total: 30 fname_total: 102 hit_total: 3635 miss_total: 1091 swrong=2 ewrong=2 total: 900 open/close in 8.98 seconds: 100.17 ops/second llite.lustre-ffff98c0b7170800.statahead_max=0 1001 real 0m16.374s user 0m0.035s sys 0m2.755s 'ls -l' 1000 files without statahead: 16 sec llite.lustre-ffff98c0b7170800.statahead_max=32 llite.lustre-ffff98c0b7170800.statahead_max=32 1001 real 0m7.244s user 0m0.028s sys 0m1.270s 'ls -l' 1000 files with statahead: 7 sec statahead total: 135 statahead wrong: 2 agl total: 130 list_total: 31 fname_total: 102 hit_total: 4632 miss_total: 1094 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff98c0b7170800.statahead_stats= statahead total: 135 statahead wrong: 2 agl total: 130 list_total: 31 fname_total: 102 hit_total: 4632 miss_total: 1094 mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788395228.517273888 secs.nsecs start_time: 1788395158.552643886 secs.nsecs elapsed_time: 69.964630002 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 2 5 13 8: 2 5 19 16: 20 55 75 32: 9 25 100 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788395228.517359258 secs.nsecs start_time: 1788395158.554150735 secs.nsecs elapsed_time: 69.963208523 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 10 10 16: 14 35 46 32: 21 53 100 - open/close 941 (time 1788395241.48 total 10.01 last 94.05) - open/close 1892 (time 1788395251.50 total 20.02 last 94.92) - open/close 2984 (time 1788395261.51 total 30.03 last 109.16) - open/close 4072 (time 1788395271.51 total 40.03 last 108.74) - open/close 4989 (time 1788395281.52 total 50.04 last 91.67) - open/close 5985 (time 1788395291.52 total 60.04 last 99.58) - open/close 7034 (time 1788395301.52 total 70.04 last 104.85) - open/close 8018 (time 1788395311.53 total 80.06 last 98.28) - open/close 9015 (time 1788395321.54 total 90.06 last 99.64) - open/close 9995 (time 1788395331.55 total 100.07 last 97.96) total: 10000 open/close in 100.10 seconds: 99.90 ops/second llite.lustre-ffff98c0b7170800.statahead_batch_max=0 llite.lustre-ffff98c0b7170800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98c0b7170800.stats=clear mdc.lustre-MDT0001-mdc-ffff98c0b7170800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.010s user 0m0.322s sys 0m13.571s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98c0b7170800.statahead_batch_max=32 llite.lustre-ffff98c0b7170800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98c0b7170800.stats=clear mdc.lustre-MDT0001-mdc-ffff98c0b7170800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m39.177s user 0m0.256s sys 0m12.981s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788395512.864675320 secs.nsecs start_time: 1788395403.669506744 secs.nsecs elapsed_time: 109.195168576 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788395512.864749043 secs.nsecs start_time: 1788395403.669631387 secs.nsecs elapsed_time: 109.195117656 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats=0 llite.lustre-ffff98c0b7170800.statahead_max=2048 llite.lustre-ffff98c0b7170800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.12 seconds: 47.27 ops/second llite.lustre-ffff98c0b7170800.statahead_max=0 101 real 0m2.226s user 0m0.004s sys 0m0.303s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff98c0b7170800.statahead_max=2048 llite.lustre-ffff98c0b7170800.statahead_max=2048 101 real 0m0.761s user 0m0.004s sys 0m0.148s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 total: 900 open/close in 8.96 seconds: 100.46 ops/second llite.lustre-ffff98c0b7170800.statahead_max=0 1001 real 0m25.086s user 0m0.030s sys 0m2.942s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff98c0b7170800.statahead_max=2048 llite.lustre-ffff98c0b7170800.statahead_max=2048 1001 real 0m8.719s user 0m0.019s sys 0m1.287s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff98c0b7170800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788395848.457566862 secs.nsecs start_time: 1788395515.512807248 secs.nsecs elapsed_time: 332.944759614 secs.nsecs subreqs per batch batches %% cum %% 1: 1 12 12 2: 0 0 12 4: 0 0 12 8: 2 25 37 16: 2 25 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788395848.457631491 secs.nsecs start_time: 1788395515.513940196 secs.nsecs elapsed_time: 332.943691295 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 2 25 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 887 (time 1788395861.09 total 10.00 last 88.70) - open/close 1800 (time 1788395871.10 total 20.01 last 91.25) - open/close 2708 (time 1788395881.10 total 30.01 last 90.75) - open/close 3613 (time 1788395891.10 total 40.01 last 90.49) - open/close 4703 (time 1788395901.11 total 50.02 last 108.96) - open/close 5802 (time 1788395911.11 total 60.02 last 109.83) - open/close 6616 (time 1788395921.12 total 70.03 last 81.33) - open/close 7464 (time 1788395931.13 total 80.04 last 84.70) - open/close 8319 (time 1788395941.14 total 90.05 last 85.43) - open/close 9344 (time 1788395951.15 total 100.06 last 102.41) total: 10000 open/close in 106.92 seconds: 93.53 ops/second llite.lustre-ffff98c0b7170800.statahead_batch_max=0 llite.lustre-ffff98c0b7170800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98c0b7170800.stats=clear mdc.lustre-MDT0001-mdc-ffff98c0b7170800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.644s user 0m0.284s sys 0m13.730s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98c0b7170800.statahead_batch_max=256 llite.lustre-ffff98c0b7170800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98c0b7170800.stats=clear mdc.lustre-MDT0001-mdc-ffff98c0b7170800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.247s user 0m0.282s sys 0m13.362s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788396096.056332781 secs.nsecs start_time: 1788396027.727393736 secs.nsecs elapsed_time: 68.328939045 secs.nsecs subreqs per batch batches %% cum %% 1: 156 49 49 2: 38 12 61 4: 31 9 71 8: 18 5 77 16: 12 3 81 32: 18 5 86 64: 5 1 88 128: 2 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff98c0b7170800.batch_stats= snapshot_time: 1788396096.056762122 secs.nsecs start_time: 1788396027.727538181 secs.nsecs elapsed_time: 68.329223941 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff98c0b7170800.statahead_batch_max=64 llite.lustre-ffff98c0b7170800.statahead_max=128