== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:04:44 (1788591884) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats=0 llite.lustre-ffff91d091dc2000.statahead_max=32 llite.lustre-ffff91d091dc2000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.07 seconds: 32.59 ops/second llite.lustre-ffff91d091dc2000.statahead_max=0 101 real 0m2.784s user 0m0.004s sys 0m0.316s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff91d091dc2000.statahead_max=32 llite.lustre-ffff91d091dc2000.statahead_max=32 101 real 0m1.458s user 0m0.009s sys 0m0.192s 'ls -l' 100 files with statahead: 2 sec statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 30 fname_total: 100 hit_total: 3937 miss_total: 1568 swrong=2 ewrong=2 - open/close 703 (time 1788591915.36 total 10.00 last 70.28) total: 900 open/close in 13.06 seconds: 68.89 ops/second llite.lustre-ffff91d091dc2000.statahead_max=0 1001 real 0m24.383s user 0m0.046s sys 0m3.234s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff91d091dc2000.statahead_max=32 llite.lustre-ffff91d091dc2000.statahead_max=32 1001 real 0m9.424s user 0m0.044s sys 0m1.592s 'ls -l' 1000 files with statahead: 9 sec statahead total: 132 statahead wrong: 2 agl total: 126 list_total: 31 fname_total: 100 hit_total: 4935 miss_total: 1570 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff91d091dc2000.statahead_stats= statahead total: 132 statahead wrong: 2 agl total: 126 list_total: 31 fname_total: 100 hit_total: 4935 miss_total: 1570 mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788591992.335741785 secs.nsecs start_time: 1788591889.557648821 secs.nsecs elapsed_time: 102.778092964 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 2 5 10 4: 2 5 16 8: 1 2 18 16: 24 64 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788591992.335852342 secs.nsecs start_time: 1788591889.557720193 secs.nsecs elapsed_time: 102.778132149 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 1 2 12 16: 14 35 47 32: 21 52 100 - open/close 611 (time 1788592005.40 total 10.01 last 61.05) - open/close 1288 (time 1788592015.40 total 20.01 last 67.69) - open/close 2021 (time 1788592025.40 total 30.01 last 73.28) - open/close 2905 (time 1788592035.41 total 40.02 last 88.28) - open/close 3576 (time 1788592045.43 total 50.04 last 67.02) - open/close 4428 (time 1788592055.43 total 60.04 last 85.16) - open/close 5072 (time 1788592065.43 total 70.04 last 64.40) - open/close 5815 (time 1788592075.43 total 80.04 last 74.29) - open/close 6460 (time 1788592085.43 total 90.04 last 64.50) - open/close 7268 (time 1788592095.44 total 100.05 last 80.75) - open/close 7792 (time 1788592105.45 total 110.06 last 52.36) - open/close 8471 (time 1788592115.47 total 120.08 last 67.75) - open/close 9091 (time 1788592125.47 total 130.08 last 61.98) - open/close 9758 (time 1788592135.47 total 140.08 last 66.70) total: 10000 open/close in 143.37 seconds: 69.75 ops/second llite.lustre-ffff91d091dc2000.statahead_batch_max=0 llite.lustre-ffff91d091dc2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91d091dc2000.stats=clear mdc.lustre-MDT0001-mdc-ffff91d091dc2000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m13.642s user 0m0.398s sys 0m16.246s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff91d091dc2000.statahead_batch_max=32 llite.lustre-ffff91d091dc2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff91d091dc2000.stats=clear mdc.lustre-MDT0001-mdc-ffff91d091dc2000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m2.082s user 0m0.372s sys 0m15.060s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788592359.743966691 secs.nsecs start_time: 1788592224.924254004 secs.nsecs elapsed_time: 134.819712687 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788592359.744035943 secs.nsecs start_time: 1788592224.924364954 secs.nsecs elapsed_time: 134.819670989 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats=0 llite.lustre-ffff91d091dc2000.statahead_max=2048 llite.lustre-ffff91d091dc2000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.99 seconds: 33.44 ops/second llite.lustre-ffff91d091dc2000.statahead_max=0 101 real 0m2.665s user 0m0.012s sys 0m0.364s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff91d091dc2000.statahead_max=2048 llite.lustre-ffff91d091dc2000.statahead_max=2048 101 real 0m1.080s user 0m0.005s sys 0m0.186s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 800 (time 1788592691.57 total 10.01 last 79.96) total: 900 open/close in 11.33 seconds: 79.43 ops/second llite.lustre-ffff91d091dc2000.statahead_max=0 1001 real 0m22.242s user 0m0.046s sys 0m3.377s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff91d091dc2000.statahead_max=2048 llite.lustre-ffff91d091dc2000.statahead_max=2048 1001 real 0m6.773s user 0m0.036s sys 0m1.596s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff91d091dc2000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788592758.735382840 secs.nsecs start_time: 1788592362.606497609 secs.nsecs elapsed_time: 396.128885231 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 0 0 11 4: 3 33 44 8: 1 11 55 16: 0 0 55 32: 2 22 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788592758.736599006 secs.nsecs start_time: 1788592362.606782160 secs.nsecs elapsed_time: 396.129816846 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 764 (time 1788592771.63 total 10.01 last 76.33) - open/close 1603 (time 1788592781.63 total 20.01 last 83.90) - open/close 2205 (time 1788592791.64 total 30.02 last 60.13) - open/close 2805 (time 1788592801.65 total 40.03 last 59.96) - open/close 3512 (time 1788592811.66 total 50.04 last 70.64) - open/close 4265 (time 1788592821.66 total 60.04 last 75.26) - open/close 4849 (time 1788592831.73 total 70.11 last 57.98) - open/close 5502 (time 1788592841.75 total 80.13 last 65.20) - open/close 6122 (time 1788592851.75 total 90.13 last 61.99) - open/close 6797 (time 1788592861.76 total 100.14 last 67.42) - open/close 7538 (time 1788592871.79 total 110.17 last 73.93) - open/close 8335 (time 1788592881.79 total 120.17 last 79.68) - open/close 9073 (time 1788592891.79 total 130.17 last 73.78) - open/close 9819 (time 1788592901.79 total 140.17 last 74.59) total: 10000 open/close in 143.00 seconds: 69.93 ops/second llite.lustre-ffff91d091dc2000.statahead_batch_max=0 llite.lustre-ffff91d091dc2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91d091dc2000.stats=clear mdc.lustre-MDT0001-mdc-ffff91d091dc2000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.461s user 0m0.340s sys 0m15.281s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff91d091dc2000.statahead_batch_max=256 llite.lustre-ffff91d091dc2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff91d091dc2000.stats=clear mdc.lustre-MDT0001-mdc-ffff91d091dc2000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.750s user 0m0.334s sys 0m14.604s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788593045.053036593 secs.nsecs start_time: 1788592976.747750070 secs.nsecs elapsed_time: 68.305286523 secs.nsecs subreqs per batch batches %% cum %% 1: 77 43 43 2: 19 10 54 4: 13 7 61 8: 3 1 63 16: 4 2 65 32: 18 10 76 64: 5 2 78 128: 2 1 80 256: 35 19 100 mdc.lustre-MDT0001-mdc-ffff91d091dc2000.batch_stats= snapshot_time: 1788593045.053093200 secs.nsecs start_time: 1788592976.747937964 secs.nsecs elapsed_time: 68.305155236 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff91d091dc2000.statahead_batch_max=64 llite.lustre-ffff91d091dc2000.statahead_max=128