== sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:03:50 (1788642230) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats=0 llite.lustre-ffff89cb055a1800.statahead_max=32 llite.lustre-ffff89cb055a1800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.79 seconds: 35.83 ops/second llite.lustre-ffff89cb055a1800.statahead_max=0 101 real 0m2.383s user 0m0.005s sys 0m0.308s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff89cb055a1800.statahead_max=32 llite.lustre-ffff89cb055a1800.statahead_max=32 101 real 0m1.119s user 0m0.005s sys 0m0.150s 'ls -l' 100 files with statahead: 1 sec statahead total: 127 statahead wrong: 2 agl total: 121 list_total: 31 fname_total: 96 hit_total: 4542 miss_total: 1282 swrong=2 ewrong=2 - open/close 722 (time 1788642257.54 total 10.00 last 72.17) total: 900 open/close in 12.60 seconds: 71.41 ops/second llite.lustre-ffff89cb055a1800.statahead_max=0 1001 real 0m21.665s user 0m0.027s sys 0m2.853s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff89cb055a1800.statahead_max=32 llite.lustre-ffff89cb055a1800.statahead_max=32 1001 real 0m9.599s user 0m0.026s sys 0m1.396s 'ls -l' 1000 files with statahead: 10 sec statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 32 fname_total: 96 hit_total: 5539 miss_total: 1285 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff89cb055a1800.statahead_stats= statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 32 fname_total: 96 hit_total: 5539 miss_total: 1285 mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788642325.975221489 secs.nsecs start_time: 1788642234.627029250 secs.nsecs elapsed_time: 91.348192239 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 1 2 10 4: 2 5 16 8: 1 2 18 16: 23 62 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788642325.975315228 secs.nsecs start_time: 1788642234.627105670 secs.nsecs elapsed_time: 91.348209558 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 1 2 7 8: 3 7 15 16: 10 25 40 32: 24 60 100 - open/close 691 (time 1788642338.43 total 10.01 last 69.03) - open/close 1409 (time 1788642348.45 total 20.03 last 71.68) - open/close 2193 (time 1788642358.46 total 30.03 last 78.34) - open/close 2892 (time 1788642368.47 total 40.05 last 69.81) - open/close 3702 (time 1788642378.47 total 50.05 last 80.98) - open/close 4456 (time 1788642388.48 total 60.05 last 75.38) - open/close 5148 (time 1788642398.49 total 70.06 last 69.13) - open/close 5892 (time 1788642408.49 total 80.07 last 74.38) - open/close 6660 (time 1788642418.50 total 90.08 last 76.70) - open/close 7374 (time 1788642428.51 total 100.08 last 71.35) - open/close 8002 (time 1788642438.52 total 110.09 last 62.75) - open/close 8688 (time 1788642448.52 total 120.10 last 68.55) - open/close 9458 (time 1788642458.53 total 130.11 last 76.94) total: 10000 open/close in 138.21 seconds: 72.35 ops/second llite.lustre-ffff89cb055a1800.statahead_batch_max=0 llite.lustre-ffff89cb055a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89cb055a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff89cb055a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.469s user 0m0.275s sys 0m15.383s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89cb055a1800.statahead_batch_max=32 llite.lustre-ffff89cb055a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89cb055a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff89cb055a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m55.129s user 0m0.292s sys 0m13.757s batched statahead RPC= hit count=9979 mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788642662.603855857 secs.nsecs start_time: 1788642536.906422102 secs.nsecs elapsed_time: 125.697433755 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788642662.603925868 secs.nsecs start_time: 1788642536.906508933 secs.nsecs elapsed_time: 125.697416935 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats=0 llite.lustre-ffff89cb055a1800.statahead_max=2048 llite.lustre-ffff89cb055a1800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.77 seconds: 36.15 ops/second llite.lustre-ffff89cb055a1800.statahead_max=0 101 real 0m3.102s user 0m0.005s sys 0m0.337s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff89cb055a1800.statahead_max=2048 llite.lustre-ffff89cb055a1800.statahead_max=2048 101 real 0m0.676s user 0m0.002s sys 0m0.147s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10078 miss_total: 22 swrong=0 ewrong=0 - open/close 745 (time 1788642978.90 total 10.00 last 74.48) total: 900 open/close in 12.17 seconds: 73.92 ops/second llite.lustre-ffff89cb055a1800.statahead_max=0 1001 real 0m24.559s user 0m0.032s sys 0m2.872s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff89cb055a1800.statahead_max=2048 llite.lustre-ffff89cb055a1800.statahead_max=2048 1001 real 0m7.722s user 0m0.027s sys 0m1.374s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff89cb055a1800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788643052.984460789 secs.nsecs start_time: 1788642665.084441314 secs.nsecs elapsed_time: 387.900019475 secs.nsecs subreqs per batch batches %% cum %% 1: 3 37 37 2: 1 12 50 4: 1 12 62 8: 1 12 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788643052.984539536 secs.nsecs start_time: 1788642665.084624701 secs.nsecs elapsed_time: 387.899914835 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 3 30 40 16: 1 10 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 728 (time 1788643066.02 total 10.00 last 72.79) - open/close 1384 (time 1788643076.03 total 20.01 last 65.54) - open/close 2085 (time 1788643086.04 total 30.02 last 70.00) - open/close 2722 (time 1788643096.04 total 40.03 last 63.69) - open/close 3388 (time 1788643106.04 total 50.03 last 66.60) - open/close 4157 (time 1788643116.05 total 60.03 last 76.86) - open/close 4868 (time 1788643126.06 total 70.04 last 71.02) - open/close 5633 (time 1788643136.06 total 80.04 last 76.50) - open/close 6311 (time 1788643146.07 total 90.05 last 67.74) - open/close 7022 (time 1788643156.09 total 100.07 last 70.98) - open/close 7651 (time 1788643166.09 total 110.07 last 62.87) - open/close 8301 (time 1788643176.10 total 120.08 last 64.97) - open/close 9053 (time 1788643186.11 total 130.10 last 75.06) - open/close 9869 (time 1788643196.12 total 140.10 last 81.58) total: 10000 open/close in 141.97 seconds: 70.44 ops/second llite.lustre-ffff89cb055a1800.statahead_batch_max=0 llite.lustre-ffff89cb055a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89cb055a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff89cb055a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.603s user 0m0.312s sys 0m15.775s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89cb055a1800.statahead_batch_max=256 llite.lustre-ffff89cb055a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89cb055a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff89cb055a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.905s user 0m0.299s sys 0m15.398s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788643322.580543321 secs.nsecs start_time: 1788643263.205029863 secs.nsecs elapsed_time: 59.375513458 secs.nsecs subreqs per batch batches %% cum %% 1: 204 48 48 2: 57 13 61 4: 56 13 75 8: 37 8 84 16: 9 2 86 32: 16 3 90 64: 9 2 92 128: 0 0 92 256: 33 7 100 mdc.lustre-MDT0001-mdc-ffff89cb055a1800.batch_stats= snapshot_time: 1788643322.580761057 secs.nsecs start_time: 1788643263.205252306 secs.nsecs elapsed_time: 59.375508751 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 1 seconds: 0.000000 unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff89cb055a1800.statahead_batch_max=64 llite.lustre-ffff89cb055a1800.statahead_max=128