== sanity test 123ad: Verify batching statahead works correctly ========================================================== 11:36:54 (1787585814) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats=0 llite.lustre-ffff99b3c51b7000.statahead_max=32 llite.lustre-ffff99b3c51b7000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.60 seconds: 27.80 ops/second llite.lustre-ffff99b3c51b7000.statahead_max=0 101 real 0m2.559s user 0m0.006s sys 0m0.371s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff99b3c51b7000.statahead_max=32 llite.lustre-ffff99b3c51b7000.statahead_max=32 101 real 0m1.014s user 0m0.006s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 169 statahead wrong: 2 agl total: 163 list_total: 31 fname_total: 134 hit_total: 3959 miss_total: 1712 swrong=2 ewrong=2 - open/close 605 (time 1787585844.67 total 10.01 last 60.46) total: 900 open/close in 15.15 seconds: 59.39 ops/second llite.lustre-ffff99b3c51b7000.statahead_max=0 1001 real 0m32.414s user 0m0.047s sys 0m3.314s 'ls -l' 1000 files without statahead: 32 sec llite.lustre-ffff99b3c51b7000.statahead_max=32 llite.lustre-ffff99b3c51b7000.statahead_max=32 1001 real 0m12.421s user 0m0.031s sys 0m1.542s 'ls -l' 1000 files with statahead: 12 sec statahead total: 170 statahead wrong: 2 agl total: 164 list_total: 32 fname_total: 134 hit_total: 4957 miss_total: 1714 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 27 seconds llite.lustre-ffff99b3c51b7000.statahead_stats= statahead total: 170 statahead wrong: 2 agl total: 164 list_total: 32 fname_total: 134 hit_total: 4957 miss_total: 1714 mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787585936.733691613 secs.nsecs start_time: 1787585820.278976379 secs.nsecs elapsed_time: 116.454715234 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 2 5 12 16: 22 56 69 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787585936.733760751 secs.nsecs start_time: 1787585820.279168851 secs.nsecs elapsed_time: 116.454591900 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 5 12 17 16: 14 35 53 32: 18 46 100 - open/close 549 (time 1787585950.58 total 10.01 last 54.86) - open/close 1022 (time 1787585960.58 total 20.01 last 47.29) - open/close 1491 (time 1787585970.61 total 30.03 last 46.79) - open/close 2033 (time 1787585980.62 total 40.04 last 54.13) - open/close 2565 (time 1787585990.62 total 50.05 last 53.19) - open/close 3186 (time 1787586000.63 total 60.05 last 62.06) - open/close 3824 (time 1787586010.63 total 70.06 last 63.77) - open/close 4449 (time 1787586020.64 total 80.06 last 62.46) - open/close 5168 (time 1787586030.64 total 90.07 last 71.87) - open/close 5912 (time 1787586040.65 total 100.08 last 74.33) - open/close 6600 (time 1787586050.66 total 110.09 last 68.74) - open/close 7247 (time 1787586060.66 total 120.09 last 64.67) - open/close 7857 (time 1787586070.67 total 130.09 last 60.99) - open/close 8527 (time 1787586080.67 total 140.10 last 66.97) - open/close 9121 (time 1787586090.68 total 150.10 last 59.36) - open/close 9801 (time 1787586100.68 total 160.11 last 67.99) total: 10000 open/close in 162.52 seconds: 61.53 ops/second llite.lustre-ffff99b3c51b7000.statahead_batch_max=0 llite.lustre-ffff99b3c51b7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.stats=clear mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.417s user 0m0.340s sys 0m15.254s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99b3c51b7000.statahead_batch_max=32 llite.lustre-ffff99b3c51b7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.stats=clear mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m4.281s user 0m0.335s sys 0m14.574s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787586314.922574653 secs.nsecs start_time: 1787586178.076124691 secs.nsecs elapsed_time: 136.846449962 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787586314.922655599 secs.nsecs start_time: 1787586178.076218861 secs.nsecs elapsed_time: 136.846436738 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats=0 llite.lustre-ffff99b3c51b7000.statahead_max=2048 llite.lustre-ffff99b3c51b7000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.55 seconds: 28.13 ops/second llite.lustre-ffff99b3c51b7000.statahead_max=0 101 real 0m2.674s user 0m0.007s sys 0m0.328s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff99b3c51b7000.statahead_max=2048 llite.lustre-ffff99b3c51b7000.statahead_max=2048 101 real 0m0.990s user 0m0.006s sys 0m0.190s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 - open/close 660 (time 1787586705.33 total 10.02 last 65.84) total: 900 open/close in 13.85 seconds: 64.99 ops/second llite.lustre-ffff99b3c51b7000.statahead_max=0 1001 real 0m25.308s user 0m0.040s sys 0m3.066s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff99b3c51b7000.statahead_max=2048 llite.lustre-ffff99b3c51b7000.statahead_max=2048 1001 real 0m9.448s user 0m0.029s sys 0m1.373s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 28 seconds llite.lustre-ffff99b3c51b7000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787586789.846102951 secs.nsecs start_time: 1787586318.102052683 secs.nsecs elapsed_time: 471.744050268 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 33 33 8: 1 11 44 16: 2 22 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787586789.846171746 secs.nsecs start_time: 1787586318.102192414 secs.nsecs elapsed_time: 471.743979332 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 5 55 66 16: 0 0 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 614 (time 1787586802.93 total 10.01 last 61.33) - open/close 1275 (time 1787586812.93 total 20.02 last 66.08) - open/close 1851 (time 1787586822.95 total 30.04 last 57.48) - open/close 2481 (time 1787586832.97 total 40.05 last 62.90) - open/close 3117 (time 1787586842.98 total 50.06 last 63.53) - open/close 3717 (time 1787586852.99 total 60.07 last 59.95) - open/close 4386 (time 1787586862.99 total 70.07 last 66.90) - open/close 5087 (time 1787586873.00 total 80.08 last 70.01) - open/close 5713 (time 1787586883.01 total 90.09 last 62.54) - open/close 6330 (time 1787586893.02 total 100.10 last 61.65) - open/close 7010 (time 1787586903.02 total 110.10 last 67.99) - open/close 7691 (time 1787586913.03 total 120.12 last 68.02) - open/close 8345 (time 1787586923.04 total 130.12 last 65.36) - open/close 8972 (time 1787586933.04 total 140.13 last 62.66) - open/close 9628 (time 1787586943.06 total 150.14 last 65.51) total: 10000 open/close in 155.67 seconds: 64.24 ops/second llite.lustre-ffff99b3c51b7000.statahead_batch_max=0 llite.lustre-ffff99b3c51b7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.stats=clear mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.732s user 0m0.326s sys 0m15.614s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99b3c51b7000.statahead_batch_max=256 llite.lustre-ffff99b3c51b7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.stats=clear mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.588s user 0m0.316s sys 0m15.617s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787587088.674020609 secs.nsecs start_time: 1787587025.285206349 secs.nsecs elapsed_time: 63.388814260 secs.nsecs subreqs per batch batches %% cum %% 1: 131 54 54 2: 28 11 66 4: 11 4 71 8: 6 2 73 16: 5 2 75 32: 15 6 82 64: 7 2 84 128: 1 0 85 256: 35 14 100 mdc.lustre-MDT0001-mdc-ffff99b3c51b7000.batch_stats= snapshot_time: 1787587088.674088524 secs.nsecs start_time: 1787587025.285298451 secs.nsecs elapsed_time: 63.388790073 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff99b3c51b7000.statahead_batch_max=64 llite.lustre-ffff99b3c51b7000.statahead_max=128