== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 10:06:44 (1787321204)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.84 seconds: 11.95 ops/second
 - mkdir 152 (time 1787321226.24 total 10.02 last 15.17)
 - mkdir 303 (time 1787321236.24 total 20.02 last 15.09)
 - mkdir 442 (time 1787321246.26 total 30.04 last 13.88)
 - mkdir 573 (time 1787321256.27 total 40.05 last 13.08)
 - mkdir 712 (time 1787321266.29 total 50.07 last 13.88)
 - mkdir 867 (time 1787321276.30 total 60.08 last 15.49)
total: 1000 mkdir in 68.16 seconds: 14.67 ops/second
 - mkdir 152 (time 1787321227.09 total 10.04 last 15.14)
 - mkdir 298 (time 1787321237.15 total 20.10 last 14.50)
 - mkdir 433 (time 1787321247.17 total 30.12 last 13.47)
 - mkdir 573 (time 1787321257.22 total 40.17 last 13.93)
 - mkdir 729 (time 1787321267.23 total 50.18 last 15.58)
 - mkdir 874 (time 1787321277.26 total 60.21 last 14.46)
total: 1000 mkdir in 67.80 seconds: 14.75 ops/second
 - mkdir 160 (time 1787321228.14 total 10.03 last 15.95)
 - mkdir 311 (time 1787321238.18 total 20.07 last 15.05)
 - mkdir 445 (time 1787321248.24 total 30.13 last 13.32)
 - mkdir 587 (time 1787321258.29 total 40.18 last 14.13)
 - mkdir 737 (time 1787321268.35 total 50.24 last 14.90)
 - mkdir 882 (time 1787321278.39 total 60.28 last 14.44)
total: 1000 mkdir in 66.93 seconds: 14.94 ops/second
 - mkdir 158 (time 1787321227.95 total 10.00 last 15.79)
 - mkdir 303 (time 1787321237.99 total 20.04 last 14.45)
 - mkdir 441 (time 1787321248.09 total 30.14 last 13.67)
 - mkdir 576 (time 1787321258.13 total 40.18 last 13.44)
 - mkdir 730 (time 1787321268.17 total 50.22 last 15.34)
 - mkdir 873 (time 1787321278.19 total 60.23 last 14.28)
total: 1000 mkdir in 67.61 seconds: 14.79 ops/second
 - mkdir 162 (time 1787321228.17 total 10.02 last 16.17)
 - mkdir 309 (time 1787321238.21 total 20.06 last 14.64)
 - mkdir 443 (time 1787321248.22 total 30.06 last 13.39)
 - mkdir 586 (time 1787321258.28 total 40.12 last 14.21)
 - mkdir 736 (time 1787321268.38 total 50.23 last 14.85)
 - mkdir 881 (time 1787321278.43 total 60.27 last 14.44)
total: 1000 mkdir in 67.86 seconds: 14.74 ops/second
 - mkdir 158 (time 1787321228.31 total 10.01 last 15.79)
 - mkdir 301 (time 1787321238.36 total 20.06 last 14.23)
 - mkdir 430 (time 1787321248.43 total 30.12 last 12.81)
 - mkdir 574 (time 1787321258.51 total 40.20 last 14.29)
 - mkdir 728 (time 1787321268.52 total 50.21 last 15.39)
 - mkdir 875 (time 1787321278.52 total 60.22 last 14.69)
total: 1000 mkdir in 67.90 seconds: 14.73 ops/second
 - mkdir 163 (time 1787321228.76 total 10.01 last 16.28)
 - mkdir 304 (time 1787321238.84 total 20.09 last 13.99)
 - mkdir 437 (time 1787321248.88 total 30.13 last 13.25)
 - mkdir 581 (time 1787321258.93 total 40.18 last 14.32)
 - mkdir 735 (time 1787321268.96 total 50.21 last 15.35)
 - mkdir 888 (time 1787321278.98 total 60.23 last 15.28)
total: 1000 mkdir in 67.56 seconds: 14.80 ops/second
 - mkdir 145 (time 1787321226.83 total 10.05 last 14.43)
 - mkdir 289 (time 1787321236.85 total 20.06 last 14.37)
 - mkdir 422 (time 1787321246.99 total 30.21 last 13.11)
 - mkdir 556 (time 1787321257.01 total 40.23 last 13.37)
 - mkdir 709 (time 1787321267.04 total 50.25 last 15.26)
 - mkdir 856 (time 1787321277.04 total 60.25 last 14.70)
total: 1000 mkdir in 70.06 seconds: 14.27 ops/second
 - mkdir 155 (time 1787321228.02 total 10.00 last 15.49)
 - mkdir 305 (time 1787321238.08 total 20.06 last 14.91)
 - mkdir 436 (time 1787321248.09 total 30.08 last 13.08)
 - mkdir 572 (time 1787321258.14 total 40.12 last 13.54)
 - mkdir 723 (time 1787321268.14 total 50.13 last 15.10)
 - mkdir 865 (time 1787321278.15 total 60.13 last 14.19)
total: 1000 mkdir in 69.44 seconds: 14.40 ops/second
 - mkdir 154 (time 1787321228.35 total 10.00 last 15.39)
 - mkdir 295 (time 1787321238.36 total 20.01 last 14.08)
 - mkdir 418 (time 1787321248.36 total 30.02 last 12.30)
 - mkdir 555 (time 1787321258.42 total 40.08 last 13.61)
 - mkdir 706 (time 1787321268.44 total 50.10 last 15.08)
 - mkdir 855 (time 1787321278.44 total 60.10 last 14.90)
total: 1000 mkdir in 69.53 seconds: 14.38 ops/second
Time for file creation 81 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 1 seconds: 0.000000 unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 1 seconds: 0.000000 unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 14 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.55 seconds: 18.17 ops/second
 - mkdir 93 (time 1787321855.82 total 10.02 last 9.28)
 - mkdir 220 (time 1787321865.96 total 20.16 last 12.53)
 - mkdir 368 (time 1787321876.00 total 30.20 last 14.74)
 - mkdir 536 (time 1787321886.01 total 40.20 last 16.80)
 - mkdir 728 (time 1787321896.05 total 50.25 last 19.11)
 - mkdir 906 (time 1787321906.06 total 60.26 last 17.79)
total: 1000 mkdir in 64.78 seconds: 15.44 ops/second
 - mkdir 91 (time 1787321855.75 total 10.12 last 8.99)
 - mkdir 214 (time 1787321865.79 total 20.16 last 12.25)
 - mkdir 356 (time 1787321875.81 total 30.18 last 14.17)
 - mkdir 524 (time 1787321885.86 total 40.23 last 16.72)
 - mkdir 715 (time 1787321895.93 total 50.30 last 18.96)
 - mkdir 893 (time 1787321905.95 total 60.32 last 17.77)
total: 1000 mkdir in 67.22 seconds: 14.88 ops/second
 - mkdir 99 (time 1787321854.87 total 10.01 last 9.89)
 - mkdir 215 (time 1787321864.88 total 20.02 last 11.59)
 - mkdir 357 (time 1787321874.88 total 30.02 last 14.20)
 - mkdir 521 (time 1787321884.92 total 40.06 last 16.33)
 - mkdir 708 (time 1787321894.94 total 50.08 last 18.67)
 - mkdir 875 (time 1787321904.95 total 60.08 last 16.69)
total: 1000 mkdir in 68.41 seconds: 14.62 ops/second
 - mkdir 87 (time 1787321856.75 total 10.02 last 8.68)
 - mkdir 203 (time 1787321866.78 total 20.06 last 11.56)
 - mkdir 353 (time 1787321876.82 total 30.09 last 14.95)
 - mkdir 523 (time 1787321886.82 total 40.09 last 17.00)
 - mkdir 711 (time 1787321896.88 total 50.15 last 18.69)
 - mkdir 881 (time 1787321906.90 total 60.17 last 16.96)
 - mkdir 998 (time 1787321917.01 total 70.28 last 11.58)
total: 1000 mkdir in 70.44 seconds: 14.20 ops/second
 - mkdir 86 (time 1787321856.43 total 10.20 last 8.43)
 - mkdir 206 (time 1787321866.54 total 20.31 last 11.87)
 - mkdir 350 (time 1787321876.57 total 30.34 last 14.35)
 - mkdir 522 (time 1787321886.59 total 40.36 last 17.17)
 - mkdir 704 (time 1787321896.62 total 50.39 last 18.14)
 - mkdir 879 (time 1787321906.66 total 60.42 last 17.44)
 - mkdir 995 (time 1787321916.72 total 70.48 last 11.53)
total: 1000 mkdir in 71.09 seconds: 14.07 ops/second
 - mkdir 88 (time 1787321856.62 total 10.01 last 8.79)
 - mkdir 210 (time 1787321866.67 total 20.06 last 12.14)
 - mkdir 360 (time 1787321876.67 total 30.06 last 15.00)
 - mkdir 529 (time 1787321886.68 total 40.07 last 16.88)
 - mkdir 711 (time 1787321896.73 total 50.12 last 18.11)
 - mkdir 876 (time 1787321906.74 total 60.13 last 16.49)
 - mkdir 989 (time 1787321916.88 total 70.27 last 11.15)
total: 1000 mkdir in 71.69 seconds: 13.95 ops/second
 - mkdir 89 (time 1787321856.95 total 10.05 last 8.86)
 - mkdir 208 (time 1787321866.97 total 20.07 last 11.87)
 - mkdir 360 (time 1787321876.98 total 30.07 last 15.19)
 - mkdir 526 (time 1787321887.02 total 40.12 last 16.52)
 - mkdir 710 (time 1787321897.08 total 50.18 last 18.29)
 - mkdir 883 (time 1787321907.16 total 60.25 last 17.17)
 - mkdir 991 (time 1787321917.19 total 70.29 last 10.76)
total: 1000 mkdir in 71.52 seconds: 13.98 ops/second
 - mkdir 82 (time 1787321856.39 total 10.01 last 8.19)
 - mkdir 201 (time 1787321866.43 total 20.05 last 11.85)
 - mkdir 342 (time 1787321876.46 total 30.08 last 14.07)
 - mkdir 510 (time 1787321886.49 total 40.11 last 16.75)
 - mkdir 693 (time 1787321896.51 total 50.13 last 18.26)
 - mkdir 862 (time 1787321906.52 total 60.14 last 16.87)
 - mkdir 979 (time 1787321916.73 total 70.35 last 11.46)
total: 1000 mkdir in 72.97 seconds: 13.70 ops/second
 - mkdir 80 (time 1787321856.20 total 10.03 last 7.97)
 - mkdir 197 (time 1787321866.21 total 20.05 last 11.68)
 - mkdir 337 (time 1787321876.25 total 30.09 last 13.95)
 - mkdir 504 (time 1787321886.27 total 40.11 last 16.67)
 - mkdir 691 (time 1787321896.28 total 50.11 last 18.69)
 - mkdir 854 (time 1787321906.28 total 60.12 last 16.29)
 - mkdir 975 (time 1787321916.39 total 70.23 last 11.97)
total: 1000 mkdir in 73.43 seconds: 13.62 ops/second
 - mkdir 79 (time 1787321855.99 total 10.03 last 7.88)
 - mkdir 191 (time 1787321866.13 total 20.17 last 11.04)
 - mkdir 326 (time 1787321876.13 total 30.17 last 13.50)
 - mkdir 484 (time 1787321886.17 total 40.21 last 15.74)
 - mkdir 667 (time 1787321896.22 total 50.26 last 18.21)
 - mkdir 830 (time 1787321906.23 total 60.27 last 16.29)
 - mkdir 951 (time 1787321916.24 total 70.28 last 12.08)
total: 1000 mkdir in 75.28 seconds: 13.28 ops/second
Time for file creation 84 sec for 1 RPC sent at a time
 - unlinked 0 (time 1787321929 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1787321929 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1787321929 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1787321930 ; total 1 ; last 1)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1787321929 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1787321929 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1787321930 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1787321930 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1787321930 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1787321930 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
Time for file removal 48 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8