== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 14:28:22 (1782239302)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.47 seconds: 21.44 ops/second
 - mkdir 180 (time 1782239320.89 total 10.01 last 17.98)
 - mkdir 358 (time 1782239330.96 total 20.08 last 17.67)
 - mkdir 497 (time 1782239340.97 total 30.09 last 13.89)
 - mkdir 623 (time 1782239351.01 total 40.14 last 12.54)
 - mkdir 761 (time 1782239361.11 total 50.24 last 13.67)
 - mkdir 933 (time 1782239371.13 total 60.26 last 17.17)
total: 1000 mkdir in 64.11 seconds: 15.60 ops/second
 - mkdir 182 (time 1782239321.86 total 10.01 last 18.19)
 - mkdir 350 (time 1782239331.87 total 20.02 last 16.78)
 - mkdir 482 (time 1782239341.91 total 30.05 last 13.15)
 - mkdir 601 (time 1782239351.92 total 40.07 last 11.88)
 - mkdir 741 (time 1782239361.93 total 50.07 last 13.99)
 - mkdir 914 (time 1782239371.95 total 60.10 last 17.26)
total: 1000 mkdir in 67.65 seconds: 14.78 ops/second
 - mkdir 196 (time 1782239323.32 total 10.04 last 19.53)
 - mkdir 358 (time 1782239333.37 total 20.09 last 16.12)
 - mkdir 492 (time 1782239343.44 total 30.15 last 13.31)
 - mkdir 609 (time 1782239353.51 total 40.23 last 11.61)
 - mkdir 759 (time 1782239363.57 total 50.28 last 14.92)
 - mkdir 934 (time 1782239373.67 total 60.38 last 17.33)
total: 1000 mkdir in 67.39 seconds: 14.84 ops/second
 - mkdir 187 (time 1782239322.29 total 10.02 last 18.66)
 - mkdir 351 (time 1782239332.31 total 20.05 last 16.36)
 - mkdir 484 (time 1782239342.36 total 30.09 last 13.24)
 - mkdir 600 (time 1782239352.42 total 40.15 last 11.53)
 - mkdir 744 (time 1782239362.49 total 50.23 last 14.29)
 - mkdir 920 (time 1782239372.56 total 60.29 last 17.49)
total: 1000 mkdir in 68.88 seconds: 14.52 ops/second
 - mkdir 201 (time 1782239323.76 total 10.00 last 20.10)
 - mkdir 358 (time 1782239333.79 total 20.03 last 15.66)
 - mkdir 486 (time 1782239343.81 total 30.05 last 12.78)
 - mkdir 605 (time 1782239353.85 total 40.09 last 11.85)
 - mkdir 754 (time 1782239364.00 total 50.24 last 14.67)
 - mkdir 931 (time 1782239374.04 total 60.28 last 17.64)
total: 1000 mkdir in 67.94 seconds: 14.72 ops/second
 - mkdir 195 (time 1782239322.98 total 10.05 last 19.40)
 - mkdir 363 (time 1782239333.06 total 20.13 last 16.68)
 - mkdir 490 (time 1782239343.12 total 30.19 last 12.62)
 - mkdir 608 (time 1782239353.19 total 40.26 last 11.71)
 - mkdir 753 (time 1782239363.25 total 50.32 last 14.41)
 - mkdir 921 (time 1782239373.27 total 60.34 last 16.77)
total: 1000 mkdir in 69.37 seconds: 14.41 ops/second
 - mkdir 194 (time 1782239323.41 total 10.06 last 19.29)
 - mkdir 352 (time 1782239333.45 total 20.09 last 15.75)
 - mkdir 488 (time 1782239343.50 total 30.15 last 13.53)
 - mkdir 607 (time 1782239353.55 total 40.19 last 11.84)
 - mkdir 751 (time 1782239363.57 total 50.21 last 14.37)
 - mkdir 923 (time 1782239373.58 total 60.23 last 17.17)
total: 1000 mkdir in 69.02 seconds: 14.49 ops/second
 - mkdir 183 (time 1782239322.74 total 10.03 last 18.25)
 - mkdir 350 (time 1782239332.77 total 20.05 last 16.66)
 - mkdir 481 (time 1782239342.78 total 30.07 last 13.08)
 - mkdir 595 (time 1782239352.80 total 40.09 last 11.38)
 - mkdir 738 (time 1782239362.85 total 50.14 last 14.23)
 - mkdir 913 (time 1782239372.92 total 60.21 last 17.38)
total: 1000 mkdir in 69.83 seconds: 14.32 ops/second
 - mkdir 183 (time 1782239323.27 total 10.02 last 18.26)
 - mkdir 339 (time 1782239333.28 total 20.02 last 15.60)
 - mkdir 466 (time 1782239343.41 total 30.15 last 12.54)
 - mkdir 581 (time 1782239353.47 total 40.22 last 11.43)
 - mkdir 728 (time 1782239363.54 total 50.29 last 14.59)
 - mkdir 901 (time 1782239373.65 total 60.39 last 17.12)
 - mkdir 989 (time 1782239383.75 total 70.50 last 8.71)
total: 1000 mkdir in 71.28 seconds: 14.03 ops/second
 - mkdir 185 (time 1782239323.55 total 10.01 last 18.48)
 - mkdir 343 (time 1782239333.57 total 20.03 last 15.76)
 - mkdir 467 (time 1782239343.65 total 30.11 last 12.31)
 - mkdir 579 (time 1782239353.68 total 40.14 last 11.16)
 - mkdir 724 (time 1782239363.73 total 50.19 last 14.43)
 - mkdir 894 (time 1782239373.78 total 60.24 last 16.92)
 - mkdir 982 (time 1782239383.84 total 70.30 last 8.75)
total: 1000 mkdir in 71.64 seconds: 13.96 ops/second
Time for file creation 81 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 11 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.71 seconds: 14.17 ops/second
 - mkdir 141 (time 1782239936.68 total 10.03 last 14.05)
 - mkdir 295 (time 1782239946.72 total 20.07 last 15.34)
 - mkdir 425 (time 1782239956.75 total 30.10 last 12.96)
 - mkdir 567 (time 1782239966.75 total 40.11 last 14.19)
 - mkdir 721 (time 1782239976.79 total 50.15 last 15.34)
 - mkdir 872 (time 1782239986.83 total 60.19 last 15.04)
total: 1000 mkdir in 68.35 seconds: 14.63 ops/second
 - mkdir 133 (time 1782239935.06 total 10.03 last 13.25)
 - mkdir 277 (time 1782239945.07 total 20.04 last 14.39)
 - mkdir 406 (time 1782239955.07 total 30.05 last 12.90)
 - mkdir 543 (time 1782239965.12 total 40.09 last 13.64)
 - mkdir 694 (time 1782239975.15 total 50.12 last 15.05)
 - mkdir 849 (time 1782239985.16 total 60.13 last 15.48)
 - mkdir 998 (time 1782239995.21 total 70.18 last 14.83)
total: 1000 mkdir in 70.24 seconds: 14.24 ops/second
 - mkdir 140 (time 1782239937.22 total 10.00 last 13.99)
 - mkdir 284 (time 1782239947.25 total 20.03 last 14.36)
 - mkdir 413 (time 1782239957.26 total 30.04 last 12.89)
 - mkdir 555 (time 1782239967.27 total 40.05 last 14.18)
 - mkdir 711 (time 1782239977.33 total 50.11 last 15.51)
 - mkdir 864 (time 1782239987.34 total 60.13 last 15.27)
total: 1000 mkdir in 70.07 seconds: 14.27 ops/second
 - mkdir 148 (time 1782239937.92 total 10.06 last 14.71)
 - mkdir 284 (time 1782239947.95 total 20.09 last 13.56)
 - mkdir 415 (time 1782239957.96 total 30.10 last 13.09)
 - mkdir 557 (time 1782239968.06 total 40.20 last 14.07)
 - mkdir 712 (time 1782239978.06 total 50.20 last 15.49)
 - mkdir 863 (time 1782239988.08 total 60.22 last 15.07)
 - mkdir 990 (time 1782239998.23 total 70.37 last 12.52)
total: 1000 mkdir in 71.97 seconds: 13.90 ops/second
 - mkdir 135 (time 1782239936.66 total 10.09 last 13.37)
 - mkdir 279 (time 1782239946.68 total 20.12 last 14.36)
 - mkdir 403 (time 1782239956.71 total 30.14 last 12.37)
 - mkdir 542 (time 1782239966.74 total 40.18 last 13.85)
 - mkdir 694 (time 1782239976.77 total 50.21 last 15.16)
 - mkdir 845 (time 1782239986.80 total 60.24 last 15.05)
 - mkdir 978 (time 1782239996.91 total 70.35 last 13.16)
total: 1000 mkdir in 73.39 seconds: 13.63 ops/second
 - mkdir 142 (time 1782239938.19 total 10.01 last 14.19)
 - mkdir 284 (time 1782239948.21 total 20.03 last 14.17)
 - mkdir 413 (time 1782239958.26 total 30.07 last 12.84)
 - mkdir 550 (time 1782239968.27 total 40.09 last 13.68)
 - mkdir 710 (time 1782239978.29 total 50.11 last 15.97)
 - mkdir 857 (time 1782239988.30 total 60.11 last 14.69)
 - mkdir 986 (time 1782239998.36 total 70.18 last 12.82)
total: 1000 mkdir in 72.78 seconds: 13.74 ops/second
 - mkdir 134 (time 1782239937.53 total 10.08 last 13.29)
 - mkdir 279 (time 1782239947.63 total 20.18 last 14.36)
 - mkdir 409 (time 1782239957.75 total 30.30 last 12.84)
 - mkdir 551 (time 1782239967.77 total 40.32 last 14.17)
 - mkdir 706 (time 1782239977.81 total 50.36 last 15.44)
 - mkdir 849 (time 1782239987.86 total 60.41 last 14.23)
 - mkdir 980 (time 1782239997.91 total 70.47 last 13.03)
total: 1000 mkdir in 74.32 seconds: 13.45 ops/second
 - mkdir 141 (time 1782239938.06 total 10.08 last 13.99)
 - mkdir 288 (time 1782239948.11 total 20.13 last 14.62)
 - mkdir 412 (time 1782239958.17 total 30.19 last 12.33)
 - mkdir 550 (time 1782239968.17 total 40.19 last 13.80)
 - mkdir 707 (time 1782239978.17 total 50.19 last 15.70)
 - mkdir 852 (time 1782239988.20 total 60.23 last 14.46)
 - mkdir 980 (time 1782239998.22 total 70.24 last 12.78)
total: 1000 mkdir in 74.04 seconds: 13.51 ops/second
 - mkdir 133 (time 1782239937.75 total 10.04 last 13.25)
 - mkdir 269 (time 1782239947.78 total 20.06 last 13.57)
 - mkdir 398 (time 1782239957.82 total 30.11 last 12.84)
 - mkdir 534 (time 1782239967.86 total 40.14 last 13.56)
 - mkdir 692 (time 1782239977.90 total 50.19 last 15.73)
 - mkdir 843 (time 1782239987.90 total 60.19 last 15.10)
 - mkdir 973 (time 1782239998.04 total 70.33 last 12.82)
total: 1000 mkdir in 74.59 seconds: 13.41 ops/second
 - mkdir 132 (time 1782239937.75 total 10.05 last 13.13)
 - mkdir 269 (time 1782239947.75 total 20.05 last 13.69)
 - mkdir 390 (time 1782239957.77 total 30.08 last 12.07)
 - mkdir 532 (time 1782239967.81 total 40.11 last 14.16)
 - mkdir 690 (time 1782239977.84 total 50.14 last 15.75)
 - mkdir 837 (time 1782239987.91 total 60.22 last 14.59)
 - mkdir 959 (time 1782239998.00 total 70.30 last 12.10)
total: 1000 mkdir in 76.43 seconds: 13.08 ops/second
Time for file creation 89 sec for 1 RPC sent at a time
 - unlinked 0 (time 1782240013 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1782240015 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1782240016 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1782240016 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1782240015 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1782240016 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1782240017 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1782240017 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1782240016 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1782240016 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
Time for file removal 52 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8