== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 19:45:27 (1788565527)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.51 seconds: 19.71 ops/second
 - mkdir 78 (time 1788565557.37 total 10.05 last 7.76)
 - mkdir 214 (time 1788565567.44 total 20.12 last 13.51)
 - mkdir 355 (time 1788565577.47 total 30.15 last 14.06)
 - mkdir 463 (time 1788565587.48 total 40.15 last 10.80)
 - mkdir 568 (time 1788565597.50 total 50.18 last 10.48)
 - mkdir 691 (time 1788565607.53 total 60.20 last 12.27)
 - mkdir 825 (time 1788565617.55 total 70.22 last 13.37)
 - mkdir 984 (time 1788565627.61 total 80.29 last 15.80)
total: 1000 mkdir in 81.14 seconds: 12.32 ops/second
 - mkdir 83 (time 1788565557.93 total 10.01 last 8.29)
 - mkdir 220 (time 1788565567.94 total 20.02 last 13.68)
 - mkdir 359 (time 1788565577.95 total 30.03 last 13.89)
 - mkdir 472 (time 1788565588.04 total 40.12 last 11.20)
 - mkdir 579 (time 1788565598.05 total 50.13 last 10.69)
 - mkdir 702 (time 1788565608.07 total 60.15 last 12.27)
 - mkdir 827 (time 1788565618.10 total 70.18 last 12.45)
 - mkdir 986 (time 1788565628.14 total 80.22 last 15.84)
total: 1000 mkdir in 81.14 seconds: 12.32 ops/second
 - mkdir 84 (time 1788565558.28 total 10.06 last 8.35)
 - mkdir 224 (time 1788565568.33 total 20.11 last 13.93)
 - mkdir 361 (time 1788565578.46 total 30.24 last 13.52)
 - mkdir 469 (time 1788565588.62 total 40.40 last 10.63)
 - mkdir 580 (time 1788565598.63 total 50.41 last 11.09)
 - mkdir 701 (time 1788565608.70 total 60.47 last 12.02)
 - mkdir 833 (time 1788565618.81 total 70.59 last 13.05)
 - mkdir 994 (time 1788565628.87 total 80.65 last 16.00)
total: 1000 mkdir in 81.21 seconds: 12.31 ops/second
 - mkdir 80 (time 1788565557.66 total 10.01 last 7.99)
 - mkdir 214 (time 1788565567.72 total 20.07 last 13.32)
 - mkdir 351 (time 1788565577.80 total 30.14 last 13.60)
 - mkdir 465 (time 1788565587.87 total 40.21 last 11.32)
 - mkdir 571 (time 1788565597.95 total 50.30 last 10.51)
 - mkdir 693 (time 1788565607.99 total 60.33 last 12.15)
 - mkdir 819 (time 1788565618.07 total 70.41 last 12.50)
 - mkdir 977 (time 1788565628.09 total 80.44 last 15.76)
total: 1000 mkdir in 82.29 seconds: 12.15 ops/second
 - mkdir 72 (time 1788565554.99 total 10.07 last 7.15)
 - mkdir 182 (time 1788565565.07 total 20.15 last 10.91)
 - mkdir 325 (time 1788565575.11 total 30.19 last 14.24)
 - mkdir 441 (time 1788565585.15 total 40.23 last 11.56)
 - mkdir 550 (time 1788565595.17 total 50.25 last 10.87)
 - mkdir 668 (time 1788565605.20 total 60.28 last 11.76)
 - mkdir 794 (time 1788565615.25 total 70.33 last 12.54)
 - mkdir 924 (time 1788565625.26 total 80.34 last 12.98)
total: 1000 mkdir in 85.05 seconds: 11.76 ops/second
 - mkdir 84 (time 1788565558.01 total 10.02 last 8.39)
 - mkdir 217 (time 1788565568.05 total 20.06 last 13.24)
 - mkdir 352 (time 1788565578.05 total 30.07 last 13.50)
 - mkdir 461 (time 1788565588.12 total 40.13 last 10.83)
 - mkdir 566 (time 1788565598.19 total 50.20 last 10.42)
 - mkdir 687 (time 1788565608.27 total 60.28 last 12.00)
 - mkdir 818 (time 1788565618.27 total 70.29 last 13.09)
 - mkdir 976 (time 1788565628.28 total 80.29 last 15.80)
total: 1000 mkdir in 82.61 seconds: 12.10 ops/second
 - mkdir 67 (time 1788565554.53 total 10.13 last 6.61)
 - mkdir 168 (time 1788565564.54 total 20.14 last 10.09)
 - mkdir 303 (time 1788565574.57 total 30.17 last 13.47)
 - mkdir 420 (time 1788565584.63 total 40.23 last 11.63)
 - mkdir 534 (time 1788565594.68 total 50.29 last 11.34)
 - mkdir 650 (time 1788565604.71 total 60.31 last 11.57)
 - mkdir 778 (time 1788565614.73 total 70.33 last 12.78)
 - mkdir 906 (time 1788565624.75 total 80.35 last 12.78)
total: 1000 mkdir in 86.49 seconds: 11.56 ops/second
 - mkdir 78 (time 1788565557.99 total 10.03 last 7.78)
 - mkdir 212 (time 1788565568.08 total 20.12 last 13.29)
 - mkdir 348 (time 1788565578.17 total 30.21 last 13.47)
 - mkdir 461 (time 1788565588.20 total 40.24 last 11.27)
 - mkdir 566 (time 1788565598.22 total 50.26 last 10.48)
 - mkdir 688 (time 1788565608.26 total 60.29 last 12.16)
 - mkdir 814 (time 1788565618.35 total 70.39 last 12.48)
 - mkdir 972 (time 1788565628.38 total 80.42 last 15.76)
total: 1000 mkdir in 83.02 seconds: 12.05 ops/second
 - mkdir 84 (time 1788565558.54 total 10.10 last 8.32)
 - mkdir 217 (time 1788565568.58 total 20.14 last 13.25)
 - mkdir 349 (time 1788565578.59 total 30.15 last 13.19)
 - mkdir 458 (time 1788565588.63 total 40.18 last 10.86)
 - mkdir 567 (time 1788565598.64 total 50.20 last 10.88)
 - mkdir 685 (time 1788565608.65 total 60.21 last 11.79)
 - mkdir 811 (time 1788565618.72 total 70.27 last 12.52)
 - mkdir 971 (time 1788565628.79 total 80.35 last 15.88)
total: 1000 mkdir in 82.91 seconds: 12.06 ops/second
 - mkdir 71 (time 1788565555.50 total 10.07 last 7.05)
 - mkdir 180 (time 1788565565.51 total 20.08 last 10.89)
 - mkdir 311 (time 1788565575.53 total 30.10 last 13.08)
 - mkdir 422 (time 1788565585.54 total 40.11 last 11.08)
 - mkdir 531 (time 1788565595.59 total 50.16 last 10.85)
 - mkdir 654 (time 1788565605.62 total 60.19 last 12.27)
 - mkdir 775 (time 1788565615.62 total 70.19 last 12.09)
 - mkdir 901 (time 1788565625.67 total 80.24 last 12.54)
total: 1000 mkdir in 87.08 seconds: 11.48 ops/second
Time for file creation 103 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 15 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.60 seconds: 16.60 ops/second
 - mkdir 113 (time 1788566364.97 total 10.01 last 11.29)
 - mkdir 236 (time 1788566374.99 total 20.02 last 12.28)
 - mkdir 349 (time 1788566385.01 total 30.05 last 11.27)
 - mkdir 469 (time 1788566395.05 total 40.09 last 11.95)
 - mkdir 580 (time 1788566405.25 total 50.29 last 10.89)
 - mkdir 704 (time 1788566415.30 total 60.34 last 12.34)
 - mkdir 822 (time 1788566425.36 total 70.39 last 11.73)
 - mkdir 933 (time 1788566435.39 total 80.43 last 11.06)
total: 1000 mkdir in 84.52 seconds: 11.83 ops/second
 - mkdir 122 (time 1788566368.04 total 10.07 last 12.12)
 - mkdir 245 (time 1788566378.11 total 20.14 last 12.22)
 - mkdir 363 (time 1788566388.17 total 30.20 last 11.73)
 - mkdir 488 (time 1788566398.21 total 40.24 last 12.44)
 - mkdir 594 (time 1788566408.23 total 50.26 last 10.58)
 - mkdir 722 (time 1788566418.35 total 60.38 last 12.65)
 - mkdir 842 (time 1788566428.36 total 70.39 last 11.99)
 - mkdir 985 (time 1788566438.36 total 80.39 last 14.29)
total: 1000 mkdir in 81.61 seconds: 12.25 ops/second
 - mkdir 121 (time 1788566368.57 total 10.03 last 12.06)
 - mkdir 239 (time 1788566378.61 total 20.07 last 11.75)
 - mkdir 353 (time 1788566388.62 total 30.09 last 11.38)
 - mkdir 475 (time 1788566398.71 total 40.17 last 12.10)
 - mkdir 581 (time 1788566408.73 total 50.19 last 10.57)
 - mkdir 707 (time 1788566418.76 total 60.23 last 12.56)
 - mkdir 830 (time 1788566428.78 total 70.24 last 12.28)
 - mkdir 969 (time 1788566438.85 total 80.31 last 13.81)
total: 1000 mkdir in 85.06 seconds: 11.76 ops/second
 - mkdir 115 (time 1788566367.20 total 10.07 last 11.42)
 - mkdir 234 (time 1788566377.22 total 20.09 last 11.87)
 - mkdir 344 (time 1788566387.27 total 30.14 last 10.94)
 - mkdir 466 (time 1788566397.34 total 40.21 last 12.13)
 - mkdir 573 (time 1788566407.50 total 50.37 last 10.52)
 - mkdir 702 (time 1788566417.57 total 60.44 last 12.82)
 - mkdir 816 (time 1788566427.61 total 70.48 last 11.35)
 - mkdir 949 (time 1788566437.61 total 80.48 last 13.29)
total: 1000 mkdir in 86.59 seconds: 11.55 ops/second
 - mkdir 114 (time 1788566367.83 total 10.03 last 11.37)
 - mkdir 240 (time 1788566377.88 total 20.07 last 12.54)
 - mkdir 353 (time 1788566388.04 total 30.23 last 11.12)
 - mkdir 475 (time 1788566398.07 total 40.26 last 12.16)
 - mkdir 580 (time 1788566408.08 total 50.27 last 10.49)
 - mkdir 709 (time 1788566418.13 total 60.32 last 12.84)
 - mkdir 826 (time 1788566428.14 total 70.34 last 11.68)
 - mkdir 958 (time 1788566438.20 total 80.39 last 13.12)
total: 1000 mkdir in 86.16 seconds: 11.61 ops/second
 - mkdir 116 (time 1788566368.45 total 10.01 last 11.58)
 - mkdir 233 (time 1788566378.45 total 20.02 last 11.69)
 - mkdir 346 (time 1788566388.55 total 30.12 last 11.19)
 - mkdir 467 (time 1788566398.62 total 40.19 last 12.02)
 - mkdir 573 (time 1788566408.63 total 50.20 last 10.59)
 - mkdir 696 (time 1788566418.66 total 60.22 last 12.27)
 - mkdir 813 (time 1788566428.67 total 70.24 last 11.69)
 - mkdir 950 (time 1788566438.72 total 80.29 last 13.63)
total: 1000 mkdir in 88.50 seconds: 11.30 ops/second
 - mkdir 112 (time 1788566368.46 total 10.07 last 11.13)
 - mkdir 229 (time 1788566378.50 total 20.10 last 11.66)
 - mkdir 338 (time 1788566388.53 total 30.13 last 10.87)
 - mkdir 461 (time 1788566398.66 total 40.27 last 12.13)
 - mkdir 570 (time 1788566408.73 total 50.33 last 10.83)
 - mkdir 697 (time 1788566418.75 total 60.36 last 12.67)
 - mkdir 812 (time 1788566428.78 total 70.39 last 11.46)
 - mkdir 949 (time 1788566438.79 total 80.40 last 13.69)
total: 1000 mkdir in 88.88 seconds: 11.25 ops/second
 - mkdir 119 (time 1788566368.14 total 10.09 last 11.79)
 - mkdir 238 (time 1788566378.24 total 20.19 last 11.79)
 - mkdir 345 (time 1788566388.30 total 30.24 last 10.64)
 - mkdir 465 (time 1788566398.38 total 40.33 last 11.90)
 - mkdir 570 (time 1788566408.43 total 50.37 last 10.45)
 - mkdir 692 (time 1788566418.44 total 60.38 last 12.19)
 - mkdir 807 (time 1788566428.66 total 70.61 last 11.25)
 - mkdir 941 (time 1788566438.72 total 80.67 last 13.32)
total: 1000 mkdir in 90.11 seconds: 11.10 ops/second
 - mkdir 110 (time 1788566368.29 total 10.01 last 10.99)
 - mkdir 223 (time 1788566378.39 total 20.12 last 11.18)
 - mkdir 329 (time 1788566388.42 total 30.14 last 10.58)
 - mkdir 452 (time 1788566398.52 total 40.24 last 12.17)
 - mkdir 560 (time 1788566408.54 total 50.27 last 10.77)
 - mkdir 684 (time 1788566418.62 total 60.34 last 12.31)
 - mkdir 797 (time 1788566428.62 total 70.34 last 11.29)
 - mkdir 929 (time 1788566438.64 total 80.36 last 13.17)
 - mkdir 994 (time 1788566448.71 total 90.43 last 6.46)
total: 1000 mkdir in 90.90 seconds: 11.00 ops/second
 - mkdir 111 (time 1788566368.78 total 10.02 last 11.07)
 - mkdir 221 (time 1788566378.79 total 20.03 last 11.00)
 - mkdir 330 (time 1788566388.80 total 30.04 last 10.89)
 - mkdir 452 (time 1788566398.86 total 40.10 last 12.12)
 - mkdir 558 (time 1788566408.94 total 50.18 last 10.51)
 - mkdir 681 (time 1788566419.01 total 60.25 last 12.22)
 - mkdir 790 (time 1788566429.04 total 70.28 last 10.87)
 - mkdir 921 (time 1788566439.11 total 80.35 last 13.00)
 - mkdir 984 (time 1788566449.21 total 90.45 last 6.24)
total: 1000 mkdir in 91.81 seconds: 10.89 ops/second
Time for file creation 105 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788566459 ; total 0 ; last 0)
total: 1000 unlinks in 42 seconds: 23.809525 unlinks/second
 - unlinked 0 (time 1788566461 ; total 0 ; last 0)
total: 1000 unlinks in 40 seconds: 25.000000 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 40 seconds: 25.000000 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 40 seconds: 25.000000 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 41 seconds: 24.390244 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 41 seconds: 24.390244 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 41 seconds: 24.390244 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 42 seconds: 23.809525 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 42 seconds: 23.809525 unlinks/second
 - unlinked 0 (time 1788566463 ; total 0 ; last 0)
total: 1000 unlinks in 42 seconds: 23.809525 unlinks/second
Time for file removal 58 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8