== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:44:14 (1739533454) Stopping clients: oleg230-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg230-client.virtnet /mnt/lustre2 (opts:-f) oleg230-server: oleg230-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg230-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg230-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1200 (time 1739533516.96 total 10.01 last 119.91) - open/close 2388 (time 1739533526.96 total 20.01 last 118.76) - open/close 3760 (time 1739533536.97 total 30.02 last 137.14) - open/close 5057 (time 1739533546.97 total 40.02 last 129.66) - open/close 6416 (time 1739533556.98 total 50.02 last 135.84) - open/close 7699 (time 1739533566.98 total 60.03 last 128.26) - open/close 9119 (time 1739533576.98 total 70.03 last 141.95) - open/close 10000 (time 1739533583.89 total 76.94 last 127.57) - open/close 11371 (time 1739533593.89 total 86.94 last 137.08) - open/close 12798 (time 1739533603.89 total 96.94 last 142.70) - open/close 14383 (time 1739533613.89 total 106.94 last 158.49) - open/close 15829 (time 1739533623.89 total 116.94 last 144.56) - open/close 17583 (time 1739533633.90 total 126.94 last 175.36) - open/close 19101 (time 1739533643.90 total 136.94 last 151.78) - open/close 20000 (time 1739533650.80 total 143.84 last 130.28) - open/close 21301 (time 1739533660.80 total 153.85 last 130.08) - open/close 22832 (time 1739533670.80 total 163.85 last 153.04) - open/close 24443 (time 1739533680.81 total 173.85 last 161.05) - open/close 26049 (time 1739533690.81 total 183.86 last 160.52) - open/close 27818 (time 1739533700.82 total 193.86 last 176.81) - open/close 29814 (time 1739533710.82 total 203.86 last 199.57) - open/close 31522 (time 1739533720.82 total 213.87 last 170.75) - open/close 33256 (time 1739533730.82 total 223.87 last 173.40) - open/close 34878 (time 1739533740.82 total 233.87 last 162.13) - open/close 37053 (time 1739533750.83 total 243.88 last 217.32) - open/close 38892 (time 1739533760.83 total 253.88 last 183.88) - open/close 40000 (time 1739533765.08 total 258.13 last 260.75) - open/close 41623 (time 1739533775.09 total 268.13 last 162.25) - open/close 43105 (time 1739533785.09 total 278.14 last 148.15) - open/close 45180 (time 1739533795.10 total 288.14 last 207.35) - open/close 47233 (time 1739533805.10 total 298.15 last 205.28) - open/close 49285 (time 1739533815.10 total 308.15 last 205.18) - open/close 50000 (time 1739533818.39 total 311.44 last 216.97) - open/close 52243 (time 1739533828.40 total 321.44 last 224.29) - open/close 54220 (time 1739533838.40 total 331.45 last 197.65) - open/close 55901 (time 1739533848.40 total 341.45 last 168.00) - open/close 57351 (time 1739533858.41 total 351.46 last 144.93) - open/close 58845 (time 1739533868.41 total 361.46 last 149.34) - open/close 60000 (time 1739533875.51 total 368.56 last 162.78) - open/close 61514 (time 1739533885.51 total 378.56 last 151.30) - open/close 63341 (time 1739533895.52 total 388.56 last 182.65) total: 64768 open/close in 397.41 seconds: 162.98 ops/second - /unlink 2415 (time 1739533915.95 total 10.01 last 241.34) - /unlink 4813 (time 1739533925.95 total 20.01 last 239.77) - /unlink 7601 (time 1739533935.95 total 30.01 last 278.77) - /unlink 10000 (time 1739533944.89 total 38.95 last 268.45) - /unlink 12552 (time 1739533954.89 total 48.95 last 255.11) - /unlink 15299 (time 1739533964.89 total 58.95 last 274.67) - /unlink 18110 (time 1739533974.89 total 68.95 last 281.06) - /unlink 20000 (time 1739533980.91 total 74.97 last 314.18) - /unlink 23191 (time 1739533990.91 total 84.97 last 319.07) - /unlink 26266 (time 1739534000.91 total 94.97 last 307.46) - /unlink 28989 (time 1739534010.91 total 104.97 last 272.25) - /unlink 30000 (time 1739534014.34 total 108.40 last 294.76) - /unlink 33958 (time 1739534024.35 total 118.40 last 395.71) - /unlink 38031 (time 1739534034.35 total 128.40 last 407.26) - /unlink 40000 (time 1739534037.91 total 131.97 last 552.98) - /unlink 44111 (time 1739534047.91 total 141.97 last 411.09) - /unlink 47064 (time 1739534057.91 total 151.97 last 295.25) - /unlink 49620 (time 1739534067.91 total 161.97 last 255.56) - /unlink 52970 (time 1739534077.91 total 171.97 last 334.95) - /unlink 56291 (time 1739534087.91 total 181.97 last 332.04) - /unlink 60000 (time 1739534097.38 total 191.43 last 392.00) - /unlink 63436 (time 1739534107.39 total 201.44 last 343.26) total: 64768 /unlink in 206.06 seconds: 314.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1612 (time 1739534140.34 total 10.00 last 161.16) - open/close 3636 (time 1739534150.35 total 20.00 last 202.36) - open/close 5326 (time 1739534160.35 total 30.00 last 168.99) - open/close 7349 (time 1739534170.35 total 40.01 last 202.21) - open/close 9261 (time 1739534180.35 total 50.01 last 191.16) - open/close 10000 (time 1739534184.67 total 54.33 last 171.25) - open/close 12052 (time 1739534194.67 total 64.33 last 205.18) - open/close 13966 (time 1739534204.67 total 74.33 last 191.33) - open/close 15708 (time 1739534214.68 total 84.34 last 174.09) - open/close 17638 (time 1739534224.68 total 94.34 last 192.95) - open/close 19260 (time 1739534234.68 total 104.34 last 162.19) - open/close 20000 (time 1739534238.48 total 108.14 last 195.00) - open/close 22065 (time 1739534248.48 total 118.14 last 206.48) - open/close 24311 (time 1739534258.48 total 128.14 last 224.50) - open/close 26231 (time 1739534268.49 total 138.15 last 191.89) - open/close 28016 (time 1739534278.49 total 148.15 last 178.39) - open/close 30000 (time 1739534286.77 total 156.43 last 239.63) - open/close 32188 (time 1739534296.78 total 166.44 last 218.68) - open/close 33950 (time 1739534306.78 total 176.44 last 176.10) - open/close 35628 (time 1739534316.80 total 186.45 last 167.61) - open/close 38072 (time 1739534326.80 total 196.46 last 244.38) - open/close 40000 (time 1739534332.77 total 202.43 last 322.51) - open/close 43233 (time 1739534342.78 total 212.44 last 323.19) - open/close 45473 (time 1739534352.78 total 222.44 last 223.95) - open/close 47678 (time 1739534362.78 total 232.44 last 220.50) - open/close 49775 (time 1739534372.78 total 242.44 last 209.68) - open/close 51827 (time 1739534382.78 total 252.44 last 205.13) - open/close 53795 (time 1739534392.79 total 262.45 last 196.75) - open/close 57332 (time 1739534402.79 total 272.45 last 353.69) - open/close 60000 (time 1739534409.74 total 279.40 last 383.50) - open/close 64077 (time 1739534419.75 total 289.40 last 407.63) total: 64768 open/close in 292.02 seconds: 221.80 ops/second - /unlink 6443 (time 1739534433.33 total 10.00 last 644.23) - /unlink 10000 (time 1739534438.87 total 15.55 last 641.57) - /unlink 15900 (time 1739534448.87 total 25.55 last 589.93) - /unlink 20000 (time 1739534456.25 total 32.93 last 555.65) - /unlink 25333 (time 1739534466.25 total 42.93 last 533.29) - /unlink 30000 (time 1739534473.82 total 50.49 last 616.88) - /unlink 35879 (time 1739534483.82 total 60.49 last 587.88) - /unlink 40000 (time 1739534489.81 total 66.48 last 687.79) - /unlink 46072 (time 1739534499.81 total 76.48 last 607.16) - /unlink 50000 (time 1739534505.29 total 81.96 last 717.26) - /unlink 57480 (time 1739534515.29 total 91.96 last 747.99) - /unlink 60000 (time 1739534518.29 total 94.97 last 838.14) total: 64768 /unlink in 103.40 seconds: 626.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2660 (time 1739534556.16 total 10.00 last 265.95) - open/close 5655 (time 1739534566.16 total 20.00 last 299.49) - open/close 9158 (time 1739534576.16 total 30.00 last 350.30) - open/close 10000 (time 1739534578.79 total 32.63 last 320.35) - open/close 14495 (time 1739534588.79 total 42.63 last 449.41) - open/close 19414 (time 1739534598.79 total 52.63 last 491.90) - open/close 24689 (time 1739534608.79 total 62.63 last 527.50) - open/close 29371 (time 1739534618.79 total 72.63 last 468.15) - open/close 33881 (time 1739534628.80 total 82.64 last 450.91) - open/close 37984 (time 1739534638.80 total 92.64 last 410.26) - open/close 40000 (time 1739534643.93 total 97.77 last 392.61) - open/close 44084 (time 1739534653.93 total 107.77 last 408.36) - open/close 48019 (time 1739534663.93 total 117.77 last 393.44) - open/close 50000 (time 1739534669.22 total 123.06 last 374.99) - open/close 53764 (time 1739534679.22 total 133.06 last 376.29) - open/close 58000 (time 1739534689.22 total 143.06 last 423.60) - open/close 60000 (time 1739534693.47 total 147.31 last 470.97) - open/close 64251 (time 1739534703.47 total 157.31 last 425.07) total: 64768 open/close in 158.66 seconds: 408.23 ops/second - /unlink 5797 (time 1739534715.71 total 10.00 last 579.69) - /unlink 10000 (time 1739534721.75 total 16.04 last 695.97) - /unlink 16017 (time 1739534731.75 total 26.04 last 601.63) - /unlink 20000 (time 1739534738.84 total 33.13 last 561.84) - /unlink 27915 (time 1739534748.84 total 43.13 last 791.45) - /unlink 30000 (time 1739534751.74 total 46.02 last 720.48) - /unlink 33130 (time 1739534761.74 total 56.03 last 312.93) - /unlink 35294 (time 1739534771.74 total 66.03 last 216.39) - /unlink 37153 (time 1739534781.74 total 76.03 last 185.87) - /unlink 39980 (time 1739534791.74 total 86.03 last 282.64) - /unlink 42959 (time 1739534801.74 total 96.03 last 297.86) - /unlink 46532 (time 1739534811.74 total 106.03 last 357.30) - /unlink 49144 (time 1739534821.75 total 116.04 last 261.04) - /unlink 50000 (time 1739534825.50 total 119.79 last 228.03) - /unlink 52500 (time 1739534835.51 total 129.79 last 249.95) - /unlink 55320 (time 1739534845.51 total 139.80 last 281.97) - /unlink 58130 (time 1739534855.51 total 149.80 last 280.95) - /unlink 60000 (time 1739534861.65 total 155.93 last 304.70) - /unlink 63609 (time 1739534871.65 total 165.94 last 360.84) total: 64768 /unlink in 169.32 seconds: 382.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1724 (time 1739534905.50 total 10.01 last 172.31) - open/close 3230 (time 1739534915.50 total 20.01 last 150.59) - open/close 4844 (time 1739534925.51 total 30.01 last 161.34) - open/close 6282 (time 1739534935.51 total 40.01 last 143.80) - open/close 7797 (time 1739534945.51 total 50.01 last 151.47) - open/close 9171 (time 1739534955.51 total 60.01 last 137.39) - open/close 10000 (time 1739534961.33 total 65.84 last 142.36) - open/close 11398 (time 1739534971.34 total 75.84 last 139.79) - open/close 12784 (time 1739534981.34 total 85.84 last 138.59) - open/close 14168 (time 1739534991.34 total 95.84 last 138.37) - open/close 15513 (time 1739535001.34 total 105.84 last 134.47) - open/close 16862 (time 1739535011.35 total 115.85 last 134.83) - open/close 18275 (time 1739535021.35 total 125.85 last 141.30) - open/close 19877 (time 1739535031.35 total 135.85 last 160.19) - open/close 21461 (time 1739535041.35 total 145.85 last 158.36) - open/close 22924 (time 1739535051.35 total 155.85 last 146.29) - open/close 24516 (time 1739535061.35 total 165.85 last 159.20) - open/close 26010 (time 1739535071.35 total 175.85 last 149.38) - open/close 27438 (time 1739535081.35 total 185.86 last 142.76) - open/close 28794 (time 1739535091.36 total 195.86 last 135.57) - open/close 30000 (time 1739535099.68 total 204.18 last 144.96) - open/close 31589 (time 1739535109.68 total 214.18 last 158.84) - open/close 33352 (time 1739535119.68 total 224.18 last 176.28) - open/close 35291 (time 1739535129.69 total 234.19 last 193.76) - open/close 37008 (time 1739535139.69 total 244.19 last 171.62) - open/close 39072 (time 1739535149.69 total 254.20 last 206.38) - open/close 40000 (time 1739535154.65 total 259.15 last 187.16) - open/close 42237 (time 1739535164.65 total 269.16 last 223.66) - open/close 44806 (time 1739535174.66 total 279.16 last 256.81) - open/close 47285 (time 1739535184.66 total 289.16 last 247.84) - open/close 49242 (time 1739535194.66 total 299.16 last 195.64) - open/close 50000 (time 1739535199.38 total 303.88 last 160.78) - open/close 51662 (time 1739535209.38 total 313.88 last 166.14) - open/close 53172 (time 1739535219.39 total 323.89 last 150.85) - open/close 54769 (time 1739535229.40 total 333.90 last 159.62) - open/close 56490 (time 1739535239.40 total 343.90 last 172.07) - open/close 58172 (time 1739535249.40 total 353.90 last 168.20) - open/close 59905 (time 1739535259.40 total 363.90 last 173.27) - open/close 61655 (time 1739535269.41 total 373.91 last 174.91) - open/close 63423 (time 1739535279.41 total 383.91 last 176.71) total: 64768 open/close in 391.88 seconds: 165.27 ops/second - /unlink 2935 (time 1739535298.86 total 10.00 last 293.43) - /unlink 5799 (time 1739535308.86 total 20.00 last 286.38) - /unlink 8319 (time 1739535318.86 total 30.00 last 251.97) - /unlink 10000 (time 1739535325.06 total 36.20 last 271.22) - /unlink 12806 (time 1739535335.06 total 46.21 last 280.53) - /unlink 15399 (time 1739535345.07 total 56.21 last 259.24) - /unlink 17989 (time 1739535355.07 total 66.21 last 258.97) - /unlink 20000 (time 1739535363.24 total 74.39 last 245.93) - /unlink 22185 (time 1739535373.24 total 84.39 last 218.49) - /unlink 24669 (time 1739535383.25 total 94.39 last 248.32) - /unlink 27272 (time 1739535393.25 total 104.39 last 260.22) - /unlink 29920 (time 1739535403.25 total 114.39 last 264.76) - /unlink 32462 (time 1739535413.25 total 124.40 last 254.17) - /unlink 34989 (time 1739535423.26 total 134.40 last 252.59) - /unlink 37389 (time 1739535433.26 total 144.40 last 239.98) - /unlink 39843 (time 1739535443.26 total 154.40 last 245.31) - /unlink 42152 (time 1739535453.27 total 164.41 last 230.84) - /unlink 44518 (time 1739535463.27 total 174.41 last 236.58) - /unlink 47159 (time 1739535473.27 total 184.41 last 264.09) - /unlink 49992 (time 1739535483.27 total 194.41 last 283.28) - /unlink 52975 (time 1739535493.27 total 204.41 last 298.30) - /unlink 55714 (time 1739535503.27 total 214.41 last 273.88) - /unlink 58215 (time 1739535513.28 total 224.42 last 249.93) - /unlink 60000 (time 1739535520.59 total 231.73 last 244.14) - /unlink 62484 (time 1739535530.59 total 241.73 last 248.38) total: 64768 /unlink in 251.51 seconds: 257.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1618 (time 1739535571.70 total 10.01 last 161.58) - open/close 3456 (time 1739535581.71 total 20.02 last 183.71) - open/close 5414 (time 1739535591.71 total 30.02 last 195.74) - open/close 7290 (time 1739535601.72 total 40.03 last 187.39) - open/close 9470 (time 1739535611.72 total 50.04 last 217.96) - open/close 10000 (time 1739535614.52 total 52.83 last 189.56) - open/close 12110 (time 1739535624.52 total 62.83 last 210.97) - open/close 13962 (time 1739535634.52 total 72.84 last 185.15) - open/close 16039 (time 1739535644.53 total 82.84 last 207.62) - open/close 17982 (time 1739535654.53 total 92.85 last 194.18) - open/close 19960 (time 1739535664.54 total 102.85 last 197.74) - open/close 21572 (time 1739535674.54 total 112.85 last 161.12) - open/close 23108 (time 1739535684.55 total 122.86 last 153.56) - open/close 24904 (time 1739535694.55 total 132.86 last 179.54) - open/close 26731 (time 1739535704.55 total 142.86 last 182.65) - open/close 28649 (time 1739535714.56 total 152.87 last 191.61) - open/close 30000 (time 1739535722.63 total 160.94 last 167.46) - open/close 31606 (time 1739535732.64 total 170.95 last 160.46) - open/close 33535 (time 1739535742.64 total 180.95 last 192.83) - open/close 35622 (time 1739535752.65 total 190.96 last 208.63) - open/close 37651 (time 1739535762.65 total 200.96 last 202.83) - open/close 39691 (time 1739535772.65 total 210.97 last 203.89) - open/close 41761 (time 1739535782.66 total 220.97 last 206.94) - open/close 43801 (time 1739535792.66 total 230.97 last 203.96) - open/close 45859 (time 1739535802.66 total 240.97 last 205.75) - open/close 47675 (time 1739535812.67 total 250.98 last 181.52) - open/close 49808 (time 1739535822.67 total 260.98 last 213.28) - open/close 51844 (time 1739535832.67 total 270.98 last 203.56) - open/close 53474 (time 1739535842.67 total 280.98 last 162.96) - open/close 55398 (time 1739535852.67 total 290.98 last 192.38) - open/close 57477 (time 1739535862.67 total 300.98 last 207.88) - open/close 59471 (time 1739535872.67 total 310.98 last 199.40) - open/close 60000 (time 1739535874.93 total 313.24 last 234.72) - open/close 62480 (time 1739535884.93 total 323.24 last 247.99) total: 64768 open/close in 328.95 seconds: 196.89 ops/second - /unlink 5858 (time 1739535901.55 total 10.00 last 585.71) - /unlink 10000 (time 1739535909.60 total 18.06 last 514.16) - /unlink 16497 (time 1739535919.60 total 28.06 last 649.67) - /unlink 20000 (time 1739535925.99 total 34.45 last 548.22) - /unlink 25447 (time 1739535935.99 total 44.45 last 544.63) - /unlink 29487 (time 1739535946.00 total 54.45 last 403.94) - /unlink 34017 (time 1739535956.00 total 64.45 last 452.93) - /unlink 38761 (time 1739535966.00 total 74.45 last 474.32) - /unlink 40000 (time 1739535969.00 total 77.45 last 413.16) - /unlink 44310 (time 1739535979.00 total 87.45 last 430.96) - /unlink 49564 (time 1739535989.00 total 97.45 last 525.39) - /unlink 54587 (time 1739535999.01 total 107.46 last 501.75) - /unlink 59119 (time 1739536009.01 total 117.47 last 453.16) - /unlink 64004 (time 1739536019.01 total 127.47 last 488.37) total: 64768 /unlink in 129.17 seconds: 501.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server unloading modules via unload_modules_local on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2