== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:46:43 (1744883203) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) oleg360-server: oleg360-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg360-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg360-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2871 (time 1744883241.04 total 10.00 last 287.04) - open/close 6065 (time 1744883251.05 total 20.01 last 319.25) - open/close 8668 (time 1744883261.05 total 30.01 last 260.29) - open/close 10000 (time 1744883265.12 total 34.08 last 327.14) - open/close 12459 (time 1744883275.12 total 44.08 last 245.87) - open/close 15018 (time 1744883285.12 total 54.08 last 255.84) - open/close 17250 (time 1744883295.12 total 64.08 last 223.18) - open/close 19753 (time 1744883305.12 total 74.08 last 250.28) - open/close 22711 (time 1744883315.12 total 84.09 last 295.77) - open/close 25824 (time 1744883325.13 total 94.09 last 311.25) - open/close 29211 (time 1744883335.13 total 104.09 last 338.66) - open/close 30000 (time 1744883337.15 total 106.11 last 390.89) - open/close 33780 (time 1744883347.15 total 116.11 last 377.90) - open/close 37280 (time 1744883357.15 total 126.11 last 349.99) - open/close 40000 (time 1744883365.10 total 134.06 last 342.20) - open/close 43842 (time 1744883375.10 total 144.06 last 384.20) - open/close 47787 (time 1744883385.10 total 154.06 last 394.49) - open/close 50000 (time 1744883390.75 total 159.71 last 391.81) - open/close 53548 (time 1744883400.75 total 169.71 last 354.72) - open/close 56489 (time 1744883410.75 total 179.71 last 294.09) - open/close 59207 (time 1744883420.75 total 189.71 last 271.78) - open/close 60000 (time 1744883423.27 total 192.24 last 313.96) - open/close 62393 (time 1744883433.28 total 202.24 last 239.16) - open/close 64454 (time 1744883443.28 total 212.25 last 206.02) total: 64768 open/close in 213.90 seconds: 302.79 ops/second - /unlink 3229 (time 1744883456.05 total 10.00 last 322.84) - /unlink 6726 (time 1744883466.05 total 20.00 last 349.67) - /unlink 10000 (time 1744883471.78 total 25.73 last 571.49) - /unlink 15216 (time 1744883481.78 total 35.73 last 521.52) - /unlink 20000 (time 1744883489.12 total 43.07 last 651.64) - /unlink 28020 (time 1744883499.12 total 53.08 last 801.91) - /unlink 30000 (time 1744883501.84 total 55.79 last 728.29) - /unlink 37328 (time 1744883511.84 total 65.80 last 732.74) - /unlink 40000 (time 1744883515.65 total 69.60 last 701.74) - /unlink 47875 (time 1744883525.65 total 79.60 last 787.44) - /unlink 50000 (time 1744883528.22 total 82.18 last 826.34) - /unlink 57373 (time 1744883538.22 total 92.18 last 737.21) - /unlink 60000 (time 1744883541.90 total 95.85 last 715.08) total: 64768 /unlink in 103.19 seconds: 627.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3616 (time 1744883578.67 total 10.00 last 361.56) - open/close 7754 (time 1744883588.67 total 20.00 last 413.71) - open/close 10000 (time 1744883594.04 total 25.37 last 418.23) - open/close 14231 (time 1744883604.04 total 35.38 last 423.03) - open/close 18387 (time 1744883614.05 total 45.38 last 415.54) - open/close 20000 (time 1744883617.73 total 49.06 last 437.35) - open/close 23511 (time 1744883627.73 total 59.07 last 351.06) - open/close 26822 (time 1744883637.74 total 69.07 last 331.06) - open/close 30000 (time 1744883647.01 total 78.34 last 342.81) - open/close 33507 (time 1744883657.01 total 88.34 last 350.63) - open/close 37831 (time 1744883667.01 total 98.34 last 432.33) - open/close 40000 (time 1744883671.63 total 102.96 last 469.51) - open/close 44327 (time 1744883681.63 total 112.96 last 432.68) - open/close 48160 (time 1744883691.63 total 122.96 last 383.23) - open/close 50000 (time 1744883696.01 total 127.34 last 420.03) - open/close 54879 (time 1744883706.01 total 137.34 last 487.85) - open/close 59845 (time 1744883716.02 total 147.35 last 496.53) total: 64768 open/close in 156.84 seconds: 412.95 ops/second - /unlink 8178 (time 1744883736.31 total 10.00 last 817.75) - /unlink 10000 (time 1744883738.62 total 12.31 last 789.28) - /unlink 18203 (time 1744883748.62 total 22.31 last 820.28) - /unlink 27287 (time 1744883758.62 total 32.31 last 908.31) - /unlink 30000 (time 1744883761.76 total 35.45 last 863.45) - /unlink 37301 (time 1744883771.76 total 45.45 last 730.04) - /unlink 40000 (time 1744883775.14 total 48.83 last 799.78) - /unlink 48396 (time 1744883785.14 total 58.83 last 839.59) - /unlink 57725 (time 1744883795.14 total 68.83 last 932.83) - /unlink 60000 (time 1744883797.55 total 71.24 last 942.27) total: 64768 /unlink in 77.27 seconds: 838.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4731 (time 1744883833.61 total 10.00 last 473.03) - open/close 9829 (time 1744883843.61 total 20.00 last 509.72) - open/close 14371 (time 1744883853.62 total 30.01 last 454.09) - open/close 17696 (time 1744883863.62 total 40.01 last 332.43) - open/close 20000 (time 1744883870.05 total 46.44 last 358.43) - open/close 24555 (time 1744883880.05 total 56.44 last 455.44) - open/close 28416 (time 1744883890.05 total 66.44 last 385.96) - open/close 30000 (time 1744883894.16 total 70.55 last 385.75) - open/close 34216 (time 1744883904.16 total 80.55 last 421.55) - open/close 38447 (time 1744883914.16 total 90.55 last 423.04) - open/close 40000 (time 1744883917.44 total 93.83 last 474.05) - open/close 44523 (time 1744883927.44 total 103.83 last 452.15) - open/close 49303 (time 1744883937.44 total 113.83 last 477.93) - open/close 53798 (time 1744883947.44 total 123.83 last 449.50) - open/close 58630 (time 1744883957.44 total 133.83 last 483.17) - open/close 60000 (time 1744883960.00 total 136.39 last 534.93) total: 64768 open/close in 145.37 seconds: 445.53 ops/second - /unlink 3107 (time 1744883979.86 total 10.00 last 310.62) - /unlink 4642 (time 1744883989.86 total 20.01 last 153.44) - /unlink 7432 (time 1744883999.86 total 30.01 last 278.97) - /unlink 10000 (time 1744884006.96 total 37.10 last 361.85) - /unlink 13954 (time 1744884016.96 total 47.11 last 395.29) - /unlink 17085 (time 1744884026.97 total 57.11 last 312.89) - /unlink 20000 (time 1744884036.64 total 66.78 last 301.50) - /unlink 25569 (time 1744884046.64 total 76.78 last 556.75) - /unlink 30000 (time 1744884055.19 total 85.34 last 517.91) - /unlink 36710 (time 1744884065.20 total 95.34 last 670.90) - /unlink 40000 (time 1744884073.06 total 103.21 last 418.12) - /unlink 45015 (time 1744884083.07 total 113.21 last 501.40) - /unlink 48687 (time 1744884093.07 total 123.21 last 367.13) - /unlink 50000 (time 1744884097.31 total 127.46 last 309.41) - /unlink 53296 (time 1744884107.31 total 137.46 last 329.58) - /unlink 56374 (time 1744884117.31 total 147.46 last 307.75) - /unlink 59111 (time 1744884127.32 total 157.46 last 273.63) - /unlink 60000 (time 1744884130.72 total 160.86 last 261.37) - /unlink 62700 (time 1744884140.72 total 170.87 last 269.95) total: 64768 /unlink in 178.27 seconds: 363.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2544 (time 1744884174.15 total 10.00 last 254.30) - open/close 5070 (time 1744884184.15 total 20.01 last 252.55) - open/close 7183 (time 1744884194.16 total 30.01 last 211.23) - open/close 9481 (time 1744884204.17 total 40.02 last 229.50) - open/close 10000 (time 1744884206.38 total 42.23 last 234.96) - open/close 12336 (time 1744884216.38 total 52.23 last 233.58) - open/close 15209 (time 1744884226.38 total 62.24 last 287.21) - open/close 17867 (time 1744884236.39 total 72.24 last 265.66) - open/close 20000 (time 1744884245.29 total 81.15 last 239.50) - open/close 22850 (time 1744884255.30 total 91.15 last 284.89) - open/close 25762 (time 1744884265.30 total 101.15 last 291.15) - open/close 28159 (time 1744884275.30 total 111.15 last 239.70) - open/close 30000 (time 1744884281.07 total 116.93 last 318.79) - open/close 33046 (time 1744884291.08 total 126.93 last 304.55) - open/close 36638 (time 1744884301.08 total 136.93 last 359.14) - open/close 39371 (time 1744884311.08 total 146.93 last 273.27) - open/close 40000 (time 1744884313.80 total 149.66 last 230.78) - open/close 43130 (time 1744884323.81 total 159.66 last 312.93) - open/close 46035 (time 1744884333.81 total 169.66 last 290.41) - open/close 49577 (time 1744884343.81 total 179.67 last 354.10) - open/close 52853 (time 1744884353.81 total 189.67 last 327.53) - open/close 56489 (time 1744884363.82 total 199.67 last 363.41) - open/close 60000 (time 1744884372.77 total 208.62 last 392.31) - open/close 64643 (time 1744884382.77 total 218.62 last 464.22) total: 64768 open/close in 218.86 seconds: 295.94 ops/second - /unlink 7306 (time 1744884393.87 total 10.00 last 730.54) - /unlink 10000 (time 1744884397.82 total 13.95 last 681.77) - /unlink 17422 (time 1744884407.82 total 23.95 last 742.17) - /unlink 20000 (time 1744884411.39 total 27.52 last 723.17) - /unlink 26574 (time 1744884421.39 total 37.52 last 657.38) - /unlink 30000 (time 1744884426.58 total 42.71 last 660.23) - /unlink 37053 (time 1744884436.58 total 52.71 last 705.23) - /unlink 40000 (time 1744884440.98 total 57.11 last 670.01) - /unlink 46609 (time 1744884450.98 total 67.11 last 660.88) - /unlink 50000 (time 1744884456.23 total 72.36 last 645.03) - /unlink 56888 (time 1744884466.24 total 82.36 last 688.75) - /unlink 60000 (time 1744884470.98 total 87.11 last 655.76) total: 64768 /unlink in 94.95 seconds: 682.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3351 (time 1744884509.69 total 10.00 last 335.08) - open/close 6854 (time 1744884519.69 total 20.00 last 350.20) - open/close 10000 (time 1744884528.67 total 28.98 last 350.38) - open/close 14054 (time 1744884538.67 total 38.98 last 405.34) - open/close 17979 (time 1744884548.67 total 48.98 last 392.48) - open/close 20000 (time 1744884554.11 total 54.42 last 371.94) - open/close 23794 (time 1744884564.11 total 64.42 last 379.37) - open/close 27186 (time 1744884574.11 total 74.42 last 339.13) - open/close 30000 (time 1744884582.00 total 82.31 last 356.68) - open/close 33506 (time 1744884592.00 total 92.31 last 350.51) - open/close 36951 (time 1744884602.00 total 102.32 last 344.40) - open/close 40000 (time 1744884611.23 total 111.54 last 330.37) - open/close 43012 (time 1744884621.24 total 121.55 last 301.11) - open/close 46319 (time 1744884631.24 total 131.55 last 330.67) - open/close 49338 (time 1744884641.24 total 141.55 last 301.88) - open/close 52694 (time 1744884651.24 total 151.55 last 335.56) - open/close 55881 (time 1744884661.24 total 161.55 last 318.67) - open/close 58823 (time 1744884671.24 total 171.55 last 294.18) - open/close 60000 (time 1744884674.66 total 174.97 last 344.45) - open/close 63105 (time 1744884684.66 total 184.97 last 310.41) total: 64768 open/close in 189.92 seconds: 341.02 ops/second - /unlink 6326 (time 1744884700.53 total 10.00 last 632.57) - /unlink 10000 (time 1744884706.32 total 15.79 last 634.53) - /unlink 16886 (time 1744884716.32 total 25.79 last 688.54) - /unlink 20000 (time 1744884720.78 total 30.25 last 699.08) - /unlink 27420 (time 1744884730.78 total 40.25 last 741.96) - /unlink 30000 (time 1744884734.13 total 43.60 last 768.99) - /unlink 37182 (time 1744884744.13 total 53.60 last 718.11) - /unlink 40000 (time 1744884748.24 total 57.70 last 687.22) - /unlink 48201 (time 1744884758.24 total 67.70 last 820.08) - /unlink 50000 (time 1744884761.13 total 70.60 last 620.80) - /unlink 57388 (time 1744884771.13 total 80.60 last 738.75) - /unlink 60000 (time 1744884774.95 total 84.42 last 684.67) total: 64768 /unlink in 90.27 seconds: 717.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2