== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:25:59 (1722237959) Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) oleg122-server: oleg122-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg122-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg122-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock oleg122-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2606 (time 1722237998.29 total 10.00 last 260.51) - open/close 5293 (time 1722238008.29 total 20.00 last 268.69) - open/close 8596 (time 1722238018.29 total 30.00 last 330.28) - open/close 10000 (time 1722238022.62 total 34.34 last 323.91) - open/close 13047 (time 1722238032.62 total 44.34 last 304.64) - open/close 16107 (time 1722238042.62 total 54.34 last 305.96) - open/close 19437 (time 1722238052.63 total 64.34 last 332.96) - open/close 23148 (time 1722238062.63 total 74.34 last 371.07) - open/close 27310 (time 1722238072.63 total 84.34 last 416.17) - open/close 30000 (time 1722238078.88 total 90.60 last 430.32) - open/close 34157 (time 1722238088.88 total 100.60 last 415.62) - open/close 38536 (time 1722238098.88 total 110.60 last 437.88) - open/close 40000 (time 1722238101.98 total 113.70 last 471.66) - open/close 44920 (time 1722238111.99 total 123.70 last 491.96) - open/close 49516 (time 1722238121.99 total 133.70 last 459.54) - open/close 53950 (time 1722238131.99 total 143.71 last 443.33) - open/close 58239 (time 1722238141.99 total 153.71 last 428.84) - open/close 60000 (time 1722238146.04 total 157.76 last 434.70) - open/close 64252 (time 1722238156.04 total 167.76 last 425.18) total: 64768 open/close in 168.95 seconds: 383.35 ops/second - /unlink 7142 (time 1722238168.15 total 10.00 last 714.19) - /unlink 10000 (time 1722238172.15 total 14.00 last 714.36) - /unlink 17505 (time 1722238182.15 total 24.00 last 750.46) - /unlink 20000 (time 1722238185.55 total 27.41 last 732.84) - /unlink 27124 (time 1722238195.55 total 37.41 last 712.37) - /unlink 30000 (time 1722238199.53 total 41.39 last 722.37) - /unlink 37620 (time 1722238209.54 total 51.39 last 761.63) - /unlink 40000 (time 1722238212.48 total 54.33 last 809.77) - /unlink 47783 (time 1722238222.48 total 64.33 last 778.27) - /unlink 50000 (time 1722238225.39 total 67.24 last 761.38) - /unlink 57828 (time 1722238235.39 total 77.24 last 782.79) - /unlink 60000 (time 1722238237.99 total 79.84 last 835.28) total: 64768 /unlink in 85.42 seconds: 758.23 ops/second - open/close 4631 (time 1722238254.42 total 10.00 last 463.06) - open/close 9233 (time 1722238264.42 total 20.00 last 460.18) - open/close 13622 (time 1722238274.42 total 30.00 last 438.85) - open/close 18302 (time 1722238284.42 total 40.00 last 467.92) - open/close 20000 (time 1722238288.04 total 43.62 last 469.68) - open/close 24768 (time 1722238298.04 total 53.62 last 476.77) - open/close 29553 (time 1722238308.04 total 63.62 last 478.43) - open/close 34066 (time 1722238318.04 total 73.62 last 451.25) - open/close 38556 (time 1722238328.04 total 83.62 last 448.99) - open/close 40000 (time 1722238331.32 total 86.91 last 439.67) - open/close 44542 (time 1722238341.32 total 96.91 last 454.19) - open/close 48953 (time 1722238351.33 total 106.91 last 441.03) - open/close 50000 (time 1722238353.70 total 109.29 last 440.26) - open/close 54469 (time 1722238363.70 total 119.29 last 446.86) - open/close 59067 (time 1722238373.71 total 129.29 last 459.76) - open/close 63624 (time 1722238383.74 total 139.33 last 453.96) total: 64768 open/close in 141.65 seconds: 457.25 ops/second - /unlink 8019 (time 1722238396.95 total 10.00 last 801.87) - /unlink 10000 (time 1722238399.57 total 12.62 last 755.61) - /unlink 18393 (time 1722238409.57 total 22.62 last 839.21) - /unlink 20000 (time 1722238411.59 total 24.64 last 795.68) - /unlink 28372 (time 1722238421.59 total 34.64 last 837.11) - /unlink 36737 (time 1722238431.59 total 44.64 last 836.44) - /unlink 40000 (time 1722238435.69 total 48.74 last 796.79) - /unlink 47712 (time 1722238445.69 total 58.74 last 771.17) - /unlink 50000 (time 1722238448.78 total 61.83 last 739.74) - /unlink 57982 (time 1722238458.78 total 71.83 last 798.13) - /unlink 60000 (time 1722238461.32 total 74.37 last 794.94) total: 64768 /unlink in 79.92 seconds: 810.40 ops/second - open/close 4925 (time 1722238477.72 total 10.00 last 492.46) - open/close 9605 (time 1722238487.72 total 20.00 last 467.96) - open/close 14708 (time 1722238497.72 total 30.00 last 510.25) - open/close 19810 (time 1722238507.73 total 40.01 last 510.08) - open/close 24635 (time 1722238517.73 total 50.01 last 482.43) - open/close 29928 (time 1722238527.73 total 60.01 last 529.27) - open/close 35167 (time 1722238537.73 total 70.01 last 523.84) - open/close 39978 (time 1722238547.73 total 80.01 last 481.06) - open/close 44799 (time 1722238557.73 total 90.01 last 482.02) - open/close 49784 (time 1722238567.73 total 100.01 last 498.43) - open/close 54204 (time 1722238577.74 total 110.01 last 441.96) - open/close 58646 (time 1722238587.74 total 120.02 last 444.10) - open/close 60000 (time 1722238591.88 total 124.16 last 326.93) - open/close 61647 (time 1722238601.88 total 134.16 last 164.70) - open/close 62839 (time 1722238611.88 total 144.16 last 119.19) total: 64768 open/close in 153.08 seconds: 423.10 ops/second - /unlink 4304 (time 1722238631.94 total 10.00 last 430.32) - /unlink 8538 (time 1722238641.94 total 20.00 last 423.30) - /unlink 10000 (time 1722238647.07 total 25.14 last 284.79) - /unlink 13541 (time 1722238657.07 total 35.14 last 354.10) - /unlink 18252 (time 1722238667.07 total 45.14 last 471.01) - /unlink 20000 (time 1722238669.96 total 48.03 last 604.70) - /unlink 25825 (time 1722238679.97 total 58.03 last 582.50) - /unlink 30000 (time 1722238687.70 total 65.77 last 539.55) - /unlink 36030 (time 1722238697.70 total 75.77 last 602.97) - /unlink 40000 (time 1722238704.54 total 82.61 last 580.59) - /unlink 44423 (time 1722238714.54 total 92.61 last 442.20) - /unlink 49884 (time 1722238724.54 total 102.61 last 546.02) - /unlink 55344 (time 1722238734.55 total 112.61 last 545.95) - /unlink 60000 (time 1722238742.22 total 120.29 last 606.36) - /unlink 64638 (time 1722238752.23 total 130.29 last 463.75) total: 64768 /unlink in 130.56 seconds: 496.08 ops/second - open/close 2690 (time 1722238763.55 total 10.00 last 268.96) - open/close 5759 (time 1722238773.55 total 20.00 last 306.82) - open/close 8878 (time 1722238783.55 total 30.00 last 311.89) - open/close 10000 (time 1722238786.68 total 33.13 last 359.06) - open/close 12925 (time 1722238796.68 total 43.13 last 292.46) - open/close 16042 (time 1722238806.68 total 53.14 last 311.56) - open/close 19113 (time 1722238816.68 total 63.14 last 307.03) - open/close 20000 (time 1722238819.77 total 66.23 last 287.29) - open/close 22838 (time 1722238829.77 total 76.23 last 283.74) - open/close 25919 (time 1722238839.78 total 86.23 last 308.05) - open/close 28986 (time 1722238849.78 total 96.23 last 306.65) - open/close 30000 (time 1722238853.27 total 99.72 last 290.62) - open/close 32672 (time 1722238863.27 total 109.72 last 267.16) - open/close 35773 (time 1722238873.27 total 119.72 last 310.09) - open/close 38769 (time 1722238883.27 total 129.72 last 299.57) - open/close 40000 (time 1722238886.33 total 132.78 last 402.22) - open/close 43528 (time 1722238896.33 total 142.79 last 352.72) - open/close 46185 (time 1722238906.34 total 152.79 last 265.61) - open/close 49109 (time 1722238916.34 total 162.79 last 292.32) - open/close 50000 (time 1722238919.48 total 165.93 last 283.73) - open/close 53312 (time 1722238929.48 total 175.93 last 331.18) - open/close 57080 (time 1722238939.48 total 185.93 last 376.73) - open/close 60000 (time 1722238947.88 total 194.34 last 347.47) - open/close 62947 (time 1722238957.89 total 204.34 last 294.66) total: 64768 open/close in 210.39 seconds: 307.85 ops/second - /unlink 5161 (time 1722238974.94 total 10.00 last 516.02) - /unlink 10000 (time 1722238984.17 total 19.24 last 524.06) - /unlink 14897 (time 1722238994.17 total 29.24 last 489.67) - /unlink 20000 (time 1722239002.71 total 37.78 last 597.47) - /unlink 26491 (time 1722239012.72 total 47.78 last 649.06) - /unlink 30000 (time 1722239017.76 total 52.82 last 695.92) - /unlink 35580 (time 1722239027.76 total 62.82 last 557.91) - /unlink 40000 (time 1722239035.10 total 70.16 last 602.47) - /unlink 45215 (time 1722239045.10 total 80.16 last 521.43) - /unlink 49813 (time 1722239055.10 total 90.16 last 459.79) - /unlink 53503 (time 1722239065.10 total 100.16 last 368.97) - /unlink 57501 (time 1722239075.10 total 110.16 last 399.78) - /unlink 60000 (time 1722239080.76 total 115.82 last 441.51) - /unlink 64177 (time 1722239090.76 total 125.82 last 417.61) total: 64768 /unlink in 127.00 seconds: 509.97 ops/second - open/close 2297 (time 1722239102.95 total 10.00 last 229.63) - open/close 4675 (time 1722239112.95 total 20.01 last 237.75) - open/close 6876 (time 1722239122.95 total 30.01 last 220.09) - open/close 9576 (time 1722239132.95 total 40.01 last 269.98) - open/close 12592 (time 1722239142.95 total 50.01 last 301.59) - open/close 15864 (time 1722239152.95 total 60.01 last 327.18) - open/close 18896 (time 1722239162.95 total 70.01 last 303.10) - open/close 20000 (time 1722239166.93 total 73.99 last 277.47) - open/close 22648 (time 1722239176.93 total 83.99 last 264.80) - open/close 25466 (time 1722239186.93 total 93.99 last 281.79) - open/close 28734 (time 1722239196.94 total 103.99 last 326.70) - open/close 30000 (time 1722239201.47 total 108.53 last 278.87) - open/close 32708 (time 1722239211.48 total 118.53 last 270.75) - open/close 35833 (time 1722239221.48 total 128.54 last 312.45) - open/close 39138 (time 1722239231.48 total 138.54 last 330.48) - open/close 40000 (time 1722239234.12 total 141.17 last 326.86) - open/close 42759 (time 1722239244.12 total 151.17 last 275.87) - open/close 45879 (time 1722239254.12 total 161.18 last 311.89) - open/close 48438 (time 1722239264.12 total 171.18 last 255.83) - open/close 50000 (time 1722239269.22 total 176.27 last 306.70) - open/close 53424 (time 1722239279.22 total 186.28 last 342.32) - open/close 56229 (time 1722239289.22 total 196.28 last 280.46) - open/close 59478 (time 1722239299.23 total 206.29 last 324.65) - open/close 62864 (time 1722239309.23 total 216.29 last 338.51) total: 64768 open/close in 221.80 seconds: 292.01 ops/second - /unlink 6415 (time 1722239325.64 total 10.00 last 641.47) - /unlink 10000 (time 1722239330.67 total 15.04 last 711.66) - /unlink 17017 (time 1722239340.68 total 25.04 last 701.63) - /unlink 20000 (time 1722239344.94 total 29.30 last 700.17) - /unlink 28501 (time 1722239354.94 total 39.30 last 850.06) - /unlink 37192 (time 1722239364.94 total 49.30 last 869.05) - /unlink 40000 (time 1722239368.53 total 52.89 last 781.83) - /unlink 48564 (time 1722239378.53 total 62.89 last 856.34) - /unlink 55646 (time 1722239388.53 total 72.89 last 708.17) - /unlink 60000 (time 1722239394.12 total 78.48 last 778.81) total: 64768 /unlink in 84.59 seconds: 765.67 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server LNET ready to unload unloading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: LNET ready to unload modules unloaded. pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2