== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:17:04 (1738966624) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg113-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg113-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5622 (time 1738966664.60 total 10.00 last 562.15) - open/close 10000 (time 1738966672.45 total 17.84 last 558.18) - open/close 15553 (time 1738966682.45 total 27.84 last 555.29) - open/close 20000 (time 1738966690.38 total 35.78 last 560.32) - open/close 25518 (time 1738966700.38 total 45.78 last 551.80) - open/close 30000 (time 1738966708.69 total 54.09 last 539.42) - open/close 35356 (time 1738966718.69 total 64.09 last 535.55) - open/close 40000 (time 1738966727.26 total 72.66 last 542.17) - open/close 45308 (time 1738966737.26 total 82.66 last 530.72) - open/close 50000 (time 1738966745.87 total 91.27 last 544.99) - open/close 55455 (time 1738966755.87 total 101.27 last 545.42) - open/close 60000 (time 1738966764.24 total 109.63 last 543.36) total: 64768 open/close in 118.26 seconds: 547.70 ops/second - /unlink 9451 (time 1738966783.77 total 10.00 last 945.05) - /unlink 19046 (time 1738966793.77 total 20.00 last 959.40) - /unlink 28770 (time 1738966803.77 total 30.00 last 972.40) - /unlink 38598 (time 1738966813.77 total 40.00 last 982.71) - /unlink 48765 (time 1738966823.77 total 50.00 last 1016.69) - /unlink 58297 (time 1738966833.77 total 60.00 last 953.11) total: 64768 /unlink in 67.38 seconds: 961.26 ops/second Waiting for MDT destroys to complete oleg113-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg113-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 - open/close 4902 (time 1738966943.75 total 10.00 last 490.09) - open/close 9508 (time 1738966953.75 total 20.00 last 460.55) - open/close 13231 (time 1738966963.75 total 30.01 last 372.21) - open/close 16121 (time 1738966973.75 total 40.01 last 288.94) - open/close 19392 (time 1738966983.75 total 50.01 last 327.04) - open/close 20000 (time 1738966986.13 total 52.39 last 255.94) - open/close 23141 (time 1738966996.13 total 62.39 last 314.10) - open/close 26261 (time 1738967006.13 total 72.39 last 311.91) - open/close 29886 (time 1738967016.13 total 82.39 last 362.42) - open/close 33178 (time 1738967026.14 total 92.39 last 329.14) - open/close 36392 (time 1738967036.14 total 102.39 last 321.31) - open/close 39375 (time 1738967046.14 total 112.40 last 298.29) - open/close 40000 (time 1738967048.29 total 114.54 last 291.05) - open/close 43142 (time 1738967058.29 total 124.54 last 314.17) - open/close 45883 (time 1738967068.29 total 134.55 last 274.04) - open/close 48721 (time 1738967078.29 total 144.55 last 283.78) - open/close 50000 (time 1738967083.13 total 149.39 last 264.23) - open/close 52652 (time 1738967093.13 total 159.39 last 265.15) - open/close 55230 (time 1738967103.13 total 169.39 last 257.75) - open/close 57752 (time 1738967113.14 total 179.39 last 252.17) - open/close 60000 (time 1738967120.64 total 186.90 last 299.60) - open/close 63308 (time 1738967130.64 total 196.90 last 330.76) total: 64768 open/close in 202.37 seconds: 320.05 ops/second - /unlink 5458 (time 1738967147.43 total 10.00 last 545.76) - /unlink 10000 (time 1738967156.67 total 19.24 last 491.80) - /unlink 14720 (time 1738967166.67 total 29.24 last 471.94) - /unlink 20000 (time 1738967175.72 total 38.28 last 583.61) - /unlink 24894 (time 1738967185.72 total 48.29 last 489.35) - /unlink 29809 (time 1738967195.72 total 58.29 last 491.39) - /unlink 34737 (time 1738967205.72 total 68.29 last 492.72) - /unlink 39442 (time 1738967215.72 total 78.29 last 470.41) - /unlink 44189 (time 1738967225.72 total 88.29 last 474.69) - /unlink 48858 (time 1738967235.73 total 98.29 last 466.83) - /unlink 50000 (time 1738967237.93 total 100.50 last 517.75) - /unlink 54607 (time 1738967247.93 total 110.50 last 460.69) - /unlink 59212 (time 1738967257.93 total 120.50 last 460.45) - /unlink 63694 (time 1738967267.93 total 130.50 last 448.16) total: 64768 /unlink in 132.93 seconds: 487.24 ops/second Waiting for MDT destroys to complete oleg113-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg113-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 - open/close 2609 (time 1738967289.24 total 10.00 last 260.87) - open/close 5487 (time 1738967299.24 total 20.00 last 287.73) - open/close 8166 (time 1738967309.25 total 30.01 last 267.85) - open/close 10000 (time 1738967316.07 total 36.83 last 268.91) - open/close 12559 (time 1738967326.07 total 46.83 last 255.88) - open/close 15104 (time 1738967336.07 total 56.83 last 254.42) - open/close 17730 (time 1738967346.07 total 66.83 last 262.49) - open/close 20000 (time 1738967354.57 total 75.33 last 267.21) - open/close 22623 (time 1738967364.57 total 85.33 last 262.25) - open/close 25167 (time 1738967374.57 total 95.33 last 254.34) - open/close 27909 (time 1738967384.57 total 105.33 last 274.17) - open/close 30000 (time 1738967391.94 total 112.70 last 283.78) - open/close 32534 (time 1738967401.94 total 122.70 last 253.36) - open/close 35274 (time 1738967411.94 total 132.70 last 274.00) - open/close 38069 (time 1738967421.94 total 142.70 last 279.49) - open/close 40000 (time 1738967428.52 total 149.28 last 293.63) - open/close 42686 (time 1738967438.53 total 159.28 last 268.49) - open/close 45478 (time 1738967448.53 total 169.29 last 279.14) - open/close 48017 (time 1738967458.53 total 179.29 last 253.83) - open/close 50000 (time 1738967465.92 total 186.68 last 268.21) - open/close 52615 (time 1738967475.93 total 196.69 last 261.44) - open/close 55180 (time 1738967485.93 total 206.69 last 256.49) - open/close 57954 (time 1738967495.93 total 216.69 last 277.31) - open/close 60000 (time 1738967503.80 total 224.56 last 259.98) - open/close 62633 (time 1738967513.80 total 234.56 last 263.29) total: 64768 open/close in 241.99 seconds: 267.64 ops/second - /unlink 4704 (time 1738967532.61 total 10.00 last 470.36) - /unlink 9165 (time 1738967542.61 total 20.00 last 446.01) - /unlink 13799 (time 1738967552.61 total 30.01 last 463.30) - /unlink 18272 (time 1738967562.62 total 40.01 last 447.20) - /unlink 20000 (time 1738967566.31 total 43.70 last 467.72) - /unlink 25066 (time 1738967576.31 total 53.70 last 506.54) - /unlink 30000 (time 1738967586.22 total 63.61 last 497.82) - /unlink 36128 (time 1738967596.22 total 73.61 last 612.77) - /unlink 40000 (time 1738967604.69 total 82.08 last 457.54) - /unlink 44542 (time 1738967614.69 total 92.08 last 454.16) - /unlink 49484 (time 1738967624.69 total 102.08 last 494.17) - /unlink 54120 (time 1738967634.69 total 112.08 last 463.60) - /unlink 58738 (time 1738967644.69 total 122.08 last 461.73) - /unlink 60000 (time 1738967647.47 total 124.86 last 453.63) - /unlink 64683 (time 1738967657.47 total 134.86 last 468.24) total: 64768 /unlink in 135.04 seconds: 479.64 ops/second Waiting for MDT destroys to complete oleg113-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg113-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 - open/close 2849 (time 1738967679.64 total 10.00 last 284.84) - open/close 5720 (time 1738967689.64 total 20.01 last 287.00) - open/close 8488 (time 1738967699.65 total 30.01 last 276.70) - open/close 10000 (time 1738967705.72 total 36.09 last 248.71) - open/close 12539 (time 1738967715.73 total 46.09 last 253.85) - open/close 15162 (time 1738967725.73 total 56.09 last 262.26) - open/close 17750 (time 1738967735.73 total 66.09 last 258.79) - open/close 20000 (time 1738967744.53 total 74.90 last 255.57) - open/close 22681 (time 1738967754.53 total 84.90 last 268.09) - open/close 25592 (time 1738967764.54 total 94.90 last 291.03) - open/close 28306 (time 1738967774.54 total 104.90 last 271.29) - open/close 30000 (time 1738967780.98 total 111.35 last 262.94) - open/close 32600 (time 1738967790.98 total 121.35 last 259.91) - open/close 35272 (time 1738967800.99 total 131.35 last 267.18) - open/close 37699 (time 1738967810.99 total 141.35 last 242.69) - open/close 40000 (time 1738967819.87 total 150.23 last 258.99) - open/close 42910 (time 1738967829.87 total 160.24 last 290.91) - open/close 45508 (time 1738967839.88 total 170.24 last 259.69) - open/close 48115 (time 1738967849.88 total 180.25 last 260.61) - open/close 50000 (time 1738967855.56 total 185.93 last 331.67) - open/close 52500 (time 1738967865.56 total 195.93 last 250.00) - open/close 55607 (time 1738967875.57 total 205.93 last 310.59) - open/close 59394 (time 1738967885.57 total 215.93 last 378.67) - open/close 64143 (time 1738967895.57 total 225.93 last 474.84) total: 64768 open/close in 227.29 seconds: 284.96 ops/second - /unlink 8519 (time 1738967907.81 total 10.00 last 851.77) - /unlink 17038 (time 1738967917.81 total 20.00 last 851.87) - /unlink 20000 (time 1738967921.04 total 23.23 last 917.02) - /unlink 29036 (time 1738967931.04 total 33.23 last 903.58) - /unlink 37480 (time 1738967941.04 total 43.23 last 844.38) - /unlink 40000 (time 1738967943.75 total 45.94 last 930.94) - /unlink 49375 (time 1738967953.75 total 55.94 last 937.49) - /unlink 58377 (time 1738967963.75 total 65.94 last 900.19) - /unlink 60000 (time 1738967965.92 total 68.11 last 748.28) total: 64768 /unlink in 76.30 seconds: 848.84 ops/second Waiting for MDT destroys to complete oleg113-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 oleg113-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 - open/close 2599 (time 1738968065.66 total 10.00 last 259.88) - open/close 7185 (time 1738968075.66 total 20.00 last 458.57) - open/close 10000 (time 1738968085.41 total 29.74 last 288.96) - open/close 13183 (time 1738968095.41 total 39.74 last 318.29) - open/close 17939 (time 1738968105.41 total 49.74 last 475.59) - open/close 20000 (time 1738968109.30 total 53.64 last 528.77) - open/close 25045 (time 1738968119.31 total 63.64 last 504.43) - open/close 30000 (time 1738968128.67 total 73.00 last 529.32) - open/close 35114 (time 1738968138.67 total 83.00 last 511.39) - open/close 40000 (time 1738968148.07 total 92.41 last 519.72) - open/close 45290 (time 1738968158.07 total 102.41 last 528.90) - open/close 48802 (time 1738968168.07 total 112.41 last 351.19) - open/close 50000 (time 1738968172.38 total 116.71 last 278.25) - open/close 54049 (time 1738968182.38 total 126.71 last 404.84) - open/close 58931 (time 1738968192.38 total 136.72 last 488.11) - open/close 60000 (time 1738968194.43 total 138.76 last 521.98) total: 64768 open/close in 148.11 seconds: 437.30 ops/second - /unlink 8943 (time 1738968214.59 total 10.00 last 894.27) - /unlink 17651 (time 1738968224.59 total 20.00 last 870.78) - /unlink 20000 (time 1738968227.21 total 22.63 last 894.11) - /unlink 27787 (time 1738968237.21 total 32.63 last 778.59) - /unlink 30000 (time 1738968240.40 total 35.82 last 693.85) - /unlink 34977 (time 1738968250.41 total 45.82 last 497.64) - /unlink 40000 (time 1738968260.33 total 55.74 last 506.33) - /unlink 46193 (time 1738968270.33 total 65.74 last 619.21) - /unlink 50000 (time 1738968277.24 total 72.65 last 551.07) - /unlink 55323 (time 1738968287.24 total 82.65 last 532.25) - /unlink 60000 (time 1738968297.02 total 92.43 last 478.16) - /unlink 64670 (time 1738968307.02 total 102.43 last 466.96) total: 64768 /unlink in 102.65 seconds: 630.95 ops/second Waiting for MDT destroys to complete oleg113-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg113-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2