== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:10:38 (1758823838) Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) oleg201-server: oleg201-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg201-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg201-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1097 (time 1758823916.21 total 10.00 last 109.70) - open/close 2586 (time 1758823926.22 total 20.00 last 148.84) - open/close 4425 (time 1758823936.22 total 30.01 last 183.86) - open/close 7155 (time 1758823946.22 total 40.01 last 272.89) - open/close 8787 (time 1758823956.23 total 50.02 last 163.12) - open/close 10000 (time 1758823963.80 total 57.59 last 160.14) - open/close 11246 (time 1758823973.80 total 67.59 last 124.60) - open/close 12838 (time 1758823983.81 total 77.59 last 159.13) - open/close 14903 (time 1758823993.81 total 87.60 last 206.35) - open/close 16999 (time 1758824003.81 total 97.60 last 209.59) - open/close 19599 (time 1758824013.82 total 107.61 last 259.71) - open/close 20000 (time 1758824016.50 total 110.29 last 149.78) - open/close 22676 (time 1758824026.51 total 120.30 last 267.41) - open/close 24052 (time 1758824036.51 total 130.30 last 137.53) - open/close 25459 (time 1758824046.52 total 140.31 last 140.60) - open/close 26796 (time 1758824056.53 total 150.31 last 133.63) - open/close 28146 (time 1758824066.53 total 160.32 last 134.91) - open/close 29441 (time 1758824076.54 total 170.33 last 129.45) - open/close 30000 (time 1758824081.20 total 174.99 last 119.79) - open/close 31255 (time 1758824091.21 total 184.99 last 125.47) - open/close 32648 (time 1758824101.21 total 195.00 last 139.27) - open/close 33934 (time 1758824111.21 total 205.00 last 128.53) - open/close 35380 (time 1758824121.21 total 215.00 last 144.59) - open/close 37076 (time 1758824131.21 total 225.00 last 169.58) - open/close 39176 (time 1758824141.22 total 235.01 last 209.93) - open/close 40000 (time 1758824145.81 total 239.60 last 179.43) - open/close 42284 (time 1758824155.81 total 249.60 last 228.38) - open/close 44231 (time 1758824165.81 total 259.60 last 194.66) - open/close 45863 (time 1758824175.81 total 269.60 last 163.19) - open/close 47379 (time 1758824185.82 total 279.61 last 151.53) - open/close 49099 (time 1758824195.83 total 289.61 last 171.87) - open/close 50000 (time 1758824201.64 total 295.43 last 154.90) - open/close 51640 (time 1758824211.65 total 305.44 last 163.93) - open/close 53297 (time 1758824221.65 total 315.44 last 165.64) - open/close 54823 (time 1758824231.65 total 325.44 last 152.55) - open/close 56434 (time 1758824241.66 total 335.45 last 161.02) - open/close 57930 (time 1758824251.66 total 345.45 last 149.54) - open/close 59552 (time 1758824261.67 total 355.46 last 162.15) - open/close 60000 (time 1758824264.10 total 357.88 last 184.44) - open/close 61752 (time 1758824274.10 total 367.89 last 175.17) - open/close 63841 (time 1758824284.10 total 377.89 last 208.88) total: 64768 open/close in 382.66 seconds: 169.26 ops/second - /unlink 3216 (time 1758824300.52 total 10.00 last 321.59) - /unlink 6875 (time 1758824310.52 total 20.00 last 365.86) - /unlink 10000 (time 1758824319.52 total 29.00 last 347.16) - /unlink 13237 (time 1758824329.53 total 39.00 last 323.69) - /unlink 17038 (time 1758824339.53 total 49.00 last 380.05) - /unlink 20000 (time 1758824346.32 total 55.80 last 435.69) - /unlink 23986 (time 1758824356.33 total 65.81 last 398.44) - /unlink 27523 (time 1758824366.33 total 75.81 last 353.49) - /unlink 30000 (time 1758824373.25 total 82.73 last 357.98) - /unlink 34419 (time 1758824383.26 total 92.73 last 441.81) - /unlink 39003 (time 1758824393.26 total 102.74 last 458.37) - /unlink 40000 (time 1758824395.49 total 104.96 last 447.34) - /unlink 44198 (time 1758824405.49 total 114.96 last 419.77) - /unlink 48424 (time 1758824415.49 total 124.97 last 422.56) - /unlink 50000 (time 1758824420.04 total 129.51 last 346.43) - /unlink 54720 (time 1758824430.04 total 139.52 last 471.96) - /unlink 59469 (time 1758824440.04 total 149.52 last 474.83) - /unlink 64731 (time 1758824450.04 total 159.52 last 526.13) total: 64768 /unlink in 159.61 seconds: 405.79 ops/second Waiting for MDT destroys to complete oleg201-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 - open/close 2635 (time 1758824472.72 total 10.00 last 263.47) - open/close 5543 (time 1758824482.73 total 20.00 last 290.72) - open/close 7788 (time 1758824492.73 total 30.01 last 224.43) - open/close 10000 (time 1758824501.30 total 38.58 last 258.11) - open/close 12933 (time 1758824511.30 total 48.58 last 293.27) - open/close 16027 (time 1758824521.30 total 58.58 last 309.37) - open/close 18023 (time 1758824531.30 total 68.58 last 199.55) - open/close 19985 (time 1758824541.31 total 78.58 last 196.15) - open/close 21902 (time 1758824551.31 total 88.59 last 191.58) - open/close 23850 (time 1758824561.32 total 98.59 last 194.73) - open/close 26437 (time 1758824571.32 total 108.60 last 258.63) - open/close 28866 (time 1758824581.32 total 118.60 last 242.86) - open/close 30000 (time 1758824587.10 total 124.38 last 196.04) - open/close 32592 (time 1758824597.11 total 134.38 last 259.15) - open/close 35475 (time 1758824607.11 total 144.39 last 288.22) - open/close 38490 (time 1758824617.11 total 154.39 last 301.45) - open/close 40000 (time 1758824622.63 total 159.91 last 273.65) - open/close 42834 (time 1758824632.63 total 169.91 last 283.27) - open/close 45147 (time 1758824642.64 total 179.92 last 231.20) - open/close 47275 (time 1758824652.64 total 189.92 last 212.74) - open/close 49504 (time 1758824662.64 total 199.92 last 222.85) - open/close 52536 (time 1758824672.64 total 209.92 last 303.20) - open/close 55346 (time 1758824682.64 total 219.92 last 280.96) - open/close 58083 (time 1758824692.65 total 229.93 last 273.63) - open/close 60000 (time 1758824700.19 total 237.47 last 254.05) - open/close 63454 (time 1758824710.20 total 247.47 last 345.30) total: 64768 open/close in 250.67 seconds: 258.37 ops/second - /unlink 5878 (time 1758824724.82 total 10.00 last 587.71) - /unlink 10000 (time 1758824731.44 total 16.62 last 622.57) - /unlink 17767 (time 1758824741.44 total 26.62 last 776.64) - /unlink 20000 (time 1758824744.08 total 29.27 last 844.84) - /unlink 28314 (time 1758824754.08 total 39.27 last 831.28) - /unlink 30000 (time 1758824756.18 total 41.36 last 805.77) - /unlink 38739 (time 1758824766.18 total 51.36 last 873.90) - /unlink 46666 (time 1758824776.18 total 61.36 last 792.61) - /unlink 50000 (time 1758824780.14 total 65.32 last 842.17) - /unlink 58229 (time 1758824790.14 total 75.32 last 822.83) - /unlink 60000 (time 1758824792.38 total 77.56 last 790.37) total: 64768 /unlink in 84.10 seconds: 770.16 ops/second Waiting for MDT destroys to complete oleg201-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 - open/close 3777 (time 1758824879.52 total 10.00 last 377.65) - open/close 7977 (time 1758824889.52 total 20.00 last 419.95) - open/close 10000 (time 1758824893.92 total 24.41 last 459.44) - open/close 13949 (time 1758824903.92 total 34.41 last 394.83) - open/close 17468 (time 1758824913.93 total 44.41 last 351.84) - open/close 20000 (time 1758824922.31 total 52.79 last 301.98) - open/close 23941 (time 1758824932.31 total 62.79 last 394.09) - open/close 27487 (time 1758824942.31 total 72.79 last 354.58) - open/close 30000 (time 1758824947.98 total 78.46 last 443.70) - open/close 34823 (time 1758824957.98 total 88.46 last 482.28) - open/close 39106 (time 1758824967.98 total 98.46 last 428.25) - open/close 44018 (time 1758824977.98 total 108.46 last 491.19) - open/close 47835 (time 1758824987.98 total 118.46 last 381.69) - open/close 50000 (time 1758824992.58 total 123.07 last 470.02) - open/close 54965 (time 1758825002.58 total 133.07 last 496.46) - open/close 59787 (time 1758825012.58 total 143.07 last 482.18) total: 64768 open/close in 152.83 seconds: 423.78 ops/second - /unlink 8444 (time 1758825033.74 total 10.00 last 844.33) - /unlink 16035 (time 1758825043.74 total 20.00 last 759.08) - /unlink 20000 (time 1758825048.15 total 24.41 last 899.53) - /unlink 28362 (time 1758825058.15 total 34.41 last 836.10) - /unlink 37591 (time 1758825068.15 total 44.41 last 922.82) - /unlink 40000 (time 1758825070.71 total 46.97 last 941.08) - /unlink 48756 (time 1758825080.71 total 56.97 last 875.58) - /unlink 57325 (time 1758825090.71 total 66.97 last 856.90) - /unlink 60000 (time 1758825093.79 total 70.05 last 869.15) total: 64768 /unlink in 75.67 seconds: 855.95 ops/second Waiting for MDT destroys to complete oleg201-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 - open/close 4677 (time 1758825158.64 total 10.00 last 467.66) - open/close 9597 (time 1758825168.64 total 20.00 last 491.94) - open/close 14696 (time 1758825178.64 total 30.00 last 509.83) - open/close 20000 (time 1758825188.60 total 39.96 last 532.64) - open/close 25224 (time 1758825198.60 total 49.96 last 522.34) - open/close 30000 (time 1758825207.46 total 58.82 last 539.10) - open/close 34454 (time 1758825217.46 total 68.82 last 445.33) - open/close 39078 (time 1758825227.46 total 78.82 last 462.36) - open/close 40000 (time 1758825229.48 total 80.84 last 458.18) - open/close 44333 (time 1758825239.48 total 90.84 last 433.28) - open/close 48494 (time 1758825249.48 total 100.84 last 416.08) - open/close 50000 (time 1758825252.81 total 104.17 last 451.62) - open/close 54582 (time 1758825262.81 total 114.17 last 458.19) - open/close 59080 (time 1758825272.81 total 124.17 last 449.76) - open/close 60000 (time 1758825274.91 total 126.27 last 439.06) - open/close 64528 (time 1758825284.91 total 136.27 last 452.68) total: 64768 open/close in 136.89 seconds: 473.15 ops/second - /unlink 7722 (time 1758825297.00 total 10.00 last 772.19) - /unlink 10000 (time 1758825299.86 total 12.86 last 795.52) - /unlink 17956 (time 1758825309.86 total 22.86 last 795.57) - /unlink 20000 (time 1758825312.38 total 25.38 last 813.07) - /unlink 27739 (time 1758825322.38 total 35.38 last 773.85) - /unlink 30000 (time 1758825325.40 total 38.40 last 748.22) - /unlink 38332 (time 1758825335.40 total 48.40 last 833.16) - /unlink 40000 (time 1758825337.42 total 50.42 last 825.59) - /unlink 47981 (time 1758825347.42 total 60.42 last 798.09) - /unlink 50000 (time 1758825349.80 total 62.80 last 847.72) - /unlink 58757 (time 1758825359.80 total 72.80 last 875.63) total: 64768 /unlink in 79.18 seconds: 817.94 ops/second Waiting for MDT destroys to complete oleg201-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 - open/close 4987 (time 1758825458.00 total 10.00 last 498.64) - open/close 9781 (time 1758825468.00 total 20.00 last 479.30) - open/close 14495 (time 1758825478.00 total 30.00 last 471.33) - open/close 19296 (time 1758825488.00 total 40.01 last 480.03) - open/close 24559 (time 1758825498.00 total 50.01 last 526.29) - open/close 29221 (time 1758825508.00 total 60.01 last 466.11) - open/close 33923 (time 1758825518.01 total 70.01 last 470.14) - open/close 38590 (time 1758825528.01 total 80.01 last 466.69) - open/close 40000 (time 1758825531.04 total 83.04 last 465.25) - open/close 44482 (time 1758825541.04 total 93.04 last 448.20) - open/close 48935 (time 1758825551.04 total 103.04 last 445.28) - open/close 50000 (time 1758825553.62 total 105.63 last 411.80) - open/close 54576 (time 1758825563.62 total 115.63 last 457.56) - open/close 59925 (time 1758825573.62 total 125.63 last 534.86) total: 64768 open/close in 134.97 seconds: 479.85 ops/second - /unlink 8644 (time 1758825594.32 total 10.00 last 864.35) - /unlink 16337 (time 1758825604.32 total 20.00 last 769.29) - /unlink 20000 (time 1758825608.69 total 24.37 last 838.57) - /unlink 28539 (time 1758825618.69 total 34.37 last 853.88) - /unlink 37108 (time 1758825628.69 total 44.37 last 856.86) - /unlink 40000 (time 1758825632.19 total 47.86 last 827.43) - /unlink 47858 (time 1758825642.19 total 57.87 last 785.76) - /unlink 50000 (time 1758825645.16 total 60.84 last 720.09) - /unlink 58391 (time 1758825655.16 total 70.84 last 839.02) - /unlink 60000 (time 1758825657.21 total 72.89 last 785.00) total: 64768 /unlink in 78.93 seconds: 820.61 ops/second Waiting for MDT destroys to complete oleg201-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg201-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2