== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:32:53 (1771439573) Stopping clients: oleg405-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg405-client.virtnet /mnt/lustre2 (opts:-f) oleg405-server: oleg405-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg405-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg405-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2504 (time 1771439633.66 total 10.00 last 250.40) - open/close 4323 (time 1771439643.66 total 20.00 last 181.87) - open/close 6042 (time 1771439653.67 total 30.01 last 171.81) - open/close 7744 (time 1771439663.67 total 40.01 last 170.16) - open/close 9633 (time 1771439673.67 total 50.01 last 188.87) - open/close 10000 (time 1771439675.68 total 52.02 last 182.58) - open/close 12294 (time 1771439685.68 total 62.02 last 229.38) - open/close 15118 (time 1771439695.68 total 72.02 last 282.40) - open/close 18354 (time 1771439705.69 total 82.03 last 323.47) - open/close 20000 (time 1771439711.52 total 87.86 last 282.10) - open/close 22628 (time 1771439721.52 total 97.86 last 262.75) - open/close 24913 (time 1771439731.53 total 107.87 last 228.41) - open/close 26660 (time 1771439741.53 total 117.87 last 174.69) - open/close 28885 (time 1771439751.53 total 127.87 last 222.43) - open/close 30000 (time 1771439756.96 total 133.30 last 205.47) - open/close 32457 (time 1771439766.96 total 143.30 last 245.60) - open/close 34617 (time 1771439776.97 total 153.30 last 215.97) - open/close 37101 (time 1771439786.97 total 163.31 last 248.34) - open/close 40000 (time 1771439796.29 total 172.63 last 310.98) - open/close 43143 (time 1771439806.29 total 182.63 last 314.23) - open/close 46922 (time 1771439816.29 total 192.63 last 377.86) - open/close 50000 (time 1771439825.96 total 202.29 last 318.50) - open/close 53731 (time 1771439835.96 total 212.30 last 373.06) - open/close 57292 (time 1771439845.96 total 222.30 last 356.02) - open/close 60000 (time 1771439853.67 total 230.01 last 351.32) - open/close 63701 (time 1771439863.67 total 240.01 last 370.05) total: 64768 open/close in 242.38 seconds: 267.22 ops/second - /unlink 5429 (time 1771439877.43 total 10.00 last 542.89) - /unlink 10000 (time 1771439884.69 total 17.26 last 629.87) - /unlink 15461 (time 1771439894.69 total 27.26 last 546.02) - /unlink 19798 (time 1771439904.69 total 37.26 last 433.64) - /unlink 24952 (time 1771439914.69 total 47.26 last 515.34) - /unlink 30000 (time 1771439923.10 total 55.67 last 600.51) - /unlink 35688 (time 1771439933.10 total 65.67 last 568.75) - /unlink 40000 (time 1771439939.67 total 72.23 last 656.77) - /unlink 47010 (time 1771439949.67 total 82.23 last 700.95) - /unlink 50000 (time 1771439954.58 total 87.15 last 608.40) - /unlink 55368 (time 1771439964.58 total 97.15 last 536.74) - /unlink 60000 (time 1771439972.66 total 105.22 last 573.74) total: 64768 /unlink in 112.17 seconds: 577.43 ops/second Waiting for MDT destroys to complete oleg405-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 - open/close 3894 (time 1771440000.57 total 10.00 last 389.37) - open/close 8117 (time 1771440010.57 total 20.00 last 422.21) - open/close 10000 (time 1771440015.48 total 24.91 last 383.74) - open/close 13827 (time 1771440025.48 total 34.91 last 382.68) - open/close 17674 (time 1771440035.48 total 44.91 last 384.69) - open/close 20000 (time 1771440042.13 total 51.56 last 350.06) - open/close 24223 (time 1771440052.13 total 61.56 last 422.22) - open/close 28316 (time 1771440062.13 total 71.56 last 409.26) - open/close 30000 (time 1771440066.35 total 75.77 last 399.43) - open/close 33838 (time 1771440076.35 total 85.78 last 383.73) - open/close 37444 (time 1771440086.35 total 95.78 last 360.55) - open/close 40000 (time 1771440094.24 total 103.66 last 324.09) - open/close 43724 (time 1771440104.24 total 113.67 last 372.33) - open/close 48131 (time 1771440114.24 total 123.67 last 440.68) - open/close 50000 (time 1771440118.31 total 127.74 last 459.06) - open/close 54221 (time 1771440128.31 total 137.74 last 422.02) - open/close 58801 (time 1771440138.31 total 147.74 last 457.93) - open/close 60000 (time 1771440140.77 total 150.20 last 488.58) - open/close 64728 (time 1771440150.77 total 160.20 last 472.78) total: 64768 open/close in 160.28 seconds: 404.09 ops/second - /unlink 8229 (time 1771440162.19 total 10.00 last 822.81) - /unlink 10000 (time 1771440164.47 total 12.28 last 776.57) - /unlink 18077 (time 1771440174.47 total 22.28 last 807.62) - /unlink 20000 (time 1771440176.97 total 24.78 last 770.43) - /unlink 28291 (time 1771440186.97 total 34.78 last 829.02) - /unlink 30000 (time 1771440189.11 total 36.92 last 799.26) - /unlink 38110 (time 1771440199.11 total 46.92 last 810.99) - /unlink 40000 (time 1771440201.36 total 49.17 last 839.15) - /unlink 48064 (time 1771440211.36 total 59.17 last 806.37) - /unlink 50000 (time 1771440213.75 total 61.56 last 810.23) - /unlink 58297 (time 1771440223.75 total 71.56 last 829.63) - /unlink 60000 (time 1771440225.91 total 73.72 last 788.37) total: 64768 /unlink in 79.11 seconds: 818.76 ops/second Waiting for MDT destroys to complete oleg405-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 - open/close 4764 (time 1771440247.83 total 10.00 last 476.39) - open/close 9771 (time 1771440257.83 total 20.00 last 500.65) - open/close 15237 (time 1771440267.83 total 30.00 last 546.56) - open/close 20000 (time 1771440276.89 total 39.06 last 525.70) - open/close 25385 (time 1771440286.89 total 49.06 last 538.43) - open/close 30000 (time 1771440296.21 total 58.39 last 494.97) - open/close 34963 (time 1771440306.21 total 68.39 last 496.23) - open/close 40000 (time 1771440316.21 total 78.38 last 503.99) - open/close 44949 (time 1771440326.21 total 88.38 last 494.89) - open/close 49789 (time 1771440336.21 total 98.39 last 483.77) - open/close 54657 (time 1771440346.21 total 108.39 last 486.73) - open/close 60000 (time 1771440356.10 total 118.27 last 540.49) total: 64768 open/close in 127.33 seconds: 508.66 ops/second - /unlink 9148 (time 1771440376.44 total 10.00 last 914.72) - /unlink 18253 (time 1771440386.44 total 20.00 last 910.47) - /unlink 20000 (time 1771440388.62 total 22.17 last 803.87) - /unlink 28903 (time 1771440398.62 total 32.17 last 890.26) - /unlink 37844 (time 1771440408.62 total 42.18 last 894.03) - /unlink 40000 (time 1771440410.98 total 44.54 last 912.01) - /unlink 48930 (time 1771440420.98 total 54.54 last 892.99) - /unlink 57579 (time 1771440430.98 total 64.54 last 864.78) - /unlink 60000 (time 1771440433.77 total 67.33 last 867.52) total: 64768 /unlink in 72.62 seconds: 891.91 ops/second Waiting for MDT destroys to complete oleg405-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 - open/close 5347 (time 1771440456.54 total 10.00 last 534.67) - open/close 10000 (time 1771440465.97 total 19.43 last 493.31) - open/close 15317 (time 1771440475.97 total 29.43 last 531.65) - open/close 20000 (time 1771440484.88 total 38.35 last 525.33) - open/close 25329 (time 1771440494.88 total 48.35 last 532.84) - open/close 30000 (time 1771440503.34 total 56.81 last 552.20) - open/close 35740 (time 1771440513.35 total 66.81 last 573.89) - open/close 38698 (time 1771440523.35 total 76.81 last 295.77) - open/close 40000 (time 1771440531.78 total 85.24 last 154.48) - open/close 41711 (time 1771440541.78 total 95.25 last 170.99) - open/close 43300 (time 1771440551.78 total 105.25 last 158.89) - open/close 44751 (time 1771440561.78 total 115.25 last 145.08) - open/close 45873 (time 1771440571.79 total 125.25 last 112.14) - open/close 47008 (time 1771440581.80 total 135.26 last 113.43) - open/close 48230 (time 1771440591.80 total 145.26 last 122.19) - open/close 49583 (time 1771440601.80 total 155.26 last 135.26) - open/close 50000 (time 1771440604.62 total 158.08 last 147.91) - open/close 51623 (time 1771440614.62 total 168.08 last 162.28) - open/close 53248 (time 1771440624.62 total 178.09 last 162.47) - open/close 54653 (time 1771440634.63 total 188.09 last 140.44) - open/close 55882 (time 1771440644.64 total 198.10 last 122.78) - open/close 57256 (time 1771440654.64 total 208.10 last 137.37) - open/close 58879 (time 1771440664.65 total 218.11 last 162.18) - open/close 60000 (time 1771440671.02 total 224.49 last 175.80) - open/close 62198 (time 1771440681.02 total 234.49 last 219.75) - open/close 63540 (time 1771440691.03 total 244.49 last 134.12) total: 64768 open/close in 250.43 seconds: 258.63 ops/second - /unlink 2830 (time 1771440708.61 total 10.01 last 282.79) - /unlink 4726 (time 1771440718.61 total 20.01 last 189.52) - /unlink 6804 (time 1771440728.62 total 30.01 last 207.75) - /unlink 9113 (time 1771440738.63 total 40.02 last 230.65) - /unlink 10000 (time 1771440743.16 total 44.56 last 195.56) - /unlink 12172 (time 1771440753.16 total 54.56 last 217.17) - /unlink 14484 (time 1771440763.21 total 64.61 last 230.15) - /unlink 16595 (time 1771440773.21 total 74.61 last 211.04) - /unlink 18785 (time 1771440783.21 total 84.61 last 218.95) - /unlink 20000 (time 1771440789.11 total 90.50 last 206.25) - /unlink 22298 (time 1771440799.11 total 100.51 last 229.76) - /unlink 24452 (time 1771440809.11 total 110.51 last 215.40) - /unlink 27292 (time 1771440819.11 total 120.51 last 283.94) - /unlink 30000 (time 1771440827.88 total 129.28 last 308.85) - /unlink 33806 (time 1771440837.88 total 139.28 last 380.53) - /unlink 36865 (time 1771440847.88 total 149.28 last 305.86) - /unlink 39900 (time 1771440857.88 total 159.28 last 303.44) - /unlink 42975 (time 1771440867.88 total 169.28 last 307.45) - /unlink 46323 (time 1771440877.89 total 179.28 last 334.74) - /unlink 49401 (time 1771440887.89 total 189.28 last 307.78) - /unlink 50000 (time 1771440890.12 total 191.52 last 268.56) - /unlink 53047 (time 1771440900.12 total 201.52 last 304.69) - /unlink 55650 (time 1771440910.12 total 211.52 last 260.19) - /unlink 58155 (time 1771440920.13 total 221.52 last 250.40) - /unlink 60000 (time 1771440927.51 total 228.91 last 249.88) - /unlink 62911 (time 1771440937.51 total 238.91 last 291.09) total: 64768 /unlink in 246.14 seconds: 263.14 ops/second Waiting for MDT destroys to complete oleg405-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 - open/close 1583 (time 1771440965.89 total 10.01 last 158.21) - open/close 3277 (time 1771440975.89 total 20.01 last 169.28) - open/close 5120 (time 1771440985.90 total 30.02 last 184.20) - open/close 7265 (time 1771440995.90 total 40.02 last 214.47) - open/close 9484 (time 1771441005.90 total 50.02 last 221.84) - open/close 10000 (time 1771441008.32 total 52.43 last 213.98) - open/close 11873 (time 1771441018.33 total 62.44 last 187.12) - open/close 13502 (time 1771441028.33 total 72.44 last 162.88) - open/close 15210 (time 1771441038.33 total 82.45 last 170.78) - open/close 16769 (time 1771441048.33 total 92.45 last 155.88) - open/close 18629 (time 1771441058.33 total 102.45 last 185.99) - open/close 20000 (time 1771441065.93 total 110.05 last 180.43) - open/close 21726 (time 1771441075.93 total 120.05 last 172.56) - open/close 23527 (time 1771441085.93 total 130.05 last 180.07) - open/close 25801 (time 1771441095.93 total 140.05 last 227.38) - open/close 27803 (time 1771441105.94 total 150.06 last 200.09) - open/close 29697 (time 1771441115.94 total 160.06 last 189.31) - open/close 31248 (time 1771441125.96 total 170.07 last 154.89) - open/close 32920 (time 1771441135.96 total 180.08 last 167.19) - open/close 34859 (time 1771441145.96 total 190.08 last 193.84) - open/close 37075 (time 1771441155.97 total 200.08 last 221.49) - open/close 39370 (time 1771441165.97 total 210.09 last 229.43) - open/close 40000 (time 1771441168.67 total 212.79 last 233.41) - open/close 42091 (time 1771441178.67 total 222.79 last 209.01) - open/close 43974 (time 1771441188.67 total 232.79 last 188.28) - open/close 45900 (time 1771441198.68 total 242.79 last 192.53) - open/close 47628 (time 1771441208.68 total 252.80 last 172.69) - open/close 49389 (time 1771441218.69 total 262.80 last 176.06) - open/close 50000 (time 1771441222.16 total 266.28 last 175.86) - open/close 51914 (time 1771441232.16 total 276.28 last 191.33) - open/close 53858 (time 1771441242.17 total 286.28 last 194.35) - open/close 55930 (time 1771441252.17 total 296.29 last 207.15) - open/close 59105 (time 1771441262.17 total 306.29 last 317.48) - open/close 60000 (time 1771441265.94 total 310.06 last 237.16) - open/close 62744 (time 1771441275.95 total 320.06 last 274.34) total: 64768 open/close in 329.43 seconds: 196.61 ops/second - /unlink 4033 (time 1771441296.98 total 10.00 last 403.18) - /unlink 8325 (time 1771441306.99 total 20.02 last 428.48) - /unlink 10000 (time 1771441310.30 total 23.33 last 506.01) - /unlink 14922 (time 1771441320.30 total 33.33 last 492.16) - /unlink 19577 (time 1771441330.30 total 43.33 last 465.49) - /unlink 23830 (time 1771441340.31 total 53.33 last 425.24) - /unlink 28136 (time 1771441350.32 total 63.35 last 430.00) - /unlink 30000 (time 1771441354.24 total 67.26 last 475.71) - /unlink 34129 (time 1771441364.24 total 77.27 last 412.69) - /unlink 38616 (time 1771441374.25 total 87.27 last 448.61) - /unlink 40000 (time 1771441378.54 total 91.57 last 322.14) - /unlink 44737 (time 1771441388.54 total 101.57 last 473.69) - /unlink 50000 (time 1771441397.89 total 110.92 last 563.03) - /unlink 55117 (time 1771441407.89 total 120.92 last 511.45) - /unlink 60000 (time 1771441417.21 total 130.24 last 524.16) total: 64768 /unlink in 139.94 seconds: 462.83 ops/second Waiting for MDT destroys to complete oleg405-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: modules unloaded. pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2