== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:31:05 (1773941465) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) oleg430-server: oleg430-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg430-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4711 (time 1773941517.10 total 10.00 last 471.01) - open/close 9031 (time 1773941527.10 total 20.00 last 431.94) - open/close 10000 (time 1773941530.09 total 22.99 last 324.23) - open/close 14259 (time 1773941540.09 total 32.99 last 425.83) - open/close 18334 (time 1773941550.09 total 42.99 last 407.44) - open/close 20000 (time 1773941555.03 total 47.93 last 337.24) - open/close 23874 (time 1773941565.04 total 57.94 last 387.32) - open/close 27409 (time 1773941575.04 total 67.94 last 353.44) - open/close 30000 (time 1773941581.12 total 74.02 last 425.75) - open/close 33881 (time 1773941591.12 total 84.03 last 388.05) - open/close 37682 (time 1773941601.12 total 94.03 last 380.09) - open/close 40000 (time 1773941609.16 total 102.06 last 288.37) - open/close 43267 (time 1773941619.17 total 112.07 last 326.61) - open/close 46100 (time 1773941629.17 total 122.07 last 283.29) - open/close 48844 (time 1773941639.17 total 132.07 last 274.37) - open/close 50000 (time 1773941642.28 total 135.18 last 370.87) - open/close 53885 (time 1773941652.29 total 145.19 last 388.38) - open/close 57614 (time 1773941662.29 total 155.19 last 372.78) - open/close 60000 (time 1773941670.00 total 162.90 last 309.56) - open/close 64232 (time 1773941680.00 total 172.90 last 423.14) total: 64768 open/close in 174.11 seconds: 372.00 ops/second - /unlink 7453 (time 1773941692.54 total 10.00 last 745.30) - /unlink 10000 (time 1773941696.86 total 14.32 last 588.98) - /unlink 15926 (time 1773941706.86 total 24.32 last 592.58) - /unlink 20000 (time 1773941713.11 total 30.58 last 651.44) - /unlink 26171 (time 1773941723.12 total 40.58 last 616.98) - /unlink 30000 (time 1773941730.28 total 47.75 last 534.25) - /unlink 34907 (time 1773941740.28 total 57.75 last 490.63) - /unlink 40000 (time 1773941749.59 total 67.06 last 547.06) - /unlink 45368 (time 1773941759.60 total 77.06 last 536.72) - /unlink 50000 (time 1773941768.11 total 85.58 last 543.82) - /unlink 54799 (time 1773941778.11 total 95.58 last 479.85) - /unlink 59803 (time 1773941788.11 total 105.58 last 500.38) total: 64768 /unlink in 114.51 seconds: 565.60 ops/second Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 - open/close 2323 (time 1773941819.00 total 10.00 last 232.24) - open/close 5169 (time 1773941829.00 total 20.00 last 284.55) - open/close 8003 (time 1773941839.00 total 30.01 last 283.28) - open/close 10000 (time 1773941847.09 total 38.09 last 247.06) - open/close 12307 (time 1773941857.09 total 48.09 last 230.67) - open/close 14518 (time 1773941867.09 total 58.09 last 221.07) - open/close 16865 (time 1773941877.09 total 68.09 last 234.70) - open/close 19208 (time 1773941887.09 total 78.10 last 234.29) - open/close 20000 (time 1773941890.36 total 81.37 last 241.96) - open/close 22415 (time 1773941900.37 total 91.37 last 241.40) - open/close 24696 (time 1773941910.37 total 101.38 last 228.01) - open/close 27579 (time 1773941920.37 total 111.38 last 288.21) - open/close 29950 (time 1773941930.38 total 121.38 last 237.02) - open/close 32950 (time 1773941940.38 total 131.38 last 299.96) - open/close 35573 (time 1773941950.38 total 141.39 last 262.25) - open/close 38602 (time 1773941960.38 total 151.39 last 302.89) - open/close 40000 (time 1773941965.12 total 156.13 last 294.80) - open/close 42652 (time 1773941975.13 total 166.13 last 265.17) - open/close 45598 (time 1773941985.13 total 176.13 last 294.52) - open/close 49401 (time 1773941995.13 total 186.13 last 380.26) - open/close 53726 (time 1773942005.13 total 196.14 last 432.42) - open/close 57247 (time 1773942015.13 total 206.14 last 351.99) - open/close 60000 (time 1773942025.14 total 216.14 last 275.21) - open/close 62820 (time 1773942035.14 total 226.14 last 281.98) total: 64768 open/close in 233.66 seconds: 277.19 ops/second - /unlink 5019 (time 1773942054.55 total 10.00 last 501.82) - /unlink 9949 (time 1773942064.55 total 20.00 last 492.96) - /unlink 14567 (time 1773942074.55 total 30.00 last 461.79) - /unlink 20000 (time 1773942084.42 total 39.87 last 550.53) - /unlink 25004 (time 1773942094.42 total 49.87 last 500.36) - /unlink 30000 (time 1773942103.94 total 59.39 last 524.81) - /unlink 35087 (time 1773942113.94 total 69.39 last 508.65) - /unlink 40000 (time 1773942123.90 total 79.36 last 493.01) - /unlink 45350 (time 1773942133.90 total 89.36 last 535.00) - /unlink 50000 (time 1773942142.28 total 97.74 last 554.89) - /unlink 55518 (time 1773942152.28 total 107.74 last 551.78) - /unlink 60000 (time 1773942160.76 total 116.22 last 528.74) total: 64768 /unlink in 125.41 seconds: 516.44 ops/second Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 - open/close 2590 (time 1773942190.49 total 10.00 last 258.93) - open/close 5141 (time 1773942200.49 total 20.00 last 255.07) - open/close 7577 (time 1773942210.50 total 30.01 last 243.51) - open/close 10000 (time 1773942220.45 total 39.96 last 243.50) - open/close 12331 (time 1773942230.45 total 49.96 last 233.06) - open/close 14856 (time 1773942240.45 total 59.96 last 252.43) - open/close 18304 (time 1773942250.45 total 69.96 last 344.80) - open/close 20000 (time 1773942256.06 total 75.58 last 302.07) - open/close 22661 (time 1773942266.07 total 85.58 last 266.02) - open/close 25553 (time 1773942276.07 total 95.58 last 289.18) - open/close 28170 (time 1773942286.07 total 105.58 last 261.65) - open/close 30000 (time 1773942293.38 total 112.90 last 250.24) - open/close 32610 (time 1773942303.38 total 122.90 last 260.98) - open/close 35168 (time 1773942313.38 total 132.90 last 255.78) - open/close 38756 (time 1773942323.39 total 142.90 last 358.66) - open/close 40000 (time 1773942327.42 total 146.93 last 308.52) - open/close 44005 (time 1773942337.42 total 156.93 last 400.46) - open/close 48759 (time 1773942347.42 total 166.93 last 475.39) - open/close 50000 (time 1773942350.05 total 169.57 last 471.41) - open/close 54408 (time 1773942360.06 total 179.57 last 440.72) - open/close 58237 (time 1773942370.06 total 189.57 last 382.74) - open/close 60000 (time 1773942374.77 total 194.28 last 374.57) - open/close 64144 (time 1773942384.77 total 204.28 last 414.37) total: 64768 open/close in 205.73 seconds: 314.83 ops/second - /unlink 5490 (time 1773942397.65 total 10.00 last 548.93) - /unlink 10000 (time 1773942405.75 total 18.11 last 556.29) - /unlink 15291 (time 1773942415.76 total 28.11 last 529.07) - /unlink 20000 (time 1773942423.26 total 35.62 last 627.28) - /unlink 25201 (time 1773942433.26 total 45.62 last 520.05) - /unlink 29762 (time 1773942443.26 total 55.62 last 456.05) - /unlink 34956 (time 1773942453.27 total 65.62 last 519.31) - /unlink 40000 (time 1773942461.49 total 73.84 last 613.29) - /unlink 45408 (time 1773942471.49 total 83.84 last 540.78) - /unlink 50000 (time 1773942480.94 total 93.30 last 485.78) - /unlink 54864 (time 1773942490.94 total 103.30 last 486.38) - /unlink 59400 (time 1773942500.94 total 113.30 last 453.56) - /unlink 64623 (time 1773942510.94 total 123.30 last 522.30) total: 64768 /unlink in 123.56 seconds: 524.16 ops/second Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 - open/close 3016 (time 1773942531.04 total 10.00 last 301.48) - open/close 5634 (time 1773942541.04 total 20.01 last 261.75) - open/close 7912 (time 1773942551.04 total 30.01 last 227.79) - open/close 10000 (time 1773942559.31 total 38.28 last 252.42) - open/close 12371 (time 1773942569.32 total 48.28 last 237.03) - open/close 14860 (time 1773942579.32 total 58.28 last 248.82) - open/close 17208 (time 1773942589.32 total 68.29 last 234.75) - open/close 19453 (time 1773942599.33 total 78.29 last 224.43) - open/close 20000 (time 1773942601.91 total 80.87 last 211.58) - open/close 22279 (time 1773942611.91 total 90.88 last 227.89) - open/close 24895 (time 1773942621.91 total 100.88 last 261.52) - open/close 27331 (time 1773942631.92 total 110.88 last 243.59) - open/close 29590 (time 1773942641.92 total 120.88 last 225.86) - open/close 32122 (time 1773942651.92 total 130.88 last 253.12) - open/close 34354 (time 1773942661.92 total 140.89 last 223.14) - open/close 36969 (time 1773942671.93 total 150.89 last 261.38) - open/close 39286 (time 1773942681.93 total 160.90 last 231.60) - open/close 40000 (time 1773942685.16 total 164.12 last 221.36) - open/close 42440 (time 1773942695.16 total 174.13 last 243.90) - open/close 44644 (time 1773942705.16 total 184.13 last 220.37) - open/close 46741 (time 1773942715.17 total 194.13 last 209.65) - open/close 49015 (time 1773942725.17 total 204.13 last 227.33) - open/close 50000 (time 1773942729.52 total 208.49 last 226.09) - open/close 52273 (time 1773942739.53 total 218.49 last 227.24) - open/close 54779 (time 1773942749.53 total 228.49 last 250.56) - open/close 57079 (time 1773942759.53 total 238.50 last 229.92) - open/close 59173 (time 1773942769.53 total 248.50 last 209.38) - open/close 60000 (time 1773942773.50 total 252.47 last 208.33) - open/close 62417 (time 1773942783.51 total 262.47 last 241.61) - open/close 64613 (time 1773942793.51 total 272.47 last 219.53) total: 64768 open/close in 273.19 seconds: 237.08 ops/second - /unlink 4467 (time 1773942806.29 total 10.00 last 446.63) - /unlink 9075 (time 1773942816.30 total 20.00 last 460.71) - /unlink 13506 (time 1773942826.30 total 30.00 last 443.05) - /unlink 17766 (time 1773942836.30 total 40.01 last 425.97) - /unlink 20000 (time 1773942841.20 total 44.90 last 455.98) - /unlink 24948 (time 1773942851.20 total 54.90 last 494.79) - /unlink 29525 (time 1773942861.20 total 64.91 last 457.66) - /unlink 34056 (time 1773942871.20 total 74.91 last 453.08) - /unlink 38353 (time 1773942881.20 total 84.91 last 429.70) - /unlink 40000 (time 1773942885.46 total 89.16 last 386.81) - /unlink 44233 (time 1773942895.46 total 99.16 last 423.26) - /unlink 48617 (time 1773942905.46 total 109.16 last 438.40) - /unlink 50000 (time 1773942908.77 total 112.48 last 417.66) - /unlink 54667 (time 1773942918.77 total 122.48 last 466.63) - /unlink 60000 (time 1773942928.08 total 131.79 last 572.92) - /unlink 61762 (time 1773942938.08 total 141.79 last 176.13) - /unlink 63494 (time 1773942948.09 total 151.80 last 173.05) total: 64768 /unlink in 159.80 seconds: 405.30 ops/second Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 - open/close 3818 (time 1773942977.59 total 10.00 last 381.79) - open/close 8658 (time 1773942987.59 total 20.00 last 483.91) - open/close 10000 (time 1773942990.80 total 23.21 last 418.07) - open/close 13380 (time 1773943000.81 total 33.22 last 337.82) - open/close 16201 (time 1773943010.81 total 43.22 last 282.07) - open/close 20000 (time 1773943019.63 total 52.04 last 430.74) - open/close 24510 (time 1773943029.63 total 62.04 last 450.92) - open/close 29563 (time 1773943039.63 total 72.04 last 505.21) - open/close 34559 (time 1773943049.63 total 82.04 last 499.57) - open/close 38475 (time 1773943059.63 total 92.04 last 391.51) - open/close 40000 (time 1773943063.75 total 96.16 last 370.24) - open/close 44100 (time 1773943073.75 total 106.16 last 409.97) - open/close 48035 (time 1773943083.76 total 116.16 last 393.47) - open/close 50000 (time 1773943089.07 total 121.48 last 369.63) - open/close 53310 (time 1773943099.07 total 131.48 last 330.98) - open/close 56369 (time 1773943109.08 total 141.48 last 305.81) - open/close 59057 (time 1773943119.08 total 151.49 last 268.71) - open/close 60000 (time 1773943122.51 total 154.92 last 274.44) - open/close 62313 (time 1773943132.51 total 164.92 last 231.29) total: 64768 open/close in 172.18 seconds: 376.16 ops/second - /unlink 5802 (time 1773943151.39 total 10.00 last 580.13) - /unlink 10000 (time 1773943157.35 total 15.96 last 704.98) - /unlink 15989 (time 1773943167.35 total 25.96 last 598.90) - /unlink 20000 (time 1773943176.44 total 35.05 last 441.03) - /unlink 25127 (time 1773943186.45 total 45.05 last 512.63) - /unlink 29498 (time 1773943196.45 total 55.05 last 437.06) - /unlink 33843 (time 1773943206.45 total 65.06 last 434.42) - /unlink 38181 (time 1773943216.45 total 75.06 last 433.76) - /unlink 40000 (time 1773943220.21 total 78.82 last 483.73) - /unlink 44638 (time 1773943230.21 total 88.82 last 463.75) - /unlink 49377 (time 1773943240.21 total 98.82 last 473.83) - /unlink 53942 (time 1773943250.21 total 108.82 last 456.47) - /unlink 58393 (time 1773943260.21 total 118.82 last 445.10) - /unlink 60000 (time 1773943263.78 total 122.38 last 450.78) - /unlink 64649 (time 1773943273.78 total 132.39 last 464.81) total: 64768 /unlink in 132.66 seconds: 488.21 ops/second Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local oleg430-server: modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2