== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:09:12 (1761962952) Stopping clients: oleg326-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg326-client.virtnet /mnt/lustre2 (opts:-f) oleg326-server: oleg326-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg326-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg326-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg326-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3755 (time 1761963004.76 total 10.00 last 375.45) - open/close 8317 (time 1761963014.76 total 20.00 last 456.14) - open/close 10000 (time 1761963019.53 total 24.78 last 352.24) - open/close 13384 (time 1761963029.54 total 34.78 last 338.35) - open/close 17680 (time 1761963039.54 total 44.78 last 429.50) - open/close 20000 (time 1761963044.20 total 49.45 last 497.34) - open/close 24091 (time 1761963054.20 total 59.45 last 409.05) - open/close 28069 (time 1761963064.21 total 69.45 last 397.75) - open/close 30000 (time 1761963068.36 total 73.61 last 464.61) - open/close 34164 (time 1761963078.36 total 83.61 last 416.40) - open/close 38308 (time 1761963088.36 total 93.61 last 414.36) - open/close 40000 (time 1761963091.84 total 97.09 last 486.47) - open/close 44810 (time 1761963101.84 total 107.09 last 480.99) - open/close 48913 (time 1761963111.84 total 117.09 last 410.21) - open/close 50000 (time 1761963114.27 total 119.52 last 447.10) - open/close 54033 (time 1761963124.28 total 129.52 last 403.25) - open/close 57633 (time 1761963134.28 total 139.52 last 359.94) - open/close 60000 (time 1761963140.72 total 145.97 last 367.11) - open/close 64306 (time 1761963150.73 total 155.97 last 430.55) total: 64768 open/close in 156.87 seconds: 412.87 ops/second - /unlink 5961 (time 1761963163.02 total 10.02 last 594.70) - /unlink 10000 (time 1761963168.76 total 15.76 last 703.97) - /unlink 16130 (time 1761963178.76 total 25.76 last 612.98) - /unlink 20000 (time 1761963183.45 total 30.45 last 825.36) - /unlink 24497 (time 1761963193.45 total 40.45 last 449.65) - /unlink 29675 (time 1761963203.45 total 50.45 last 517.74) - /unlink 35093 (time 1761963213.45 total 60.45 last 541.77) - /unlink 39889 (time 1761963223.45 total 70.45 last 479.52) - /unlink 46438 (time 1761963233.45 total 80.46 last 654.86) - /unlink 50000 (time 1761963237.93 total 84.93 last 796.61) - /unlink 56912 (time 1761963247.93 total 94.93 last 691.14) - /unlink 60000 (time 1761963252.00 total 99.00 last 758.04) total: 64768 /unlink in 105.25 seconds: 615.39 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 3938 (time 1761963285.50 total 10.00 last 393.75) - open/close 8565 (time 1761963295.50 total 20.00 last 462.60) - open/close 10000 (time 1761963298.51 total 23.01 last 477.74) - open/close 14772 (time 1761963308.51 total 33.01 last 477.18) - open/close 18754 (time 1761963318.51 total 43.01 last 398.15) - open/close 20000 (time 1761963321.97 total 46.47 last 360.33) - open/close 24444 (time 1761963331.97 total 56.47 last 444.34) - open/close 28685 (time 1761963341.97 total 66.47 last 424.02) - open/close 30000 (time 1761963344.68 total 69.18 last 485.19) - open/close 34321 (time 1761963354.68 total 79.18 last 432.08) - open/close 38402 (time 1761963364.68 total 89.18 last 408.06) - open/close 40000 (time 1761963368.56 total 93.06 last 411.90) - open/close 44591 (time 1761963378.56 total 103.06 last 459.09) - open/close 48883 (time 1761963388.56 total 113.06 last 429.18) - open/close 50000 (time 1761963391.25 total 115.75 last 415.63) - open/close 53806 (time 1761963401.25 total 125.75 last 380.53) - open/close 58070 (time 1761963411.25 total 135.75 last 426.37) - open/close 60000 (time 1761963415.82 total 140.32 last 422.44) - open/close 63216 (time 1761963425.82 total 150.32 last 321.56) total: 64768 open/close in 155.09 seconds: 417.62 ops/second - /unlink 7100 (time 1761963442.05 total 10.00 last 709.98) - /unlink 10000 (time 1761963446.12 total 14.07 last 712.81) - /unlink 17623 (time 1761963456.12 total 24.07 last 762.23) - /unlink 20000 (time 1761963459.18 total 27.13 last 776.11) - /unlink 26858 (time 1761963469.18 total 37.13 last 685.72) - /unlink 30000 (time 1761963473.05 total 41.00 last 811.73) - /unlink 38003 (time 1761963483.05 total 51.00 last 800.27) - /unlink 40000 (time 1761963485.50 total 53.45 last 816.00) - /unlink 48633 (time 1761963495.50 total 63.45 last 863.22) - /unlink 56713 (time 1761963505.50 total 73.45 last 807.97) - /unlink 60000 (time 1761963509.21 total 77.16 last 887.39) total: 64768 /unlink in 82.29 seconds: 787.08 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 4873 (time 1761963604.53 total 10.00 last 487.29) - open/close 9231 (time 1761963614.53 total 20.00 last 435.79) - open/close 14387 (time 1761963624.53 total 30.00 last 515.54) - open/close 18882 (time 1761963634.53 total 40.00 last 449.41) - open/close 20000 (time 1761963637.01 total 42.48 last 450.90) - open/close 24461 (time 1761963647.01 total 52.48 last 446.08) - open/close 28872 (time 1761963657.01 total 62.49 last 441.02) - open/close 30000 (time 1761963659.67 total 65.14 last 424.92) - open/close 34852 (time 1761963669.67 total 75.14 last 485.12) - open/close 39699 (time 1761963679.67 total 85.14 last 484.60) - open/close 44257 (time 1761963689.68 total 95.15 last 455.72) - open/close 48570 (time 1761963699.68 total 105.15 last 431.26) - open/close 50000 (time 1761963702.90 total 108.38 last 442.89) - open/close 55028 (time 1761963712.91 total 118.38 last 502.78) - open/close 60000 (time 1761963722.03 total 127.50 last 544.74) total: 64768 open/close in 136.27 seconds: 475.29 ops/second - /unlink 7669 (time 1761963742.34 total 10.00 last 766.84) - /unlink 10000 (time 1761963745.75 total 13.42 last 682.40) - /unlink 18334 (time 1761963755.75 total 23.42 last 833.35) - /unlink 20000 (time 1761963757.80 total 25.46 last 814.39) - /unlink 27983 (time 1761963767.80 total 35.46 last 798.27) - /unlink 30000 (time 1761963770.16 total 37.83 last 853.12) - /unlink 37649 (time 1761963780.17 total 47.83 last 764.83) - /unlink 40000 (time 1761963783.10 total 50.77 last 800.16) - /unlink 47531 (time 1761963793.10 total 60.77 last 753.02) - /unlink 50000 (time 1761963795.97 total 63.64 last 860.62) - /unlink 57186 (time 1761963805.98 total 73.64 last 718.48) - /unlink 60000 (time 1761963809.43 total 77.10 last 813.69) total: 64768 /unlink in 82.82 seconds: 782.08 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 4824 (time 1761963871.07 total 10.00 last 482.37) - open/close 9496 (time 1761963881.08 total 20.00 last 467.13) - open/close 14231 (time 1761963891.08 total 30.00 last 473.50) - open/close 18893 (time 1761963901.08 total 40.00 last 466.15) - open/close 20000 (time 1761963903.47 total 42.40 last 462.59) - open/close 24554 (time 1761963913.47 total 52.40 last 455.38) - open/close 28837 (time 1761963923.47 total 62.40 last 428.22) - open/close 30000 (time 1761963926.51 total 65.44 last 382.21) - open/close 34392 (time 1761963936.52 total 75.44 last 439.18) - open/close 39071 (time 1761963946.52 total 85.44 last 467.89) - open/close 43828 (time 1761963956.52 total 95.44 last 475.63) - open/close 48298 (time 1761963966.52 total 105.44 last 446.95) - open/close 50000 (time 1761963970.99 total 109.92 last 380.35) - open/close 54548 (time 1761963980.99 total 119.92 last 454.79) - open/close 59285 (time 1761963990.99 total 129.92 last 473.66) - open/close 64050 (time 1761964001.00 total 139.92 last 476.44) total: 64768 open/close in 141.63 seconds: 457.31 ops/second - /unlink 7297 (time 1761964014.04 total 10.00 last 729.65) - /unlink 10000 (time 1761964017.43 total 13.39 last 797.17) - /unlink 17751 (time 1761964027.43 total 23.39 last 775.09) - /unlink 20000 (time 1761964030.30 total 26.26 last 783.09) - /unlink 28217 (time 1761964040.30 total 36.26 last 821.66) - /unlink 30000 (time 1761964042.45 total 38.42 last 828.63) - /unlink 38352 (time 1761964052.46 total 48.42 last 835.13) - /unlink 46735 (time 1761964062.46 total 58.42 last 838.21) - /unlink 50000 (time 1761964066.35 total 62.31 last 837.97) - /unlink 58341 (time 1761964076.35 total 72.32 last 834.01) - /unlink 60000 (time 1761964078.42 total 74.38 last 804.23) total: 64768 /unlink in 80.13 seconds: 808.34 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 - open/close 4850 (time 1761964177.94 total 10.00 last 484.92) - open/close 9658 (time 1761964187.94 total 20.00 last 480.72) - open/close 14457 (time 1761964197.94 total 30.00 last 479.84) - open/close 19299 (time 1761964207.94 total 40.01 last 484.15) - open/close 23959 (time 1761964217.94 total 50.01 last 465.94) - open/close 28581 (time 1761964227.94 total 60.01 last 462.16) - open/close 30000 (time 1761964230.91 total 62.97 last 478.30) - open/close 34666 (time 1761964240.91 total 72.98 last 466.48) - open/close 39165 (time 1761964250.92 total 82.98 last 449.83) - open/close 43750 (time 1761964260.92 total 92.98 last 458.47) - open/close 48360 (time 1761964270.92 total 102.98 last 460.98) - open/close 50000 (time 1761964274.33 total 106.39 last 480.53) - open/close 54713 (time 1761964284.33 total 116.39 last 471.23) - open/close 59441 (time 1761964294.33 total 126.40 last 472.72) - open/close 64169 (time 1761964304.33 total 136.40 last 472.80) total: 64768 open/close in 137.65 seconds: 470.54 ops/second - /unlink 8465 (time 1761964316.90 total 10.00 last 846.45) - /unlink 10000 (time 1761964318.96 total 12.07 last 742.03) - /unlink 17823 (time 1761964328.96 total 22.07 last 782.28) - /unlink 20000 (time 1761964331.63 total 24.74 last 816.48) - /unlink 27919 (time 1761964341.63 total 34.74 last 791.82) - /unlink 30000 (time 1761964344.02 total 37.12 last 872.26) - /unlink 37517 (time 1761964354.02 total 47.12 last 751.67) - /unlink 40000 (time 1761964357.14 total 50.25 last 794.31) - /unlink 48140 (time 1761964367.14 total 60.25 last 813.97) - /unlink 50000 (time 1761964369.29 total 62.39 last 868.46) - /unlink 58423 (time 1761964379.29 total 72.39 last 842.21) - /unlink 60000 (time 1761964381.57 total 74.67 last 691.36) total: 64768 /unlink in 80.76 seconds: 802.03 ops/second Waiting for MDT destroys to complete oleg326-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg326-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg326-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing unload_modules_local oleg326-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg326-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2