== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:09:57 (1760137797) Stopping clients: oleg353-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg353-client.virtnet /mnt/lustre2 (opts:-f) oleg353-server: oleg353-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg353-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg353-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg353-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4806 (time 1760137849.32 total 10.00 last 480.54) - open/close 9297 (time 1760137859.32 total 20.00 last 449.03) - open/close 13549 (time 1760137869.32 total 30.01 last 425.07) - open/close 18034 (time 1760137879.33 total 40.01 last 448.45) - open/close 20000 (time 1760137883.76 total 44.45 last 442.95) - open/close 24470 (time 1760137893.76 total 54.45 last 446.99) - open/close 28975 (time 1760137903.77 total 64.45 last 450.42) - open/close 30000 (time 1760137906.00 total 66.68 last 459.50) - open/close 34430 (time 1760137916.00 total 76.68 last 442.95) - open/close 38680 (time 1760137926.00 total 86.68 last 424.99) - open/close 40000 (time 1760137929.11 total 89.79 last 423.77) - open/close 44449 (time 1760137939.11 total 99.80 last 444.86) - open/close 48911 (time 1760137949.12 total 109.80 last 446.06) - open/close 50000 (time 1760137951.64 total 112.32 last 432.21) - open/close 54069 (time 1760137961.64 total 122.32 last 406.82) - open/close 58340 (time 1760137971.64 total 132.32 last 427.02) - open/close 60000 (time 1760137975.93 total 136.61 last 387.25) - open/close 64084 (time 1760137985.93 total 146.61 last 408.35) total: 64768 open/close in 148.28 seconds: 436.78 ops/second - /unlink 7951 (time 1760137999.11 total 10.00 last 795.03) - /unlink 10000 (time 1760138001.67 total 12.57 last 798.63) - /unlink 17695 (time 1760138011.67 total 22.57 last 769.49) - /unlink 20000 (time 1760138014.94 total 25.84 last 704.87) - /unlink 27852 (time 1760138024.94 total 35.84 last 785.12) - /unlink 30000 (time 1760138027.33 total 38.22 last 900.28) - /unlink 38282 (time 1760138037.33 total 48.22 last 828.11) - /unlink 40000 (time 1760138039.84 total 50.74 last 683.37) - /unlink 48743 (time 1760138049.84 total 60.74 last 874.22) - /unlink 57407 (time 1760138059.84 total 70.74 last 866.38) - /unlink 60000 (time 1760138063.12 total 74.02 last 791.39) total: 64768 /unlink in 79.63 seconds: 813.37 ops/second Waiting for MDT destroys to complete oleg353-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 - open/close 4415 (time 1760138127.82 total 10.00 last 441.46) - open/close 8947 (time 1760138137.82 total 20.00 last 453.16) - open/close 10000 (time 1760138140.09 total 22.27 last 464.00) - open/close 14456 (time 1760138150.09 total 32.27 last 445.53) - open/close 18894 (time 1760138160.10 total 42.27 last 443.75) - open/close 20000 (time 1760138162.47 total 44.65 last 464.80) - open/close 24398 (time 1760138172.48 total 54.65 last 439.75) - open/close 28805 (time 1760138182.48 total 64.66 last 440.59) - open/close 30000 (time 1760138185.12 total 67.29 last 453.21) - open/close 34633 (time 1760138195.12 total 77.30 last 463.23) - open/close 39275 (time 1760138205.12 total 87.30 last 464.11) - open/close 43876 (time 1760138215.12 total 97.30 last 460.03) - open/close 48488 (time 1760138225.12 total 107.30 last 461.10) - open/close 50000 (time 1760138228.53 total 110.70 last 444.25) - open/close 54492 (time 1760138238.53 total 120.71 last 449.14) - open/close 59073 (time 1760138248.53 total 130.71 last 458.07) - open/close 63663 (time 1760138258.53 total 140.71 last 458.95) total: 64768 open/close in 143.22 seconds: 452.24 ops/second - /unlink 7746 (time 1760138272.55 total 10.00 last 774.58) - /unlink 10000 (time 1760138275.45 total 12.90 last 776.76) - /unlink 17721 (time 1760138285.45 total 22.90 last 772.05) - /unlink 20000 (time 1760138288.16 total 25.61 last 842.25) - /unlink 28110 (time 1760138298.16 total 35.61 last 810.95) - /unlink 30000 (time 1760138300.63 total 38.08 last 766.07) - /unlink 38228 (time 1760138310.63 total 48.08 last 822.78) - /unlink 40000 (time 1760138312.72 total 50.17 last 846.34) - /unlink 48055 (time 1760138322.72 total 60.17 last 805.43) - /unlink 50000 (time 1760138325.06 total 62.51 last 832.15) - /unlink 58368 (time 1760138335.06 total 72.51 last 836.79) total: 64768 /unlink in 80.27 seconds: 806.85 ops/second Waiting for MDT destroys to complete oleg353-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 - open/close 4822 (time 1760138424.97 total 10.00 last 482.19) - open/close 9510 (time 1760138434.97 total 20.00 last 468.80) - open/close 14140 (time 1760138444.97 total 30.00 last 462.91) - open/close 18600 (time 1760138454.97 total 40.00 last 445.97) - open/close 20000 (time 1760138457.79 total 42.83 last 495.89) - open/close 25034 (time 1760138467.79 total 52.83 last 503.33) - open/close 30000 (time 1760138477.11 total 62.15 last 532.79) - open/close 35021 (time 1760138487.12 total 72.15 last 502.03) - open/close 39403 (time 1760138497.12 total 82.15 last 438.16) - open/close 43584 (time 1760138507.12 total 92.15 last 418.03) - open/close 48021 (time 1760138517.12 total 102.15 last 443.66) - open/close 50000 (time 1760138521.50 total 106.54 last 451.35) - open/close 54784 (time 1760138531.50 total 116.54 last 478.34) - open/close 59322 (time 1760138541.51 total 126.54 last 453.74) - open/close 64076 (time 1760138551.51 total 136.54 last 475.28) total: 64768 open/close in 138.05 seconds: 469.16 ops/second - /unlink 7702 (time 1760138564.56 total 10.00 last 770.17) - /unlink 10000 (time 1760138567.58 total 13.02 last 761.48) - /unlink 17621 (time 1760138577.58 total 23.02 last 762.06) - /unlink 20000 (time 1760138581.80 total 27.24 last 564.18) - /unlink 26443 (time 1760138591.80 total 37.24 last 644.11) - /unlink 28288 (time 1760138601.80 total 47.24 last 184.47) - /unlink 30000 (time 1760138611.01 total 56.45 last 185.95) - /unlink 31778 (time 1760138621.01 total 66.45 last 177.76) - /unlink 34317 (time 1760138631.01 total 76.45 last 253.85) - /unlink 38574 (time 1760138641.01 total 86.45 last 425.67) - /unlink 40000 (time 1760138643.27 total 88.71 last 632.25) - /unlink 46125 (time 1760138653.27 total 98.71 last 612.47) - /unlink 50000 (time 1760138658.77 total 104.20 last 705.04) - /unlink 54680 (time 1760138668.77 total 114.20 last 467.99) - /unlink 58950 (time 1760138678.77 total 124.21 last 426.95) total: 64768 /unlink in 132.59 seconds: 488.48 ops/second Waiting for MDT destroys to complete oleg353-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 - open/close 4917 (time 1760138708.55 total 10.00 last 491.69) - open/close 9284 (time 1760138718.55 total 20.00 last 436.69) - open/close 12417 (time 1760138728.55 total 30.00 last 313.30) - open/close 15824 (time 1760138738.55 total 40.00 last 340.70) - open/close 20000 (time 1760138747.67 total 49.12 last 458.05) - open/close 24677 (time 1760138757.67 total 59.12 last 467.62) - open/close 29299 (time 1760138767.67 total 69.12 last 462.19) - open/close 33318 (time 1760138777.67 total 79.12 last 401.81) - open/close 37624 (time 1760138787.67 total 89.12 last 430.56) - open/close 40000 (time 1760138792.71 total 94.16 last 471.23) - open/close 44267 (time 1760138802.72 total 104.17 last 426.65) - open/close 48536 (time 1760138812.72 total 114.17 last 426.88) - open/close 50000 (time 1760138815.88 total 117.33 last 462.99) - open/close 54366 (time 1760138825.88 total 127.33 last 436.56) - open/close 58890 (time 1760138835.88 total 137.33 last 452.36) - open/close 60000 (time 1760138838.28 total 139.73 last 461.59) - open/close 64652 (time 1760138848.28 total 149.74 last 465.18) total: 64768 open/close in 149.99 seconds: 431.82 ops/second - /unlink 7820 (time 1760138859.90 total 10.00 last 781.92) - /unlink 10000 (time 1760138862.65 total 12.75 last 794.17) - /unlink 17707 (time 1760138872.65 total 22.75 last 770.64) - /unlink 20000 (time 1760138875.59 total 25.69 last 779.76) - /unlink 28089 (time 1760138885.59 total 35.69 last 808.88) - /unlink 30000 (time 1760138887.84 total 37.93 last 851.46) - /unlink 38317 (time 1760138897.84 total 47.93 last 831.67) - /unlink 40000 (time 1760138899.93 total 50.03 last 803.29) - /unlink 48091 (time 1760138909.93 total 60.03 last 809.09) - /unlink 50000 (time 1760138912.09 total 62.19 last 883.88) - /unlink 58307 (time 1760138922.09 total 72.19 last 830.66) total: 64768 /unlink in 80.07 seconds: 808.89 ops/second Waiting for MDT destroys to complete oleg353-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 - open/close 4317 (time 1760139006.08 total 10.00 last 431.65) - open/close 8984 (time 1760139016.08 total 20.00 last 466.68) - open/close 10000 (time 1760139018.35 total 22.27 last 447.09) - open/close 14118 (time 1760139028.35 total 32.28 last 411.76) - open/close 18803 (time 1760139038.35 total 42.28 last 468.49) - open/close 20000 (time 1760139041.03 total 44.95 last 448.04) - open/close 23889 (time 1760139051.03 total 54.95 last 388.90) - open/close 27768 (time 1760139061.03 total 64.95 last 387.83) - open/close 30000 (time 1760139066.76 total 70.68 last 389.23) - open/close 33685 (time 1760139076.77 total 80.69 last 368.33) - open/close 37279 (time 1760139086.77 total 90.69 last 359.37) - open/close 40000 (time 1760139093.46 total 97.39 last 406.33) - open/close 44128 (time 1760139103.47 total 107.39 last 412.77) - open/close 47159 (time 1760139113.47 total 117.39 last 303.03) - open/close 50000 (time 1760139121.48 total 125.40 last 354.53) - open/close 53153 (time 1760139131.48 total 135.40 last 315.28) - open/close 56998 (time 1760139141.48 total 145.40 last 384.48) - open/close 60000 (time 1760139150.33 total 154.25 last 339.30) - open/close 63560 (time 1760139160.33 total 164.25 last 355.98) total: 64768 open/close in 167.17 seconds: 387.43 ops/second - /unlink 7121 (time 1760139174.65 total 10.00 last 712.09) - /unlink 10000 (time 1760139178.83 total 14.18 last 689.41) - /unlink 17335 (time 1760139188.83 total 24.18 last 733.47) - /unlink 20000 (time 1760139194.09 total 29.43 last 506.88) - /unlink 26960 (time 1760139204.09 total 39.43 last 695.97) - /unlink 30000 (time 1760139208.90 total 44.25 last 631.66) - /unlink 36174 (time 1760139218.90 total 54.25 last 617.34) - /unlink 40000 (time 1760139224.04 total 59.39 last 744.14) - /unlink 46277 (time 1760139234.04 total 69.39 last 627.66) - /unlink 50000 (time 1760139240.86 total 76.20 last 546.37) - /unlink 56718 (time 1760139250.86 total 86.21 last 671.68) - /unlink 60000 (time 1760139255.63 total 90.98 last 688.19) total: 64768 /unlink in 97.62 seconds: 663.49 ops/second Waiting for MDT destroys to complete oleg353-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg353-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local oleg353-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg353-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2