== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:00:52 (1755039652) Stopping clients: oleg320-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg320-client.virtnet /mnt/lustre2 (opts:-f) oleg320-server: oleg320-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg320-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg320-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg320-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1439 (time 1755039715.26 total 10.00 last 143.84) - open/close 2645 (time 1755039725.26 total 20.01 last 120.51) - open/close 3713 (time 1755039735.27 total 30.01 last 106.78) - open/close 4959 (time 1755039745.27 total 40.02 last 124.51) - open/close 6328 (time 1755039755.28 total 50.02 last 136.85) - open/close 7978 (time 1755039765.28 total 60.03 last 164.93) - open/close 10000 (time 1755039774.94 total 69.69 last 209.32) - open/close 12283 (time 1755039784.95 total 79.69 last 228.22) - open/close 13942 (time 1755039794.95 total 89.69 last 165.84) - open/close 16118 (time 1755039804.95 total 99.70 last 217.56) - open/close 17632 (time 1755039814.96 total 109.70 last 151.32) - open/close 19861 (time 1755039824.96 total 119.70 last 222.85) - open/close 22988 (time 1755039834.96 total 129.71 last 312.62) - open/close 26239 (time 1755039844.96 total 139.71 last 325.08) - open/close 28639 (time 1755039854.96 total 149.71 last 239.98) - open/close 30000 (time 1755039861.24 total 155.99 last 216.78) - open/close 32803 (time 1755039871.24 total 165.99 last 280.28) - open/close 35590 (time 1755039881.24 total 175.99 last 278.68) - open/close 37820 (time 1755039891.24 total 185.99 last 222.93) - open/close 40000 (time 1755039899.90 total 194.65 last 251.80) - open/close 41827 (time 1755039909.91 total 204.65 last 182.61) - open/close 43682 (time 1755039919.91 total 214.66 last 185.42) - open/close 45823 (time 1755039929.91 total 224.66 last 214.09) - open/close 48348 (time 1755039939.91 total 234.66 last 252.47) - open/close 50000 (time 1755039946.87 total 241.62 last 237.30) - open/close 52143 (time 1755039956.88 total 251.62 last 214.28) - open/close 54474 (time 1755039966.88 total 261.62 last 233.10) - open/close 57581 (time 1755039976.88 total 271.62 last 310.61) - open/close 60000 (time 1755039986.48 total 281.23 last 251.84) - open/close 63128 (time 1755039996.49 total 291.23 last 312.71) total: 64768 open/close in 296.72 seconds: 218.28 ops/second - /unlink 5951 (time 1755040013.43 total 10.00 last 594.99) - /unlink 10000 (time 1755040021.79 total 18.37 last 484.13) - /unlink 14153 (time 1755040031.79 total 28.37 last 415.22) - /unlink 18285 (time 1755040041.79 total 38.37 last 413.11) - /unlink 20000 (time 1755040045.62 total 42.20 last 447.95) - /unlink 24973 (time 1755040055.62 total 52.20 last 497.29) - /unlink 30000 (time 1755040063.14 total 59.71 last 668.88) - /unlink 36270 (time 1755040073.14 total 69.71 last 626.92) - /unlink 40000 (time 1755040078.69 total 75.26 last 672.30) - /unlink 45142 (time 1755040088.69 total 85.26 last 514.16) - /unlink 50000 (time 1755040096.91 total 93.49 last 590.82) - /unlink 55542 (time 1755040106.91 total 103.49 last 554.15) - /unlink 60000 (time 1755040115.08 total 111.65 last 546.08) total: 64768 /unlink in 119.12 seconds: 543.72 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 3199 (time 1755040139.87 total 10.00 last 319.89) - open/close 6544 (time 1755040149.87 total 20.00 last 334.44) - open/close 9689 (time 1755040159.87 total 30.00 last 314.44) - open/close 12630 (time 1755040169.89 total 40.02 last 293.62) - open/close 15803 (time 1755040179.89 total 50.02 last 317.28) - open/close 18473 (time 1755040189.89 total 60.02 last 266.91) - open/close 20000 (time 1755040196.42 total 66.56 last 233.73) - open/close 22868 (time 1755040206.43 total 76.56 last 286.74) - open/close 25848 (time 1755040216.43 total 86.56 last 297.99) - open/close 29300 (time 1755040226.43 total 96.56 last 345.16) - open/close 32942 (time 1755040236.43 total 106.56 last 364.15) - open/close 36596 (time 1755040246.43 total 116.57 last 365.30) - open/close 39687 (time 1755040256.43 total 126.57 last 309.06) - open/close 43112 (time 1755040266.43 total 136.57 last 342.48) - open/close 46051 (time 1755040276.44 total 146.57 last 293.83) - open/close 49230 (time 1755040286.44 total 156.57 last 317.82) - open/close 50000 (time 1755040289.33 total 159.46 last 266.41) - open/close 53040 (time 1755040299.33 total 169.47 last 303.90) - open/close 56077 (time 1755040309.33 total 179.47 last 303.69) - open/close 58603 (time 1755040319.34 total 189.47 last 252.52) - open/close 60000 (time 1755040324.67 total 194.80 last 261.84) - open/close 62864 (time 1755040334.67 total 204.81 last 286.33) total: 64768 open/close in 211.63 seconds: 306.04 ops/second - /unlink 5639 (time 1755040353.09 total 10.00 last 563.83) - /unlink 10000 (time 1755040360.78 total 17.69 last 567.04) - /unlink 15082 (time 1755040370.79 total 27.69 last 508.12) - /unlink 20000 (time 1755040378.98 total 35.89 last 599.92) - /unlink 25662 (time 1755040388.98 total 45.89 last 566.16) - /unlink 30000 (time 1755040395.00 total 51.91 last 720.49) - /unlink 36228 (time 1755040405.01 total 61.91 last 622.70) - /unlink 40000 (time 1755040411.41 total 68.32 last 588.78) - /unlink 46054 (time 1755040421.41 total 78.32 last 605.33) - /unlink 50000 (time 1755040428.22 total 85.12 last 580.16) - /unlink 55674 (time 1755040438.22 total 95.12 last 567.37) - /unlink 60000 (time 1755040444.99 total 101.90 last 638.66) total: 64768 /unlink in 107.68 seconds: 601.48 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 4593 (time 1755040504.32 total 10.00 last 459.15) - open/close 9073 (time 1755040514.32 total 20.00 last 447.93) - open/close 14082 (time 1755040524.32 total 30.01 last 500.83) - open/close 19110 (time 1755040534.32 total 40.01 last 502.73) - open/close 24376 (time 1755040544.32 total 50.01 last 526.56) - open/close 29179 (time 1755040554.32 total 60.01 last 480.29) - open/close 33614 (time 1755040564.32 total 70.01 last 443.38) - open/close 38721 (time 1755040574.33 total 80.01 last 510.66) - open/close 40000 (time 1755040576.98 total 82.66 last 482.17) - open/close 44062 (time 1755040586.98 total 92.67 last 406.12) - open/close 48467 (time 1755040596.98 total 102.67 last 440.42) - open/close 50000 (time 1755040600.08 total 105.76 last 495.17) - open/close 54252 (time 1755040610.08 total 115.77 last 425.10) - open/close 58852 (time 1755040620.08 total 125.77 last 459.95) - open/close 60000 (time 1755040622.58 total 128.27 last 459.38) total: 64768 open/close in 137.76 seconds: 470.16 ops/second - /unlink 8469 (time 1755040643.44 total 10.00 last 846.88) - /unlink 16441 (time 1755040653.44 total 20.00 last 797.18) - /unlink 20000 (time 1755040657.83 total 24.40 last 809.87) - /unlink 27709 (time 1755040667.84 total 34.40 last 770.20) - /unlink 30000 (time 1755040670.74 total 37.30 last 791.81) - /unlink 37646 (time 1755040680.74 total 47.30 last 764.52) - /unlink 40000 (time 1755040684.01 total 50.57 last 718.50) - /unlink 48725 (time 1755040694.02 total 60.58 last 872.43) - /unlink 57927 (time 1755040704.02 total 70.58 last 920.14) - /unlink 60000 (time 1755040706.70 total 73.26 last 773.25) total: 64768 /unlink in 78.30 seconds: 827.12 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 5062 (time 1755040764.93 total 10.00 last 506.16) - open/close 10000 (time 1755040774.24 total 19.30 last 530.74) - open/close 15318 (time 1755040784.24 total 29.31 last 531.74) - open/close 20000 (time 1755040793.38 total 38.45 last 511.93) - open/close 25429 (time 1755040803.38 total 48.45 last 542.82) - open/close 30000 (time 1755040812.97 total 58.04 last 476.92) - open/close 35452 (time 1755040822.97 total 68.04 last 545.11) - open/close 40000 (time 1755040831.80 total 76.87 last 515.24) - open/close 45127 (time 1755040841.80 total 86.87 last 512.67) - open/close 50000 (time 1755040850.89 total 95.96 last 535.97) - open/close 55239 (time 1755040860.89 total 105.96 last 523.87) - open/close 60000 (time 1755040869.72 total 114.79 last 539.30) total: 64768 open/close in 124.11 seconds: 521.86 ops/second - /unlink 9544 (time 1755040890.35 total 10.00 last 954.38) - /unlink 18667 (time 1755040900.35 total 20.00 last 912.29) - /unlink 27890 (time 1755040910.35 total 30.00 last 922.28) - /unlink 30000 (time 1755040913.27 total 32.93 last 720.56) - /unlink 38730 (time 1755040923.28 total 42.93 last 872.95) - /unlink 47128 (time 1755040933.28 total 52.93 last 839.78) - /unlink 50000 (time 1755040936.45 total 56.11 last 904.42) - /unlink 59382 (time 1755040946.45 total 66.11 last 938.18) total: 64768 /unlink in 71.51 seconds: 905.68 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 - open/close 5376 (time 1755041012.23 total 10.00 last 537.51) - open/close 9873 (time 1755041022.23 total 20.00 last 449.61) - open/close 15037 (time 1755041032.23 total 30.01 last 516.33) - open/close 20000 (time 1755041041.64 total 39.41 last 527.51) - open/close 25225 (time 1755041051.64 total 49.41 last 522.48) - open/close 30000 (time 1755041060.56 total 58.34 last 535.16) - open/close 35232 (time 1755041070.56 total 68.34 last 523.17) - open/close 40000 (time 1755041079.20 total 76.97 last 552.25) - open/close 44869 (time 1755041089.20 total 86.97 last 486.84) - open/close 49493 (time 1755041099.20 total 96.97 last 462.30) - open/close 54236 (time 1755041109.20 total 106.97 last 474.28) - open/close 59567 (time 1755041119.20 total 116.98 last 533.00) - open/close 64318 (time 1755041129.20 total 126.98 last 475.10) total: 64768 open/close in 127.85 seconds: 506.58 ops/second - /unlink 9075 (time 1755041141.46 total 10.00 last 907.42) - /unlink 18197 (time 1755041151.47 total 20.00 last 912.07) - /unlink 20000 (time 1755041153.57 total 22.10 last 858.45) - /unlink 29151 (time 1755041163.57 total 32.10 last 915.00) - /unlink 37344 (time 1755041173.57 total 42.10 last 819.27) - /unlink 40000 (time 1755041176.70 total 45.24 last 846.66) - /unlink 48360 (time 1755041186.71 total 55.24 last 835.94) - /unlink 50000 (time 1755041188.71 total 57.25 last 816.68) - /unlink 58083 (time 1755041198.71 total 67.25 last 808.28) - /unlink 60000 (time 1755041200.90 total 69.43 last 877.55) total: 64768 /unlink in 74.99 seconds: 863.71 ops/second Waiting for MDT destroys to complete oleg320-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg320-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2