-----============= acceptance-small: conf-sanity ============----- Fri Jul 25 06:49:37 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg227-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg227-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg227-server oleg227-server: oleg227-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg227-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg227-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:51:53 (1753440713) Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) oleg227-server: oleg227-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg227-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg227-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1082 (time 1753440786.61 total 10.00 last 108.16) - open/close 2141 (time 1753440796.62 total 20.02 last 105.77) - open/close 3470 (time 1753440806.63 total 30.02 last 132.79) - open/close 4718 (time 1753440816.63 total 40.03 last 124.76) - open/close 6119 (time 1753440826.64 total 50.03 last 140.06) - open/close 7195 (time 1753440836.65 total 60.04 last 107.48) - open/close 8188 (time 1753440846.65 total 70.04 last 99.26) - open/close 9139 (time 1753440856.65 total 80.05 last 95.06) - open/close 10000 (time 1753440864.99 total 88.38 last 103.29) - open/close 11340 (time 1753440874.99 total 98.39 last 133.96) - open/close 12403 (time 1753440885.01 total 108.40 last 106.15) - open/close 13454 (time 1753440895.01 total 118.40 last 105.07) - open/close 14605 (time 1753440905.02 total 128.42 last 114.94) - open/close 15867 (time 1753440915.02 total 138.42 last 126.20) - open/close 17144 (time 1753440925.02 total 148.42 last 127.70) - open/close 18348 (time 1753440935.03 total 158.42 last 120.38) - open/close 19571 (time 1753440945.03 total 168.42 last 122.26) - open/close 20000 (time 1753440948.05 total 171.44 last 142.04) - open/close 21262 (time 1753440958.05 total 181.45 last 126.16) - open/close 22505 (time 1753440968.06 total 191.45 last 124.23) - open/close 23721 (time 1753440978.07 total 201.46 last 121.47) - open/close 24955 (time 1753440988.07 total 211.47 last 123.36) - open/close 26438 (time 1753440998.08 total 221.47 last 148.22) - open/close 27862 (time 1753441008.08 total 231.47 last 142.39) - open/close 28494 (time 1753441018.08 total 241.47 last 63.18) - open/close 28962 (time 1753441028.10 total 251.50 last 46.70) - open/close 29454 (time 1753441038.12 total 261.51 last 49.11) - open/close 30000 (time 1753441047.75 total 271.15 last 56.68) - open/close 30544 (time 1753441057.78 total 281.18 last 54.24) - open/close 31041 (time 1753441067.79 total 291.18 last 49.68) - open/close 31759 (time 1753441077.79 total 301.19 last 71.76) - open/close 32549 (time 1753441087.79 total 311.19 last 78.99) - open/close 33320 (time 1753441097.80 total 321.19 last 77.09) - open/close 34114 (time 1753441107.80 total 331.20 last 79.35) - open/close 34943 (time 1753441117.81 total 341.20 last 82.83) - open/close 35754 (time 1753441127.83 total 351.23 last 80.93) - open/close 36538 (time 1753441137.84 total 361.23 last 78.37) - open/close 37290 (time 1753441147.85 total 371.24 last 75.11) - open/close 38070 (time 1753441157.86 total 381.25 last 77.94) - open/close 38901 (time 1753441167.86 total 391.26 last 83.04) - open/close 39670 (time 1753441177.87 total 401.27 last 76.83) - open/close 40000 (time 1753441182.22 total 405.62 last 75.81) - open/close 40855 (time 1753441192.23 total 415.63 last 85.43) - open/close 41667 (time 1753441202.23 total 425.63 last 81.19) - open/close 42495 (time 1753441212.23 total 435.63 last 82.79) - open/close 43394 (time 1753441222.24 total 445.64 last 89.84) - open/close 44171 (time 1753441232.26 total 455.65 last 77.56) - open/close 44999 (time 1753441242.27 total 465.66 last 82.72) - open/close 45799 (time 1753441252.27 total 475.67 last 79.97) - open/close 46602 (time 1753441262.27 total 485.67 last 80.30) - open/close 47478 (time 1753441272.27 total 495.67 last 87.60) - open/close 48240 (time 1753441282.29 total 505.68 last 76.09) - open/close 48959 (time 1753441292.30 total 515.69 last 71.83) - open/close 49710 (time 1753441302.31 total 525.70 last 75.01) - open/close 50000 (time 1753441306.64 total 530.03 last 66.99) - open/close 50804 (time 1753441316.65 total 540.04 last 80.34) - open/close 51586 (time 1753441326.65 total 550.04 last 78.17) - open/close 52355 (time 1753441336.65 total 560.05 last 76.86) - open/close 53150 (time 1753441346.66 total 570.06 last 79.42) - open/close 53925 (time 1753441356.68 total 580.07 last 77.40) - open/close 54682 (time 1753441366.68 total 590.08 last 75.64) - open/close 55422 (time 1753441376.69 total 600.08 last 73.95) - open/close 56189 (time 1753441386.69 total 610.09 last 76.67) - open/close 57001 (time 1753441396.70 total 620.10 last 81.13) - open/close 58043 (time 1753441406.71 total 630.11 last 104.12) - open/close 58990 (time 1753441416.72 total 640.11 last 94.65) - open/close 59935 (time 1753441426.72 total 650.11 last 94.48) - open/close 60929 (time 1753441436.73 total 660.12 last 99.33) - open/close 61935 (time 1753441446.73 total 670.12 last 100.55) - open/close 62949 (time 1753441456.74 total 680.13 last 101.35) - open/close 63914 (time 1753441466.74 total 690.14 last 96.43) total: 64768 open/close in 699.61 seconds: 92.58 ops/second - /unlink 843 (time 1753441488.88 total 10.02 last 84.15) - /unlink 1848 (time 1753441498.89 total 20.02 last 100.46) - /unlink 2911 (time 1753441508.89 total 30.03 last 106.23) - /unlink 3844 (time 1753441518.90 total 40.04 last 93.23) - /unlink 4621 (time 1753441528.90 total 50.04 last 77.70) - /unlink 5542 (time 1753441538.90 total 60.04 last 92.09) - /unlink 6682 (time 1753441548.91 total 70.05 last 113.89) - /unlink 7643 (time 1753441558.92 total 80.06 last 95.99) - /unlink 8597 (time 1753441568.93 total 90.06 last 95.36) - /unlink 9572 (time 1753441578.93 total 100.06 last 97.50) - /unlink 10000 (time 1753441583.34 total 104.48 last 96.95) - /unlink 11077 (time 1753441593.34 total 114.48 last 107.66) - /unlink 12026 (time 1753441603.35 total 124.49 last 94.83) - /unlink 13020 (time 1753441613.36 total 134.50 last 99.32) - /unlink 13993 (time 1753441623.37 total 144.50 last 97.23) - /unlink 15098 (time 1753441633.38 total 154.52 last 110.36) - /unlink 15998 (time 1753441643.39 total 164.53 last 89.89) - /unlink 16938 (time 1753441653.40 total 174.54 last 93.88) - /unlink 18246 (time 1753441663.41 total 184.54 last 130.76) - /unlink 19505 (time 1753441673.41 total 194.54 last 125.90) - /unlink 20000 (time 1753441678.79 total 199.93 last 91.90) - /unlink 21219 (time 1753441688.80 total 209.93 last 121.86) - /unlink 22321 (time 1753441698.81 total 219.94 last 110.07) - /unlink 23530 (time 1753441708.82 total 229.95 last 120.80) - /unlink 24971 (time 1753441718.83 total 239.96 last 143.97) - /unlink 26218 (time 1753441728.83 total 249.96 last 124.70) - /unlink 27494 (time 1753441738.83 total 259.96 last 127.58) - /unlink 28748 (time 1753441748.83 total 269.96 last 125.39) - /unlink 29667 (time 1753441758.83 total 279.97 last 91.85) - /unlink 30000 (time 1753441762.34 total 283.47 last 95.05) - /unlink 30901 (time 1753441772.35 total 293.49 last 89.95) - /unlink 32265 (time 1753441782.35 total 303.49 last 136.40) - /unlink 33673 (time 1753441792.36 total 313.50 last 140.68) - /unlink 35033 (time 1753441802.37 total 323.51 last 135.91) - /unlink 36552 (time 1753441812.37 total 333.51 last 151.86) - /unlink 37869 (time 1753441822.37 total 343.51 last 131.68) - /unlink 39421 (time 1753441832.38 total 353.52 last 155.10) - /unlink 40000 (time 1753441835.59 total 356.73 last 180.31) - /unlink 41354 (time 1753441845.59 total 366.73 last 135.39) - /unlink 42852 (time 1753441855.60 total 376.73 last 149.74) - /unlink 44433 (time 1753441865.60 total 386.73 last 158.08) - /unlink 46110 (time 1753441875.60 total 396.74 last 167.63) - /unlink 47681 (time 1753441885.60 total 406.74 last 157.06) - /unlink 49527 (time 1753441895.60 total 416.74 last 184.58) - /unlink 52715 (time 1753441905.61 total 426.74 last 318.74) - /unlink 56876 (time 1753441915.61 total 436.74 last 416.07) - /unlink 60000 (time 1753441923.19 total 444.33 last 411.97) - /unlink 63021 (time 1753441933.19 total 454.33 last 302.10) total: 64768 /unlink in 460.47 seconds: 140.66 ops/second Waiting for MDT destroys to complete oleg227-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 - open/close 1423 (time 1753441961.38 total 10.00 last 142.27) - open/close 2916 (time 1753441971.39 total 20.00 last 149.29) - open/close 4382 (time 1753441981.39 total 30.00 last 146.57) - open/close 5567 (time 1753441991.39 total 40.01 last 118.47) - open/close 7085 (time 1753442001.39 total 50.01 last 151.80) - open/close 8460 (time 1753442011.41 total 60.02 last 137.28) - open/close 9748 (time 1753442021.41 total 70.03 last 128.75) - open/close 10000 (time 1753442023.95 total 72.57 last 99.18) - open/close 11208 (time 1753442033.96 total 82.57 last 120.75) - open/close 12664 (time 1753442043.97 total 92.58 last 145.47) - open/close 13817 (time 1753442053.97 total 102.58 last 115.30) - open/close 15215 (time 1753442063.97 total 112.59 last 139.76) - open/close 16410 (time 1753442073.98 total 122.59 last 119.42) - open/close 17927 (time 1753442083.98 total 132.60 last 151.64) - open/close 19284 (time 1753442093.99 total 142.61 last 135.53) - open/close 20000 (time 1753442099.78 total 148.40 last 123.74) - open/close 21324 (time 1753442109.79 total 158.41 last 132.26) - open/close 22821 (time 1753442119.79 total 168.41 last 149.67) - open/close 24576 (time 1753442129.80 total 178.42 last 175.26) - open/close 26202 (time 1753442139.81 total 188.43 last 162.51) - open/close 27738 (time 1753442149.82 total 198.43 last 153.50) - open/close 29479 (time 1753442159.82 total 208.44 last 174.02) - open/close 30000 (time 1753442163.60 total 212.22 last 137.90) - open/close 31644 (time 1753442173.60 total 222.22 last 164.36) - open/close 33348 (time 1753442183.61 total 232.22 last 170.32) - open/close 34792 (time 1753442193.61 total 242.23 last 144.36) - open/close 36401 (time 1753442203.61 total 252.23 last 160.85) - open/close 38123 (time 1753442213.61 total 262.23 last 172.18) - open/close 40000 (time 1753442221.24 total 269.86 last 246.15) - open/close 42186 (time 1753442231.24 total 279.86 last 218.56) - open/close 44071 (time 1753442241.24 total 289.86 last 188.48) - open/close 45880 (time 1753442251.25 total 299.86 last 180.83) - open/close 47858 (time 1753442261.25 total 309.87 last 197.70) - open/close 49671 (time 1753442271.26 total 319.87 last 181.22) - open/close 50000 (time 1753442273.36 total 321.98 last 156.50) - open/close 51526 (time 1753442283.37 total 331.98 last 152.48) - open/close 53161 (time 1753442293.37 total 341.99 last 163.47) - open/close 55421 (time 1753442303.37 total 351.99 last 225.97) - open/close 57549 (time 1753442313.37 total 361.99 last 212.68) - open/close 59994 (time 1753442323.38 total 371.99 last 244.46) - open/close 62583 (time 1753442333.38 total 382.00 last 258.85) total: 64768 open/close in 391.44 seconds: 165.46 ops/second - /unlink 3200 (time 1753442354.66 total 10.00 last 319.98) - /unlink 8084 (time 1753442364.66 total 20.00 last 488.33) - /unlink 10000 (time 1753442368.59 total 23.94 last 486.97) - /unlink 14475 (time 1753442378.59 total 33.94 last 447.46) - /unlink 17728 (time 1753442388.60 total 43.94 last 325.27) - /unlink 20000 (time 1753442396.39 total 51.73 last 291.51) - /unlink 23756 (time 1753442406.39 total 61.73 last 375.52) - /unlink 28132 (time 1753442416.39 total 71.74 last 437.55) - /unlink 30000 (time 1753442420.98 total 76.33 last 406.98) - /unlink 36071 (time 1753442430.98 total 86.33 last 607.04) - /unlink 40000 (time 1753442437.69 total 93.03 last 585.73) - /unlink 47325 (time 1753442447.69 total 103.03 last 732.45) - /unlink 50000 (time 1753442451.30 total 106.64 last 741.12) - /unlink 57595 (time 1753442461.30 total 116.65 last 759.41) - /unlink 60000 (time 1753442465.06 total 120.40 last 640.25) total: 64768 /unlink in 129.79 seconds: 499.04 ops/second Waiting for MDT destroys to complete oleg227-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 - open/close 4105 (time 1753442507.21 total 10.00 last 410.46) - open/close 6959 (time 1753442517.21 total 20.00 last 285.35) - open/close 10000 (time 1753442526.20 total 28.99 last 338.23) - open/close 13568 (time 1753442536.20 total 39.00 last 356.72) - open/close 17857 (time 1753442546.21 total 49.00 last 428.85) - open/close 20000 (time 1753442550.43 total 53.22 last 507.58) - open/close 24580 (time 1753442560.43 total 63.22 last 457.94) - open/close 28974 (time 1753442570.43 total 73.22 last 439.34) - open/close 30000 (time 1753442572.48 total 75.27 last 500.93) - open/close 34171 (time 1753442582.48 total 85.27 last 417.02) - open/close 37491 (time 1753442592.48 total 95.27 last 331.96) - open/close 40000 (time 1753442598.69 total 101.48 last 404.17) - open/close 43489 (time 1753442608.69 total 111.48 last 348.83) - open/close 47718 (time 1753442618.69 total 121.48 last 422.86) - open/close 50000 (time 1753442624.71 total 127.50 last 379.18) - open/close 53631 (time 1753442634.71 total 137.50 last 363.06) - open/close 56992 (time 1753442644.71 total 147.50 last 336.09) - open/close 59902 (time 1753442654.71 total 157.51 last 290.95) - open/close 62702 (time 1753442664.72 total 167.51 last 279.86) total: 64768 open/close in 174.39 seconds: 371.40 ops/second - /unlink 5445 (time 1753442683.22 total 10.00 last 544.49) - /unlink 10000 (time 1753442691.54 total 18.32 last 547.31) - /unlink 16605 (time 1753442701.54 total 28.32 last 660.48) - /unlink 20000 (time 1753442706.91 total 33.69 last 632.30) - /unlink 25611 (time 1753442716.91 total 43.69 last 561.01) - /unlink 30000 (time 1753442722.92 total 49.70 last 731.07) - /unlink 35267 (time 1753442732.92 total 59.70 last 526.66) - /unlink 40000 (time 1753442741.56 total 68.34 last 547.68) - /unlink 45422 (time 1753442751.56 total 78.34 last 542.12) - /unlink 50000 (time 1753442759.53 total 86.31 last 574.67) - /unlink 56753 (time 1753442769.53 total 96.31 last 675.27) - /unlink 60000 (time 1753442775.30 total 102.08 last 562.74) total: 64768 /unlink in 110.54 seconds: 585.91 ops/second Waiting for MDT destroys to complete oleg227-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 - open/close 4076 (time 1753442811.35 total 10.00 last 407.56) - open/close 8723 (time 1753442821.35 total 20.00 last 464.66) - open/close 10000 (time 1753442823.84 total 22.50 last 511.50) - open/close 14979 (time 1753442833.84 total 32.50 last 497.88) - open/close 19247 (time 1753442843.84 total 42.50 last 426.74) - open/close 23559 (time 1753442853.85 total 52.50 last 431.12) - open/close 28205 (time 1753442863.85 total 62.50 last 464.54) - open/close 30000 (time 1753442868.04 total 66.69 last 428.43) - open/close 34561 (time 1753442878.04 total 76.69 last 456.05) - open/close 38392 (time 1753442888.04 total 86.70 last 382.98) - open/close 40000 (time 1753442892.59 total 91.24 last 353.82) - open/close 42993 (time 1753442902.59 total 101.24 last 299.30) - open/close 46347 (time 1753442912.59 total 111.24 last 335.32) - open/close 50000 (time 1753442921.71 total 120.36 last 400.68) - open/close 53342 (time 1753442931.71 total 130.36 last 334.17) - open/close 56273 (time 1753442941.71 total 140.37 last 293.02) - open/close 59624 (time 1753442951.71 total 150.37 last 335.01) - open/close 62363 (time 1753442961.71 total 160.37 last 273.90) total: 64768 open/close in 166.67 seconds: 388.59 ops/second - /unlink 7312 (time 1753442979.53 total 10.00 last 731.16) - /unlink 10000 (time 1753442983.81 total 14.28 last 627.98) - /unlink 16473 (time 1753442993.81 total 24.28 last 647.24) - /unlink 20000 (time 1753442998.49 total 28.95 last 754.73) - /unlink 27948 (time 1753443008.49 total 38.96 last 794.78) - /unlink 30000 (time 1753443011.25 total 41.72 last 742.10) - /unlink 37353 (time 1753443021.25 total 51.72 last 735.29) - /unlink 40000 (time 1753443024.72 total 55.19 last 763.57) - /unlink 47205 (time 1753443034.72 total 65.19 last 720.47) - /unlink 50000 (time 1753443040.65 total 71.12 last 471.01) - /unlink 55576 (time 1753443050.65 total 81.12 last 557.58) - /unlink 60000 (time 1753443059.32 total 89.79 last 510.13) - /unlink 62457 (time 1753443069.33 total 99.80 last 245.48) - /unlink 64313 (time 1753443079.34 total 109.80 last 185.57) total: 64768 /unlink in 113.04 seconds: 572.99 ops/second Waiting for MDT destroys to complete oleg227-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 - open/close 842 (time 1753443116.79 total 10.01 last 84.13) - open/close 1700 (time 1753443126.80 total 20.02 last 85.69) - open/close 2759 (time 1753443136.81 total 30.03 last 105.81) - open/close 3771 (time 1753443146.82 total 40.04 last 101.14) - open/close 4770 (time 1753443156.83 total 50.04 last 99.84) - open/close 5766 (time 1753443166.83 total 60.05 last 99.57) - open/close 6864 (time 1753443176.84 total 70.05 last 109.71) - open/close 7851 (time 1753443186.84 total 80.06 last 98.64) - open/close 8840 (time 1753443196.85 total 90.06 last 98.86) - open/close 9764 (time 1753443206.85 total 100.06 last 92.39) - open/close 10000 (time 1753443209.14 total 102.36 last 102.77) - open/close 10941 (time 1753443219.16 total 112.37 last 94.00) - open/close 11941 (time 1753443229.16 total 122.38 last 99.95) - open/close 12914 (time 1753443239.16 total 132.38 last 97.29) - open/close 13899 (time 1753443249.16 total 142.38 last 98.49) - open/close 15024 (time 1753443259.16 total 152.38 last 112.48) - open/close 16016 (time 1753443269.16 total 162.38 last 99.19) - open/close 16963 (time 1753443279.17 total 172.38 last 94.68) - open/close 17950 (time 1753443289.17 total 182.38 last 98.68) - open/close 19008 (time 1753443299.17 total 192.39 last 105.78) - open/close 20000 (time 1753443308.85 total 202.07 last 102.45) - open/close 20941 (time 1753443318.86 total 212.08 last 94.02) - open/close 21831 (time 1753443328.86 total 222.08 last 88.97) - open/close 22783 (time 1753443338.87 total 232.09 last 95.16) - open/close 23737 (time 1753443348.87 total 242.09 last 95.36) - open/close 24714 (time 1753443358.88 total 252.09 last 97.68) - open/close 25623 (time 1753443368.88 total 262.09 last 90.88) - open/close 26555 (time 1753443378.88 total 272.10 last 93.17) - open/close 27486 (time 1753443388.89 total 282.10 last 93.06) - open/close 28248 (time 1753443398.90 total 292.11 last 76.10) - open/close 28717 (time 1753443408.90 total 302.12 last 46.90) - open/close 29204 (time 1753443418.90 total 312.12 last 48.68) - open/close 29649 (time 1753443428.92 total 322.13 last 44.44) - open/close 30000 (time 1753443436.37 total 329.58 last 47.12) - open/close 30421 (time 1753443446.37 total 339.58 last 42.09) - open/close 31108 (time 1753443456.37 total 349.59 last 68.68) - open/close 31913 (time 1753443466.37 total 359.59 last 80.49) - open/close 32758 (time 1753443476.38 total 369.59 last 84.46) - open/close 33612 (time 1753443486.43 total 379.64 last 84.98) - open/close 34415 (time 1753443496.43 total 389.65 last 80.27) - open/close 35179 (time 1753443506.43 total 399.65 last 76.39) - open/close 36076 (time 1753443516.44 total 409.66 last 89.65) - open/close 36971 (time 1753443526.45 total 419.66 last 89.42) - open/close 37884 (time 1753443536.46 total 429.67 last 91.23) - open/close 38713 (time 1753443546.47 total 439.68 last 82.81) - open/close 39451 (time 1753443556.50 total 449.71 last 73.59) - open/close 40000 (time 1753443563.52 total 456.73 last 78.20) - open/close 40915 (time 1753443573.52 total 466.74 last 91.48) - open/close 41813 (time 1753443583.54 total 476.76 last 89.59) - open/close 42682 (time 1753443593.56 total 486.78 last 86.75) - open/close 43556 (time 1753443603.57 total 496.78 last 87.35) - open/close 44414 (time 1753443613.57 total 506.78 last 85.79) - open/close 45320 (time 1753443623.57 total 516.79 last 90.57) - open/close 46204 (time 1753443633.57 total 526.79 last 88.38) - open/close 47045 (time 1753443643.57 total 536.79 last 84.09) - open/close 47891 (time 1753443653.58 total 546.80 last 84.54) - open/close 48780 (time 1753443663.58 total 556.80 last 88.86) - open/close 49569 (time 1753443673.60 total 566.81 last 78.81) - open/close 50000 (time 1753443678.75 total 571.97 last 83.56) - open/close 50908 (time 1753443688.76 total 581.98 last 90.74) - open/close 51807 (time 1753443698.76 total 591.98 last 89.87) - open/close 52644 (time 1753443708.78 total 601.99 last 83.59) - open/close 53504 (time 1753443718.79 total 612.00 last 85.92) - open/close 54306 (time 1753443728.79 total 622.00 last 80.19) - open/close 55123 (time 1753443738.82 total 632.03 last 81.46) - open/close 55927 (time 1753443748.82 total 642.04 last 80.34) - open/close 56666 (time 1753443758.83 total 652.04 last 73.88) - open/close 57499 (time 1753443768.83 total 662.05 last 83.25) - open/close 58210 (time 1753443778.89 total 672.10 last 70.72) - open/close 58875 (time 1753443788.90 total 682.11 last 66.43) - open/close 59545 (time 1753443798.90 total 692.12 last 66.96) - open/close 60000 (time 1753443805.15 total 698.36 last 72.86) - open/close 60745 (time 1753443815.15 total 708.37 last 74.48) - open/close 61520 (time 1753443825.15 total 718.37 last 77.48) - open/close 62354 (time 1753443835.18 total 728.39 last 83.21) - open/close 63096 (time 1753443845.20 total 738.41 last 74.06) - open/close 63828 (time 1753443855.21 total 748.43 last 73.07) - open/close 64543 (time 1753443865.22 total 758.44 last 71.42) total: 64768 open/close in 762.44 seconds: 84.95 ops/second - /unlink 782 (time 1753443884.46 total 10.00 last 78.16) - /unlink 1479 (time 1753443894.47 total 20.01 last 69.63) - /unlink 2237 (time 1753443904.48 total 30.02 last 75.75) - /unlink 3025 (time 1753443914.48 total 40.02 last 78.78) - /unlink 3792 (time 1753443924.49 total 50.03 last 76.63) - /unlink 4532 (time 1753443934.49 total 60.03 last 74.00) - /unlink 5253 (time 1753443944.49 total 70.04 last 72.06) - /unlink 5952 (time 1753443954.49 total 80.04 last 69.90) - /unlink 6659 (time 1753443964.50 total 90.05 last 70.65) - /unlink 7396 (time 1753443974.51 total 100.05 last 73.64) - /unlink 8176 (time 1753443984.51 total 110.06 last 77.97) - /unlink 8963 (time 1753443994.51 total 120.06 last 78.68) - /unlink 9750 (time 1753444004.52 total 130.07 last 78.62) - /unlink 10000 (time 1753444007.92 total 133.47 last 73.64) - /unlink 10769 (time 1753444017.93 total 143.48 last 76.80) - /unlink 11554 (time 1753444027.94 total 153.48 last 78.48) - /unlink 12302 (time 1753444037.95 total 163.49 last 74.73) - /unlink 13101 (time 1753444047.95 total 173.49 last 79.89) - /unlink 13920 (time 1753444057.95 total 183.50 last 81.85) - /unlink 14704 (time 1753444067.96 total 193.51 last 78.31) - /unlink 15535 (time 1753444077.97 total 203.52 last 83.03) - /unlink 16333 (time 1753444087.98 total 213.53 last 79.72) - /unlink 17081 (time 1753444098.00 total 223.54 last 74.71) - /unlink 17939 (time 1753444108.00 total 233.55 last 85.76) - /unlink 18683 (time 1753444118.01 total 243.55 last 74.35) - /unlink 19492 (time 1753444128.01 total 253.56 last 80.88) - /unlink 20000 (time 1753444133.99 total 259.54 last 84.90) - /unlink 20847 (time 1753444143.99 total 269.54 last 84.70) - /unlink 21665 (time 1753444154.00 total 279.54 last 81.77) - /unlink 22321 (time 1753444164.00 total 289.55 last 65.54) - /unlink 23038 (time 1753444174.06 total 299.61 last 71.28) - /unlink 23811 (time 1753444184.07 total 309.62 last 77.23) - /unlink 24501 (time 1753444194.07 total 319.62 last 68.99) - /unlink 25182 (time 1753444204.08 total 329.62 last 68.08) - /unlink 25875 (time 1753444214.08 total 339.63 last 69.25) - /unlink 26618 (time 1753444224.09 total 349.63 last 74.27) - /unlink 27360 (time 1753444234.09 total 359.64 last 74.18) - /unlink 28137 (time 1753444244.10 total 369.64 last 77.67) - /unlink 29014 (time 1753444254.10 total 379.64 last 87.70) - /unlink 29785 (time 1753444264.11 total 389.66 last 76.97) - /unlink 30000 (time 1753444267.27 total 392.82 last 68.13) - /unlink 30784 (time 1753444277.28 total 402.82 last 78.33) - /unlink 31656 (time 1753444287.29 total 412.83 last 87.12) - /unlink 32472 (time 1753444297.29 total 422.83 last 81.59) - /unlink 33214 (time 1753444307.29 total 432.84 last 74.19) - /unlink 34026 (time 1753444317.30 total 442.84 last 81.16) - /unlink 34945 (time 1753444327.30 total 452.85 last 91.82) - /unlink 35773 (time 1753444337.31 total 462.86 last 82.75) - /unlink 36545 (time 1753444347.32 total 472.87 last 77.12) - /unlink 37363 (time 1753444357.33 total 482.88 last 81.69) - /unlink 38246 (time 1753444367.34 total 492.88 last 88.28) - /unlink 39167 (time 1753444377.35 total 502.89 last 91.99) - /unlink 40000 (time 1753444386.52 total 512.07 last 90.79) - /unlink 40759 (time 1753444396.52 total 522.07 last 75.90) - /unlink 41666 (time 1753444406.53 total 532.08 last 90.62) - /unlink 42569 (time 1753444416.53 total 542.08 last 90.28) - /unlink 43388 (time 1753444426.54 total 552.09 last 81.85) - /unlink 44244 (time 1753444436.55 total 562.10 last 85.51) - /unlink 45082 (time 1753444446.55 total 572.10 last 83.79) - /unlink 45921 (time 1753444456.56 total 582.10 last 83.86) - /unlink 46765 (time 1753444466.57 total 592.11 last 84.32) - /unlink 47621 (time 1753444476.57 total 602.11 last 85.59) - /unlink 48492 (time 1753444486.57 total 612.12 last 87.04) - /unlink 49328 (time 1753444496.58 total 622.12 last 83.56) - /unlink 50000 (time 1753444504.71 total 630.25 last 82.68) - /unlink 50828 (time 1753444514.71 total 640.25 last 82.78) - /unlink 51689 (time 1753444524.71 total 650.25 last 86.10) - /unlink 52527 (time 1753444534.71 total 660.26 last 83.76) - /unlink 53394 (time 1753444544.72 total 670.27 last 86.62) - /unlink 54290 (time 1753444554.72 total 680.27 last 89.57) - /unlink 55230 (time 1753444564.73 total 690.28 last 93.92) - /unlink 56106 (time 1753444574.73 total 700.28 last 87.59) - /unlink 56982 (time 1753444584.74 total 710.29 last 87.56) - /unlink 57859 (time 1753444594.75 total 720.30 last 87.56) - /unlink 58742 (time 1753444604.76 total 730.31 last 88.22) - /unlink 59625 (time 1753444614.78 total 740.32 last 88.18) - /unlink 60000 (time 1753444619.32 total 744.86 last 82.62) - /unlink 60892 (time 1753444629.32 total 754.87 last 89.15) - /unlink 61801 (time 1753444639.32 total 764.87 last 90.87) - /unlink 62697 (time 1753444649.33 total 774.87 last 89.59) - /unlink 63584 (time 1753444659.33 total 784.88 last 88.66) - /unlink 64501 (time 1753444669.33 total 794.88 last 91.68) total: 64768 /unlink in 797.76 seconds: 81.19 ops/second Waiting for MDT destroys to complete oleg227-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local oleg227-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg227-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 PASS 106 (4008s) Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 oleg227-server: oleg227-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4190 sec ========== 07:59:28 (1753444768) === conf-sanity: start cleanup 07:59:29 (1753444769) === === conf-sanity: finish cleanup 07:59:31 (1753444771) ===