-----============= acceptance-small: conf-sanity ============----- Thu Jan 1 21:28:02 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg409-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg409-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg409-server oleg409-server: oleg409-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg409-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg409-server: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg409-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:31:42 (1767321102) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) oleg409-server: oleg409-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg409-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 701 (time 1767321292.95 total 10.00 last 70.08) - open/close 1570 (time 1767321302.96 total 20.01 last 86.83) - open/close 2446 (time 1767321312.97 total 30.02 last 87.54) - open/close 3408 (time 1767321322.97 total 40.02 last 96.18) - open/close 4226 (time 1767321332.97 total 50.02 last 81.79) - open/close 5178 (time 1767321342.97 total 60.02 last 95.17) - open/close 6090 (time 1767321352.99 total 70.04 last 91.00) - open/close 6908 (time 1767321363.02 total 80.07 last 81.59) - open/close 7837 (time 1767321373.02 total 90.07 last 92.89) - open/close 8605 (time 1767321383.03 total 100.08 last 76.74) - open/close 9391 (time 1767321393.04 total 110.09 last 78.49) - open/close 10000 (time 1767321402.75 total 119.80 last 62.71) - open/close 10687 (time 1767321412.77 total 129.82 last 68.58) - open/close 11548 (time 1767321422.77 total 139.82 last 86.08) - open/close 12387 (time 1767321432.78 total 149.83 last 83.83) - open/close 13049 (time 1767321442.80 total 159.86 last 66.06) - open/close 13933 (time 1767321452.81 total 169.86 last 88.31) - open/close 14892 (time 1767321462.82 total 179.87 last 95.86) - open/close 15773 (time 1767321472.82 total 189.87 last 88.08) - open/close 16710 (time 1767321482.82 total 199.87 last 93.70) - open/close 17566 (time 1767321492.82 total 209.87 last 85.59) - open/close 18294 (time 1767321502.83 total 219.88 last 72.74) - open/close 19092 (time 1767321512.83 total 229.88 last 79.78) - open/close 20000 (time 1767321522.77 total 239.82 last 91.34) - open/close 20830 (time 1767321532.78 total 249.83 last 82.97) - open/close 21602 (time 1767321542.78 total 259.83 last 77.19) - open/close 22457 (time 1767321552.78 total 269.84 last 85.43) - open/close 23400 (time 1767321562.79 total 279.85 last 94.21) - open/close 24422 (time 1767321572.80 total 289.85 last 102.11) - open/close 25350 (time 1767321582.81 total 299.86 last 92.75) - open/close 26225 (time 1767321592.82 total 309.87 last 87.39) - open/close 27099 (time 1767321602.83 total 319.88 last 87.32) - open/close 27967 (time 1767321612.83 total 329.89 last 86.76) - open/close 28698 (time 1767321622.84 total 339.89 last 73.09) - open/close 29302 (time 1767321632.84 total 349.89 last 60.38) - open/close 29988 (time 1767321642.84 total 359.89 last 68.59) - open/close 30777 (time 1767321652.85 total 369.90 last 78.85) - open/close 31802 (time 1767321662.85 total 379.90 last 102.50) - open/close 32763 (time 1767321672.85 total 389.91 last 96.03) - open/close 33720 (time 1767321682.86 total 399.91 last 95.62) - open/close 34582 (time 1767321692.87 total 409.92 last 86.16) - open/close 35451 (time 1767321702.87 total 419.93 last 86.84) - open/close 36392 (time 1767321712.88 total 429.93 last 94.03) - open/close 37270 (time 1767321722.89 total 439.94 last 87.74) - open/close 38106 (time 1767321732.89 total 449.94 last 83.60) - open/close 38691 (time 1767321742.89 total 459.95 last 58.47) - open/close 39539 (time 1767321752.90 total 469.95 last 84.73) - open/close 40000 (time 1767321758.41 total 475.46 last 83.66) - open/close 40892 (time 1767321768.43 total 485.48 last 89.07) - open/close 41726 (time 1767321778.44 total 495.49 last 83.32) - open/close 42639 (time 1767321788.44 total 505.49 last 91.27) - open/close 43499 (time 1767321798.46 total 515.51 last 85.84) - open/close 44377 (time 1767321808.51 total 525.56 last 87.36) - open/close 45271 (time 1767321818.52 total 535.57 last 89.34) - open/close 46130 (time 1767321828.53 total 545.58 last 85.81) - open/close 47019 (time 1767321838.54 total 555.59 last 88.81) - open/close 47921 (time 1767321848.54 total 565.59 last 90.19) - open/close 48908 (time 1767321858.54 total 575.59 last 98.68) - open/close 49835 (time 1767321868.54 total 585.59 last 92.70) - open/close 50745 (time 1767321878.55 total 595.60 last 90.89) - open/close 51648 (time 1767321888.55 total 605.61 last 90.29) - open/close 52495 (time 1767321898.57 total 615.62 last 84.58) - open/close 53464 (time 1767321908.58 total 625.63 last 96.83) - open/close 54378 (time 1767321918.58 total 635.63 last 91.34) - open/close 55146 (time 1767321928.62 total 645.67 last 76.52) - open/close 55996 (time 1767321938.63 total 655.68 last 84.94) - open/close 56719 (time 1767321948.63 total 665.68 last 72.26) - open/close 57499 (time 1767321958.64 total 675.69 last 77.92) - open/close 58286 (time 1767321968.66 total 685.71 last 78.59) - open/close 58921 (time 1767321978.66 total 695.72 last 63.45) - open/close 59694 (time 1767321988.69 total 705.74 last 77.13) - open/close 60000 (time 1767321992.54 total 709.59 last 79.49) - open/close 60906 (time 1767322002.56 total 719.61 last 90.37) - open/close 61625 (time 1767322012.57 total 729.62 last 71.85) - open/close 62359 (time 1767322022.57 total 739.62 last 73.37) - open/close 63050 (time 1767322032.58 total 749.63 last 69.07) - open/close 63800 (time 1767322042.59 total 759.64 last 74.93) - open/close 64558 (time 1767322052.59 total 769.64 last 75.79) total: 64768 open/close in 772.15 seconds: 83.88 ops/second - /unlink 935 (time 1767322069.49 total 10.00 last 93.48) - /unlink 1721 (time 1767322079.50 total 20.01 last 78.57) - /unlink 2527 (time 1767322089.50 total 30.01 last 80.55) - /unlink 3423 (time 1767322099.51 total 40.02 last 89.56) - /unlink 4309 (time 1767322109.51 total 50.02 last 88.58) - /unlink 5147 (time 1767322119.52 total 60.03 last 83.74) - /unlink 5872 (time 1767322129.53 total 70.04 last 72.41) - /unlink 6613 (time 1767322139.55 total 80.06 last 73.96) - /unlink 7526 (time 1767322149.55 total 90.06 last 91.29) - /unlink 8401 (time 1767322159.55 total 100.06 last 87.48) - /unlink 9218 (time 1767322169.56 total 110.06 last 81.68) - /unlink 10000 (time 1767322178.65 total 119.16 last 85.95) - /unlink 10845 (time 1767322188.66 total 129.17 last 84.43) - /unlink 11737 (time 1767322198.67 total 139.18 last 89.10) - /unlink 12567 (time 1767322208.68 total 149.19 last 82.96) - /unlink 13305 (time 1767322218.69 total 159.20 last 73.68) - /unlink 13846 (time 1767322228.73 total 169.23 last 53.93) - /unlink 14365 (time 1767322238.74 total 179.24 last 51.84) - /unlink 14962 (time 1767322248.76 total 189.27 last 59.56) - /unlink 15750 (time 1767322258.77 total 199.28 last 78.74) - /unlink 16604 (time 1767322268.77 total 209.28 last 85.37) - /unlink 17380 (time 1767322278.78 total 219.28 last 77.56) - /unlink 18102 (time 1767322288.78 total 229.29 last 72.19) - /unlink 19022 (time 1767322298.78 total 239.29 last 91.96) - /unlink 19909 (time 1767322308.79 total 249.29 last 88.68) - /unlink 20789 (time 1767322318.79 total 259.30 last 87.93) - /unlink 21536 (time 1767322328.80 total 269.31 last 74.63) - /unlink 22296 (time 1767322338.81 total 279.32 last 75.91) - /unlink 23055 (time 1767322348.82 total 289.33 last 75.87) - /unlink 23831 (time 1767322358.82 total 299.33 last 77.56) - /unlink 24588 (time 1767322368.84 total 309.34 last 75.61) - /unlink 25406 (time 1767322378.84 total 319.35 last 81.73) - /unlink 26176 (time 1767322388.87 total 329.38 last 76.80) - /unlink 26989 (time 1767322398.89 total 339.40 last 81.10) - /unlink 27757 (time 1767322408.90 total 349.41 last 76.76) - /unlink 28650 (time 1767322418.90 total 359.41 last 89.29) - /unlink 29574 (time 1767322428.90 total 369.41 last 92.37) - /unlink 30000 (time 1767322433.92 total 374.43 last 84.90) - /unlink 30821 (time 1767322443.93 total 384.43 last 82.06) - /unlink 31660 (time 1767322453.93 total 394.44 last 83.85) - /unlink 32466 (time 1767322463.94 total 404.44 last 80.56) - /unlink 33293 (time 1767322473.95 total 414.45 last 82.63) - /unlink 34163 (time 1767322483.95 total 424.46 last 86.92) - /unlink 35084 (time 1767322493.95 total 434.46 last 92.09) - /unlink 35937 (time 1767322503.96 total 444.47 last 85.22) - /unlink 36888 (time 1767322513.98 total 454.48 last 94.99) - /unlink 37648 (time 1767322523.98 total 464.48 last 75.99) - /unlink 38271 (time 1767322533.99 total 474.49 last 62.24) - /unlink 38977 (time 1767322544.00 total 484.51 last 70.51) - /unlink 39752 (time 1767322554.01 total 494.51 last 77.45) - /unlink 40000 (time 1767322556.77 total 497.28 last 89.62) - /unlink 40907 (time 1767322566.78 total 507.29 last 90.66) - /unlink 41740 (time 1767322576.78 total 517.29 last 83.27) - /unlink 42586 (time 1767322586.79 total 527.30 last 84.50) - /unlink 43389 (time 1767322596.79 total 537.30 last 80.30) - /unlink 44337 (time 1767322606.81 total 547.31 last 94.67) - /unlink 45223 (time 1767322616.81 total 557.32 last 88.54) - /unlink 45940 (time 1767322626.81 total 567.32 last 71.70) - /unlink 46803 (time 1767322636.82 total 577.33 last 86.22) - /unlink 47641 (time 1767322646.83 total 587.33 last 83.79) - /unlink 48522 (time 1767322656.83 total 597.34 last 88.07) - /unlink 49484 (time 1767322666.84 total 607.34 last 96.13) - /unlink 50000 (time 1767322673.19 total 613.70 last 81.16) - /unlink 50866 (time 1767322683.20 total 623.71 last 86.52) - /unlink 51765 (time 1767322693.21 total 633.72 last 89.85) - /unlink 52706 (time 1767322703.23 total 643.74 last 93.91) - /unlink 53657 (time 1767322713.23 total 653.74 last 95.05) - /unlink 54528 (time 1767322723.24 total 663.75 last 87.06) - /unlink 55355 (time 1767322733.25 total 673.75 last 82.64) - /unlink 56313 (time 1767322743.26 total 683.76 last 95.71) - /unlink 57270 (time 1767322753.26 total 693.77 last 95.67) - /unlink 58153 (time 1767322763.26 total 703.77 last 88.26) - /unlink 59036 (time 1767322773.27 total 713.77 last 88.28) - /unlink 59895 (time 1767322783.27 total 723.78 last 85.87) - /unlink 60799 (time 1767322793.27 total 733.78 last 90.39) - /unlink 61807 (time 1767322803.28 total 743.78 last 100.75) - /unlink 62730 (time 1767322813.30 total 753.81 last 92.08) - /unlink 63532 (time 1767322823.30 total 763.81 last 80.19) - /unlink 64401 (time 1767322833.31 total 773.82 last 86.83) total: 64768 /unlink in 777.53 seconds: 83.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 901 (time 1767322873.15 total 10.02 last 89.91) - open/close 1816 (time 1767322883.15 total 20.03 last 91.46) - open/close 2753 (time 1767322893.16 total 30.03 last 93.67) - open/close 3749 (time 1767322903.16 total 40.04 last 99.53) - open/close 4818 (time 1767322913.17 total 50.04 last 106.88) - open/close 5947 (time 1767322923.17 total 60.04 last 112.90) - open/close 6922 (time 1767322933.17 total 70.04 last 97.44) - open/close 7895 (time 1767322943.18 total 80.05 last 97.26) - open/close 8946 (time 1767322953.18 total 90.05 last 105.06) - open/close 10000 (time 1767322963.00 total 99.87 last 107.32) - open/close 10974 (time 1767322973.00 total 109.87 last 97.40) - open/close 11961 (time 1767322983.01 total 119.88 last 98.64) - open/close 12856 (time 1767322993.02 total 129.89 last 89.41) - open/close 13772 (time 1767323003.03 total 139.90 last 91.53) - open/close 14723 (time 1767323013.03 total 149.90 last 95.09) - open/close 15689 (time 1767323023.03 total 159.90 last 96.57) - open/close 16526 (time 1767323033.03 total 169.90 last 83.69) - open/close 17464 (time 1767323043.04 total 179.91 last 93.69) - open/close 18371 (time 1767323053.04 total 189.91 last 90.69) - open/close 19292 (time 1767323063.05 total 199.93 last 92.00) - open/close 20000 (time 1767323070.04 total 206.91 last 101.29) - open/close 21043 (time 1767323080.05 total 216.92 last 104.27) - open/close 22015 (time 1767323090.05 total 226.93 last 97.11) - open/close 22860 (time 1767323100.06 total 236.93 last 84.44) - open/close 23771 (time 1767323110.07 total 246.94 last 91.07) - open/close 24630 (time 1767323120.08 total 256.95 last 85.82) - open/close 25341 (time 1767323130.10 total 266.97 last 70.93) - open/close 26165 (time 1767323140.10 total 276.98 last 82.36) - open/close 26966 (time 1767323150.11 total 286.98 last 80.08) - open/close 27899 (time 1767323160.14 total 297.01 last 93.01) - open/close 28660 (time 1767323170.14 total 307.02 last 76.06) - open/close 29637 (time 1767323180.15 total 317.02 last 97.65) - open/close 30000 (time 1767323183.72 total 320.59 last 101.70) - open/close 30964 (time 1767323193.73 total 330.60 last 96.27) - open/close 31956 (time 1767323203.73 total 340.61 last 99.17) - open/close 32738 (time 1767323213.74 total 350.61 last 78.17) - open/close 33556 (time 1767323223.74 total 360.61 last 81.77) - open/close 34477 (time 1767323233.75 total 370.62 last 92.06) - open/close 35355 (time 1767323243.75 total 380.63 last 87.73) - open/close 36198 (time 1767323253.76 total 390.63 last 84.29) - open/close 37129 (time 1767323263.76 total 400.63 last 93.08) - open/close 37997 (time 1767323273.76 total 410.63 last 86.76) - open/close 38968 (time 1767323283.76 total 420.63 last 97.10) - open/close 39909 (time 1767323293.77 total 430.64 last 94.05) - open/close 40952 (time 1767323303.77 total 440.65 last 104.24) - open/close 41962 (time 1767323313.77 total 450.65 last 100.98) - open/close 42845 (time 1767323323.78 total 460.65 last 88.25) - open/close 43807 (time 1767323333.78 total 470.65 last 96.19) - open/close 44924 (time 1767323343.78 total 480.65 last 111.68) - open/close 45938 (time 1767323353.79 total 490.66 last 101.37) - open/close 46938 (time 1767323363.79 total 500.66 last 99.96) - open/close 47877 (time 1767323373.80 total 510.67 last 93.82) - open/close 48828 (time 1767323383.81 total 520.68 last 95.03) - open/close 49765 (time 1767323393.81 total 530.68 last 93.69) - open/close 50000 (time 1767323396.45 total 533.32 last 88.94) - open/close 50954 (time 1767323406.45 total 543.32 last 95.36) - open/close 51903 (time 1767323416.46 total 553.33 last 94.81) - open/close 52719 (time 1767323426.47 total 563.34 last 81.55) - open/close 53468 (time 1767323436.47 total 573.35 last 74.87) - open/close 54341 (time 1767323446.48 total 583.35 last 87.23) - open/close 55352 (time 1767323456.48 total 593.35 last 101.08) - open/close 56335 (time 1767323466.49 total 603.36 last 98.25) - open/close 57355 (time 1767323476.49 total 613.36 last 101.98) - open/close 58328 (time 1767323486.49 total 623.37 last 97.25) - open/close 59194 (time 1767323496.50 total 633.37 last 86.56) - open/close 60000 (time 1767323504.72 total 641.59 last 98.02) - open/close 61139 (time 1767323514.74 total 651.61 last 113.76) - open/close 62052 (time 1767323524.74 total 661.61 last 91.28) - open/close 62958 (time 1767323534.74 total 671.61 last 90.59) - open/close 63897 (time 1767323544.74 total 681.61 last 93.89) total: 64768 open/close in 690.98 seconds: 93.73 ops/second - /unlink 870 (time 1767323567.11 total 10.00 last 86.96) - /unlink 1944 (time 1767323577.12 total 20.01 last 107.37) - /unlink 3052 (time 1767323587.12 total 30.01 last 110.74) - /unlink 3909 (time 1767323597.14 total 40.03 last 85.58) - /unlink 4699 (time 1767323607.14 total 50.03 last 78.99) - /unlink 5813 (time 1767323617.15 total 60.04 last 111.31) - /unlink 6832 (time 1767323627.15 total 70.04 last 101.84) - /unlink 7875 (time 1767323637.15 total 80.04 last 104.29) - /unlink 8761 (time 1767323647.16 total 90.05 last 88.54) - /unlink 9762 (time 1767323657.17 total 100.06 last 100.01) - /unlink 10000 (time 1767323659.51 total 102.40 last 101.76) - /unlink 10951 (time 1767323669.52 total 112.41 last 95.03) - /unlink 11894 (time 1767323679.52 total 122.41 last 94.27) - /unlink 12978 (time 1767323689.53 total 132.42 last 108.29) - /unlink 13992 (time 1767323699.54 total 142.43 last 101.30) - /unlink 15108 (time 1767323709.54 total 152.43 last 111.55) - /unlink 16239 (time 1767323719.55 total 162.44 last 113.03) - /unlink 17438 (time 1767323729.55 total 172.44 last 119.84) - /unlink 18655 (time 1767323739.56 total 182.45 last 121.58) - /unlink 19600 (time 1767323749.56 total 192.46 last 94.49) - /unlink 20000 (time 1767323753.54 total 196.43 last 100.67) - /unlink 21260 (time 1767323763.55 total 206.44 last 125.90) - /unlink 22590 (time 1767323773.56 total 216.45 last 132.89) - /unlink 23874 (time 1767323783.57 total 226.46 last 128.24) - /unlink 25257 (time 1767323793.57 total 236.46 last 138.26) - /unlink 26457 (time 1767323803.57 total 246.46 last 119.98) - /unlink 27708 (time 1767323813.58 total 256.47 last 125.06) - /unlink 28857 (time 1767323823.58 total 266.47 last 114.85) - /unlink 30000 (time 1767323833.16 total 276.05 last 119.29) - /unlink 31185 (time 1767323843.17 total 286.06 last 118.46) - /unlink 32341 (time 1767323853.17 total 296.06 last 115.51) - /unlink 33433 (time 1767323863.18 total 306.07 last 109.13) - /unlink 34509 (time 1767323873.19 total 316.08 last 107.45) - /unlink 35813 (time 1767323883.20 total 326.09 last 130.31) - /unlink 36971 (time 1767323893.20 total 336.09 last 115.76) - /unlink 38140 (time 1767323903.21 total 346.10 last 116.79) - /unlink 39206 (time 1767323913.22 total 356.11 last 106.56) - /unlink 40000 (time 1767323919.81 total 362.70 last 120.38) - /unlink 41260 (time 1767323929.81 total 372.70 last 126.00) - /unlink 42434 (time 1767323939.82 total 382.71 last 117.32) - /unlink 43606 (time 1767323949.82 total 392.71 last 117.15) - /unlink 44802 (time 1767323959.83 total 402.72 last 119.57) - /unlink 45896 (time 1767323969.83 total 412.72 last 109.40) - /unlink 47028 (time 1767323979.83 total 422.72 last 113.17) - /unlink 48074 (time 1767323989.84 total 432.73 last 104.51) - /unlink 49259 (time 1767323999.84 total 442.73 last 118.46) - /unlink 50000 (time 1767324006.69 total 449.58 last 108.16) - /unlink 51121 (time 1767324016.70 total 459.59 last 112.03) - /unlink 52040 (time 1767324026.71 total 469.60 last 91.80) - /unlink 53165 (time 1767324036.71 total 479.60 last 112.45) - /unlink 54460 (time 1767324046.71 total 489.61 last 129.48) - /unlink 56752 (time 1767324056.72 total 499.61 last 229.13) - /unlink 58749 (time 1767324066.72 total 509.61 last 199.65) - /unlink 60000 (time 1767324072.43 total 515.32 last 218.95) - /unlink 62099 (time 1767324082.44 total 525.33 last 209.86) - /unlink 64171 (time 1767324092.44 total 535.33 last 207.20) total: 64768 /unlink in 537.86 seconds: 120.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1274 (time 1767324131.31 total 10.01 last 127.28) - open/close 2579 (time 1767324141.31 total 20.01 last 130.43) - open/close 3911 (time 1767324151.32 total 30.02 last 133.16) - open/close 5148 (time 1767324161.32 total 40.02 last 123.69) - open/close 6294 (time 1767324171.32 total 50.02 last 114.57) - open/close 7526 (time 1767324181.32 total 60.03 last 123.14) - open/close 8653 (time 1767324191.33 total 70.03 last 112.63) - open/close 9869 (time 1767324201.34 total 80.04 last 121.49) - open/close 11128 (time 1767324211.34 total 90.04 last 125.89) - open/close 12375 (time 1767324221.35 total 100.05 last 124.57) - open/close 13554 (time 1767324231.35 total 110.06 last 117.84) - open/close 14759 (time 1767324241.36 total 120.07 last 120.39) - open/close 15913 (time 1767324251.37 total 130.07 last 115.39) - open/close 17066 (time 1767324261.37 total 140.07 last 115.23) - open/close 18306 (time 1767324271.38 total 150.08 last 123.90) - open/close 19527 (time 1767324281.38 total 160.09 last 122.04) - open/close 20000 (time 1767324285.15 total 163.86 last 125.49) - open/close 21138 (time 1767324295.16 total 173.86 last 113.71) - open/close 22367 (time 1767324305.17 total 183.87 last 122.80) - open/close 23639 (time 1767324315.17 total 193.87 last 127.18) - open/close 24947 (time 1767324325.17 total 203.87 last 130.78) - open/close 26233 (time 1767324335.18 total 213.88 last 128.54) - open/close 27423 (time 1767324345.18 total 223.88 last 118.95) - open/close 28577 (time 1767324355.19 total 233.89 last 115.33) - open/close 29895 (time 1767324365.19 total 243.90 last 131.72) - open/close 31283 (time 1767324375.20 total 253.91 last 138.65) - open/close 32647 (time 1767324385.21 total 263.91 last 136.33) - open/close 33939 (time 1767324395.21 total 273.92 last 129.14) - open/close 35273 (time 1767324405.22 total 283.92 last 133.33) - open/close 36584 (time 1767324415.22 total 293.92 last 131.07) - open/close 37999 (time 1767324425.22 total 303.92 last 141.49) - open/close 39597 (time 1767324435.23 total 313.93 last 159.72) - open/close 40000 (time 1767324437.69 total 316.39 last 163.63) - open/close 41709 (time 1767324447.69 total 326.39 last 170.85) - open/close 43108 (time 1767324457.70 total 336.40 last 139.83) - open/close 44958 (time 1767324467.70 total 346.40 last 184.96) - open/close 46849 (time 1767324477.70 total 356.40 last 189.04) - open/close 48404 (time 1767324487.71 total 366.41 last 155.45) - open/close 49841 (time 1767324497.71 total 376.41 last 143.62) - open/close 51209 (time 1767324507.72 total 386.42 last 136.69) - open/close 52907 (time 1767324517.72 total 396.42 last 169.76) - open/close 54503 (time 1767324527.72 total 406.43 last 159.56) - open/close 55818 (time 1767324537.73 total 416.43 last 131.40) - open/close 57460 (time 1767324547.73 total 426.44 last 164.16) - open/close 59261 (time 1767324557.74 total 436.44 last 180.07) - open/close 60000 (time 1767324561.00 total 439.70 last 226.46) - open/close 61753 (time 1767324571.00 total 449.70 last 175.28) - open/close 63942 (time 1767324581.01 total 459.71 last 218.75) total: 64768 open/close in 463.85 seconds: 139.63 ops/second - /unlink 3279 (time 1767324596.92 total 10.00 last 327.81) - /unlink 7432 (time 1767324606.92 total 20.00 last 415.25) - /unlink 10000 (time 1767324612.24 total 25.32 last 483.24) - /unlink 15264 (time 1767324622.24 total 35.32 last 526.29) - /unlink 18815 (time 1767324632.24 total 45.32 last 355.02) - /unlink 20000 (time 1767324635.00 total 48.08 last 430.20) - /unlink 23457 (time 1767324645.00 total 58.08 last 345.66) - /unlink 26889 (time 1767324655.00 total 68.08 last 343.19) - /unlink 30000 (time 1767324663.82 total 76.90 last 352.51) - /unlink 33710 (time 1767324673.83 total 86.91 last 370.84) - /unlink 38065 (time 1767324683.83 total 96.91 last 435.26) - /unlink 40000 (time 1767324688.96 total 102.04 last 377.63) - /unlink 43114 (time 1767324698.96 total 112.04 last 311.37) - /unlink 46799 (time 1767324708.96 total 122.04 last 368.41) - /unlink 50000 (time 1767324718.89 total 131.97 last 322.31) - /unlink 53601 (time 1767324728.89 total 141.97 last 360.05) - /unlink 56857 (time 1767324738.89 total 151.97 last 325.57) - /unlink 60000 (time 1767324745.93 total 159.02 last 446.40) - /unlink 64178 (time 1767324755.94 total 169.02 last 417.74) total: 64768 /unlink in 170.36 seconds: 380.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2025 (time 1767324790.07 total 10.00 last 202.46) - open/close 4357 (time 1767324800.07 total 20.00 last 233.17) - open/close 5943 (time 1767324810.07 total 30.01 last 158.55) - open/close 7480 (time 1767324820.08 total 40.01 last 153.59) - open/close 8830 (time 1767324830.08 total 50.02 last 134.95) - open/close 10000 (time 1767324837.91 total 57.85 last 149.41) - open/close 11812 (time 1767324847.92 total 67.85 last 181.09) - open/close 13651 (time 1767324857.92 total 77.86 last 183.86) - open/close 15771 (time 1767324867.92 total 87.86 last 212.00) - open/close 17651 (time 1767324877.93 total 97.86 last 187.94) - open/close 20000 (time 1767324886.87 total 106.80 last 262.66) - open/close 21789 (time 1767324896.87 total 116.81 last 178.83) - open/close 23431 (time 1767324906.88 total 126.81 last 164.12) - open/close 25296 (time 1767324916.88 total 136.81 last 186.45) - open/close 26768 (time 1767324926.89 total 146.82 last 147.10) - open/close 28234 (time 1767324936.89 total 156.82 last 146.60) - open/close 29677 (time 1767324946.89 total 166.83 last 144.19) - open/close 30000 (time 1767324949.22 total 169.15 last 138.96) - open/close 31342 (time 1767324959.22 total 179.16 last 134.17) - open/close 33111 (time 1767324969.23 total 189.16 last 176.77) - open/close 34574 (time 1767324979.23 total 199.16 last 146.27) - open/close 36109 (time 1767324989.23 total 209.17 last 153.50) - open/close 37635 (time 1767324999.23 total 219.17 last 152.55) - open/close 39282 (time 1767325009.24 total 229.17 last 164.63) - open/close 40000 (time 1767325013.73 total 233.66 last 159.88) - open/close 41432 (time 1767325023.73 total 243.67 last 143.10) - open/close 42925 (time 1767325033.74 total 253.67 last 149.27) - open/close 44437 (time 1767325043.74 total 263.67 last 151.19) - open/close 45918 (time 1767325053.74 total 273.68 last 148.04) - open/close 47988 (time 1767325063.75 total 283.68 last 206.89) - open/close 49634 (time 1767325073.75 total 293.68 last 164.58) - open/close 51847 (time 1767325083.75 total 303.68 last 221.26) - open/close 54533 (time 1767325093.75 total 313.69 last 268.55) - open/close 57065 (time 1767325103.76 total 323.69 last 253.10) - open/close 59371 (time 1767325113.76 total 333.69 last 230.50) - open/close 60000 (time 1767325116.19 total 336.13 last 258.51) - open/close 62906 (time 1767325126.21 total 346.15 last 290.10) total: 64768 open/close in 353.63 seconds: 183.15 ops/second - /unlink 5359 (time 1767325145.16 total 10.00 last 535.84) - /unlink 10000 (time 1767325152.30 total 17.14 last 649.67) - /unlink 15514 (time 1767325162.31 total 27.15 last 551.34) - /unlink 19355 (time 1767325172.31 total 37.15 last 383.83) - /unlink 20000 (time 1767325174.37 total 39.21 last 313.21) - /unlink 23038 (time 1767325184.37 total 49.21 last 303.77) - /unlink 27367 (time 1767325194.38 total 59.22 last 432.65) - /unlink 30000 (time 1767325200.50 total 65.34 last 430.14) - /unlink 34122 (time 1767325210.50 total 75.34 last 412.14) - /unlink 37692 (time 1767325220.50 total 85.34 last 356.95) - /unlink 40000 (time 1767325226.05 total 90.89 last 416.14) - /unlink 43698 (time 1767325236.05 total 100.89 last 369.74) - /unlink 47342 (time 1767325246.05 total 110.89 last 364.37) - /unlink 50000 (time 1767325253.01 total 117.85 last 381.95) - /unlink 53127 (time 1767325263.01 total 127.85 last 312.68) - /unlink 57592 (time 1767325273.01 total 137.85 last 446.38) - /unlink 60000 (time 1767325278.90 total 143.73 last 409.45) - /unlink 63218 (time 1767325288.90 total 153.74 last 321.78) total: 64768 /unlink in 158.23 seconds: 409.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3172 (time 1767325324.00 total 10.00 last 317.15) - open/close 6143 (time 1767325334.00 total 20.00 last 297.06) - open/close 8993 (time 1767325344.00 total 30.01 last 284.93) - open/close 10000 (time 1767325347.26 total 33.26 last 309.48) - open/close 13144 (time 1767325357.26 total 43.26 last 314.35) - open/close 15849 (time 1767325367.26 total 53.26 last 270.42) - open/close 18149 (time 1767325377.26 total 63.26 last 229.99) - open/close 20000 (time 1767325387.14 total 73.14 last 187.43) - open/close 22203 (time 1767325397.14 total 83.14 last 220.23) - open/close 24446 (time 1767325407.15 total 93.15 last 224.21) - open/close 27459 (time 1767325417.15 total 103.16 last 301.06) - open/close 29708 (time 1767325427.16 total 113.16 last 224.86) - open/close 32106 (time 1767325437.16 total 123.16 last 239.71) - open/close 33772 (time 1767325447.16 total 133.16 last 166.57) - open/close 35769 (time 1767325457.17 total 143.17 last 199.59) - open/close 37470 (time 1767325467.17 total 153.17 last 170.08) - open/close 39467 (time 1767325477.17 total 163.17 last 199.66) - open/close 40000 (time 1767325480.17 total 166.18 last 177.41) - open/close 41791 (time 1767325490.18 total 176.18 last 179.03) - open/close 43390 (time 1767325500.18 total 186.18 last 159.89) - open/close 45023 (time 1767325510.18 total 196.18 last 163.29) - open/close 47005 (time 1767325520.18 total 206.18 last 198.17) - open/close 49224 (time 1767325530.18 total 216.18 last 221.88) - open/close 50000 (time 1767325533.76 total 219.76 last 216.74) - open/close 52556 (time 1767325543.76 total 229.77 last 255.57) - open/close 54648 (time 1767325553.76 total 239.77 last 209.19) - open/close 57886 (time 1767325563.77 total 249.77 last 323.78) - open/close 60000 (time 1767325573.11 total 259.11 last 226.29) - open/close 61973 (time 1767325583.11 total 269.11 last 197.24) - open/close 64595 (time 1767325593.11 total 279.11 last 262.18) total: 64768 open/close in 279.70 seconds: 231.56 ops/second - /unlink 3755 (time 1767325605.33 total 10.00 last 375.50) - /unlink 7364 (time 1767325615.33 total 20.00 last 360.80) - /unlink 10000 (time 1767325621.70 total 26.37 last 413.70) - /unlink 14238 (time 1767325631.71 total 36.37 last 423.79) - /unlink 18445 (time 1767325641.71 total 46.38 last 420.66) - /unlink 20000 (time 1767325645.31 total 49.98 last 431.28) - /unlink 25692 (time 1767325655.31 total 59.98 last 569.18) - /unlink 29328 (time 1767325665.31 total 69.98 last 363.51) - /unlink 32939 (time 1767325675.32 total 79.99 last 361.03) - /unlink 36215 (time 1767325685.32 total 89.99 last 327.39) - /unlink 40000 (time 1767325691.73 total 96.40 last 591.03) - /unlink 44773 (time 1767325701.73 total 106.40 last 477.23) - /unlink 50000 (time 1767325711.56 total 116.23 last 531.74) - /unlink 53631 (time 1767325721.56 total 126.23 last 362.96) - /unlink 58017 (time 1767325731.56 total 136.23 last 438.55) - /unlink 60000 (time 1767325736.54 total 141.21 last 398.16) - /unlink 63180 (time 1767325746.55 total 151.22 last 317.85) total: 64768 /unlink in 155.06 seconds: 417.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 PASS 106 (4690s) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 oleg409-server: oleg409-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg409-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4972 sec ========== 22:50:56 (1767325856) === conf-sanity: start cleanup 22:50:56 (1767325856) === === conf-sanity: finish cleanup 22:50:58 (1767325858) ===