-----============= acceptance-small: conf-sanity ============----- Sun Nov 16 22:34:58 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg120-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg120-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg120-server oleg120-server: oleg120-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:37:20 (1763350640) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) oleg120-server: oleg120-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 582 (time 1763350744.91 total 10.01 last 58.17) - open/close 1195 (time 1763350754.91 total 20.01 last 61.29) - open/close 1794 (time 1763350764.94 total 30.04 last 59.72) - open/close 2444 (time 1763350774.94 total 40.04 last 65.00) - open/close 3066 (time 1763350784.96 total 50.05 last 62.11) - open/close 3715 (time 1763350794.96 total 60.06 last 64.87) - open/close 4351 (time 1763350804.97 total 70.06 last 63.57) - open/close 4952 (time 1763350814.97 total 80.06 last 60.09) - open/close 5584 (time 1763350824.98 total 90.07 last 63.14) - open/close 6190 (time 1763350834.99 total 100.08 last 60.54) - open/close 6794 (time 1763350845.00 total 110.09 last 60.37) - open/close 7408 (time 1763350855.01 total 120.10 last 61.33) - open/close 8076 (time 1763350865.01 total 130.10 last 66.80) - open/close 8688 (time 1763350875.02 total 140.11 last 61.13) - open/close 9326 (time 1763350885.05 total 150.14 last 63.60) - open/close 9907 (time 1763350895.06 total 160.16 last 58.02) - open/close 10504 (time 1763350905.07 total 170.16 last 59.66) - open/close 11123 (time 1763350915.08 total 180.17 last 61.85) - open/close 11760 (time 1763350925.10 total 190.19 last 63.58) - open/close 12390 (time 1763350935.12 total 200.21 last 62.89) - open/close 13039 (time 1763350945.15 total 210.25 last 64.65) - open/close 13671 (time 1763350955.16 total 220.26 last 63.13) - open/close 14303 (time 1763350965.18 total 230.27 last 63.11) - open/close 14936 (time 1763350975.19 total 240.28 last 63.26) - open/close 15599 (time 1763350985.20 total 250.29 last 66.23) - open/close 16232 (time 1763350995.21 total 260.30 last 63.22) - open/close 16862 (time 1763351005.22 total 270.32 last 62.91) - open/close 17504 (time 1763351015.24 total 280.33 last 64.08) - open/close 18152 (time 1763351025.24 total 290.33 last 64.80) - open/close 18811 (time 1763351035.26 total 300.35 last 65.79) - open/close 19439 (time 1763351045.27 total 310.36 last 62.73) - open/close 20000 (time 1763351054.25 total 319.34 last 62.49) - open/close 20608 (time 1763351064.25 total 329.34 last 60.78) - open/close 21267 (time 1763351074.26 total 339.35 last 65.84) - open/close 21963 (time 1763351084.28 total 349.37 last 69.46) - open/close 22588 (time 1763351094.29 total 359.38 last 62.41) - open/close 23207 (time 1763351104.30 total 369.40 last 61.82) - open/close 23835 (time 1763351114.32 total 379.41 last 62.70) - open/close 24478 (time 1763351124.32 total 389.42 last 64.28) - open/close 25134 (time 1763351134.34 total 399.43 last 65.51) - open/close 25761 (time 1763351144.35 total 409.45 last 62.59) - open/close 26384 (time 1763351154.36 total 419.45 last 62.25) - open/close 27007 (time 1763351164.36 total 429.46 last 62.29) - open/close 27578 (time 1763351174.37 total 439.46 last 57.08) - open/close 28197 (time 1763351184.38 total 449.47 last 61.82) - open/close 28801 (time 1763351194.39 total 459.48 last 60.34) - open/close 29433 (time 1763351204.40 total 469.50 last 63.13) - open/close 30000 (time 1763351213.29 total 478.38 last 63.80) - open/close 30644 (time 1763351223.29 total 488.38 last 64.39) - open/close 31239 (time 1763351233.29 total 498.38 last 59.50) - open/close 31831 (time 1763351243.31 total 508.41 last 59.08) - open/close 32400 (time 1763351253.32 total 518.42 last 56.84) - open/close 32867 (time 1763351263.33 total 528.42 last 46.68) - open/close 33445 (time 1763351273.35 total 538.44 last 57.67) - open/close 34056 (time 1763351283.36 total 548.45 last 61.04) - open/close 34645 (time 1763351293.37 total 558.46 last 58.84) - open/close 35251 (time 1763351303.40 total 568.49 last 60.44) - open/close 35869 (time 1763351313.40 total 578.49 last 61.77) - open/close 36483 (time 1763351323.40 total 588.50 last 61.38) - open/close 37127 (time 1763351333.41 total 598.50 last 64.35) - open/close 37749 (time 1763351343.42 total 608.51 last 62.15) - open/close 38450 (time 1763351353.42 total 618.51 last 70.10) - open/close 39158 (time 1763351363.44 total 628.53 last 70.67) - open/close 39869 (time 1763351373.44 total 638.53 last 71.10) - open/close 40000 (time 1763351375.57 total 640.66 last 61.44) - open/close 40669 (time 1763351385.57 total 650.67 last 66.88) - open/close 41355 (time 1763351395.57 total 660.67 last 68.60) - open/close 41959 (time 1763351405.58 total 670.67 last 60.39) - open/close 42578 (time 1763351415.58 total 680.68 last 61.86) - open/close 43229 (time 1763351425.60 total 690.69 last 65.00) - open/close 43890 (time 1763351435.60 total 700.69 last 66.09) - open/close 44576 (time 1763351445.60 total 710.70 last 68.58) - open/close 45241 (time 1763351455.62 total 720.71 last 66.42) - open/close 45877 (time 1763351465.63 total 730.72 last 63.53) - open/close 46503 (time 1763351475.63 total 740.73 last 62.56) - open/close 47060 (time 1763351485.63 total 750.73 last 55.69) - open/close 47730 (time 1763351495.65 total 760.74 last 66.93) - open/close 48442 (time 1763351505.66 total 770.75 last 71.13) - open/close 49073 (time 1763351515.67 total 780.76 last 63.02) - open/close 49706 (time 1763351525.67 total 790.76 last 63.30) - open/close 50000 (time 1763351530.24 total 795.33 last 64.30) - open/close 50655 (time 1763351540.25 total 805.34 last 65.44) - open/close 51316 (time 1763351550.26 total 815.36 last 66.00) - open/close 52007 (time 1763351560.26 total 825.36 last 69.09) - open/close 52648 (time 1763351570.27 total 835.36 last 64.09) - open/close 53305 (time 1763351580.27 total 845.36 last 65.69) - open/close 53991 (time 1763351590.28 total 855.37 last 68.51) - open/close 54583 (time 1763351600.29 total 865.38 last 59.15) - open/close 55172 (time 1763351610.30 total 875.39 last 58.86) - open/close 55841 (time 1763351620.31 total 885.40 last 66.83) - open/close 56495 (time 1763351630.33 total 895.42 last 65.27) - open/close 57158 (time 1763351640.34 total 905.43 last 66.25) - open/close 57857 (time 1763351650.34 total 915.44 last 69.85) - open/close 58485 (time 1763351660.35 total 925.44 last 62.76) - open/close 59152 (time 1763351670.35 total 935.45 last 66.67) - open/close 59829 (time 1763351680.36 total 945.45 last 67.68) - open/close 60000 (time 1763351682.84 total 947.93 last 68.89) - open/close 60685 (time 1763351692.85 total 957.94 last 68.44) - open/close 61351 (time 1763351702.86 total 967.95 last 66.54) - open/close 62028 (time 1763351712.86 total 977.95 last 67.68) - open/close 62711 (time 1763351722.86 total 987.95 last 68.30) - open/close 63406 (time 1763351732.87 total 997.96 last 69.43) - open/close 64108 (time 1763351742.88 total 1007.97 last 70.14) total: 64768 open/close in 1017.17 seconds: 63.67 ops/second - /unlink 676 (time 1763351764.81 total 10.01 last 67.52) - /unlink 1381 (time 1763351774.82 total 20.03 last 70.41) - /unlink 2077 (time 1763351784.85 total 30.05 last 69.40) - /unlink 2789 (time 1763351794.85 total 40.06 last 71.19) - /unlink 3500 (time 1763351804.86 total 50.06 last 71.04) - /unlink 4261 (time 1763351814.86 total 60.07 last 76.09) - /unlink 5148 (time 1763351824.86 total 70.07 last 88.69) - /unlink 6121 (time 1763351834.87 total 80.07 last 97.28) - /unlink 7039 (time 1763351844.87 total 90.08 last 91.73) - /unlink 7832 (time 1763351854.88 total 100.08 last 79.25) - /unlink 8574 (time 1763351864.89 total 110.09 last 74.16) - /unlink 9293 (time 1763351874.89 total 120.09 last 71.87) - /unlink 10000 (time 1763351884.18 total 129.39 last 76.08) - /unlink 10717 (time 1763351894.19 total 139.39 last 71.64) - /unlink 11486 (time 1763351904.20 total 149.40 last 76.87) - /unlink 12372 (time 1763351914.20 total 159.40 last 88.55) - /unlink 13190 (time 1763351924.21 total 169.41 last 81.75) - /unlink 14066 (time 1763351934.22 total 179.42 last 87.49) - /unlink 15056 (time 1763351944.23 total 189.43 last 98.89) - /unlink 15922 (time 1763351954.24 total 199.44 last 86.53) - /unlink 16954 (time 1763351964.24 total 209.45 last 103.15) - /unlink 17907 (time 1763351974.26 total 219.46 last 95.17) - /unlink 18871 (time 1763351984.27 total 229.47 last 96.31) - /unlink 20000 (time 1763351993.27 total 238.47 last 125.41) - /unlink 20943 (time 1763352003.27 total 248.48 last 94.26) - /unlink 21908 (time 1763352013.28 total 258.48 last 96.44) - /unlink 22822 (time 1763352023.29 total 268.49 last 91.33) - /unlink 23852 (time 1763352033.30 total 278.50 last 102.90) - /unlink 24791 (time 1763352043.30 total 288.50 last 93.86) - /unlink 25706 (time 1763352053.31 total 298.51 last 91.43) - /unlink 26577 (time 1763352063.31 total 308.51 last 87.08) - /unlink 27445 (time 1763352073.32 total 318.52 last 86.72) - /unlink 28449 (time 1763352083.33 total 328.53 last 100.35) - /unlink 29563 (time 1763352093.33 total 338.53 last 111.39) - /unlink 30000 (time 1763352097.55 total 342.75 last 103.47) - /unlink 30971 (time 1763352107.55 total 352.75 last 97.08) - /unlink 31935 (time 1763352117.55 total 362.76 last 96.39) - /unlink 32874 (time 1763352127.56 total 372.77 last 93.80) - /unlink 33765 (time 1763352137.57 total 382.77 last 89.08) - /unlink 34580 (time 1763352147.57 total 392.77 last 81.46) - /unlink 35457 (time 1763352157.58 total 402.79 last 87.58) - /unlink 36377 (time 1763352167.59 total 412.79 last 91.97) - /unlink 37282 (time 1763352177.60 total 422.80 last 90.42) - /unlink 38269 (time 1763352187.60 total 432.80 last 98.66) - /unlink 39385 (time 1763352197.60 total 442.81 last 111.56) - /unlink 40000 (time 1763352204.53 total 449.74 last 88.73) - /unlink 40949 (time 1763352214.54 total 459.74 last 94.89) - /unlink 41972 (time 1763352224.54 total 469.74 last 102.25) - /unlink 42896 (time 1763352234.56 total 479.76 last 92.23) - /unlink 43777 (time 1763352244.57 total 489.77 last 87.99) - /unlink 44724 (time 1763352254.58 total 499.78 last 94.65) - /unlink 45591 (time 1763352264.58 total 509.79 last 86.64) - /unlink 46569 (time 1763352274.59 total 519.79 last 97.75) - /unlink 47720 (time 1763352284.59 total 529.79 last 115.06) - /unlink 48720 (time 1763352294.60 total 539.80 last 99.96) - /unlink 49919 (time 1763352304.60 total 549.80 last 119.84) - /unlink 50980 (time 1763352314.61 total 559.81 last 106.05) - /unlink 52017 (time 1763352324.62 total 569.83 last 103.53) - /unlink 53046 (time 1763352334.63 total 579.83 last 102.87) - /unlink 53900 (time 1763352344.63 total 589.84 last 85.33) - /unlink 54990 (time 1763352354.64 total 599.84 last 108.97) - /unlink 55907 (time 1763352364.65 total 609.85 last 91.62) - /unlink 56934 (time 1763352374.66 total 619.86 last 102.58) - /unlink 57948 (time 1763352384.66 total 629.86 last 101.36) - /unlink 59019 (time 1763352394.67 total 639.87 last 107.01) - /unlink 60000 (time 1763352403.17 total 648.37 last 115.38) - /unlink 60944 (time 1763352413.18 total 658.38 last 94.35) - /unlink 61847 (time 1763352423.18 total 668.38 last 90.27) - /unlink 62805 (time 1763352433.18 total 678.39 last 95.76) - /unlink 63802 (time 1763352443.19 total 688.39 last 99.63) total: 64768 /unlink in 698.15 seconds: 92.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 972 (time 1763352487.53 total 10.00 last 97.16) - open/close 1977 (time 1763352497.53 total 20.01 last 100.49) - open/close 2990 (time 1763352507.53 total 30.01 last 101.27) - open/close 4051 (time 1763352517.54 total 40.01 last 106.09) - open/close 5052 (time 1763352527.54 total 50.02 last 100.03) - open/close 5910 (time 1763352537.55 total 60.03 last 85.70) - open/close 6753 (time 1763352547.56 total 70.03 last 84.29) - open/close 7578 (time 1763352557.57 total 80.04 last 82.42) - open/close 8568 (time 1763352567.57 total 90.04 last 98.96) - open/close 9721 (time 1763352577.57 total 100.05 last 115.25) - open/close 10000 (time 1763352580.25 total 102.73 last 104.13) - open/close 11061 (time 1763352590.26 total 112.73 last 106.04) - open/close 12246 (time 1763352600.26 total 122.73 last 118.48) - open/close 13327 (time 1763352610.27 total 132.74 last 108.03) - open/close 14343 (time 1763352620.27 total 142.75 last 101.54) - open/close 15510 (time 1763352630.28 total 152.75 last 116.61) - open/close 16648 (time 1763352640.28 total 162.76 last 113.75) - open/close 17578 (time 1763352650.29 total 172.76 last 92.99) - open/close 18520 (time 1763352660.29 total 182.76 last 94.14) - open/close 19642 (time 1763352670.29 total 192.77 last 112.19) - open/close 20000 (time 1763352673.94 total 196.41 last 98.13) - open/close 20915 (time 1763352683.94 total 206.42 last 91.47) - open/close 21744 (time 1763352693.95 total 216.42 last 82.88) - open/close 22591 (time 1763352703.95 total 226.42 last 84.69) - open/close 23471 (time 1763352713.96 total 236.44 last 87.87) - open/close 24374 (time 1763352723.97 total 246.44 last 90.23) - open/close 25445 (time 1763352733.98 total 256.45 last 107.01) - open/close 26651 (time 1763352743.98 total 266.46 last 120.54) - open/close 27647 (time 1763352753.98 total 276.46 last 99.59) - open/close 28545 (time 1763352763.99 total 286.46 last 89.79) - open/close 29533 (time 1763352773.99 total 296.46 last 98.74) - open/close 30000 (time 1763352779.15 total 301.63 last 90.45) - open/close 30992 (time 1763352789.17 total 311.64 last 99.06) - open/close 32054 (time 1763352799.17 total 321.65 last 106.15) - open/close 33208 (time 1763352809.19 total 331.66 last 115.27) - open/close 34486 (time 1763352819.19 total 341.66 last 127.80) - open/close 35564 (time 1763352829.19 total 351.66 last 107.74) - open/close 36474 (time 1763352839.20 total 361.67 last 90.95) - open/close 37428 (time 1763352849.20 total 371.67 last 95.36) - open/close 38666 (time 1763352859.21 total 381.68 last 123.74) - open/close 39920 (time 1763352869.21 total 391.69 last 125.31) - open/close 40968 (time 1763352879.22 total 401.69 last 104.78) - open/close 42152 (time 1763352889.22 total 411.69 last 118.35) - open/close 43383 (time 1763352899.22 total 421.69 last 123.08) - open/close 44495 (time 1763352909.23 total 431.70 last 111.13) - open/close 45442 (time 1763352919.23 total 441.71 last 94.65) - open/close 46535 (time 1763352929.24 total 451.71 last 109.24) - open/close 47693 (time 1763352939.24 total 461.72 last 115.74) - open/close 48913 (time 1763352949.25 total 471.72 last 121.91) - open/close 49997 (time 1763352959.25 total 481.73 last 108.38) - open/close 51227 (time 1763352969.26 total 491.73 last 122.91) - open/close 52428 (time 1763352979.26 total 501.73 last 120.10) - open/close 53893 (time 1763352989.27 total 511.74 last 146.44) - open/close 55047 (time 1763352999.27 total 521.74 last 115.40) - open/close 56186 (time 1763353009.27 total 531.74 last 113.84) - open/close 57581 (time 1763353019.29 total 541.76 last 139.23) - open/close 59233 (time 1763353029.29 total 551.77 last 165.14) - open/close 60000 (time 1763353033.60 total 556.07 last 178.25) - open/close 61746 (time 1763353043.60 total 566.08 last 174.49) - open/close 63372 (time 1763353053.60 total 576.08 last 162.59) total: 64768 open/close in 584.65 seconds: 110.78 ops/second - /unlink 2066 (time 1763353074.07 total 10.01 last 206.34) - /unlink 4227 (time 1763353084.07 total 20.01 last 216.05) - /unlink 6309 (time 1763353094.07 total 30.02 last 208.16) - /unlink 8601 (time 1763353104.07 total 40.02 last 229.17) - /unlink 10000 (time 1763353109.64 total 45.58 last 251.46) - /unlink 12665 (time 1763353119.64 total 55.58 last 266.49) - /unlink 15428 (time 1763353129.64 total 65.59 last 276.21) - /unlink 18248 (time 1763353139.64 total 75.59 last 281.94) - /unlink 20000 (time 1763353147.52 total 83.47 last 222.38) - /unlink 22817 (time 1763353157.56 total 93.50 last 280.70) - /unlink 27179 (time 1763353167.56 total 103.50 last 436.14) - /unlink 30000 (time 1763353177.28 total 113.22 last 290.18) - /unlink 32415 (time 1763353187.28 total 123.23 last 241.46) - /unlink 35149 (time 1763353197.28 total 133.23 last 273.35) - /unlink 38300 (time 1763353207.28 total 143.23 last 315.09) - /unlink 40000 (time 1763353213.07 total 149.02 last 293.70) - /unlink 42179 (time 1763353223.07 total 159.02 last 217.83) - /unlink 44717 (time 1763353233.08 total 169.02 last 253.75) - /unlink 47137 (time 1763353243.08 total 179.02 last 241.94) - /unlink 49583 (time 1763353253.08 total 189.03 last 244.51) - /unlink 52374 (time 1763353263.09 total 199.03 last 279.01) - /unlink 55152 (time 1763353273.09 total 209.04 last 277.66) - /unlink 57922 (time 1763353283.09 total 219.04 last 276.98) - /unlink 60000 (time 1763353290.57 total 226.52 last 277.71) - /unlink 62992 (time 1763353300.58 total 236.52 last 299.11) total: 64768 /unlink in 242.40 seconds: 267.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1637 (time 1763353335.49 total 10.00 last 163.63) - open/close 3276 (time 1763353345.49 total 20.01 last 163.86) - open/close 5009 (time 1763353355.49 total 30.01 last 173.22) - open/close 6318 (time 1763353365.50 total 40.02 last 130.83) - open/close 8051 (time 1763353375.51 total 50.02 last 173.21) - open/close 9679 (time 1763353385.51 total 60.03 last 162.69) - open/close 10000 (time 1763353387.80 total 62.31 last 140.45) - open/close 11413 (time 1763353397.80 total 72.32 last 141.25) - open/close 13095 (time 1763353407.81 total 82.33 last 168.03) - open/close 14828 (time 1763353417.81 total 92.33 last 173.28) - open/close 16154 (time 1763353427.82 total 102.33 last 132.54) - open/close 17523 (time 1763353437.82 total 112.34 last 136.78) - open/close 19118 (time 1763353447.83 total 122.35 last 159.44) - open/close 20000 (time 1763353453.14 total 127.66 last 165.94) - open/close 21548 (time 1763353463.16 total 137.68 last 154.56) - open/close 23226 (time 1763353473.16 total 147.68 last 167.71) - open/close 24892 (time 1763353483.17 total 157.69 last 166.51) - open/close 25989 (time 1763353493.18 total 167.69 last 109.65) - open/close 27344 (time 1763353503.18 total 177.70 last 135.41) - open/close 28933 (time 1763353513.19 total 187.70 last 158.83) - open/close 30000 (time 1763353520.48 total 195.00 last 146.23) - open/close 31360 (time 1763353530.49 total 205.00 last 135.96) - open/close 33040 (time 1763353540.49 total 215.00 last 167.97) - open/close 34768 (time 1763353550.49 total 225.01 last 172.79) - open/close 36477 (time 1763353560.49 total 235.01 last 170.85) - open/close 38139 (time 1763353570.50 total 245.01 last 166.14) - open/close 40000 (time 1763353580.05 total 254.57 last 194.80) - open/close 41801 (time 1763353590.05 total 264.57 last 180.04) - open/close 43257 (time 1763353600.06 total 274.57 last 145.55) - open/close 44742 (time 1763353610.06 total 284.58 last 148.44) - open/close 46137 (time 1763353620.06 total 294.58 last 139.43) - open/close 47676 (time 1763353630.07 total 304.59 last 153.80) - open/close 49127 (time 1763353640.08 total 314.60 last 144.95) - open/close 50000 (time 1763353646.68 total 321.20 last 132.28) - open/close 51612 (time 1763353656.71 total 331.23 last 160.66) - open/close 53512 (time 1763353666.72 total 341.23 last 190.00) - open/close 55076 (time 1763353676.72 total 351.23 last 156.39) - open/close 56992 (time 1763353686.72 total 361.23 last 191.55) - open/close 58508 (time 1763353696.72 total 371.24 last 151.58) - open/close 59631 (time 1763353706.72 total 381.24 last 112.29) - open/close 60000 (time 1763353709.55 total 384.07 last 130.41) - open/close 61436 (time 1763353719.56 total 394.08 last 143.43) - open/close 62765 (time 1763353729.57 total 404.08 last 132.82) - open/close 64247 (time 1763353739.57 total 414.09 last 148.12) total: 64768 open/close in 417.82 seconds: 155.01 ops/second - /unlink 2773 (time 1763353755.32 total 10.00 last 277.20) - /unlink 5346 (time 1763353765.32 total 20.01 last 257.21) - /unlink 7904 (time 1763353775.32 total 30.01 last 255.74) - /unlink 10000 (time 1763353781.96 total 36.65 last 315.64) - /unlink 12634 (time 1763353791.96 total 46.65 last 263.36) - /unlink 15189 (time 1763353801.96 total 56.65 last 255.50) - /unlink 18360 (time 1763353811.97 total 66.65 last 317.01) - /unlink 20000 (time 1763353817.65 total 72.33 last 288.78) - /unlink 23248 (time 1763353827.65 total 82.34 last 324.75) - /unlink 26525 (time 1763353837.65 total 92.34 last 327.64) - /unlink 29595 (time 1763353847.65 total 102.34 last 306.92) - /unlink 33062 (time 1763353857.66 total 112.34 last 346.59) - /unlink 35900 (time 1763353867.66 total 122.34 last 283.80) - /unlink 39090 (time 1763353877.66 total 132.34 last 318.99) - /unlink 40000 (time 1763353880.83 total 135.52 last 286.33) - /unlink 43071 (time 1763353890.83 total 145.52 last 307.10) - /unlink 45932 (time 1763353900.84 total 155.52 last 286.05) - /unlink 49170 (time 1763353910.84 total 165.52 last 323.78) - /unlink 50000 (time 1763353913.98 total 168.67 last 264.11) - /unlink 52607 (time 1763353923.98 total 178.67 last 260.67) - /unlink 55753 (time 1763353933.98 total 188.67 last 314.55) - /unlink 58525 (time 1763353943.98 total 198.67 last 277.14) - /unlink 60000 (time 1763353947.55 total 202.24 last 413.88) - /unlink 64135 (time 1763353957.55 total 212.24 last 413.45) total: 64768 /unlink in 213.41 seconds: 303.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2348 (time 1763353988.90 total 10.00 last 234.80) - open/close 4698 (time 1763353998.90 total 20.00 last 234.99) - open/close 6368 (time 1763354008.91 total 30.00 last 166.94) - open/close 8138 (time 1763354018.91 total 40.01 last 176.98) - open/close 9956 (time 1763354028.91 total 50.01 last 181.76) - open/close 11884 (time 1763354038.91 total 60.01 last 192.74) - open/close 13869 (time 1763354048.91 total 70.01 last 198.48) - open/close 15816 (time 1763354058.92 total 80.01 last 194.64) - open/close 17974 (time 1763354068.92 total 90.02 last 215.78) - open/close 19995 (time 1763354078.92 total 100.02 last 202.06) - open/close 21645 (time 1763354088.93 total 110.03 last 164.87) - open/close 23522 (time 1763354098.93 total 120.03 last 187.64) - open/close 25183 (time 1763354108.94 total 130.03 last 166.01) - open/close 27009 (time 1763354118.94 total 140.04 last 182.51) - open/close 28738 (time 1763354128.94 total 150.04 last 172.88) - open/close 30000 (time 1763354135.67 total 156.77 last 187.43) - open/close 32105 (time 1763354145.68 total 166.78 last 210.46) - open/close 33757 (time 1763354155.68 total 176.78 last 165.15) - open/close 35673 (time 1763354165.68 total 186.78 last 191.55) - open/close 37588 (time 1763354175.68 total 196.78 last 191.49) - open/close 39470 (time 1763354185.69 total 206.78 last 188.16) - open/close 40000 (time 1763354188.51 total 209.61 last 187.47) - open/close 41778 (time 1763354198.51 total 219.61 last 177.78) - open/close 43752 (time 1763354208.52 total 229.61 last 197.36) - open/close 45558 (time 1763354218.52 total 239.62 last 180.52) - open/close 47530 (time 1763354228.52 total 249.62 last 197.19) - open/close 49267 (time 1763354238.53 total 259.62 last 173.61) - open/close 50000 (time 1763354242.72 total 263.82 last 174.71) - open/close 51596 (time 1763354252.72 total 273.82 last 159.56) - open/close 53430 (time 1763354262.72 total 283.82 last 183.38) - open/close 55197 (time 1763354272.72 total 293.82 last 176.70) - open/close 56981 (time 1763354282.73 total 303.83 last 178.29) - open/close 58642 (time 1763354292.74 total 313.83 last 166.03) - open/close 60000 (time 1763354300.68 total 321.78 last 170.90) - open/close 61623 (time 1763354310.68 total 331.78 last 162.30) - open/close 63150 (time 1763354320.69 total 341.78 last 152.65) - open/close 64597 (time 1763354330.69 total 351.79 last 144.69) total: 64768 open/close in 352.89 seconds: 183.54 ops/second - /unlink 2568 (time 1763354343.73 total 10.00 last 256.70) - /unlink 5224 (time 1763354353.73 total 20.00 last 265.58) - /unlink 7767 (time 1763354363.74 total 30.01 last 254.26) - /unlink 10000 (time 1763354371.07 total 37.34 last 304.58) - /unlink 13590 (time 1763354381.07 total 47.34 last 358.91) - /unlink 16822 (time 1763354391.07 total 57.34 last 323.18) - /unlink 20000 (time 1763354399.35 total 65.62 last 384.07) - /unlink 24394 (time 1763354409.35 total 75.62 last 439.39) - /unlink 29210 (time 1763354419.35 total 85.62 last 481.55) - /unlink 30000 (time 1763354421.44 total 87.71 last 377.14) - /unlink 33473 (time 1763354431.44 total 97.71 last 347.27) - /unlink 37276 (time 1763354441.44 total 107.71 last 380.27) - /unlink 40000 (time 1763354447.96 total 114.23 last 418.23) - /unlink 43235 (time 1763354457.96 total 124.23 last 323.46) - /unlink 46611 (time 1763354467.96 total 134.23 last 337.44) - /unlink 50000 (time 1763354477.43 total 143.70 last 357.79) - /unlink 54075 (time 1763354487.44 total 153.71 last 407.43) - /unlink 58139 (time 1763354497.44 total 163.71 last 406.32) - /unlink 60000 (time 1763354501.82 total 168.09 last 424.34) - /unlink 63887 (time 1763354511.82 total 178.09 last 388.69) total: 64768 /unlink in 180.58 seconds: 358.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2005 (time 1763354545.81 total 10.00 last 200.48) - open/close 3758 (time 1763354555.81 total 20.01 last 175.20) - open/close 5413 (time 1763354565.82 total 30.01 last 165.43) - open/close 6944 (time 1763354575.82 total 40.01 last 153.10) - open/close 8522 (time 1763354585.82 total 50.02 last 157.72) - open/close 10000 (time 1763354594.12 total 58.31 last 178.09) - open/close 11239 (time 1763354604.13 total 68.32 last 123.86) - open/close 12508 (time 1763354614.13 total 78.32 last 126.85) - open/close 14401 (time 1763354624.13 total 88.32 last 189.24) - open/close 16202 (time 1763354634.14 total 98.33 last 179.95) - open/close 17909 (time 1763354644.15 total 108.34 last 170.61) - open/close 19425 (time 1763354654.15 total 118.34 last 151.54) - open/close 20000 (time 1763354657.93 total 122.12 last 152.33) - open/close 21941 (time 1763354667.93 total 132.12 last 194.06) - open/close 23776 (time 1763354677.94 total 142.13 last 183.33) - open/close 25410 (time 1763354687.94 total 152.13 last 163.33) - open/close 27576 (time 1763354697.94 total 162.14 last 216.52) - open/close 29503 (time 1763354707.95 total 172.14 last 192.66) - open/close 30000 (time 1763354710.45 total 174.64 last 198.72) - open/close 32180 (time 1763354720.45 total 184.64 last 217.89) - open/close 34077 (time 1763354730.46 total 194.65 last 189.62) - open/close 35800 (time 1763354740.46 total 204.65 last 172.21) - open/close 37787 (time 1763354750.46 total 214.66 last 198.65) - open/close 39656 (time 1763354760.46 total 224.66 last 186.89) - open/close 41499 (time 1763354770.47 total 234.66 last 184.29) - open/close 43100 (time 1763354780.47 total 244.66 last 160.08) - open/close 45015 (time 1763354790.47 total 254.66 last 191.39) - open/close 47495 (time 1763354800.47 total 264.67 last 247.94) - open/close 50000 (time 1763354809.01 total 273.20 last 293.52) - open/close 52658 (time 1763354819.01 total 283.20 last 265.78) - open/close 55181 (time 1763354829.01 total 293.20 last 252.24) - open/close 57987 (time 1763354839.02 total 303.21 last 280.53) - open/close 60000 (time 1763354845.51 total 309.71 last 309.71) - open/close 63493 (time 1763354855.53 total 319.72 last 348.94) total: 64768 open/close in 324.19 seconds: 199.79 ops/second - /unlink 6134 (time 1763354871.67 total 10.00 last 613.32) - /unlink 10000 (time 1763354878.96 total 17.30 last 530.04) - /unlink 15061 (time 1763354888.96 total 27.30 last 506.06) - /unlink 19908 (time 1763354898.96 total 37.30 last 484.62) - /unlink 23616 (time 1763354908.96 total 47.30 last 370.79) - /unlink 27667 (time 1763354918.97 total 57.30 last 405.00) - /unlink 30000 (time 1763354924.50 total 62.84 last 421.44) - /unlink 35160 (time 1763354934.50 total 72.84 last 515.85) - /unlink 40000 (time 1763354943.65 total 81.99 last 528.99) - /unlink 44397 (time 1763354953.65 total 91.99 last 439.66) - /unlink 48846 (time 1763354963.66 total 101.99 last 444.85) - /unlink 50000 (time 1763354966.53 total 104.86 last 401.48) - /unlink 53820 (time 1763354976.53 total 114.87 last 381.97) - /unlink 58006 (time 1763354986.53 total 124.87 last 418.51) - /unlink 60000 (time 1763354991.61 total 129.94 last 393.08) - /unlink 63917 (time 1763355001.61 total 139.94 last 391.65) total: 64768 /unlink in 142.13 seconds: 455.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 PASS 106 (4396s) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: oleg120-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4568 sec ========== 23:51:07 (1763355067) === conf-sanity: start cleanup 23:51:07 (1763355067) === === conf-sanity: finish cleanup 23:51:09 (1763355069) ===