-----============= acceptance-small: conf-sanity ============----- Thu Nov 13 19:33:21 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg220-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg220-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg220-server oleg220-server: oleg220-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg220-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg220-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg220-server: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg220-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:35:48 (1763080548) Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f) oleg220-server: oleg220-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg220-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg220-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 707 (time 1763080653.91 total 10.00 last 70.68) - open/close 1412 (time 1763080663.93 total 20.02 last 70.35) - open/close 2122 (time 1763080673.93 total 30.03 last 70.98) - open/close 2832 (time 1763080683.95 total 40.04 last 70.88) - open/close 3541 (time 1763080693.96 total 50.05 last 70.87) - open/close 4335 (time 1763080703.96 total 60.06 last 79.33) - open/close 5101 (time 1763080713.97 total 70.07 last 76.53) - open/close 5870 (time 1763080723.98 total 80.07 last 76.86) - open/close 6604 (time 1763080733.98 total 90.07 last 73.39) - open/close 7277 (time 1763080743.98 total 100.08 last 67.28) - open/close 7948 (time 1763080754.00 total 110.09 last 67.01) - open/close 8535 (time 1763080764.02 total 120.11 last 58.56) - open/close 9162 (time 1763080774.03 total 130.12 last 62.65) - open/close 9895 (time 1763080784.03 total 140.12 last 73.29) - open/close 10615 (time 1763080794.04 total 150.14 last 71.90) - open/close 11367 (time 1763080804.05 total 160.14 last 75.16) - open/close 12123 (time 1763080814.06 total 170.15 last 75.53) - open/close 12906 (time 1763080824.06 total 180.15 last 78.28) - open/close 13653 (time 1763080834.07 total 190.16 last 74.67) - open/close 14479 (time 1763080844.07 total 200.16 last 82.59) - open/close 15251 (time 1763080854.07 total 210.17 last 77.14) - open/close 16054 (time 1763080864.08 total 220.18 last 80.23) - open/close 16768 (time 1763080874.10 total 230.19 last 71.32) - open/close 17499 (time 1763080884.10 total 240.19 last 73.08) - open/close 18225 (time 1763080894.10 total 250.19 last 72.57) - open/close 18930 (time 1763080904.11 total 260.20 last 70.43) - open/close 19730 (time 1763080914.11 total 270.20 last 79.99) - open/close 20000 (time 1763080917.38 total 273.47 last 82.69) - open/close 20739 (time 1763080927.39 total 283.49 last 73.77) - open/close 21436 (time 1763080937.40 total 293.50 last 69.63) - open/close 22162 (time 1763080947.42 total 303.51 last 72.51) - open/close 22916 (time 1763080957.44 total 313.53 last 75.26) - open/close 23764 (time 1763080967.44 total 323.53 last 84.80) - open/close 24491 (time 1763080977.44 total 333.54 last 72.64) - open/close 25212 (time 1763080987.44 total 343.54 last 72.10) - open/close 25928 (time 1763080997.45 total 353.54 last 71.58) - open/close 26640 (time 1763081007.46 total 363.55 last 71.14) - open/close 27309 (time 1763081017.47 total 373.56 last 66.81) - open/close 28002 (time 1763081027.47 total 383.56 last 69.28) - open/close 28731 (time 1763081037.47 total 393.57 last 72.89) - open/close 29410 (time 1763081047.48 total 403.57 last 67.86) - open/close 30000 (time 1763081056.26 total 412.36 last 67.17) - open/close 30708 (time 1763081066.27 total 422.36 last 70.75) - open/close 31441 (time 1763081076.28 total 432.37 last 73.22) - open/close 32181 (time 1763081086.29 total 442.39 last 73.91) - open/close 32929 (time 1763081096.30 total 452.39 last 74.79) - open/close 33744 (time 1763081106.30 total 462.39 last 81.46) - open/close 34430 (time 1763081116.31 total 472.40 last 68.55) - open/close 35160 (time 1763081126.31 total 482.40 last 72.99) - open/close 35779 (time 1763081136.31 total 492.40 last 61.89) - open/close 36361 (time 1763081146.32 total 502.41 last 58.13) - open/close 36937 (time 1763081156.35 total 512.44 last 57.45) - open/close 37501 (time 1763081166.36 total 522.46 last 56.31) - open/close 38122 (time 1763081176.37 total 532.46 last 62.06) - open/close 38797 (time 1763081186.38 total 542.47 last 67.43) - open/close 39505 (time 1763081196.39 total 552.48 last 70.73) - open/close 40000 (time 1763081203.68 total 559.77 last 67.93) - open/close 40734 (time 1763081213.68 total 569.78 last 73.36) - open/close 41537 (time 1763081223.70 total 579.79 last 80.15) - open/close 42260 (time 1763081233.71 total 589.80 last 72.27) - open/close 42916 (time 1763081243.71 total 599.80 last 65.60) - open/close 43604 (time 1763081253.72 total 609.81 last 68.71) - open/close 44313 (time 1763081263.72 total 619.81 last 70.90) - open/close 44998 (time 1763081273.73 total 629.82 last 68.44) - open/close 45727 (time 1763081283.73 total 639.82 last 72.87) - open/close 46305 (time 1763081293.76 total 649.85 last 57.67) - open/close 47022 (time 1763081303.77 total 659.87 last 71.58) - open/close 47676 (time 1763081313.78 total 669.87 last 65.35) - open/close 48332 (time 1763081323.79 total 679.88 last 65.54) - open/close 49103 (time 1763081333.80 total 689.90 last 77.00) - open/close 49765 (time 1763081343.81 total 699.90 last 66.18) - open/close 50000 (time 1763081347.20 total 703.29 last 69.25) - open/close 50811 (time 1763081357.20 total 713.29 last 81.09) - open/close 51492 (time 1763081367.20 total 723.29 last 68.10) - open/close 52204 (time 1763081377.24 total 733.33 last 70.93) - open/close 52895 (time 1763081387.24 total 743.34 last 69.05) - open/close 53573 (time 1763081397.25 total 753.34 last 67.75) - open/close 54325 (time 1763081407.26 total 763.36 last 75.11) - open/close 54971 (time 1763081417.26 total 773.36 last 64.60) - open/close 55604 (time 1763081427.28 total 783.37 last 63.23) - open/close 56266 (time 1763081437.28 total 793.37 last 66.20) - open/close 57003 (time 1763081447.30 total 803.39 last 73.55) - open/close 57677 (time 1763081457.31 total 813.40 last 67.33) - open/close 58391 (time 1763081467.31 total 823.41 last 71.36) - open/close 59185 (time 1763081477.32 total 833.41 last 79.38) - open/close 60000 (time 1763081486.80 total 842.90 last 85.90) - open/close 60778 (time 1763081496.81 total 852.90 last 77.73) - open/close 61491 (time 1763081506.82 total 862.92 last 71.22) - open/close 62293 (time 1763081516.84 total 872.93 last 80.06) - open/close 62969 (time 1763081526.85 total 882.94 last 67.55) - open/close 63595 (time 1763081536.85 total 892.95 last 62.56) - open/close 64326 (time 1763081546.86 total 902.96 last 73.04) total: 64768 open/close in 908.37 seconds: 71.30 ops/second - /unlink 728 (time 1763081565.06 total 10.01 last 72.73) - /unlink 1397 (time 1763081575.06 total 20.02 last 66.87) - /unlink 2072 (time 1763081585.06 total 30.02 last 67.48) - /unlink 2770 (time 1763081595.06 total 40.02 last 69.80) - /unlink 3437 (time 1763081605.08 total 50.03 last 66.60) - /unlink 4122 (time 1763081615.08 total 60.04 last 68.46) - /unlink 4758 (time 1763081625.10 total 70.05 last 63.51) - /unlink 5440 (time 1763081635.11 total 80.06 last 68.14) - /unlink 6129 (time 1763081645.11 total 90.07 last 68.86) - /unlink 6845 (time 1763081655.11 total 100.07 last 71.58) - /unlink 7538 (time 1763081665.12 total 110.08 last 69.24) - /unlink 8259 (time 1763081675.13 total 120.08 last 72.09) - /unlink 9003 (time 1763081685.13 total 130.09 last 74.34) - /unlink 9694 (time 1763081695.14 total 140.10 last 69.04) - /unlink 10000 (time 1763081699.53 total 144.48 last 69.73) - /unlink 10702 (time 1763081709.54 total 154.50 last 70.11) - /unlink 11357 (time 1763081719.55 total 164.50 last 65.48) - /unlink 12094 (time 1763081729.55 total 174.50 last 73.68) - /unlink 12956 (time 1763081739.55 total 184.51 last 86.15) - /unlink 13824 (time 1763081749.56 total 194.51 last 86.75) - /unlink 14614 (time 1763081759.57 total 204.52 last 78.92) - /unlink 15340 (time 1763081769.58 total 214.54 last 72.50) - /unlink 16071 (time 1763081779.59 total 224.55 last 73.02) - /unlink 16751 (time 1763081789.60 total 234.56 last 67.93) - /unlink 17422 (time 1763081799.62 total 244.57 last 67.02) - /unlink 18107 (time 1763081809.62 total 254.58 last 68.46) - /unlink 18756 (time 1763081819.63 total 264.58 last 64.88) - /unlink 19390 (time 1763081829.64 total 274.60 last 63.30) - /unlink 19997 (time 1763081839.65 total 284.60 last 60.66) - /unlink 20682 (time 1763081849.67 total 294.62 last 68.38) - /unlink 21365 (time 1763081859.67 total 304.62 last 68.29) - /unlink 22066 (time 1763081869.68 total 314.63 last 70.02) - /unlink 22792 (time 1763081879.69 total 324.65 last 72.49) - /unlink 23480 (time 1763081889.70 total 334.65 last 68.77) - /unlink 24207 (time 1763081899.71 total 344.67 last 72.59) - /unlink 24876 (time 1763081909.73 total 354.68 last 66.82) - /unlink 25580 (time 1763081919.73 total 364.68 last 70.39) - /unlink 26247 (time 1763081929.73 total 374.69 last 66.66) - /unlink 27002 (time 1763081939.75 total 384.70 last 75.38) - /unlink 27713 (time 1763081949.76 total 394.71 last 71.03) - /unlink 28419 (time 1763081959.77 total 404.72 last 70.54) - /unlink 29060 (time 1763081969.78 total 414.73 last 64.05) - /unlink 29678 (time 1763081979.78 total 424.73 last 61.79) - /unlink 30000 (time 1763081984.60 total 429.55 last 66.79) - /unlink 30719 (time 1763081994.60 total 439.55 last 71.89) - /unlink 31370 (time 1763082004.61 total 449.57 last 65.02) - /unlink 32014 (time 1763082014.63 total 459.58 last 64.31) - /unlink 32746 (time 1763082024.63 total 469.59 last 73.16) - /unlink 33501 (time 1763082034.63 total 479.59 last 75.48) - /unlink 34164 (time 1763082044.64 total 489.59 last 66.26) - /unlink 34783 (time 1763082054.64 total 499.60 last 61.88) - /unlink 35453 (time 1763082064.65 total 509.61 last 66.94) - /unlink 36142 (time 1763082074.66 total 519.61 last 68.89) - /unlink 36800 (time 1763082084.68 total 529.64 last 65.61) - /unlink 37461 (time 1763082094.70 total 539.65 last 66.02) - /unlink 38133 (time 1763082104.71 total 549.66 last 67.11) - /unlink 38859 (time 1763082114.73 total 559.69 last 72.43) - /unlink 39531 (time 1763082124.73 total 569.69 last 67.19) - /unlink 40000 (time 1763082131.15 total 576.11 last 73.08) - /unlink 40694 (time 1763082141.16 total 586.12 last 69.32) - /unlink 41378 (time 1763082151.17 total 596.12 last 68.38) - /unlink 42095 (time 1763082161.17 total 606.12 last 71.68) - /unlink 42848 (time 1763082171.19 total 616.14 last 75.16) - /unlink 43494 (time 1763082181.19 total 626.14 last 64.59) - /unlink 44165 (time 1763082191.20 total 636.15 last 67.05) - /unlink 44830 (time 1763082201.21 total 646.16 last 66.41) - /unlink 45517 (time 1763082211.21 total 656.17 last 68.67) - /unlink 46228 (time 1763082221.22 total 666.18 last 71.05) - /unlink 46935 (time 1763082231.23 total 676.18 last 70.66) - /unlink 47618 (time 1763082241.23 total 686.18 last 68.28) - /unlink 48325 (time 1763082251.25 total 696.20 last 70.57) - /unlink 49062 (time 1763082261.25 total 706.21 last 73.65) - /unlink 49570 (time 1763082271.26 total 716.21 last 50.76) - /unlink 50000 (time 1763082278.96 total 723.92 last 55.82) - /unlink 50636 (time 1763082288.97 total 733.92 last 63.56) - /unlink 51247 (time 1763082298.99 total 743.95 last 60.96) - /unlink 51909 (time 1763082309.00 total 753.96 last 66.14) - /unlink 52554 (time 1763082319.01 total 763.96 last 64.47) - /unlink 53255 (time 1763082329.02 total 773.97 last 70.05) - /unlink 53957 (time 1763082339.02 total 783.98 last 70.13) - /unlink 54648 (time 1763082349.03 total 793.99 last 69.05) - /unlink 55330 (time 1763082359.04 total 803.99 last 68.17) - /unlink 56129 (time 1763082369.04 total 814.00 last 79.85) - /unlink 56835 (time 1763082379.05 total 824.00 last 70.55) - /unlink 57479 (time 1763082389.06 total 834.02 last 64.31) - /unlink 58154 (time 1763082399.07 total 844.02 last 67.47) - /unlink 58876 (time 1763082409.09 total 854.04 last 72.07) - /unlink 59580 (time 1763082419.10 total 864.06 last 70.27) - /unlink 60000 (time 1763082425.35 total 870.30 last 67.27) - /unlink 60752 (time 1763082435.36 total 880.31 last 75.14) - /unlink 61455 (time 1763082445.36 total 890.31 last 70.30) - /unlink 62090 (time 1763082455.37 total 900.32 last 63.41) - /unlink 62781 (time 1763082465.37 total 910.33 last 69.07) - /unlink 63500 (time 1763082475.37 total 920.33 last 71.90) - /unlink 64108 (time 1763082485.38 total 930.33 last 60.78) total: 64768 /unlink in 940.16 seconds: 68.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 744 (time 1763082533.59 total 10.01 last 74.31) - open/close 1401 (time 1763082543.60 total 20.03 last 65.61) - open/close 2068 (time 1763082553.61 total 30.03 last 66.67) - open/close 2751 (time 1763082563.61 total 40.04 last 68.26) - open/close 3601 (time 1763082573.62 total 50.04 last 84.95) - open/close 4353 (time 1763082583.63 total 60.06 last 75.10) - open/close 5052 (time 1763082593.64 total 70.06 last 69.84) - open/close 5738 (time 1763082603.66 total 80.08 last 68.51) - open/close 6485 (time 1763082613.66 total 90.09 last 74.63) - open/close 7192 (time 1763082623.69 total 100.11 last 70.55) - open/close 7882 (time 1763082633.70 total 110.12 last 68.90) - open/close 8641 (time 1763082643.72 total 120.14 last 75.75) - open/close 9371 (time 1763082653.73 total 130.15 last 72.95) - open/close 10000 (time 1763082660.67 total 137.09 last 90.60) - open/close 11104 (time 1763082670.67 total 147.10 last 110.35) - open/close 11987 (time 1763082680.68 total 157.10 last 88.27) - open/close 12773 (time 1763082690.69 total 167.11 last 78.50) - open/close 13511 (time 1763082700.70 total 177.12 last 73.73) - open/close 14279 (time 1763082710.70 total 187.12 last 76.78) - open/close 15072 (time 1763082720.71 total 197.13 last 79.27) - open/close 15749 (time 1763082730.71 total 207.13 last 67.66) - open/close 16471 (time 1763082740.74 total 217.16 last 71.98) - open/close 17257 (time 1763082750.75 total 227.17 last 78.58) - open/close 18147 (time 1763082760.75 total 237.17 last 88.95) - open/close 18809 (time 1763082770.76 total 247.18 last 66.13) - open/close 19443 (time 1763082780.77 total 257.19 last 63.37) - open/close 20000 (time 1763082787.98 total 264.40 last 77.20) - open/close 20699 (time 1763082798.00 total 274.42 last 69.77) - open/close 21480 (time 1763082808.00 total 284.43 last 78.08) - open/close 22159 (time 1763082818.01 total 294.43 last 67.89) - open/close 22900 (time 1763082828.02 total 304.44 last 74.01) - open/close 23602 (time 1763082838.02 total 314.44 last 70.18) - open/close 24302 (time 1763082848.02 total 324.45 last 69.96) - open/close 24951 (time 1763082858.03 total 334.45 last 64.90) - open/close 25698 (time 1763082868.03 total 344.45 last 74.65) - open/close 26327 (time 1763082878.04 total 354.46 last 62.88) - open/close 27041 (time 1763082888.05 total 364.47 last 71.31) - open/close 27704 (time 1763082898.06 total 374.48 last 66.21) - open/close 28394 (time 1763082908.07 total 384.49 last 68.93) - open/close 29051 (time 1763082918.08 total 394.50 last 65.68) - open/close 29733 (time 1763082928.09 total 404.51 last 68.10) - open/close 30000 (time 1763082932.29 total 408.71 last 63.56) - open/close 30649 (time 1763082942.30 total 418.72 last 64.83) - open/close 31333 (time 1763082952.30 total 428.73 last 68.38) - open/close 32043 (time 1763082962.31 total 438.73 last 70.94) - open/close 32649 (time 1763082972.32 total 448.74 last 60.55) - open/close 33168 (time 1763082982.33 total 458.76 last 51.83) - open/close 33761 (time 1763082992.34 total 468.76 last 59.27) - open/close 34449 (time 1763083002.35 total 478.77 last 68.74) - open/close 35074 (time 1763083012.35 total 488.77 last 62.50) - open/close 35744 (time 1763083022.36 total 498.79 last 66.90) - open/close 36422 (time 1763083032.37 total 508.79 last 67.79) - open/close 37050 (time 1763083042.37 total 518.80 last 62.75) - open/close 37747 (time 1763083052.38 total 528.80 last 69.66) - open/close 38468 (time 1763083062.40 total 538.82 last 71.99) - open/close 39212 (time 1763083072.40 total 548.82 last 74.38) - open/close 39958 (time 1763083082.40 total 558.82 last 74.58) - open/close 40621 (time 1763083092.42 total 568.85 last 66.14) - open/close 41268 (time 1763083102.43 total 578.85 last 64.64) - open/close 41969 (time 1763083112.45 total 588.87 last 70.01) - open/close 42683 (time 1763083122.46 total 598.88 last 71.33) - open/close 43395 (time 1763083132.47 total 608.89 last 71.13) - open/close 44065 (time 1763083142.47 total 618.89 last 66.96) - open/close 44715 (time 1763083152.47 total 628.90 last 64.99) - open/close 45343 (time 1763083162.49 total 638.91 last 62.68) - open/close 45992 (time 1763083172.50 total 648.92 last 64.84) - open/close 46611 (time 1763083182.52 total 658.94 last 61.81) - open/close 47317 (time 1763083192.52 total 668.94 last 70.58) - open/close 47922 (time 1763083202.53 total 678.96 last 60.41) - open/close 48536 (time 1763083212.53 total 688.96 last 61.39) - open/close 49203 (time 1763083222.54 total 698.96 last 66.69) - open/close 49845 (time 1763083232.54 total 708.96 last 64.16) - open/close 50000 (time 1763083234.65 total 711.08 last 73.38) - open/close 50767 (time 1763083244.66 total 721.08 last 76.65) - open/close 51493 (time 1763083254.66 total 731.09 last 72.58) - open/close 52235 (time 1763083264.68 total 741.10 last 74.09) - open/close 53038 (time 1763083274.69 total 751.11 last 80.19) - open/close 53896 (time 1763083284.70 total 761.12 last 85.73) - open/close 54691 (time 1763083294.71 total 771.13 last 79.47) - open/close 55437 (time 1763083304.71 total 781.14 last 74.54) - open/close 56143 (time 1763083314.72 total 791.14 last 70.59) - open/close 56868 (time 1763083324.72 total 801.14 last 72.46) - open/close 57667 (time 1763083334.73 total 811.15 last 79.82) - open/close 58444 (time 1763083344.73 total 821.16 last 77.67) - open/close 59171 (time 1763083354.74 total 831.16 last 72.67) - open/close 59943 (time 1763083364.75 total 841.17 last 77.12) - open/close 60784 (time 1763083374.75 total 851.17 last 84.08) - open/close 61514 (time 1763083384.75 total 861.18 last 72.98) - open/close 62357 (time 1763083394.77 total 871.19 last 84.19) - open/close 63098 (time 1763083404.77 total 881.20 last 74.05) - open/close 63815 (time 1763083414.78 total 891.20 last 71.65) - open/close 64529 (time 1763083424.78 total 901.21 last 71.38) total: 64768 open/close in 904.63 seconds: 71.60 ops/second - /unlink 783 (time 1763083441.40 total 10.01 last 78.23) - /unlink 1536 (time 1763083451.41 total 20.02 last 75.24) - /unlink 2289 (time 1763083461.41 total 30.02 last 75.30) - /unlink 3030 (time 1763083471.41 total 40.02 last 74.08) - /unlink 3799 (time 1763083481.42 total 50.03 last 76.82) - /unlink 4599 (time 1763083491.43 total 60.04 last 79.94) - /unlink 5358 (time 1763083501.43 total 70.04 last 75.89) - /unlink 6245 (time 1763083511.43 total 80.04 last 88.70) - /unlink 7193 (time 1763083521.43 total 90.04 last 94.78) - /unlink 8119 (time 1763083531.43 total 100.05 last 92.55) - /unlink 8935 (time 1763083541.45 total 110.06 last 81.49) - /unlink 9754 (time 1763083551.45 total 120.06 last 81.89) - /unlink 10000 (time 1763083554.52 total 123.13 last 80.12) - /unlink 10797 (time 1763083564.53 total 133.14 last 79.66) - /unlink 11557 (time 1763083574.53 total 143.15 last 75.93) - /unlink 12401 (time 1763083584.54 total 153.15 last 84.35) - /unlink 13296 (time 1763083594.55 total 163.16 last 89.44) - /unlink 14173 (time 1763083604.55 total 173.16 last 87.68) - /unlink 15069 (time 1763083614.56 total 183.17 last 89.55) - /unlink 15875 (time 1763083624.57 total 193.18 last 80.51) - /unlink 16632 (time 1763083634.59 total 203.20 last 75.56) - /unlink 17437 (time 1763083644.59 total 213.20 last 80.49) - /unlink 18393 (time 1763083654.59 total 223.21 last 95.53) - /unlink 19470 (time 1763083664.60 total 233.21 last 107.68) - /unlink 20000 (time 1763083670.26 total 238.87 last 93.53) - /unlink 20961 (time 1763083680.26 total 248.87 last 96.09) - /unlink 21834 (time 1763083690.27 total 258.88 last 87.22) - /unlink 22692 (time 1763083700.28 total 268.89 last 85.77) - /unlink 23484 (time 1763083710.28 total 278.89 last 79.20) - /unlink 24587 (time 1763083720.28 total 288.89 last 110.25) - /unlink 25524 (time 1763083730.29 total 298.90 last 93.60) - /unlink 26366 (time 1763083740.30 total 308.91 last 84.18) - /unlink 27241 (time 1763083750.30 total 318.91 last 87.48) - /unlink 28072 (time 1763083760.31 total 328.92 last 83.01) - /unlink 28896 (time 1763083770.31 total 338.92 last 82.39) - /unlink 29693 (time 1763083780.32 total 348.93 last 79.64) - /unlink 30000 (time 1763083783.61 total 352.22 last 93.32) - /unlink 30910 (time 1763083793.61 total 362.22 last 90.97) - /unlink 31710 (time 1763083803.61 total 372.22 last 79.99) - /unlink 32564 (time 1763083813.61 total 382.22 last 85.38) - /unlink 33652 (time 1763083823.62 total 392.23 last 108.75) - /unlink 34721 (time 1763083833.63 total 402.24 last 106.80) - /unlink 35644 (time 1763083843.63 total 412.24 last 92.29) - /unlink 36585 (time 1763083853.64 total 422.25 last 94.03) - /unlink 37477 (time 1763083863.64 total 432.25 last 89.13) - /unlink 38371 (time 1763083873.64 total 442.25 last 89.39) - /unlink 39221 (time 1763083883.65 total 452.26 last 84.95) - /unlink 40000 (time 1763083891.63 total 460.24 last 97.56) - /unlink 40911 (time 1763083901.64 total 470.25 last 91.07) - /unlink 41834 (time 1763083911.64 total 480.25 last 92.29) - /unlink 42688 (time 1763083921.65 total 490.26 last 85.33) - /unlink 43533 (time 1763083931.65 total 500.26 last 84.47) - /unlink 44394 (time 1763083941.65 total 510.26 last 86.08) - /unlink 45420 (time 1763083951.66 total 520.27 last 102.54) - /unlink 46512 (time 1763083961.66 total 530.27 last 109.15) - /unlink 47411 (time 1763083971.67 total 540.28 last 89.86) - /unlink 48387 (time 1763083981.67 total 550.28 last 97.59) - /unlink 49345 (time 1763083991.68 total 560.29 last 95.72) - /unlink 50000 (time 1763083999.27 total 567.88 last 86.33) - /unlink 50835 (time 1763084009.28 total 577.89 last 83.39) - /unlink 51713 (time 1763084019.28 total 587.89 last 87.76) - /unlink 52516 (time 1763084029.29 total 597.90 last 80.27) - /unlink 53378 (time 1763084039.30 total 607.91 last 86.12) - /unlink 54557 (time 1763084049.30 total 617.91 last 117.88) - /unlink 55680 (time 1763084059.30 total 627.91 last 112.26) - /unlink 56853 (time 1763084069.30 total 637.92 last 117.25) - /unlink 57824 (time 1763084079.31 total 647.92 last 97.08) - /unlink 58885 (time 1763084089.34 total 657.95 last 105.79) - /unlink 59982 (time 1763084099.34 total 667.95 last 109.67) - /unlink 61069 (time 1763084109.34 total 677.95 last 108.65) - /unlink 62253 (time 1763084119.35 total 687.96 last 118.34) - /unlink 63447 (time 1763084129.36 total 697.97 last 119.32) - /unlink 64742 (time 1763084139.36 total 707.97 last 129.43) total: 64768 /unlink in 708.17 seconds: 91.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1374 (time 1763084173.82 total 10.00 last 137.37) - open/close 2714 (time 1763084183.82 total 20.00 last 133.97) - open/close 4320 (time 1763084193.83 total 30.01 last 160.46) - open/close 6119 (time 1763084203.83 total 40.01 last 179.89) - open/close 7482 (time 1763084213.84 total 50.02 last 136.20) - open/close 8673 (time 1763084223.86 total 60.04 last 118.92) - open/close 9960 (time 1763084233.86 total 70.04 last 128.63) - open/close 11229 (time 1763084243.87 total 80.05 last 126.83) - open/close 12725 (time 1763084253.87 total 90.05 last 149.57) - open/close 14380 (time 1763084263.87 total 100.05 last 165.47) - open/close 15998 (time 1763084273.87 total 110.06 last 161.74) - open/close 17860 (time 1763084283.88 total 120.06 last 186.15) - open/close 19497 (time 1763084293.88 total 130.06 last 163.66) - open/close 20000 (time 1763084296.79 total 132.97 last 172.87) - open/close 21555 (time 1763084306.79 total 142.97 last 155.49) - open/close 23291 (time 1763084316.79 total 152.97 last 173.55) - open/close 24717 (time 1763084326.79 total 162.97 last 142.58) - open/close 26360 (time 1763084336.80 total 172.98 last 164.24) - open/close 27946 (time 1763084346.80 total 182.98 last 158.49) - open/close 29331 (time 1763084356.81 total 192.99 last 138.44) - open/close 30000 (time 1763084361.54 total 197.72 last 141.36) - open/close 31131 (time 1763084371.55 total 207.73 last 112.99) - open/close 32365 (time 1763084381.56 total 217.74 last 123.32) - open/close 33759 (time 1763084391.56 total 227.74 last 139.37) - open/close 35212 (time 1763084401.56 total 237.74 last 145.27) - open/close 36631 (time 1763084411.56 total 247.74 last 141.90) - open/close 38024 (time 1763084421.56 total 257.74 last 139.28) - open/close 39564 (time 1763084431.56 total 267.75 last 153.97) - open/close 40000 (time 1763084434.28 total 270.46 last 160.71) - open/close 41685 (time 1763084444.28 total 280.46 last 168.45) - open/close 43453 (time 1763084454.28 total 290.46 last 176.76) - open/close 45356 (time 1763084464.29 total 300.47 last 190.22) - open/close 46974 (time 1763084474.29 total 310.47 last 161.72) - open/close 48727 (time 1763084484.29 total 320.47 last 175.30) - open/close 50000 (time 1763084493.36 total 329.54 last 140.39) - open/close 51486 (time 1763084503.37 total 339.55 last 148.51) - open/close 52845 (time 1763084513.37 total 349.55 last 135.80) - open/close 54157 (time 1763084523.38 total 359.56 last 131.15) - open/close 55473 (time 1763084533.38 total 369.56 last 131.58) - open/close 57223 (time 1763084543.38 total 379.56 last 174.94) - open/close 58839 (time 1763084553.39 total 389.57 last 161.53) - open/close 60000 (time 1763084561.18 total 397.36 last 149.05) - open/close 61456 (time 1763084571.18 total 407.36 last 145.59) - open/close 63087 (time 1763084581.18 total 417.36 last 163.01) - open/close 64619 (time 1763084591.18 total 427.36 last 153.20) total: 64768 open/close in 428.50 seconds: 151.15 ops/second - /unlink 3139 (time 1763084604.30 total 10.00 last 313.77) - /unlink 6503 (time 1763084614.31 total 20.01 last 336.34) - /unlink 9767 (time 1763084624.31 total 30.01 last 326.36) - /unlink 13211 (time 1763084634.31 total 40.01 last 344.40) - /unlink 16415 (time 1763084644.31 total 50.01 last 320.34) - /unlink 19500 (time 1763084654.31 total 60.01 last 308.45) - /unlink 20000 (time 1763084656.31 total 62.01 last 249.83) - /unlink 22936 (time 1763084666.31 total 72.01 last 293.54) - /unlink 25643 (time 1763084676.32 total 82.02 last 270.62) - /unlink 28564 (time 1763084686.32 total 92.02 last 292.07) - /unlink 30000 (time 1763084691.64 total 97.33 last 270.08) - /unlink 32829 (time 1763084701.64 total 107.34 last 282.83) - /unlink 36218 (time 1763084711.64 total 117.34 last 338.88) - /unlink 39695 (time 1763084721.64 total 127.34 last 347.65) - /unlink 42767 (time 1763084731.64 total 137.34 last 307.15) - /unlink 45747 (time 1763084741.64 total 147.34 last 297.93) - /unlink 48962 (time 1763084751.65 total 157.35 last 321.40) - /unlink 50000 (time 1763084755.19 total 160.89 last 292.90) - /unlink 52916 (time 1763084765.19 total 170.89 last 291.50) - /unlink 55871 (time 1763084775.20 total 180.89 last 295.47) - /unlink 58991 (time 1763084785.20 total 190.90 last 311.92) - /unlink 60000 (time 1763084788.87 total 194.57 last 274.65) - /unlink 62472 (time 1763084798.87 total 204.57 last 247.19) total: 64768 /unlink in 212.80 seconds: 304.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2002 (time 1763084836.09 total 10.01 last 200.10) - open/close 3657 (time 1763084846.09 total 20.01 last 165.40) - open/close 5320 (time 1763084856.10 total 30.01 last 166.25) - open/close 6908 (time 1763084866.10 total 40.02 last 158.78) - open/close 8610 (time 1763084876.11 total 50.02 last 170.11) - open/close 10000 (time 1763084884.16 total 58.07 last 172.59) - open/close 11950 (time 1763084894.16 total 68.08 last 194.97) - open/close 14128 (time 1763084904.16 total 78.08 last 217.76) - open/close 16388 (time 1763084914.17 total 88.08 last 225.93) - open/close 18597 (time 1763084924.17 total 98.08 last 220.82) - open/close 20000 (time 1763084930.29 total 104.20 last 229.26) - open/close 22311 (time 1763084940.29 total 114.21 last 231.06) - open/close 25156 (time 1763084950.29 total 124.21 last 284.41) - open/close 27680 (time 1763084960.29 total 134.21 last 252.40) - open/close 30000 (time 1763084967.99 total 141.91 last 301.44) - open/close 32965 (time 1763084977.99 total 151.91 last 296.45) - open/close 35887 (time 1763084987.99 total 161.91 last 292.19) - open/close 39053 (time 1763084998.00 total 171.91 last 316.49) - open/close 40000 (time 1763085001.44 total 175.36 last 274.71) - open/close 43914 (time 1763085011.44 total 185.36 last 391.39) - open/close 47436 (time 1763085021.44 total 195.36 last 352.15) - open/close 50000 (time 1763085029.86 total 203.78 last 304.63) - open/close 52563 (time 1763085039.86 total 213.78 last 256.29) - open/close 55470 (time 1763085049.86 total 223.78 last 290.62) - open/close 58363 (time 1763085059.87 total 233.78 last 289.25) - open/close 60000 (time 1763085064.88 total 238.79 last 326.70) - open/close 63717 (time 1763085074.88 total 248.79 last 371.63) total: 64768 open/close in 252.43 seconds: 256.58 ops/second - /unlink 5118 (time 1763085090.14 total 10.00 last 511.74) - /unlink 10000 (time 1763085098.26 total 18.12 last 601.52) - /unlink 16185 (time 1763085108.26 total 28.12 last 618.49) - /unlink 20000 (time 1763085114.23 total 34.09 last 639.06) - /unlink 24801 (time 1763085124.23 total 44.09 last 480.05) - /unlink 30000 (time 1763085133.46 total 53.32 last 563.23) - /unlink 36541 (time 1763085143.46 total 63.32 last 654.04) - /unlink 40000 (time 1763085148.50 total 68.36 last 686.14) - /unlink 46820 (time 1763085158.50 total 78.36 last 681.99) - /unlink 50000 (time 1763085164.96 total 84.82 last 492.41) - /unlink 55061 (time 1763085174.96 total 94.82 last 506.05) - /unlink 60000 (time 1763085183.39 total 103.25 last 585.79) total: 64768 /unlink in 111.75 seconds: 579.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3539 (time 1763085221.61 total 10.00 last 353.87) - open/close 7655 (time 1763085231.62 total 20.00 last 411.50) - open/close 10000 (time 1763085238.32 total 26.71 last 349.77) - open/close 13911 (time 1763085248.32 total 36.71 last 391.08) - open/close 18352 (time 1763085258.32 total 46.71 last 444.03) - open/close 20000 (time 1763085262.83 total 51.22 last 365.36) - open/close 24045 (time 1763085272.84 total 61.22 last 404.39) - open/close 28256 (time 1763085282.84 total 71.22 last 421.03) - open/close 30000 (time 1763085287.32 total 75.71 last 388.98) - open/close 34751 (time 1763085297.32 total 85.71 last 475.03) - open/close 39447 (time 1763085307.32 total 95.71 last 469.54) - open/close 44120 (time 1763085317.32 total 105.71 last 467.25) - open/close 48760 (time 1763085327.32 total 115.71 last 463.99) - open/close 50000 (time 1763085329.89 total 118.28 last 483.01) - open/close 54853 (time 1763085339.89 total 128.28 last 485.23) - open/close 59345 (time 1763085349.89 total 138.28 last 449.14) - open/close 64049 (time 1763085359.89 total 148.28 last 470.40) total: 64768 open/close in 149.84 seconds: 432.25 ops/second - /unlink 7850 (time 1763085372.73 total 10.00 last 784.96) - /unlink 10000 (time 1763085375.10 total 12.37 last 906.54) - /unlink 17506 (time 1763085385.10 total 22.37 last 750.59) - /unlink 20000 (time 1763085389.01 total 26.28 last 637.90) - /unlink 27541 (time 1763085399.01 total 36.28 last 754.03) - /unlink 30000 (time 1763085402.35 total 39.63 last 735.72) - /unlink 37634 (time 1763085412.35 total 49.63 last 763.36) - /unlink 40000 (time 1763085415.06 total 52.33 last 874.56) - /unlink 47434 (time 1763085425.06 total 62.33 last 743.40) - /unlink 50000 (time 1763085428.46 total 65.73 last 753.99) - /unlink 57320 (time 1763085438.46 total 75.74 last 731.85) - /unlink 60000 (time 1763085441.83 total 79.11 last 795.38) total: 64768 /unlink in 84.81 seconds: 763.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg220-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing unload_modules_local oleg220-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 PASS 106 (4933s) Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 oleg220-server: oleg220-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions oleg220-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg220-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 5102 sec ========== 20:58:25 (1763085505) === conf-sanity: start cleanup 20:58:26 (1763085506) === === conf-sanity: finish cleanup 20:58:27 (1763085507) ===