-----============= acceptance-small: conf-sanity ============----- Tue Dec 2 22:44:11 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg156-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg156-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg156-server oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg156-server: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg156-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:46:46 (1764733606) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 582 (time 1764733720.22 total 10.03 last 58.05) - open/close 1220 (time 1764733730.22 total 20.04 last 63.74) - open/close 1834 (time 1764733740.23 total 30.04 last 61.39) - open/close 2494 (time 1764733750.24 total 40.06 last 65.88) - open/close 3169 (time 1764733760.25 total 50.06 last 67.48) - open/close 3872 (time 1764733770.25 total 60.06 last 70.26) - open/close 4590 (time 1764733780.26 total 70.08 last 71.72) - open/close 5348 (time 1764733790.27 total 80.08 last 75.77) - open/close 6052 (time 1764733800.27 total 90.08 last 70.40) - open/close 6711 (time 1764733810.27 total 100.08 last 65.88) - open/close 7395 (time 1764733820.28 total 110.09 last 68.38) - open/close 8067 (time 1764733830.28 total 120.09 last 67.19) - open/close 8788 (time 1764733840.28 total 130.10 last 72.05) - open/close 9449 (time 1764733850.29 total 140.10 last 66.09) - open/close 10000 (time 1764733858.70 total 148.51 last 65.48) - open/close 10725 (time 1764733868.71 total 158.52 last 72.45) - open/close 11379 (time 1764733878.72 total 168.53 last 65.34) - open/close 12056 (time 1764733888.73 total 178.54 last 67.65) - open/close 12724 (time 1764733898.73 total 188.54 last 66.77) - open/close 13425 (time 1764733908.74 total 198.55 last 70.06) - open/close 14137 (time 1764733918.74 total 208.55 last 71.20) - open/close 14852 (time 1764733928.78 total 218.59 last 71.18) - open/close 15568 (time 1764733938.78 total 228.59 last 71.58) - open/close 16276 (time 1764733948.78 total 238.59 last 70.80) - open/close 16964 (time 1764733958.80 total 248.62 last 68.66) - open/close 17692 (time 1764733968.81 total 258.62 last 72.77) - open/close 18535 (time 1764733978.82 total 268.63 last 84.24) - open/close 19307 (time 1764733988.83 total 278.64 last 77.11) - open/close 20000 (time 1764733996.85 total 286.66 last 86.42) - open/close 20774 (time 1764734006.85 total 296.66 last 77.39) - open/close 21505 (time 1764734016.86 total 306.67 last 73.01) - open/close 22252 (time 1764734026.87 total 316.68 last 74.60) - open/close 23077 (time 1764734036.88 total 326.69 last 82.43) - open/close 23867 (time 1764734046.90 total 336.71 last 78.86) - open/close 24702 (time 1764734056.91 total 346.72 last 83.43) - open/close 25560 (time 1764734066.92 total 356.73 last 85.70) - open/close 26399 (time 1764734076.93 total 366.74 last 83.84) - open/close 27212 (time 1764734086.94 total 376.75 last 81.17) - open/close 28021 (time 1764734096.95 total 386.76 last 80.87) - open/close 28788 (time 1764734106.96 total 396.77 last 76.56) - open/close 29605 (time 1764734116.97 total 406.78 last 81.69) - open/close 30000 (time 1764734121.06 total 410.87 last 96.40) - open/close 30742 (time 1764734131.07 total 420.88 last 74.18) - open/close 31508 (time 1764734141.07 total 430.88 last 76.55) - open/close 32258 (time 1764734151.08 total 440.89 last 74.97) - open/close 32783 (time 1764734161.08 total 450.89 last 52.47) - open/close 33374 (time 1764734171.08 total 460.89 last 59.09) - open/close 33936 (time 1764734181.09 total 470.90 last 56.18) - open/close 34572 (time 1764734191.10 total 480.91 last 63.52) - open/close 35289 (time 1764734201.10 total 490.91 last 71.67) - open/close 35975 (time 1764734211.11 total 500.92 last 68.57) - open/close 36645 (time 1764734221.12 total 510.93 last 66.90) - open/close 37322 (time 1764734231.13 total 520.94 last 67.66) - open/close 38072 (time 1764734241.14 total 530.95 last 74.92) - open/close 38777 (time 1764734251.14 total 540.95 last 70.48) - open/close 39503 (time 1764734261.14 total 550.96 last 72.58) - open/close 40000 (time 1764734268.65 total 558.46 last 66.23) - open/close 40688 (time 1764734278.67 total 568.48 last 68.66) - open/close 41371 (time 1764734288.71 total 578.52 last 68.02) - open/close 42078 (time 1764734298.72 total 588.53 last 70.63) - open/close 42696 (time 1764734308.73 total 598.55 last 61.70) - open/close 43371 (time 1764734318.74 total 608.55 last 67.47) - open/close 44071 (time 1764734328.74 total 618.55 last 69.98) - open/close 44772 (time 1764734338.74 total 628.55 last 70.09) - open/close 45494 (time 1764734348.75 total 638.57 last 72.12) - open/close 46188 (time 1764734358.76 total 648.57 last 69.38) - open/close 46855 (time 1764734368.76 total 658.57 last 66.68) - open/close 47534 (time 1764734378.78 total 668.59 last 67.78) - open/close 48248 (time 1764734388.78 total 678.59 last 71.39) - open/close 48955 (time 1764734398.79 total 688.60 last 70.63) - open/close 49636 (time 1764734408.79 total 698.60 last 68.07) - open/close 50000 (time 1764734413.95 total 703.76 last 70.55) - open/close 50689 (time 1764734423.96 total 713.77 last 68.87) - open/close 51433 (time 1764734433.96 total 723.78 last 74.35) - open/close 52110 (time 1764734443.97 total 733.78 last 67.64) - open/close 52779 (time 1764734453.97 total 743.79 last 66.89) - open/close 53382 (time 1764734463.99 total 753.80 last 60.20) - open/close 54044 (time 1764734474.00 total 763.82 last 66.10) - open/close 54742 (time 1764734484.01 total 773.83 last 69.73) - open/close 55382 (time 1764734494.02 total 783.83 last 63.96) - open/close 55989 (time 1764734504.03 total 793.84 last 60.62) - open/close 56600 (time 1764734514.05 total 803.86 last 61.01) - open/close 57298 (time 1764734524.07 total 813.88 last 69.67) - open/close 58010 (time 1764734534.08 total 823.89 last 71.09) - open/close 58733 (time 1764734544.09 total 833.90 last 72.28) - open/close 59493 (time 1764734554.09 total 843.90 last 75.95) - open/close 60000 (time 1764734560.99 total 850.80 last 73.54) - open/close 60681 (time 1764734571.00 total 860.81 last 68.04) - open/close 61334 (time 1764734581.01 total 870.82 last 65.24) - open/close 62093 (time 1764734591.01 total 880.82 last 75.90) - open/close 62806 (time 1764734601.01 total 890.83 last 71.24) - open/close 63594 (time 1764734611.02 total 900.83 last 78.78) - open/close 64316 (time 1764734621.02 total 910.83 last 72.16) total: 64768 open/close in 917.12 seconds: 70.62 ops/second - /unlink 797 (time 1764734640.46 total 10.01 last 79.65) - /unlink 1626 (time 1764734650.47 total 20.02 last 82.78) - /unlink 2386 (time 1764734660.48 total 30.03 last 75.96) - /unlink 3078 (time 1764734670.48 total 40.03 last 69.15) - /unlink 3832 (time 1764734680.49 total 50.03 last 75.39) - /unlink 4537 (time 1764734690.50 total 60.05 last 70.41) - /unlink 5228 (time 1764734700.51 total 70.05 last 69.05) - /unlink 5938 (time 1764734710.52 total 80.07 last 70.90) - /unlink 6743 (time 1764734720.53 total 90.08 last 80.43) - /unlink 7488 (time 1764734730.54 total 100.09 last 74.43) - /unlink 8227 (time 1764734740.55 total 110.10 last 73.80) - /unlink 8936 (time 1764734750.55 total 120.10 last 70.88) - /unlink 9748 (time 1764734760.57 total 130.12 last 81.05) - /unlink 10000 (time 1764734764.10 total 133.64 last 71.53) - /unlink 10701 (time 1764734774.11 total 143.65 last 70.03) - /unlink 11426 (time 1764734784.12 total 153.67 last 72.37) - /unlink 12146 (time 1764734794.13 total 163.67 last 71.98) - /unlink 12915 (time 1764734804.13 total 173.68 last 76.83) - /unlink 13742 (time 1764734814.14 total 183.69 last 82.62) - /unlink 14588 (time 1764734824.15 total 193.70 last 84.58) - /unlink 15639 (time 1764734834.16 total 203.71 last 105.00) - /unlink 16622 (time 1764734844.16 total 213.71 last 98.27) - /unlink 17573 (time 1764734854.16 total 223.71 last 95.09) - /unlink 18476 (time 1764734864.17 total 233.72 last 90.23) - /unlink 19501 (time 1764734874.17 total 243.72 last 102.45) - /unlink 20000 (time 1764734879.52 total 249.07 last 93.33) - /unlink 21038 (time 1764734889.53 total 259.08 last 103.72) - /unlink 21974 (time 1764734899.54 total 269.08 last 93.53) - /unlink 22852 (time 1764734909.54 total 279.09 last 87.73) - /unlink 23768 (time 1764734919.55 total 289.10 last 91.56) - /unlink 24690 (time 1764734929.56 total 299.11 last 92.10) - /unlink 25682 (time 1764734939.56 total 309.11 last 99.15) - /unlink 26591 (time 1764734949.56 total 319.11 last 90.90) - /unlink 27539 (time 1764734959.58 total 329.12 last 94.69) - /unlink 28408 (time 1764734969.58 total 339.13 last 86.83) - /unlink 29236 (time 1764734979.59 total 349.14 last 82.76) - /unlink 30000 (time 1764734988.67 total 358.21 last 84.16) - /unlink 30902 (time 1764734998.67 total 368.21 last 90.20) - /unlink 31734 (time 1764735008.68 total 378.22 last 83.12) - /unlink 32691 (time 1764735018.68 total 388.23 last 95.68) - /unlink 33497 (time 1764735028.68 total 398.23 last 80.58) - /unlink 34309 (time 1764735038.70 total 408.24 last 81.07) - /unlink 35298 (time 1764735048.70 total 418.24 last 98.90) - /unlink 36327 (time 1764735058.70 total 428.24 last 102.90) - /unlink 37210 (time 1764735068.70 total 438.25 last 88.23) - /unlink 38170 (time 1764735078.71 total 448.26 last 95.93) - /unlink 39223 (time 1764735088.72 total 458.26 last 105.26) - /unlink 40000 (time 1764735095.72 total 465.27 last 110.97) - /unlink 41215 (time 1764735105.72 total 475.27 last 121.50) - /unlink 42344 (time 1764735115.72 total 485.27 last 112.83) - /unlink 43528 (time 1764735125.73 total 495.28 last 118.35) - /unlink 44569 (time 1764735135.73 total 505.28 last 104.03) - /unlink 45777 (time 1764735145.74 total 515.29 last 120.73) - /unlink 46851 (time 1764735155.74 total 525.29 last 107.36) - /unlink 48138 (time 1764735165.75 total 535.29 last 128.67) - /unlink 49353 (time 1764735175.75 total 545.29 last 121.50) - /unlink 50000 (time 1764735181.86 total 551.41 last 105.81) - /unlink 51027 (time 1764735191.86 total 561.41 last 102.69) - /unlink 52329 (time 1764735201.87 total 571.41 last 130.15) - /unlink 53282 (time 1764735211.87 total 581.42 last 95.23) - /unlink 54300 (time 1764735221.88 total 591.43 last 101.73) - /unlink 55816 (time 1764735231.88 total 601.43 last 151.57) - /unlink 57105 (time 1764735241.89 total 611.44 last 128.84) - /unlink 58410 (time 1764735251.89 total 621.44 last 130.48) - /unlink 59729 (time 1764735261.89 total 631.44 last 131.89) - /unlink 60000 (time 1764735264.09 total 633.64 last 123.17) - /unlink 61223 (time 1764735274.09 total 643.64 last 122.29) - /unlink 62491 (time 1764735284.09 total 653.64 last 126.74) - /unlink 64189 (time 1764735294.10 total 663.64 last 169.78) total: 64768 /unlink in 667.86 seconds: 96.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1196 (time 1764735329.10 total 10.00 last 119.55) - open/close 1891 (time 1764735339.10 total 20.01 last 69.49) - open/close 2523 (time 1764735349.10 total 30.01 last 63.20) - open/close 3064 (time 1764735359.11 total 40.02 last 54.04) - open/close 3713 (time 1764735369.13 total 50.04 last 64.79) - open/close 4549 (time 1764735379.14 total 60.04 last 83.54) - open/close 5417 (time 1764735389.14 total 70.05 last 86.78) - open/close 6175 (time 1764735399.14 total 80.05 last 75.80) - open/close 6945 (time 1764735409.15 total 90.06 last 76.92) - open/close 7655 (time 1764735419.15 total 100.06 last 70.99) - open/close 8386 (time 1764735429.16 total 110.07 last 73.01) - open/close 9136 (time 1764735439.17 total 120.07 last 74.97) - open/close 9867 (time 1764735449.18 total 130.08 last 73.03) - open/close 10639 (time 1764735459.18 total 140.09 last 77.18) - open/close 11420 (time 1764735469.19 total 150.10 last 78.01) - open/close 12188 (time 1764735479.20 total 160.11 last 76.70) - open/close 12943 (time 1764735489.22 total 170.13 last 75.38) - open/close 13691 (time 1764735499.23 total 180.14 last 74.73) - open/close 14450 (time 1764735509.23 total 190.14 last 75.89) - open/close 15230 (time 1764735519.23 total 200.14 last 77.97) - open/close 16074 (time 1764735529.25 total 210.15 last 84.29) - open/close 16909 (time 1764735539.25 total 220.15 last 83.50) - open/close 17709 (time 1764735549.25 total 230.16 last 79.98) - open/close 18584 (time 1764735559.25 total 240.16 last 87.47) - open/close 19367 (time 1764735569.26 total 250.17 last 78.26) - open/close 20000 (time 1764735577.01 total 257.92 last 81.65) - open/close 20794 (time 1764735587.02 total 267.92 last 79.35) - open/close 21671 (time 1764735597.02 total 277.93 last 87.69) - open/close 22707 (time 1764735607.02 total 287.93 last 103.59) - open/close 23843 (time 1764735617.02 total 297.93 last 113.58) - open/close 25084 (time 1764735627.03 total 307.93 last 124.06) - open/close 26252 (time 1764735637.03 total 317.94 last 116.74) - open/close 27402 (time 1764735647.03 total 327.94 last 114.97) - open/close 28949 (time 1764735657.04 total 337.94 last 154.65) - open/close 30000 (time 1764735664.71 total 345.62 last 136.91) - open/close 31380 (time 1764735674.71 total 355.62 last 137.98) - open/close 32826 (time 1764735684.72 total 365.63 last 144.52) - open/close 34358 (time 1764735694.72 total 375.63 last 153.14) - open/close 35758 (time 1764735704.73 total 385.64 last 139.93) - open/close 37198 (time 1764735714.73 total 395.64 last 143.95) - open/close 38596 (time 1764735724.73 total 405.64 last 139.80) - open/close 40000 (time 1764735734.02 total 414.92 last 151.23) - open/close 41608 (time 1764735744.02 total 424.93 last 160.74) - open/close 42981 (time 1764735754.02 total 434.93 last 137.28) - open/close 44346 (time 1764735764.02 total 444.93 last 136.49) - open/close 45458 (time 1764735774.03 total 454.94 last 111.14) - open/close 46545 (time 1764735784.04 total 464.94 last 108.61) - open/close 47662 (time 1764735794.04 total 474.95 last 111.63) - open/close 48848 (time 1764735804.05 total 484.96 last 118.50) - open/close 50000 (time 1764735812.94 total 493.85 last 129.61) - open/close 51144 (time 1764735822.95 total 503.85 last 114.32) - open/close 52178 (time 1764735832.95 total 513.86 last 103.37) - open/close 53370 (time 1764735842.95 total 523.86 last 119.13) - open/close 54407 (time 1764735852.96 total 533.86 last 103.68) - open/close 55379 (time 1764735862.96 total 543.87 last 97.13) - open/close 56623 (time 1764735872.97 total 553.87 last 124.35) - open/close 57600 (time 1764735882.99 total 563.89 last 97.50) - open/close 58488 (time 1764735892.99 total 573.90 last 88.78) - open/close 59587 (time 1764735902.99 total 583.90 last 109.88) - open/close 60000 (time 1764735906.91 total 587.82 last 105.29) - open/close 61160 (time 1764735916.92 total 597.83 last 115.93) - open/close 62374 (time 1764735926.93 total 607.83 last 121.33) - open/close 63601 (time 1764735936.93 total 617.84 last 122.62) total: 64768 open/close in 627.15 seconds: 103.27 ops/second - /unlink 1312 (time 1764735958.45 total 10.01 last 131.13) - /unlink 2534 (time 1764735968.46 total 20.01 last 122.13) - /unlink 3685 (time 1764735978.46 total 30.02 last 115.04) - /unlink 4970 (time 1764735988.46 total 40.02 last 128.48) - /unlink 6056 (time 1764735998.47 total 50.02 last 108.55) - /unlink 7928 (time 1764736008.47 total 60.02 last 187.15) - /unlink 10000 (time 1764736016.63 total 68.19 last 253.83) - /unlink 12351 (time 1764736026.63 total 78.19 last 235.05) - /unlink 14907 (time 1764736036.64 total 88.19 last 255.58) - /unlink 17011 (time 1764736046.64 total 98.19 last 210.36) - /unlink 19181 (time 1764736056.64 total 108.20 last 216.93) - /unlink 20000 (time 1764736060.85 total 112.40 last 194.59) - /unlink 22287 (time 1764736070.85 total 122.41 last 228.64) - /unlink 24418 (time 1764736080.85 total 132.41 last 213.08) - /unlink 26467 (time 1764736090.85 total 142.41 last 204.89) - /unlink 28868 (time 1764736100.85 total 152.41 last 240.08) - /unlink 30000 (time 1764736106.43 total 157.99 last 202.97) - /unlink 32146 (time 1764736116.44 total 167.99 last 214.50) - /unlink 34150 (time 1764736126.44 total 177.99 last 200.34) - /unlink 36350 (time 1764736136.44 total 188.00 last 219.98) - /unlink 38820 (time 1764736146.45 total 198.00 last 246.85) - /unlink 40000 (time 1764736152.10 total 203.65 last 208.84) - /unlink 41435 (time 1764736162.10 total 213.65 last 143.50) - /unlink 42780 (time 1764736172.10 total 223.65 last 134.47) - /unlink 43963 (time 1764736182.11 total 233.66 last 118.22) - /unlink 45239 (time 1764736192.12 total 243.67 last 127.46) - /unlink 46256 (time 1764736202.13 total 253.68 last 101.58) - /unlink 47262 (time 1764736212.14 total 263.69 last 100.53) - /unlink 48563 (time 1764736222.15 total 273.71 last 129.88) - /unlink 49915 (time 1764736232.15 total 283.71 last 135.20) - /unlink 51266 (time 1764736242.15 total 293.71 last 135.09) - /unlink 52608 (time 1764736252.16 total 303.71 last 134.15) - /unlink 54059 (time 1764736262.16 total 313.72 last 144.99) - /unlink 55538 (time 1764736272.16 total 323.72 last 147.89) - /unlink 57074 (time 1764736282.17 total 333.72 last 153.58) - /unlink 58541 (time 1764736292.17 total 343.73 last 146.64) - /unlink 59893 (time 1764736302.18 total 353.73 last 135.13) - /unlink 61375 (time 1764736312.18 total 363.74 last 148.09) - /unlink 62972 (time 1764736322.19 total 373.75 last 159.55) - /unlink 64400 (time 1764736332.20 total 383.76 last 142.67) total: 64768 /unlink in 386.31 seconds: 167.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 781 (time 1764736369.92 total 10.00 last 78.08) - open/close 1604 (time 1764736379.92 total 20.00 last 82.29) - open/close 2391 (time 1764736389.92 total 30.01 last 78.68) - open/close 3161 (time 1764736399.93 total 40.01 last 76.95) - open/close 3966 (time 1764736409.93 total 50.02 last 80.46) - open/close 4990 (time 1764736419.93 total 60.02 last 102.39) - open/close 5796 (time 1764736429.94 total 70.03 last 80.53) - open/close 6577 (time 1764736439.95 total 80.04 last 78.01) - open/close 7397 (time 1764736449.96 total 90.04 last 81.97) - open/close 8154 (time 1764736459.96 total 100.04 last 75.69) - open/close 8992 (time 1764736469.98 total 110.06 last 83.67) - open/close 9889 (time 1764736479.98 total 120.07 last 89.64) - open/close 10711 (time 1764736489.99 total 130.07 last 82.14) - open/close 11545 (time 1764736500.01 total 140.09 last 83.23) - open/close 12443 (time 1764736510.01 total 150.10 last 89.76) - open/close 13356 (time 1764736520.02 total 160.10 last 91.24) - open/close 14306 (time 1764736530.03 total 170.11 last 94.93) - open/close 15215 (time 1764736540.03 total 180.11 last 90.90) - open/close 16080 (time 1764736550.04 total 190.12 last 86.41) - open/close 16991 (time 1764736560.04 total 200.13 last 91.07) - open/close 17930 (time 1764736570.05 total 210.13 last 93.85) - open/close 18782 (time 1764736580.05 total 220.13 last 85.17) - open/close 19607 (time 1764736590.06 total 230.14 last 82.43) - open/close 20000 (time 1764736593.83 total 233.91 last 104.33) - open/close 20971 (time 1764736603.83 total 243.91 last 97.05) - open/close 21885 (time 1764736613.84 total 253.92 last 91.31) - open/close 22739 (time 1764736623.85 total 263.93 last 85.36) - open/close 23703 (time 1764736633.86 total 273.94 last 96.31) - open/close 24729 (time 1764736643.86 total 283.94 last 102.58) - open/close 25781 (time 1764736653.87 total 293.95 last 105.08) - open/close 26740 (time 1764736663.88 total 303.96 last 95.84) - open/close 27625 (time 1764736673.88 total 313.97 last 88.42) - open/close 28472 (time 1764736683.89 total 323.98 last 84.63) - open/close 29336 (time 1764736693.90 total 333.98 last 86.37) - open/close 30000 (time 1764736701.24 total 341.33 last 90.37) - open/close 30936 (time 1764736711.24 total 351.33 last 93.59) - open/close 31623 (time 1764736721.26 total 361.34 last 68.62) - open/close 32204 (time 1764736731.26 total 371.34 last 58.08) - open/close 32754 (time 1764736741.28 total 381.37 last 54.89) - open/close 33238 (time 1764736751.28 total 391.37 last 48.39) - open/close 33950 (time 1764736761.29 total 401.38 last 71.14) - open/close 34680 (time 1764736771.31 total 411.39 last 72.90) - open/close 35441 (time 1764736781.31 total 421.40 last 76.05) - open/close 36183 (time 1764736791.32 total 431.40 last 74.13) - open/close 36997 (time 1764736801.33 total 441.42 last 81.29) - open/close 37779 (time 1764736811.35 total 451.43 last 78.11) - open/close 38619 (time 1764736821.35 total 461.43 last 84.00) - open/close 39402 (time 1764736831.35 total 471.43 last 78.27) - open/close 40000 (time 1764736838.46 total 478.54 last 84.13) - open/close 40822 (time 1764736848.48 total 488.56 last 82.03) - open/close 41642 (time 1764736858.48 total 498.56 last 82.00) - open/close 42526 (time 1764736868.48 total 508.56 last 88.40) - open/close 43406 (time 1764736878.48 total 518.56 last 87.99) - open/close 44267 (time 1764736888.49 total 528.57 last 86.05) - open/close 45111 (time 1764736898.49 total 538.57 last 84.37) - open/close 45877 (time 1764736908.50 total 548.58 last 76.53) - open/close 46692 (time 1764736918.50 total 558.58 last 81.49) - open/close 47495 (time 1764736928.51 total 568.60 last 80.20) - open/close 48279 (time 1764736938.53 total 578.61 last 78.30) - open/close 49069 (time 1764736948.55 total 588.64 last 78.78) - open/close 49920 (time 1764736958.56 total 598.64 last 85.08) - open/close 50788 (time 1764736968.56 total 608.65 last 86.74) - open/close 51750 (time 1764736978.57 total 618.65 last 96.15) - open/close 52656 (time 1764736988.58 total 628.66 last 90.53) - open/close 53578 (time 1764736998.58 total 638.66 last 92.15) - open/close 54439 (time 1764737008.58 total 648.67 last 86.09) - open/close 55416 (time 1764737018.59 total 658.67 last 97.66) - open/close 56269 (time 1764737028.60 total 668.68 last 85.22) - open/close 57048 (time 1764737038.60 total 678.69 last 77.85) - open/close 57903 (time 1764737048.60 total 688.69 last 85.50) - open/close 58780 (time 1764737058.62 total 698.70 last 87.56) - open/close 59774 (time 1764737068.62 total 708.70 last 99.38) - open/close 60922 (time 1764737078.63 total 718.71 last 114.71) - open/close 61946 (time 1764737088.63 total 728.72 last 102.37) - open/close 62751 (time 1764737098.64 total 738.72 last 80.47) - open/close 63650 (time 1764737108.64 total 748.72 last 89.88) - open/close 64671 (time 1764737118.64 total 758.73 last 102.06) total: 64768 open/close in 759.49 seconds: 85.28 ops/second - /unlink 1166 (time 1764737131.74 total 10.01 last 116.52) - /unlink 2268 (time 1764737141.74 total 20.01 last 110.17) - /unlink 3371 (time 1764737151.76 total 30.03 last 110.08) - /unlink 4614 (time 1764737161.76 total 40.03 last 124.28) - /unlink 6091 (time 1764737171.77 total 50.04 last 147.60) - /unlink 7274 (time 1764737181.77 total 60.04 last 118.28) - /unlink 8494 (time 1764737191.78 total 70.05 last 121.91) - /unlink 9701 (time 1764737201.78 total 80.05 last 120.67) - /unlink 10000 (time 1764737203.93 total 82.20 last 138.69) - /unlink 11318 (time 1764737213.94 total 92.21 last 131.74) - /unlink 12842 (time 1764737223.94 total 102.21 last 152.34) - /unlink 14478 (time 1764737233.95 total 112.22 last 163.43) - /unlink 15734 (time 1764737243.96 total 122.23 last 125.56) - /unlink 17596 (time 1764737253.96 total 132.23 last 186.18) - /unlink 19463 (time 1764737263.96 total 142.23 last 186.60) - /unlink 20000 (time 1764737268.19 total 146.46 last 126.91) - /unlink 21502 (time 1764737278.20 total 156.48 last 150.04) - /unlink 23039 (time 1764737288.21 total 166.48 last 153.64) - /unlink 25012 (time 1764737298.21 total 176.48 last 197.23) - /unlink 27317 (time 1764737308.21 total 186.48 last 230.46) - /unlink 29414 (time 1764737318.22 total 196.49 last 209.60) - /unlink 30000 (time 1764737321.80 total 200.07 last 163.48) - /unlink 31566 (time 1764737331.80 total 210.08 last 156.58) - /unlink 33872 (time 1764737341.81 total 220.08 last 230.54) - /unlink 36431 (time 1764737351.81 total 230.08 last 255.90) - /unlink 39932 (time 1764737361.82 total 240.09 last 349.81) - /unlink 43159 (time 1764737371.82 total 250.09 last 322.62) - /unlink 46245 (time 1764737381.82 total 260.09 last 308.57) - /unlink 49498 (time 1764737391.82 total 270.09 last 325.25) - /unlink 52435 (time 1764737401.82 total 280.09 last 293.69) - /unlink 56138 (time 1764737411.82 total 290.09 last 370.26) - /unlink 59508 (time 1764737421.82 total 300.09 last 337.00) - /unlink 60000 (time 1764737424.11 total 302.38 last 214.82) - /unlink 63338 (time 1764737434.11 total 312.38 last 333.74) total: 64768 /unlink in 317.30 seconds: 204.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1420 (time 1764737474.54 total 10.01 last 141.90) - open/close 2683 (time 1764737484.54 total 20.01 last 126.28) - open/close 4082 (time 1764737494.55 total 30.01 last 139.85) - open/close 5312 (time 1764737504.55 total 40.01 last 122.99) - open/close 6370 (time 1764737514.55 total 50.01 last 105.79) - open/close 7664 (time 1764737524.55 total 60.02 last 129.38) - open/close 9142 (time 1764737534.55 total 70.02 last 147.75) - open/close 10000 (time 1764737541.56 total 77.02 last 122.52) - open/close 11207 (time 1764737551.56 total 87.03 last 120.62) - open/close 12473 (time 1764737561.57 total 97.03 last 126.59) - open/close 13493 (time 1764737571.57 total 107.04 last 101.93) - open/close 14665 (time 1764737581.58 total 117.04 last 117.12) - open/close 16112 (time 1764737591.58 total 127.05 last 144.67) - open/close 17231 (time 1764737601.58 total 137.05 last 111.88) - open/close 18234 (time 1764737611.59 total 147.05 last 100.25) - open/close 19749 (time 1764737621.59 total 157.05 last 151.47) - open/close 21641 (time 1764737631.59 total 167.06 last 189.15) - open/close 23469 (time 1764737641.60 total 177.06 last 182.71) - open/close 25270 (time 1764737651.60 total 187.07 last 180.02) - open/close 26869 (time 1764737661.61 total 197.07 last 159.80) - open/close 28257 (time 1764737671.61 total 207.08 last 138.75) - open/close 29966 (time 1764737681.61 total 217.08 last 170.88) - open/close 31775 (time 1764737691.62 total 227.08 last 180.80) - open/close 33938 (time 1764737701.62 total 237.09 last 216.21) - open/close 36014 (time 1764737711.62 total 247.09 last 207.59) - open/close 37626 (time 1764737721.63 total 257.09 last 161.13) - open/close 38992 (time 1764737731.63 total 267.10 last 136.50) - open/close 40000 (time 1764737738.64 total 274.10 last 143.89) - open/close 41609 (time 1764737748.64 total 284.11 last 160.87) - open/close 43214 (time 1764737758.64 total 294.11 last 160.49) - open/close 45031 (time 1764737768.64 total 304.11 last 181.67) - open/close 46436 (time 1764737778.65 total 314.11 last 140.44) - open/close 48207 (time 1764737788.65 total 324.11 last 177.09) - open/close 50000 (time 1764737798.38 total 333.84 last 184.27) - open/close 51896 (time 1764737808.38 total 343.84 last 189.59) - open/close 53707 (time 1764737818.38 total 353.84 last 181.10) - open/close 55509 (time 1764737828.38 total 363.85 last 180.14) - open/close 57590 (time 1764737838.39 total 373.85 last 208.03) - open/close 59513 (time 1764737848.39 total 383.86 last 192.20) - open/close 60000 (time 1764737850.81 total 386.27 last 201.52) - open/close 61920 (time 1764737860.81 total 396.28 last 191.96) - open/close 64343 (time 1764737870.81 total 406.28 last 242.25) total: 64768 open/close in 407.94 seconds: 158.77 ops/second - /unlink 4230 (time 1764737884.33 total 10.00 last 422.93) - /unlink 9233 (time 1764737894.33 total 20.00 last 500.24) - /unlink 14536 (time 1764737904.33 total 30.00 last 530.27) - /unlink 19643 (time 1764737914.33 total 40.00 last 510.63) - /unlink 23705 (time 1764737924.34 total 50.01 last 406.12) - /unlink 27237 (time 1764737934.34 total 60.01 last 353.14) - /unlink 30000 (time 1764737942.03 total 67.70 last 359.25) - /unlink 33823 (time 1764737952.03 total 77.70 last 382.26) - /unlink 37087 (time 1764737962.04 total 87.71 last 326.25) - /unlink 40000 (time 1764737969.28 total 94.95 last 402.29) - /unlink 44099 (time 1764737979.28 total 104.95 last 409.83) - /unlink 47564 (time 1764737989.28 total 114.95 last 346.47) - /unlink 50000 (time 1764737995.47 total 121.14 last 393.43) - /unlink 55445 (time 1764738005.47 total 131.14 last 544.47) - /unlink 60000 (time 1764738014.09 total 139.76 last 528.71) total: 64768 /unlink in 148.49 seconds: 436.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2533 (time 1764738055.19 total 10.00 last 253.28) - open/close 4497 (time 1764738065.19 total 20.00 last 196.35) - open/close 6497 (time 1764738075.19 total 30.01 last 199.96) - open/close 8244 (time 1764738085.20 total 40.01 last 174.63) - open/close 10000 (time 1764738093.16 total 47.97 last 220.62) - open/close 12230 (time 1764738103.16 total 57.98 last 222.85) - open/close 13876 (time 1764738113.16 total 67.98 last 164.59) - open/close 15678 (time 1764738123.17 total 77.98 last 180.13) - open/close 18236 (time 1764738133.17 total 87.98 last 255.73) - open/close 20000 (time 1764738143.06 total 97.88 last 178.31) - open/close 21346 (time 1764738153.07 total 107.88 last 134.54) - open/close 22854 (time 1764738163.07 total 117.88 last 150.77) - open/close 24034 (time 1764738173.07 total 127.88 last 117.99) - open/close 25338 (time 1764738183.07 total 137.88 last 130.39) - open/close 26735 (time 1764738193.08 total 147.89 last 139.59) - open/close 28513 (time 1764738203.08 total 157.89 last 177.76) - open/close 30000 (time 1764738209.15 total 163.96 last 245.07) - open/close 31954 (time 1764738219.15 total 173.97 last 195.30) - open/close 33394 (time 1764738229.16 total 183.97 last 143.95) - open/close 35395 (time 1764738239.16 total 193.97 last 200.07) - open/close 37505 (time 1764738249.16 total 203.97 last 210.96) - open/close 39360 (time 1764738259.16 total 213.98 last 185.45) - open/close 40000 (time 1764738262.86 total 217.67 last 173.18) - open/close 42106 (time 1764738272.86 total 227.68 last 210.50) - open/close 44247 (time 1764738282.87 total 237.68 last 214.02) - open/close 46194 (time 1764738292.87 total 247.68 last 194.65) - open/close 48974 (time 1764738302.87 total 257.68 last 277.94) - open/close 50000 (time 1764738307.78 total 262.59 last 209.23) - open/close 52207 (time 1764738317.78 total 272.59 last 220.66) - open/close 54364 (time 1764738327.78 total 282.59 last 215.65) - open/close 56828 (time 1764738337.78 total 292.59 last 246.37) - open/close 59581 (time 1764738347.78 total 302.59 last 275.29) - open/close 62052 (time 1764738357.79 total 312.60 last 247.02) total: 64768 open/close in 321.05 seconds: 201.74 ops/second - /unlink 5469 (time 1764738377.71 total 10.00 last 546.87) - /unlink 10000 (time 1764738387.37 total 19.66 last 469.31) - /unlink 15028 (time 1764738397.37 total 29.66 last 502.77) - /unlink 19401 (time 1764738407.37 total 39.66 last 437.20) - /unlink 23042 (time 1764738417.37 total 49.66 last 364.02) - /unlink 26887 (time 1764738427.37 total 59.66 last 384.43) - /unlink 30000 (time 1764738436.39 total 68.68 last 345.26) - /unlink 33326 (time 1764738446.39 total 78.68 last 332.53) - /unlink 36880 (time 1764738456.39 total 88.68 last 355.36) - /unlink 40000 (time 1764738463.82 total 96.11 last 420.20) - /unlink 44472 (time 1764738473.82 total 106.11 last 447.16) - /unlink 48338 (time 1764738483.82 total 116.11 last 386.58) - /unlink 50000 (time 1764738487.41 total 119.70 last 462.43) - /unlink 55355 (time 1764738497.41 total 129.70 last 535.40) - /unlink 60000 (time 1764738506.27 total 138.56 last 524.34) total: 64768 /unlink in 146.57 seconds: 441.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 PASS 106 (4943s) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions oleg156-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg156-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 5123 sec ========== 00:09:36 (1764738576) === conf-sanity: start cleanup 00:09:36 (1764738576) === === conf-sanity: finish cleanup 00:09:38 (1764738578) ===