-----============= acceptance-small: conf-sanity ============----- Sun Nov 16 22:26:24 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg303-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg303-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg303-server oleg303-server: oleg303-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg303-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg303-server: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg303-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:28:42 (1763350122) Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:-f) oleg303-server: oleg303-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg303-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 745 (time 1763350218.24 total 10.00 last 74.48) - open/close 1507 (time 1763350228.25 total 20.01 last 76.17) - open/close 2288 (time 1763350238.25 total 30.01 last 78.08) - open/close 3038 (time 1763350248.26 total 40.02 last 74.88) - open/close 3748 (time 1763350258.27 total 50.03 last 70.97) - open/close 4538 (time 1763350268.28 total 60.04 last 78.91) - open/close 5279 (time 1763350278.30 total 70.06 last 73.98) - open/close 6091 (time 1763350288.30 total 80.06 last 81.14) - open/close 6828 (time 1763350298.31 total 90.07 last 73.65) - open/close 7571 (time 1763350308.31 total 100.07 last 74.28) - open/close 8344 (time 1763350318.33 total 110.09 last 77.20) - open/close 9062 (time 1763350328.33 total 120.09 last 71.79) - open/close 9796 (time 1763350338.33 total 130.09 last 73.38) - open/close 10000 (time 1763350341.07 total 132.83 last 74.49) - open/close 10728 (time 1763350351.08 total 142.84 last 72.75) - open/close 11419 (time 1763350361.08 total 152.84 last 69.05) - open/close 12191 (time 1763350371.09 total 162.85 last 77.15) - open/close 12918 (time 1763350381.10 total 172.86 last 72.65) - open/close 13600 (time 1763350391.10 total 182.86 last 68.15) - open/close 14179 (time 1763350401.11 total 192.87 last 57.88) - open/close 14750 (time 1763350411.11 total 202.87 last 57.07) - open/close 15249 (time 1763350421.15 total 212.91 last 49.70) - open/close 15825 (time 1763350431.17 total 222.93 last 57.49) - open/close 16509 (time 1763350441.18 total 232.94 last 68.34) - open/close 17221 (time 1763350451.19 total 242.95 last 71.11) - open/close 17930 (time 1763350461.21 total 252.96 last 70.81) - open/close 18628 (time 1763350471.21 total 262.97 last 69.75) - open/close 19334 (time 1763350481.22 total 272.98 last 70.56) - open/close 20000 (time 1763350490.28 total 282.04 last 73.48) - open/close 20742 (time 1763350500.29 total 292.05 last 74.13) - open/close 21441 (time 1763350510.30 total 302.06 last 69.84) - open/close 22158 (time 1763350520.31 total 312.06 last 71.66) - open/close 22871 (time 1763350530.31 total 322.07 last 71.25) - open/close 23581 (time 1763350540.32 total 332.08 last 70.91) - open/close 24280 (time 1763350550.34 total 342.10 last 69.81) - open/close 24960 (time 1763350560.34 total 352.10 last 67.97) - open/close 25673 (time 1763350570.35 total 362.11 last 71.25) - open/close 26343 (time 1763350580.37 total 372.13 last 66.87) - open/close 27024 (time 1763350590.37 total 382.13 last 68.06) - open/close 27717 (time 1763350600.38 total 392.14 last 69.26) - open/close 28416 (time 1763350610.38 total 402.14 last 69.90) - open/close 29143 (time 1763350620.39 total 412.15 last 72.65) - open/close 29787 (time 1763350630.40 total 422.15 last 64.35) - open/close 30000 (time 1763350633.85 total 425.61 last 61.69) - open/close 30600 (time 1763350643.85 total 435.61 last 59.99) - open/close 31253 (time 1763350653.86 total 445.62 last 65.25) - open/close 31966 (time 1763350663.86 total 455.62 last 71.25) - open/close 32612 (time 1763350673.88 total 465.64 last 64.49) - open/close 33271 (time 1763350683.89 total 475.65 last 65.83) - open/close 33929 (time 1763350693.89 total 485.65 last 65.79) - open/close 34491 (time 1763350703.91 total 495.67 last 56.13) - open/close 35019 (time 1763350713.91 total 505.67 last 52.79) - open/close 35707 (time 1763350723.92 total 515.68 last 68.70) - open/close 36385 (time 1763350733.93 total 525.69 last 67.75) - open/close 37095 (time 1763350743.94 total 535.70 last 70.92) - open/close 37786 (time 1763350753.94 total 545.70 last 69.09) - open/close 38461 (time 1763350763.94 total 555.70 last 67.50) - open/close 39189 (time 1763350773.95 total 565.71 last 72.78) - open/close 39844 (time 1763350783.95 total 575.71 last 65.47) - open/close 40000 (time 1763350786.02 total 577.78 last 75.39) - open/close 40635 (time 1763350796.04 total 587.80 last 63.37) - open/close 41291 (time 1763350806.04 total 597.80 last 65.60) - open/close 41975 (time 1763350816.05 total 607.80 last 68.38) - open/close 42660 (time 1763350826.06 total 617.82 last 68.41) - open/close 43345 (time 1763350836.06 total 627.82 last 68.46) - open/close 44069 (time 1763350846.07 total 637.83 last 72.35) - open/close 44703 (time 1763350856.08 total 647.84 last 63.34) - open/close 45348 (time 1763350866.09 total 657.85 last 64.47) - open/close 45988 (time 1763350876.10 total 667.86 last 63.92) - open/close 46667 (time 1763350886.11 total 677.87 last 67.81) - open/close 47370 (time 1763350896.13 total 687.89 last 70.14) - open/close 48128 (time 1763350906.14 total 697.90 last 75.77) - open/close 48840 (time 1763350916.14 total 707.90 last 71.20) - open/close 49542 (time 1763350926.15 total 717.91 last 70.13) - open/close 50000 (time 1763350932.33 total 724.09 last 74.04) - open/close 50749 (time 1763350942.34 total 734.10 last 74.87) - open/close 51488 (time 1763350952.36 total 744.12 last 73.77) - open/close 52232 (time 1763350962.37 total 754.12 last 74.34) - open/close 52992 (time 1763350972.38 total 764.14 last 75.85) - open/close 53686 (time 1763350982.39 total 774.14 last 69.39) - open/close 54406 (time 1763350992.39 total 784.15 last 71.94) - open/close 55109 (time 1763351002.39 total 794.15 last 70.29) - open/close 55838 (time 1763351012.40 total 804.16 last 72.85) - open/close 56591 (time 1763351022.43 total 814.19 last 75.07) - open/close 57308 (time 1763351032.44 total 824.20 last 71.66) - open/close 58024 (time 1763351042.45 total 834.21 last 71.49) - open/close 58752 (time 1763351052.46 total 844.22 last 72.76) - open/close 59470 (time 1763351062.47 total 854.23 last 71.73) - open/close 60000 (time 1763351069.41 total 861.17 last 76.33) - open/close 60686 (time 1763351079.41 total 871.17 last 68.60) - open/close 61470 (time 1763351089.42 total 881.18 last 78.34) - open/close 62194 (time 1763351099.42 total 891.18 last 72.38) - open/close 62876 (time 1763351109.43 total 901.19 last 68.17) - open/close 63603 (time 1763351119.43 total 911.19 last 72.68) - open/close 64341 (time 1763351129.44 total 921.20 last 73.71) total: 64768 open/close in 927.30 seconds: 69.85 ops/second - /unlink 719 (time 1763351149.34 total 10.01 last 71.82) - /unlink 1400 (time 1763351159.34 total 20.01 last 68.08) - /unlink 2014 (time 1763351169.35 total 30.02 last 61.36) - /unlink 2679 (time 1763351179.36 total 40.03 last 66.42) - /unlink 3376 (time 1763351189.38 total 50.05 last 69.58) - /unlink 4041 (time 1763351199.38 total 60.06 last 66.46) - /unlink 4765 (time 1763351209.39 total 70.07 last 72.33) - /unlink 5456 (time 1763351219.40 total 80.08 last 69.01) - /unlink 6155 (time 1763351229.41 total 90.09 last 69.83) - /unlink 6848 (time 1763351239.42 total 100.09 last 69.25) - /unlink 7514 (time 1763351249.44 total 110.11 last 66.47) - /unlink 8200 (time 1763351259.44 total 120.12 last 68.58) - /unlink 8881 (time 1763351269.45 total 130.12 last 68.08) - /unlink 9544 (time 1763351279.46 total 140.13 last 66.22) - /unlink 10000 (time 1763351286.74 total 147.41 last 62.64) - /unlink 10635 (time 1763351296.75 total 157.42 last 63.44) - /unlink 11295 (time 1763351306.75 total 167.43 last 65.96) - /unlink 11987 (time 1763351316.76 total 177.43 last 69.19) - /unlink 12669 (time 1763351326.79 total 187.46 last 67.96) - /unlink 13342 (time 1763351336.80 total 197.47 last 67.25) - /unlink 14004 (time 1763351346.80 total 207.48 last 66.16) - /unlink 14712 (time 1763351356.80 total 217.48 last 70.79) - /unlink 15408 (time 1763351366.83 total 227.51 last 69.40) - /unlink 16028 (time 1763351376.84 total 237.51 last 61.97) - /unlink 16688 (time 1763351386.85 total 247.53 last 65.91) - /unlink 17319 (time 1763351396.86 total 257.53 last 63.07) - /unlink 18009 (time 1763351406.87 total 267.54 last 68.92) - /unlink 18755 (time 1763351416.88 total 277.55 last 74.54) - /unlink 19460 (time 1763351426.89 total 287.57 last 70.40) - /unlink 20000 (time 1763351434.55 total 295.22 last 70.52) - /unlink 20679 (time 1763351444.55 total 305.22 last 67.89) - /unlink 21350 (time 1763351454.56 total 315.24 last 67.02) - /unlink 21991 (time 1763351464.57 total 325.25 last 64.02) - /unlink 22683 (time 1763351474.58 total 335.26 last 69.13) - /unlink 23385 (time 1763351484.59 total 345.27 last 70.15) - /unlink 24104 (time 1763351494.60 total 355.28 last 71.81) - /unlink 24769 (time 1763351504.62 total 365.29 last 66.42) - /unlink 25452 (time 1763351514.62 total 375.29 last 68.28) - /unlink 26196 (time 1763351524.63 total 385.31 last 74.31) - /unlink 26900 (time 1763351534.64 total 395.31 last 70.37) - /unlink 27572 (time 1763351544.64 total 405.31 last 67.19) - /unlink 28286 (time 1763351554.64 total 415.32 last 71.35) - /unlink 28974 (time 1763351564.65 total 425.33 last 68.75) - /unlink 29670 (time 1763351574.65 total 435.33 last 69.59) - /unlink 30000 (time 1763351579.56 total 440.24 last 67.21) - /unlink 30630 (time 1763351589.57 total 450.24 last 62.96) - /unlink 31333 (time 1763351599.58 total 460.25 last 70.27) - /unlink 32008 (time 1763351609.59 total 470.27 last 67.37) - /unlink 32668 (time 1763351619.60 total 480.28 last 65.94) - /unlink 33355 (time 1763351629.61 total 490.29 last 68.62) - /unlink 34073 (time 1763351639.62 total 500.30 last 71.74) - /unlink 34750 (time 1763351649.63 total 510.30 last 67.65) - /unlink 35433 (time 1763351659.64 total 520.31 last 68.25) - /unlink 36193 (time 1763351669.64 total 530.31 last 75.98) - /unlink 36886 (time 1763351679.66 total 540.33 last 69.17) - /unlink 37612 (time 1763351689.67 total 550.34 last 72.55) - /unlink 38353 (time 1763351699.69 total 560.37 last 73.90) - /unlink 39077 (time 1763351709.70 total 570.37 last 72.34) - /unlink 39800 (time 1763351719.71 total 580.38 last 72.26) - /unlink 40000 (time 1763351722.60 total 583.28 last 69.06) - /unlink 40763 (time 1763351732.61 total 593.28 last 76.27) - /unlink 41512 (time 1763351742.61 total 603.29 last 74.84) - /unlink 42250 (time 1763351752.62 total 613.30 last 73.72) - /unlink 42954 (time 1763351762.63 total 623.30 last 70.37) - /unlink 43713 (time 1763351772.63 total 633.30 last 75.89) - /unlink 44450 (time 1763351782.63 total 643.31 last 73.69) - /unlink 45257 (time 1763351792.64 total 653.31 last 80.65) - /unlink 46088 (time 1763351802.64 total 663.31 last 83.08) - /unlink 47017 (time 1763351812.65 total 673.32 last 92.85) - /unlink 47774 (time 1763351822.65 total 683.33 last 75.65) - /unlink 48508 (time 1763351832.66 total 693.34 last 73.33) - /unlink 49301 (time 1763351842.67 total 703.34 last 79.24) - /unlink 50000 (time 1763351851.95 total 712.63 last 75.30) - /unlink 50710 (time 1763351861.96 total 722.63 last 70.98) - /unlink 51446 (time 1763351871.96 total 732.63 last 73.58) - /unlink 52220 (time 1763351881.97 total 742.65 last 77.31) - /unlink 53061 (time 1763351891.97 total 752.65 last 84.09) - /unlink 53831 (time 1763351901.98 total 762.65 last 76.96) - /unlink 54593 (time 1763351911.99 total 772.66 last 76.13) - /unlink 55495 (time 1763351921.99 total 782.66 last 90.19) - /unlink 56440 (time 1763351932.00 total 792.67 last 94.39) - /unlink 57289 (time 1763351942.01 total 802.68 last 84.81) - /unlink 58112 (time 1763351952.02 total 812.69 last 82.24) - /unlink 58932 (time 1763351962.02 total 822.70 last 81.94) - /unlink 59799 (time 1763351972.03 total 832.70 last 86.68) - /unlink 60000 (time 1763351974.37 total 835.05 last 85.71) - /unlink 60811 (time 1763351984.39 total 845.06 last 80.97) - /unlink 61589 (time 1763351994.39 total 855.07 last 77.75) - /unlink 62365 (time 1763352004.40 total 865.08 last 77.53) - /unlink 63095 (time 1763352014.41 total 875.08 last 72.98) - /unlink 63936 (time 1763352024.41 total 885.09 last 84.04) - /unlink 64754 (time 1763352034.42 total 895.10 last 81.73) total: 64768 /unlink in 895.28 seconds: 72.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 911 (time 1763352066.01 total 10.01 last 90.99) - open/close 1832 (time 1763352076.02 total 20.02 last 92.02) - open/close 2847 (time 1763352086.02 total 30.02 last 101.47) - open/close 3860 (time 1763352096.03 total 40.03 last 101.25) - open/close 4668 (time 1763352106.03 total 50.04 last 80.74) - open/close 5539 (time 1763352116.04 total 60.04 last 87.07) - open/close 6412 (time 1763352126.05 total 70.05 last 87.20) - open/close 7258 (time 1763352136.05 total 80.05 last 84.58) - open/close 8126 (time 1763352146.06 total 90.06 last 86.73) - open/close 9007 (time 1763352156.07 total 100.07 last 88.03) - open/close 10000 (time 1763352165.28 total 109.28 last 107.83) - open/close 10888 (time 1763352175.28 total 119.29 last 88.73) - open/close 11785 (time 1763352185.29 total 129.29 last 89.68) - open/close 12747 (time 1763352195.29 total 139.29 last 96.14) - open/close 13677 (time 1763352205.30 total 149.30 last 92.96) - open/close 14507 (time 1763352215.31 total 159.31 last 82.86) - open/close 15374 (time 1763352225.32 total 169.32 last 86.64) - open/close 16214 (time 1763352235.32 total 179.32 last 84.00) - open/close 17045 (time 1763352245.32 total 189.32 last 83.08) - open/close 17964 (time 1763352255.33 total 199.33 last 91.86) - open/close 18839 (time 1763352265.33 total 209.33 last 87.47) - open/close 19697 (time 1763352275.34 total 219.34 last 85.70) - open/close 20000 (time 1763352279.17 total 223.17 last 79.23) - open/close 20778 (time 1763352289.17 total 233.17 last 77.75) - open/close 21705 (time 1763352299.18 total 243.18 last 92.67) - open/close 22649 (time 1763352309.18 total 253.18 last 94.34) - open/close 23560 (time 1763352319.19 total 263.19 last 91.05) - open/close 24544 (time 1763352329.19 total 273.19 last 98.38) - open/close 25479 (time 1763352339.19 total 283.20 last 93.48) - open/close 26332 (time 1763352349.21 total 293.21 last 85.20) - open/close 27189 (time 1763352359.21 total 303.22 last 85.62) - open/close 28101 (time 1763352369.22 total 313.22 last 91.14) - open/close 28997 (time 1763352379.24 total 323.24 last 89.45) - open/close 29839 (time 1763352389.24 total 333.25 last 84.16) - open/close 30874 (time 1763352399.25 total 343.25 last 103.41) - open/close 31801 (time 1763352409.26 total 353.26 last 92.67) - open/close 32751 (time 1763352419.26 total 363.26 last 94.95) - open/close 33604 (time 1763352429.26 total 373.26 last 85.30) - open/close 34451 (time 1763352439.26 total 383.26 last 84.69) - open/close 35258 (time 1763352449.26 total 393.27 last 80.69) - open/close 36085 (time 1763352459.27 total 403.27 last 82.66) - open/close 36922 (time 1763352469.27 total 413.28 last 83.65) - open/close 37749 (time 1763352479.28 total 423.28 last 82.68) - open/close 38677 (time 1763352489.28 total 433.28 last 92.79) - open/close 39518 (time 1763352499.28 total 443.28 last 84.07) - open/close 40000 (time 1763352505.18 total 449.18 last 81.72) - open/close 40813 (time 1763352515.19 total 459.19 last 81.25) - open/close 41645 (time 1763352525.20 total 469.20 last 83.13) - open/close 42486 (time 1763352535.20 total 479.20 last 84.09) - open/close 43337 (time 1763352545.20 total 489.20 last 85.06) - open/close 44190 (time 1763352555.21 total 499.21 last 85.27) - open/close 44965 (time 1763352565.22 total 509.22 last 77.42) - open/close 45805 (time 1763352575.22 total 519.22 last 83.95) - open/close 46692 (time 1763352585.23 total 529.23 last 88.65) - open/close 47647 (time 1763352595.24 total 539.24 last 95.39) - open/close 48509 (time 1763352605.24 total 549.24 last 86.18) - open/close 49404 (time 1763352615.25 total 559.25 last 89.46) - open/close 50000 (time 1763352621.79 total 565.79 last 91.06) - open/close 50888 (time 1763352631.80 total 575.80 last 88.76) - open/close 51888 (time 1763352641.80 total 585.80 last 99.96) - open/close 52812 (time 1763352651.82 total 595.82 last 92.24) - open/close 53579 (time 1763352661.82 total 605.82 last 76.66) - open/close 54353 (time 1763352671.83 total 615.83 last 77.34) - open/close 55184 (time 1763352681.84 total 625.84 last 83.00) - open/close 56056 (time 1763352691.84 total 635.85 last 87.19) - open/close 56979 (time 1763352701.85 total 645.85 last 92.25) - open/close 57950 (time 1763352711.86 total 655.86 last 97.06) - open/close 59188 (time 1763352721.86 total 665.86 last 123.79) - open/close 60000 (time 1763352731.80 total 675.80 last 81.64) - open/close 60808 (time 1763352741.80 total 685.80 last 80.79) - open/close 61798 (time 1763352751.81 total 695.81 last 98.98) - open/close 62655 (time 1763352761.82 total 705.82 last 85.62) - open/close 63478 (time 1763352771.82 total 715.82 last 82.28) - open/close 64404 (time 1763352781.83 total 725.83 last 92.51) total: 64768 open/close in 729.61 seconds: 88.77 ops/second - /unlink 987 (time 1763352798.03 total 10.01 last 98.64) - /unlink 2038 (time 1763352808.04 total 20.01 last 105.06) - /unlink 3037 (time 1763352818.04 total 30.01 last 99.85) - /unlink 4126 (time 1763352828.05 total 40.02 last 108.82) - /unlink 5322 (time 1763352838.05 total 50.03 last 119.56) - /unlink 6304 (time 1763352848.06 total 60.03 last 98.16) - /unlink 7323 (time 1763352858.06 total 70.03 last 101.86) - /unlink 8338 (time 1763352868.07 total 80.04 last 101.43) - /unlink 9472 (time 1763352878.07 total 90.04 last 113.40) - /unlink 10000 (time 1763352883.19 total 95.17 last 102.97) - /unlink 11068 (time 1763352893.20 total 105.17 last 106.73) - /unlink 12333 (time 1763352903.20 total 115.18 last 126.46) - /unlink 13600 (time 1763352913.20 total 125.18 last 126.68) - /unlink 15180 (time 1763352923.21 total 135.18 last 157.95) - /unlink 16606 (time 1763352933.21 total 145.19 last 142.55) - /unlink 17863 (time 1763352943.22 total 155.19 last 125.64) - /unlink 19029 (time 1763352953.22 total 165.20 last 116.52) - /unlink 20000 (time 1763352960.88 total 172.85 last 126.90) - /unlink 21588 (time 1763352970.88 total 182.85 last 158.75) - /unlink 23022 (time 1763352980.89 total 192.86 last 143.30) - /unlink 24497 (time 1763352990.89 total 202.86 last 147.43) - /unlink 25802 (time 1763353000.89 total 212.87 last 130.45) - /unlink 27168 (time 1763353010.90 total 222.87 last 136.55) - /unlink 28361 (time 1763353020.91 total 232.88 last 119.16) - /unlink 29570 (time 1763353030.92 total 242.89 last 120.78) - /unlink 30000 (time 1763353034.01 total 245.99 last 138.99) - /unlink 31866 (time 1763353044.01 total 255.99 last 186.58) - /unlink 33289 (time 1763353054.01 total 265.99 last 142.30) - /unlink 35143 (time 1763353064.02 total 275.99 last 185.34) - /unlink 36820 (time 1763353074.02 total 285.99 last 167.65) - /unlink 38769 (time 1763353084.02 total 296.00 last 194.83) - /unlink 40000 (time 1763353088.17 total 300.14 last 296.92) - /unlink 43191 (time 1763353098.17 total 310.15 last 319.01) - /unlink 46275 (time 1763353108.18 total 320.15 last 308.32) - /unlink 49631 (time 1763353118.18 total 330.15 last 335.54) - /unlink 53218 (time 1763353128.18 total 340.15 last 358.60) - /unlink 56368 (time 1763353138.18 total 350.15 last 314.98) - /unlink 59016 (time 1763353148.18 total 360.16 last 264.73) - /unlink 60000 (time 1763353151.47 total 363.44 last 299.58) - /unlink 62799 (time 1763353161.47 total 373.45 last 279.81) total: 64768 /unlink in 379.72 seconds: 170.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1915 (time 1763353201.97 total 10.00 last 191.49) - open/close 4006 (time 1763353211.97 total 20.00 last 209.04) - open/close 5656 (time 1763353221.97 total 30.00 last 164.98) - open/close 7190 (time 1763353231.97 total 40.01 last 153.39) - open/close 8560 (time 1763353241.98 total 50.01 last 136.96) - open/close 9862 (time 1763353251.98 total 60.01 last 130.20) - open/close 11546 (time 1763353261.98 total 70.01 last 168.39) - open/close 13412 (time 1763353271.98 total 80.01 last 186.55) - open/close 15091 (time 1763353281.99 total 90.02 last 167.82) - open/close 17048 (time 1763353291.99 total 100.02 last 195.62) - open/close 19028 (time 1763353301.99 total 110.02 last 197.95) - open/close 20000 (time 1763353307.07 total 115.10 last 191.49) - open/close 22283 (time 1763353317.07 total 125.10 last 228.25) - open/close 24526 (time 1763353327.07 total 135.10 last 224.29) - open/close 26600 (time 1763353337.08 total 145.11 last 207.29) - open/close 28646 (time 1763353347.08 total 155.11 last 204.50) - open/close 30000 (time 1763353353.64 total 161.67 last 206.39) - open/close 31984 (time 1763353363.64 total 171.68 last 198.33) - open/close 33843 (time 1763353373.65 total 181.68 last 185.83) - open/close 35435 (time 1763353383.65 total 191.68 last 159.20) - open/close 37115 (time 1763353393.65 total 201.68 last 167.93) - open/close 38815 (time 1763353403.66 total 211.69 last 169.96) - open/close 40000 (time 1763353409.45 total 217.48 last 204.45) - open/close 42204 (time 1763353419.45 total 227.49 last 220.33) - open/close 44363 (time 1763353429.46 total 237.49 last 215.74) - open/close 45800 (time 1763353439.47 total 247.50 last 143.64) - open/close 47550 (time 1763353449.47 total 257.50 last 174.89) - open/close 49028 (time 1763353459.47 total 267.51 last 147.77) - open/close 50000 (time 1763353465.68 total 273.71 last 156.55) - open/close 51726 (time 1763353475.69 total 283.72 last 172.56) - open/close 53679 (time 1763353485.69 total 293.72 last 195.29) - open/close 55826 (time 1763353495.69 total 303.72 last 214.61) - open/close 57906 (time 1763353505.69 total 313.72 last 207.94) - open/close 59932 (time 1763353515.69 total 323.73 last 202.58) - open/close 61292 (time 1763353525.70 total 333.73 last 135.94) - open/close 62648 (time 1763353535.71 total 343.74 last 135.46) - open/close 63731 (time 1763353545.71 total 353.75 last 108.25) total: 64768 open/close in 361.90 seconds: 178.97 ops/second - /unlink 3647 (time 1763353565.84 total 10.00 last 364.70) - /unlink 6963 (time 1763353575.84 total 20.00 last 331.55) - /unlink 10000 (time 1763353583.86 total 28.02 last 378.73) - /unlink 13821 (time 1763353593.86 total 38.02 last 382.08) - /unlink 19294 (time 1763353603.86 total 48.02 last 547.27) - /unlink 24122 (time 1763353613.86 total 58.02 last 482.74) - /unlink 29736 (time 1763353623.86 total 68.02 last 561.39) - /unlink 34519 (time 1763353633.86 total 78.03 last 478.11) - /unlink 38606 (time 1763353643.86 total 88.03 last 408.66) - /unlink 40000 (time 1763353646.87 total 91.03 last 463.80) - /unlink 44207 (time 1763353656.87 total 101.03 last 420.65) - /unlink 48339 (time 1763353666.87 total 111.04 last 413.15) - /unlink 50000 (time 1763353671.44 total 115.61 last 363.33) - /unlink 55206 (time 1763353681.44 total 125.61 last 520.53) - /unlink 59649 (time 1763353691.44 total 135.61 last 444.27) - /unlink 64284 (time 1763353701.44 total 145.61 last 463.48) total: 64768 /unlink in 146.83 seconds: 441.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2810 (time 1763353731.01 total 10.00 last 280.89) - open/close 6018 (time 1763353741.01 total 20.01 last 320.71) - open/close 9552 (time 1763353751.01 total 30.01 last 353.33) - open/close 12913 (time 1763353761.01 total 40.01 last 336.06) - open/close 16139 (time 1763353771.01 total 50.01 last 322.58) - open/close 19468 (time 1763353781.01 total 60.01 last 332.90) - open/close 23784 (time 1763353791.01 total 70.01 last 431.56) - open/close 27755 (time 1763353801.02 total 80.01 last 397.00) - open/close 30000 (time 1763353808.76 total 87.76 last 289.83) - open/close 34503 (time 1763353818.76 total 97.76 last 450.24) - open/close 39230 (time 1763353828.76 total 107.76 last 472.69) - open/close 43694 (time 1763353838.76 total 117.76 last 446.31) - open/close 47432 (time 1763353848.77 total 127.76 last 373.75) - open/close 50000 (time 1763353855.20 total 134.20 last 398.93) - open/close 53865 (time 1763353865.20 total 144.20 last 386.48) - open/close 56934 (time 1763353875.20 total 154.20 last 306.90) - open/close 60000 (time 1763353883.01 total 162.01 last 392.54) - open/close 63646 (time 1763353893.02 total 172.01 last 364.56) total: 64768 open/close in 175.25 seconds: 369.57 ops/second - /unlink 4919 (time 1763353907.76 total 10.00 last 491.86) - /unlink 8230 (time 1763353917.76 total 20.00 last 331.06) - /unlink 10000 (time 1763353921.50 total 23.75 last 472.42) - /unlink 14855 (time 1763353931.50 total 33.75 last 485.47) - /unlink 20000 (time 1763353940.86 total 43.11 last 549.92) - /unlink 25762 (time 1763353950.86 total 53.11 last 576.17) - /unlink 30000 (time 1763353958.83 total 61.07 last 531.89) - /unlink 36747 (time 1763353968.83 total 71.08 last 674.56) - /unlink 40000 (time 1763353974.00 total 76.25 last 629.12) - /unlink 46678 (time 1763353984.00 total 86.25 last 667.77) - /unlink 50000 (time 1763353989.03 total 91.27 last 661.28) - /unlink 55977 (time 1763353999.03 total 101.28 last 597.24) - /unlink 60000 (time 1763354006.68 total 108.93 last 525.76) total: 64768 /unlink in 118.86 seconds: 544.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3900 (time 1763354050.75 total 10.00 last 389.92) - open/close 7924 (time 1763354060.76 total 20.00 last 402.34) - open/close 10000 (time 1763354065.55 total 24.80 last 432.63) - open/close 14764 (time 1763354075.56 total 34.80 last 476.39) - open/close 19162 (time 1763354085.56 total 44.80 last 439.77) - open/close 23302 (time 1763354095.56 total 54.80 last 413.96) - open/close 27353 (time 1763354105.56 total 64.80 last 405.08) - open/close 30000 (time 1763354112.54 total 71.79 last 378.99) - open/close 34236 (time 1763354122.54 total 81.79 last 423.56) - open/close 38435 (time 1763354132.54 total 91.79 last 419.88) - open/close 40000 (time 1763354136.71 total 95.96 last 375.65) - open/close 44300 (time 1763354146.71 total 105.96 last 429.97) - open/close 48069 (time 1763354156.71 total 115.96 last 376.82) - open/close 50000 (time 1763354162.41 total 121.66 last 338.66) - open/close 54381 (time 1763354172.42 total 131.66 last 438.04) - open/close 58617 (time 1763354182.42 total 141.66 last 423.57) - open/close 60000 (time 1763354185.58 total 144.83 last 437.41) - open/close 64624 (time 1763354195.58 total 154.83 last 462.40) total: 64768 open/close in 155.11 seconds: 417.55 ops/second - /unlink 8148 (time 1763354207.13 total 10.00 last 814.78) - /unlink 10000 (time 1763354209.81 total 12.68 last 692.27) - /unlink 17380 (time 1763354219.81 total 22.68 last 737.99) - /unlink 20000 (time 1763354222.87 total 25.74 last 854.60) - /unlink 28032 (time 1763354232.87 total 35.74 last 803.12) - /unlink 30000 (time 1763354235.36 total 38.23 last 790.22) - /unlink 37440 (time 1763354245.37 total 48.23 last 743.96) - /unlink 40000 (time 1763354248.67 total 51.53 last 775.75) - /unlink 46421 (time 1763354258.67 total 61.54 last 641.99) - /unlink 50000 (time 1763354264.35 total 67.22 last 629.41) - /unlink 56954 (time 1763354274.35 total 77.22 last 695.40) - /unlink 60000 (time 1763354278.54 total 81.41 last 727.24) total: 64768 /unlink in 87.31 seconds: 741.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing unload_modules_local oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 PASS 106 (4193s) Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 oleg303-server: oleg303-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg303-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4353 sec ========== 23:38:59 (1763354339) === conf-sanity: start cleanup 23:39:00 (1763354340) === === conf-sanity: finish cleanup 23:39:01 (1763354341) ===