-----============= acceptance-small: conf-sanity ============----- Fri Nov 14 15:53:39 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg344-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg344-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg344-server oleg344-server: oleg344-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg344-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg344-server: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg344-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:56:13 (1763153773) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) oleg344-server: oleg344-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg344-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 732 (time 1763153872.61 total 10.01 last 73.12) - open/close 1509 (time 1763153882.61 total 20.02 last 77.66) - open/close 2324 (time 1763153892.62 total 30.02 last 81.47) - open/close 3131 (time 1763153902.62 total 40.03 last 80.65) - open/close 3910 (time 1763153912.62 total 50.03 last 77.90) - open/close 4674 (time 1763153922.63 total 60.03 last 76.35) - open/close 5467 (time 1763153932.64 total 70.04 last 79.24) - open/close 6101 (time 1763153942.64 total 80.05 last 63.36) - open/close 6775 (time 1763153952.64 total 90.05 last 67.40) - open/close 7463 (time 1763153962.65 total 100.05 last 68.79) - open/close 8182 (time 1763153972.65 total 110.05 last 71.89) - open/close 8920 (time 1763153982.66 total 120.06 last 73.70) - open/close 9668 (time 1763153992.66 total 130.07 last 74.79) - open/close 10000 (time 1763153997.12 total 134.53 last 74.44) - open/close 10753 (time 1763154007.13 total 144.53 last 75.26) - open/close 11525 (time 1763154017.14 total 154.54 last 77.15) - open/close 12241 (time 1763154027.14 total 164.55 last 71.54) - open/close 12982 (time 1763154037.15 total 174.55 last 74.05) - open/close 13721 (time 1763154047.15 total 184.56 last 73.88) - open/close 14432 (time 1763154057.16 total 194.56 last 71.09) - open/close 15165 (time 1763154067.16 total 204.56 last 73.30) - open/close 15950 (time 1763154077.17 total 214.57 last 78.41) - open/close 16665 (time 1763154087.17 total 224.58 last 71.47) - open/close 17449 (time 1763154097.18 total 234.58 last 78.37) - open/close 18229 (time 1763154107.18 total 244.59 last 77.96) - open/close 19010 (time 1763154117.18 total 254.59 last 78.08) - open/close 19722 (time 1763154127.19 total 264.60 last 71.14) - open/close 20000 (time 1763154130.89 total 268.29 last 75.19) - open/close 20815 (time 1763154140.90 total 278.30 last 81.45) - open/close 21630 (time 1763154150.90 total 288.31 last 81.44) - open/close 22313 (time 1763154160.91 total 298.31 last 68.27) - open/close 23074 (time 1763154170.92 total 308.32 last 76.05) - open/close 23801 (time 1763154180.93 total 318.33 last 72.59) - open/close 24501 (time 1763154190.94 total 328.34 last 69.96) - open/close 25226 (time 1763154200.94 total 338.35 last 72.44) - open/close 25911 (time 1763154210.95 total 348.35 last 68.49) - open/close 26608 (time 1763154220.96 total 358.36 last 69.63) - open/close 27366 (time 1763154230.96 total 368.36 last 75.80) - open/close 28156 (time 1763154240.96 total 378.36 last 78.99) - open/close 28944 (time 1763154250.97 total 388.38 last 78.68) - open/close 29706 (time 1763154260.98 total 398.38 last 76.15) - open/close 30000 (time 1763154265.23 total 402.63 last 69.16) - open/close 30686 (time 1763154275.24 total 412.64 last 68.55) - open/close 31406 (time 1763154285.26 total 422.66 last 71.85) - open/close 32214 (time 1763154295.27 total 432.67 last 80.71) - open/close 32987 (time 1763154305.27 total 442.67 last 77.29) - open/close 33691 (time 1763154315.28 total 452.69 last 70.31) - open/close 34401 (time 1763154325.29 total 462.69 last 70.95) - open/close 35197 (time 1763154335.29 total 472.70 last 79.59) - open/close 36070 (time 1763154345.30 total 482.70 last 87.27) - open/close 36870 (time 1763154355.33 total 492.73 last 79.74) - open/close 37644 (time 1763154365.34 total 502.75 last 77.29) - open/close 38448 (time 1763154375.35 total 512.76 last 80.30) - open/close 39256 (time 1763154385.36 total 522.76 last 80.76) - open/close 40000 (time 1763154394.85 total 532.26 last 78.36) - open/close 40769 (time 1763154404.86 total 542.26 last 76.89) - open/close 41488 (time 1763154414.86 total 552.26 last 71.88) - open/close 42194 (time 1763154424.87 total 562.27 last 70.53) - open/close 42873 (time 1763154434.87 total 572.27 last 67.89) - open/close 43648 (time 1763154444.88 total 582.28 last 77.44) - open/close 44403 (time 1763154454.89 total 592.29 last 75.43) - open/close 45153 (time 1763154464.89 total 602.29 last 74.97) - open/close 45839 (time 1763154474.90 total 612.31 last 68.51) - open/close 46530 (time 1763154484.91 total 622.31 last 69.07) - open/close 47246 (time 1763154494.91 total 632.31 last 71.59) - open/close 47954 (time 1763154504.91 total 642.32 last 70.77) - open/close 48637 (time 1763154514.92 total 652.33 last 68.24) - open/close 49239 (time 1763154524.94 total 662.34 last 60.13) - open/close 49897 (time 1763154534.94 total 672.34 last 65.79) - open/close 50520 (time 1763154544.96 total 682.37 last 62.13) - open/close 51083 (time 1763154554.98 total 692.38 last 56.23) - open/close 51738 (time 1763154564.98 total 702.38 last 65.48) - open/close 52454 (time 1763154574.98 total 712.39 last 71.58) - open/close 53161 (time 1763154584.99 total 722.39 last 70.66) - open/close 53913 (time 1763154594.99 total 732.39 last 75.18) - open/close 54657 (time 1763154605.00 total 742.40 last 74.35) - open/close 55353 (time 1763154615.00 total 752.40 last 69.57) - open/close 56113 (time 1763154625.01 total 762.42 last 75.90) - open/close 56839 (time 1763154635.02 total 772.42 last 72.59) - open/close 57591 (time 1763154645.03 total 782.43 last 75.13) - open/close 58309 (time 1763154655.04 total 792.44 last 71.72) - open/close 58995 (time 1763154665.06 total 802.46 last 68.44) - open/close 59692 (time 1763154675.06 total 812.47 last 69.67) - open/close 60000 (time 1763154679.77 total 817.17 last 65.45) - open/close 60660 (time 1763154689.79 total 827.19 last 65.89) - open/close 61263 (time 1763154699.80 total 837.20 last 60.21) - open/close 61831 (time 1763154709.81 total 847.21 last 56.78) - open/close 62528 (time 1763154719.83 total 857.23 last 69.53) - open/close 63227 (time 1763154729.83 total 867.24 last 69.86) - open/close 63938 (time 1763154739.84 total 877.24 last 71.06) - open/close 64669 (time 1763154749.85 total 887.26 last 73.01) total: 64768 open/close in 888.66 seconds: 72.88 ops/second - /unlink 746 (time 1763154765.51 total 10.01 last 74.52) - /unlink 1463 (time 1763154775.52 total 20.02 last 71.61) - /unlink 2166 (time 1763154785.53 total 30.03 last 70.24) - /unlink 2759 (time 1763154795.55 total 40.05 last 59.20) - /unlink 3460 (time 1763154805.56 total 50.06 last 70.04) - /unlink 4150 (time 1763154815.57 total 60.07 last 68.95) - /unlink 4874 (time 1763154825.58 total 70.08 last 72.31) - /unlink 5557 (time 1763154835.59 total 80.09 last 68.22) - /unlink 6259 (time 1763154845.59 total 90.09 last 70.17) - /unlink 6929 (time 1763154855.62 total 100.12 last 66.86) - /unlink 7610 (time 1763154865.63 total 110.13 last 68.01) - /unlink 8319 (time 1763154875.64 total 120.14 last 70.85) - /unlink 9037 (time 1763154885.65 total 130.15 last 71.68) - /unlink 9752 (time 1763154895.68 total 140.18 last 71.29) - /unlink 10000 (time 1763154898.76 total 143.26 last 80.70) - /unlink 10740 (time 1763154908.76 total 153.26 last 73.95) - /unlink 11464 (time 1763154918.76 total 163.26 last 72.38) - /unlink 12232 (time 1763154928.78 total 173.28 last 76.71) - /unlink 12941 (time 1763154938.79 total 183.29 last 70.83) - /unlink 13696 (time 1763154948.79 total 193.29 last 75.47) - /unlink 14426 (time 1763154958.82 total 203.32 last 72.80) - /unlink 15147 (time 1763154968.83 total 213.33 last 72.01) - /unlink 15841 (time 1763154978.83 total 223.33 last 69.39) - /unlink 16511 (time 1763154988.84 total 233.34 last 66.95) - /unlink 17143 (time 1763154998.85 total 243.35 last 63.15) - /unlink 17823 (time 1763155008.85 total 253.35 last 67.97) - /unlink 18527 (time 1763155018.88 total 263.38 last 70.21) - /unlink 19210 (time 1763155028.88 total 273.38 last 68.29) - /unlink 19959 (time 1763155038.89 total 283.39 last 74.81) - /unlink 20726 (time 1763155048.90 total 293.40 last 76.68) - /unlink 21498 (time 1763155058.90 total 303.40 last 77.14) - /unlink 22245 (time 1763155068.91 total 313.41 last 74.68) - /unlink 22998 (time 1763155078.91 total 323.41 last 75.27) - /unlink 23796 (time 1763155088.92 total 333.42 last 79.75) - /unlink 24543 (time 1763155098.92 total 343.42 last 74.70) - /unlink 25302 (time 1763155108.93 total 353.43 last 75.82) - /unlink 26073 (time 1763155118.95 total 363.45 last 76.95) - /unlink 26748 (time 1763155128.95 total 373.45 last 67.45) - /unlink 27415 (time 1763155138.96 total 383.46 last 66.66) - /unlink 28124 (time 1763155148.96 total 393.46 last 70.89) - /unlink 28835 (time 1763155158.97 total 403.47 last 71.03) - /unlink 29562 (time 1763155168.98 total 413.48 last 72.67) - /unlink 30000 (time 1763155174.99 total 419.49 last 72.77) - /unlink 30774 (time 1763155185.00 total 429.50 last 77.37) - /unlink 31551 (time 1763155195.01 total 439.51 last 77.64) - /unlink 32283 (time 1763155205.01 total 449.51 last 73.18) - /unlink 32961 (time 1763155215.02 total 459.52 last 67.75) - /unlink 33632 (time 1763155225.02 total 469.52 last 67.09) - /unlink 34345 (time 1763155235.03 total 479.53 last 71.25) - /unlink 35069 (time 1763155245.03 total 489.53 last 72.34) - /unlink 35821 (time 1763155255.04 total 499.54 last 75.15) - /unlink 36576 (time 1763155265.05 total 509.55 last 75.46) - /unlink 37233 (time 1763155275.06 total 519.56 last 65.63) - /unlink 37914 (time 1763155285.06 total 529.56 last 68.08) - /unlink 38599 (time 1763155295.06 total 539.56 last 68.49) - /unlink 39289 (time 1763155305.08 total 549.58 last 68.87) - /unlink 39980 (time 1763155315.09 total 559.59 last 69.05) - /unlink 40697 (time 1763155325.10 total 569.60 last 71.64) - /unlink 41375 (time 1763155335.10 total 579.60 last 67.78) - /unlink 42105 (time 1763155345.10 total 589.60 last 72.97) - /unlink 42817 (time 1763155355.11 total 599.61 last 71.17) - /unlink 43554 (time 1763155365.12 total 609.62 last 73.61) - /unlink 44310 (time 1763155375.13 total 619.63 last 75.50) - /unlink 45076 (time 1763155385.15 total 629.65 last 76.50) - /unlink 45811 (time 1763155395.15 total 639.65 last 73.46) - /unlink 46524 (time 1763155405.16 total 649.66 last 71.25) - /unlink 47267 (time 1763155415.20 total 659.70 last 74.01) - /unlink 47996 (time 1763155425.20 total 669.70 last 72.85) - /unlink 48683 (time 1763155435.21 total 679.71 last 68.65) - /unlink 49363 (time 1763155445.22 total 689.72 last 67.95) - /unlink 50000 (time 1763155454.42 total 698.92 last 69.25) - /unlink 50710 (time 1763155464.43 total 708.93 last 70.92) - /unlink 51382 (time 1763155474.44 total 718.94 last 67.15) - /unlink 52074 (time 1763155484.44 total 728.94 last 69.16) - /unlink 52717 (time 1763155494.46 total 738.96 last 64.20) - /unlink 53427 (time 1763155504.46 total 748.96 last 70.99) - /unlink 54159 (time 1763155514.47 total 758.96 last 73.16) - /unlink 54922 (time 1763155524.47 total 768.97 last 76.24) - /unlink 55660 (time 1763155534.47 total 778.97 last 73.79) - /unlink 56431 (time 1763155544.48 total 788.98 last 77.05) - /unlink 57127 (time 1763155554.49 total 798.99 last 69.55) - /unlink 57864 (time 1763155564.49 total 808.99 last 73.65) - /unlink 58598 (time 1763155574.50 total 819.00 last 73.38) - /unlink 59343 (time 1763155584.51 total 829.01 last 74.38) - /unlink 60000 (time 1763155593.55 total 838.05 last 72.69) - /unlink 60786 (time 1763155603.55 total 848.05 last 78.59) - /unlink 61571 (time 1763155613.56 total 858.06 last 78.43) - /unlink 62365 (time 1763155623.57 total 868.06 last 79.37) - /unlink 63080 (time 1763155633.57 total 878.07 last 71.48) - /unlink 63793 (time 1763155643.59 total 888.09 last 71.13) - /unlink 64544 (time 1763155653.60 total 898.10 last 75.08) total: 64768 /unlink in 901.35 seconds: 71.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 758 (time 1763155693.32 total 10.00 last 75.79) - open/close 1459 (time 1763155703.33 total 20.01 last 70.06) - open/close 2086 (time 1763155713.33 total 30.01 last 62.66) - open/close 2769 (time 1763155723.33 total 40.01 last 68.30) - open/close 3568 (time 1763155733.35 total 50.03 last 79.76) - open/close 4315 (time 1763155743.35 total 60.03 last 74.68) - open/close 5090 (time 1763155753.35 total 70.04 last 77.48) - open/close 5749 (time 1763155763.35 total 80.04 last 65.90) - open/close 6358 (time 1763155773.36 total 90.05 last 60.84) - open/close 7105 (time 1763155783.37 total 100.05 last 74.67) - open/close 7924 (time 1763155793.37 total 110.05 last 81.90) - open/close 8711 (time 1763155803.38 total 120.06 last 78.64) - open/close 9490 (time 1763155813.38 total 130.06 last 77.89) - open/close 10000 (time 1763155819.94 total 136.63 last 77.68) - open/close 10763 (time 1763155829.94 total 146.63 last 76.29) - open/close 11572 (time 1763155839.95 total 156.63 last 80.85) - open/close 12380 (time 1763155849.95 total 166.64 last 80.77) - open/close 13139 (time 1763155859.96 total 176.64 last 75.84) - open/close 13939 (time 1763155869.97 total 186.65 last 79.96) - open/close 14751 (time 1763155879.97 total 196.65 last 81.17) - open/close 15569 (time 1763155889.99 total 206.67 last 81.65) - open/close 16342 (time 1763155900.00 total 216.68 last 77.24) - open/close 17148 (time 1763155910.00 total 226.69 last 80.54) - open/close 17891 (time 1763155920.02 total 236.70 last 74.21) - open/close 18623 (time 1763155930.02 total 246.70 last 73.15) - open/close 19393 (time 1763155940.03 total 256.71 last 76.95) - open/close 20000 (time 1763155948.31 total 264.99 last 73.33) - open/close 20818 (time 1763155958.31 total 274.99 last 81.74) - open/close 21605 (time 1763155968.32 total 285.00 last 78.68) - open/close 22371 (time 1763155978.33 total 295.01 last 76.51) - open/close 23178 (time 1763155988.34 total 305.02 last 80.61) - open/close 23982 (time 1763155998.34 total 315.03 last 80.37) - open/close 24713 (time 1763156008.34 total 325.03 last 73.10) - open/close 25470 (time 1763156018.34 total 335.03 last 75.70) - open/close 26235 (time 1763156028.35 total 345.03 last 76.45) - open/close 26969 (time 1763156038.37 total 355.05 last 73.27) - open/close 27700 (time 1763156048.38 total 365.06 last 73.05) - open/close 28418 (time 1763156058.38 total 375.07 last 71.74) - open/close 29164 (time 1763156068.39 total 385.07 last 74.57) - open/close 29899 (time 1763156078.39 total 395.08 last 73.45) - open/close 30609 (time 1763156088.40 total 405.09 last 70.93) - open/close 31341 (time 1763156098.42 total 415.10 last 73.07) - open/close 32073 (time 1763156108.43 total 425.11 last 73.15) - open/close 32868 (time 1763156118.44 total 435.12 last 79.40) - open/close 33638 (time 1763156128.45 total 445.13 last 76.93) - open/close 34328 (time 1763156138.49 total 455.17 last 68.71) - open/close 35057 (time 1763156148.50 total 465.18 last 72.83) - open/close 35860 (time 1763156158.52 total 475.20 last 80.18) - open/close 36665 (time 1763156168.52 total 485.20 last 80.46) - open/close 37492 (time 1763156178.52 total 495.20 last 82.70) - open/close 38291 (time 1763156188.53 total 505.21 last 79.87) - open/close 39117 (time 1763156198.55 total 515.23 last 82.43) - open/close 39983 (time 1763156208.55 total 525.24 last 86.54) - open/close 40791 (time 1763156218.56 total 535.24 last 80.75) - open/close 41607 (time 1763156228.56 total 545.24 last 81.60) - open/close 42417 (time 1763156238.56 total 555.24 last 80.99) - open/close 43229 (time 1763156248.57 total 565.26 last 81.08) - open/close 44022 (time 1763156258.58 total 575.26 last 79.28) - open/close 44756 (time 1763156268.58 total 585.26 last 73.39) - open/close 45473 (time 1763156278.58 total 595.27 last 71.66) - open/close 46178 (time 1763156288.60 total 605.28 last 70.42) - open/close 46996 (time 1763156298.60 total 615.29 last 81.73) - open/close 47735 (time 1763156308.62 total 625.30 last 73.80) - open/close 48446 (time 1763156318.63 total 635.31 last 71.01) - open/close 49154 (time 1763156328.66 total 645.34 last 70.59) - open/close 49905 (time 1763156338.68 total 655.36 last 74.98) - open/close 50687 (time 1763156348.68 total 665.36 last 78.16) - open/close 51482 (time 1763156358.69 total 675.37 last 79.43) - open/close 52237 (time 1763156368.70 total 685.38 last 75.45) - open/close 53067 (time 1763156378.70 total 695.38 last 82.97) - open/close 53863 (time 1763156388.71 total 705.39 last 79.51) - open/close 54620 (time 1763156398.72 total 715.40 last 75.66) - open/close 55441 (time 1763156408.72 total 725.40 last 82.06) - open/close 56210 (time 1763156418.73 total 735.41 last 76.84) - open/close 56977 (time 1763156428.73 total 745.41 last 76.69) - open/close 57752 (time 1763156438.75 total 755.43 last 77.38) - open/close 58430 (time 1763156448.76 total 765.44 last 67.73) - open/close 59197 (time 1763156458.76 total 775.44 last 76.70) - open/close 59930 (time 1763156468.76 total 785.45 last 73.26) - open/close 60701 (time 1763156478.77 total 795.45 last 77.04) - open/close 61492 (time 1763156488.78 total 805.46 last 79.04) - open/close 62293 (time 1763156498.78 total 815.46 last 80.08) - open/close 63118 (time 1763156508.80 total 825.49 last 82.31) - open/close 63963 (time 1763156518.81 total 835.49 last 84.45) total: 64768 open/close in 844.89 seconds: 76.66 ops/second - /unlink 888 (time 1763156540.99 total 10.00 last 88.78) - /unlink 1894 (time 1763156550.99 total 20.00 last 100.60) - /unlink 2837 (time 1763156560.99 total 30.00 last 94.28) - /unlink 3735 (time 1763156571.00 total 40.01 last 89.72) - /unlink 4583 (time 1763156581.01 total 50.02 last 84.76) - /unlink 5457 (time 1763156591.02 total 60.03 last 87.33) - /unlink 6316 (time 1763156601.02 total 70.03 last 85.87) - /unlink 7228 (time 1763156611.02 total 80.03 last 91.17) - /unlink 8051 (time 1763156621.03 total 90.04 last 82.23) - /unlink 8918 (time 1763156631.04 total 100.05 last 86.67) - /unlink 9878 (time 1763156641.05 total 110.06 last 95.89) - /unlink 10767 (time 1763156651.05 total 120.06 last 88.88) - /unlink 11607 (time 1763156661.06 total 130.07 last 83.93) - /unlink 12463 (time 1763156671.06 total 140.07 last 85.57) - /unlink 13552 (time 1763156681.07 total 150.09 last 108.75) - /unlink 14582 (time 1763156691.08 total 160.09 last 102.99) - /unlink 15530 (time 1763156701.08 total 170.09 last 94.74) - /unlink 16701 (time 1763156711.08 total 180.09 last 117.08) - /unlink 17930 (time 1763156721.09 total 190.10 last 122.84) - /unlink 18988 (time 1763156731.09 total 200.10 last 105.78) - /unlink 19935 (time 1763156741.10 total 210.11 last 94.61) - /unlink 20930 (time 1763156751.10 total 220.11 last 99.45) - /unlink 21854 (time 1763156761.11 total 230.12 last 92.37) - /unlink 22841 (time 1763156771.11 total 240.12 last 98.69) - /unlink 23998 (time 1763156781.11 total 250.12 last 115.63) - /unlink 25135 (time 1763156791.11 total 260.13 last 113.69) - /unlink 26055 (time 1763156801.12 total 270.13 last 91.99) - /unlink 26983 (time 1763156811.12 total 280.13 last 92.76) - /unlink 28034 (time 1763156821.13 total 290.14 last 104.96) - /unlink 29155 (time 1763156831.13 total 300.14 last 112.10) - /unlink 30000 (time 1763156839.31 total 308.32 last 103.39) - /unlink 31138 (time 1763156849.31 total 318.32 last 113.73) - /unlink 32307 (time 1763156859.32 total 328.33 last 116.86) - /unlink 33675 (time 1763156869.32 total 338.33 last 136.76) - /unlink 35040 (time 1763156879.32 total 348.33 last 136.48) - /unlink 36453 (time 1763156889.32 total 358.34 last 141.24) - /unlink 37997 (time 1763156899.33 total 368.34 last 154.33) - /unlink 39365 (time 1763156909.33 total 378.34 last 136.78) - /unlink 40000 (time 1763156914.33 total 383.34 last 126.99) - /unlink 41123 (time 1763156924.33 total 393.34 last 112.28) - /unlink 42510 (time 1763156934.34 total 403.35 last 138.65) - /unlink 44156 (time 1763156944.34 total 413.35 last 164.52) - /unlink 45453 (time 1763156954.34 total 423.36 last 129.65) - /unlink 46846 (time 1763156964.35 total 433.36 last 139.26) - /unlink 48086 (time 1763156974.35 total 443.36 last 123.93) - /unlink 49570 (time 1763156984.36 total 453.37 last 148.30) - /unlink 50000 (time 1763156987.57 total 456.58 last 134.05) - /unlink 51349 (time 1763156997.58 total 466.59 last 134.78) - /unlink 52761 (time 1763157007.58 total 476.60 last 141.08) - /unlink 53973 (time 1763157017.59 total 486.60 last 121.14) - /unlink 55430 (time 1763157027.59 total 496.60 last 145.67) - /unlink 58479 (time 1763157037.59 total 506.60 last 304.89) - /unlink 60000 (time 1763157042.09 total 511.10 last 337.90) - /unlink 62728 (time 1763157052.09 total 521.11 last 272.75) total: 64768 /unlink in 529.44 seconds: 122.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2473 (time 1763157093.36 total 10.00 last 247.25) - open/close 4835 (time 1763157103.36 total 20.00 last 236.17) - open/close 6639 (time 1763157113.36 total 30.00 last 180.39) - open/close 8770 (time 1763157123.36 total 40.00 last 213.09) - open/close 10000 (time 1763157130.21 total 46.86 last 179.46) - open/close 11738 (time 1763157140.22 total 56.86 last 173.74) - open/close 13791 (time 1763157150.22 total 66.86 last 205.25) - open/close 15575 (time 1763157160.22 total 76.86 last 178.39) - open/close 18014 (time 1763157170.23 total 86.87 last 243.69) - open/close 20000 (time 1763157178.12 total 94.76 last 251.82) - open/close 23012 (time 1763157188.12 total 104.76 last 301.17) - open/close 25660 (time 1763157198.12 total 114.76 last 264.80) - open/close 28835 (time 1763157208.12 total 124.76 last 317.41) - open/close 30000 (time 1763157213.43 total 130.07 last 219.51) - open/close 33195 (time 1763157223.43 total 140.07 last 319.50) - open/close 36204 (time 1763157233.43 total 150.07 last 300.88) - open/close 39082 (time 1763157243.43 total 160.07 last 287.73) - open/close 40000 (time 1763157247.70 total 164.34 last 215.16) - open/close 42076 (time 1763157257.70 total 174.34 last 207.57) - open/close 43930 (time 1763157267.70 total 184.35 last 185.32) - open/close 46007 (time 1763157277.70 total 194.35 last 207.69) - open/close 48080 (time 1763157287.70 total 204.35 last 207.29) - open/close 50000 (time 1763157297.29 total 213.94 last 200.19) - open/close 51845 (time 1763157307.29 total 223.94 last 184.49) - open/close 53894 (time 1763157317.30 total 233.94 last 204.82) - open/close 56394 (time 1763157327.30 total 243.94 last 249.96) - open/close 59355 (time 1763157337.30 total 253.95 last 296.03) - open/close 63261 (time 1763157347.30 total 263.95 last 390.59) total: 64768 open/close in 268.35 seconds: 241.35 ops/second - /unlink 5780 (time 1763157363.20 total 10.00 last 577.98) - /unlink 10000 (time 1763157371.91 total 18.71 last 484.78) - /unlink 14904 (time 1763157381.91 total 28.71 last 490.37) - /unlink 19405 (time 1763157391.91 total 38.71 last 450.00) - /unlink 25689 (time 1763157401.91 total 48.71 last 628.27) - /unlink 30000 (time 1763157411.78 total 58.58 last 436.76) - /unlink 33319 (time 1763157421.79 total 68.58 last 331.76) - /unlink 36068 (time 1763157431.79 total 78.59 last 274.89) - /unlink 38582 (time 1763157441.79 total 88.59 last 251.37) - /unlink 40000 (time 1763157446.92 total 93.72 last 276.19) - /unlink 44111 (time 1763157456.92 total 103.72 last 411.09) - /unlink 47950 (time 1763157466.92 total 113.72 last 383.85) - /unlink 50000 (time 1763157472.33 total 119.13 last 379.39) - /unlink 54052 (time 1763157482.33 total 129.13 last 405.15) - /unlink 57523 (time 1763157492.33 total 139.13 last 347.08) - /unlink 60000 (time 1763157498.12 total 144.92 last 427.95) - /unlink 62651 (time 1763157508.12 total 154.92 last 265.08) total: 64768 /unlink in 162.73 seconds: 398.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2116 (time 1763157546.39 total 10.00 last 211.57) - open/close 4213 (time 1763157556.39 total 20.01 last 209.63) - open/close 5849 (time 1763157566.39 total 30.01 last 163.55) - open/close 8258 (time 1763157576.40 total 40.01 last 240.87) - open/close 10000 (time 1763157583.83 total 47.45 last 234.22) - open/close 11709 (time 1763157593.83 total 57.45 last 170.87) - open/close 13698 (time 1763157603.84 total 67.45 last 198.83) - open/close 16696 (time 1763157613.84 total 77.45 last 299.76) - open/close 19225 (time 1763157623.84 total 87.45 last 252.88) - open/close 20000 (time 1763157626.44 total 90.05 last 298.47) - open/close 22922 (time 1763157636.44 total 100.06 last 292.04) - open/close 25445 (time 1763157646.49 total 110.10 last 251.18) - open/close 28145 (time 1763157656.49 total 120.10 last 269.92) - open/close 30000 (time 1763157664.80 total 128.41 last 223.22) - open/close 31755 (time 1763157674.80 total 138.42 last 175.43) - open/close 34171 (time 1763157684.81 total 148.42 last 241.57) - open/close 37433 (time 1763157694.81 total 158.42 last 326.09) - open/close 40000 (time 1763157703.20 total 166.82 last 305.79) - open/close 42879 (time 1763157713.21 total 176.82 last 287.84) - open/close 47029 (time 1763157723.21 total 186.82 last 414.94) - open/close 50000 (time 1763157730.98 total 194.59 last 382.35) - open/close 53247 (time 1763157740.98 total 204.59 last 324.65) - open/close 56200 (time 1763157750.98 total 214.59 last 295.24) - open/close 59849 (time 1763157760.98 total 224.60 last 364.86) - open/close 62204 (time 1763157770.98 total 234.60 last 235.46) total: 64768 open/close in 242.30 seconds: 267.31 ops/second - /unlink 4474 (time 1763157790.24 total 10.00 last 447.35) - /unlink 10000 (time 1763157799.32 total 19.08 last 608.80) - /unlink 16085 (time 1763157809.32 total 29.08 last 608.42) - /unlink 20000 (time 1763157816.87 total 36.64 last 518.00) - /unlink 25189 (time 1763157826.88 total 46.64 last 518.85) - /unlink 30000 (time 1763157836.21 total 55.97 last 515.35) - /unlink 34944 (time 1763157846.21 total 65.97 last 494.40) - /unlink 39984 (time 1763157856.21 total 75.98 last 503.92) - /unlink 44951 (time 1763157866.21 total 85.98 last 496.65) - /unlink 50000 (time 1763157875.28 total 95.04 last 556.93) - /unlink 56252 (time 1763157885.28 total 105.04 last 625.15) - /unlink 60000 (time 1763157891.90 total 111.66 last 566.14) total: 64768 /unlink in 119.55 seconds: 541.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4958 (time 1763157932.65 total 10.00 last 495.70) - open/close 8413 (time 1763157942.66 total 20.00 last 345.43) - open/close 10000 (time 1763157947.99 total 25.34 last 297.59) - open/close 13282 (time 1763157957.99 total 35.34 last 328.17) - open/close 17021 (time 1763157967.99 total 45.34 last 373.85) - open/close 20000 (time 1763157977.00 total 54.34 last 330.85) - open/close 23105 (time 1763157987.00 total 64.34 last 310.47) - open/close 26477 (time 1763157997.00 total 74.34 last 337.19) - open/close 29766 (time 1763158007.00 total 84.35 last 328.76) - open/close 32835 (time 1763158017.00 total 94.35 last 306.86) - open/close 36407 (time 1763158027.01 total 104.35 last 357.11) - open/close 39303 (time 1763158037.01 total 114.35 last 289.58) - open/close 40000 (time 1763158039.99 total 117.34 last 233.49) - open/close 42994 (time 1763158049.99 total 127.34 last 299.34) - open/close 46371 (time 1763158059.99 total 137.34 last 337.67) - open/close 49694 (time 1763158070.00 total 147.34 last 332.23) - open/close 53953 (time 1763158080.00 total 157.35 last 425.82) - open/close 58010 (time 1763158090.07 total 167.42 last 402.61) - open/close 60000 (time 1763158095.19 total 172.53 last 389.27) - open/close 64324 (time 1763158105.19 total 182.54 last 432.31) total: 64768 open/close in 183.77 seconds: 352.45 ops/second - /unlink 6681 (time 1763158117.89 total 10.00 last 668.06) - /unlink 10000 (time 1763158122.50 total 14.60 last 721.17) - /unlink 15581 (time 1763158132.50 total 24.60 last 558.06) - /unlink 20000 (time 1763158140.25 total 32.36 last 570.06) - /unlink 26650 (time 1763158150.25 total 42.36 last 664.99) - /unlink 30000 (time 1763158155.24 total 47.35 last 670.84) - /unlink 36895 (time 1763158165.24 total 57.35 last 689.49) - /unlink 40000 (time 1763158169.18 total 61.29 last 788.49) - /unlink 48080 (time 1763158179.18 total 71.29 last 807.96) - /unlink 50000 (time 1763158181.27 total 73.37 last 921.41) - /unlink 58334 (time 1763158191.27 total 83.37 last 833.33) total: 64768 /unlink in 91.35 seconds: 708.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local oleg344-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg344-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 PASS 106 (4462s) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 oleg344-server: oleg344-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg344-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4641 sec ========== 17:11:01 (1763158261) === conf-sanity: start cleanup 17:11:01 (1763158261) === === conf-sanity: finish cleanup 17:11:02 (1763158262) ===