-----============= acceptance-small: conf-sanity ============----- Wed Sep 24 14:06:53 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg421-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg421-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg421-server oleg421-server: oleg421-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg421-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:09:08 (1758737348) Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) oleg421-server: oleg421-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg421-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 855 (time 1758737439.59 total 10.01 last 85.44) - open/close 1800 (time 1758737449.59 total 20.01 last 94.48) - open/close 2830 (time 1758737459.61 total 30.02 last 102.84) - open/close 3769 (time 1758737469.61 total 40.03 last 93.84) - open/close 4650 (time 1758737479.63 total 50.04 last 87.98) - open/close 5454 (time 1758737489.63 total 60.05 last 80.34) - open/close 6238 (time 1758737499.64 total 70.06 last 78.36) - open/close 7016 (time 1758737509.64 total 80.06 last 77.75) - open/close 7836 (time 1758737519.65 total 90.07 last 81.92) - open/close 8587 (time 1758737529.66 total 100.08 last 75.03) - open/close 9423 (time 1758737539.67 total 110.08 last 83.58) - open/close 10000 (time 1758737546.97 total 117.38 last 79.05) - open/close 10817 (time 1758737556.97 total 127.38 last 81.69) - open/close 11591 (time 1758737566.97 total 137.39 last 77.37) - open/close 12387 (time 1758737576.97 total 147.39 last 79.56) - open/close 13192 (time 1758737586.97 total 157.39 last 80.49) - open/close 13983 (time 1758737596.98 total 167.39 last 79.09) - open/close 14816 (time 1758737606.98 total 177.39 last 83.29) - open/close 15720 (time 1758737616.98 total 187.40 last 90.34) - open/close 16504 (time 1758737627.00 total 197.41 last 78.29) - open/close 17313 (time 1758737637.00 total 207.42 last 80.85) - open/close 18145 (time 1758737647.01 total 217.43 last 83.16) - open/close 18948 (time 1758737657.02 total 227.43 last 80.24) - open/close 19753 (time 1758737667.03 total 237.45 last 80.41) - open/close 20000 (time 1758737670.66 total 241.08 last 67.93) - open/close 20772 (time 1758737680.68 total 251.10 last 77.09) - open/close 21554 (time 1758737690.68 total 261.10 last 78.16) - open/close 22341 (time 1758737700.69 total 271.11 last 78.67) - open/close 23180 (time 1758737710.70 total 281.11 last 83.82) - open/close 24035 (time 1758737720.70 total 291.12 last 85.49) - open/close 24934 (time 1758737730.71 total 301.12 last 89.83) - open/close 25714 (time 1758737740.71 total 311.13 last 77.99) - open/close 26510 (time 1758737750.72 total 321.14 last 79.51) - open/close 27347 (time 1758737760.72 total 331.14 last 83.65) - open/close 28230 (time 1758737770.73 total 341.14 last 88.28) - open/close 29065 (time 1758737780.74 total 351.16 last 83.38) - open/close 29888 (time 1758737790.75 total 361.17 last 82.24) - open/close 30663 (time 1758737800.78 total 371.20 last 77.26) - open/close 31375 (time 1758737810.79 total 381.21 last 71.14) - open/close 32117 (time 1758737820.80 total 391.22 last 74.11) - open/close 32926 (time 1758737830.81 total 401.22 last 80.86) - open/close 33730 (time 1758737840.81 total 411.23 last 80.35) - open/close 34548 (time 1758737850.82 total 421.24 last 81.75) - open/close 35362 (time 1758737860.83 total 431.24 last 81.34) - open/close 36197 (time 1758737870.84 total 441.25 last 83.40) - open/close 37020 (time 1758737880.84 total 451.26 last 82.28) - open/close 37825 (time 1758737890.85 total 461.27 last 80.43) - open/close 38613 (time 1758737900.87 total 471.28 last 78.67) - open/close 39305 (time 1758737910.88 total 481.29 last 69.12) - open/close 40000 (time 1758737920.15 total 490.57 last 74.93) - open/close 40767 (time 1758737930.16 total 500.58 last 76.64) - open/close 41573 (time 1758737940.16 total 510.58 last 80.58) - open/close 42342 (time 1758737950.18 total 520.60 last 76.77) - open/close 43107 (time 1758737960.19 total 530.60 last 76.44) - open/close 43918 (time 1758737970.19 total 540.61 last 81.08) - open/close 44776 (time 1758737980.20 total 550.62 last 85.71) - open/close 45507 (time 1758737990.22 total 560.64 last 72.97) - open/close 46296 (time 1758738000.23 total 570.65 last 78.80) - open/close 47162 (time 1758738010.24 total 580.66 last 86.49) - open/close 47982 (time 1758738020.25 total 590.67 last 81.94) - open/close 48829 (time 1758738030.26 total 600.68 last 84.62) - open/close 49674 (time 1758738040.26 total 610.68 last 84.48) - open/close 50000 (time 1758738044.26 total 614.67 last 81.57) - open/close 50850 (time 1758738054.27 total 624.68 last 84.93) - open/close 51690 (time 1758738064.27 total 634.69 last 83.96) - open/close 52460 (time 1758738074.28 total 644.70 last 76.92) - open/close 53336 (time 1758738084.29 total 654.71 last 87.53) - open/close 54192 (time 1758738094.30 total 664.72 last 85.50) - open/close 55016 (time 1758738104.30 total 674.72 last 82.38) - open/close 55814 (time 1758738114.31 total 684.73 last 79.74) - open/close 56634 (time 1758738124.32 total 694.73 last 81.96) - open/close 57497 (time 1758738134.32 total 704.74 last 86.26) - open/close 58419 (time 1758738144.33 total 714.74 last 92.14) - open/close 59416 (time 1758738154.33 total 724.75 last 99.66) - open/close 60000 (time 1758738160.91 total 731.33 last 88.72) - open/close 60769 (time 1758738170.92 total 741.34 last 76.81) - open/close 61585 (time 1758738180.93 total 751.35 last 81.56) - open/close 62403 (time 1758738190.93 total 761.35 last 81.76) - open/close 63176 (time 1758738200.94 total 771.36 last 77.26) - open/close 63815 (time 1758738210.98 total 781.39 last 63.67) - open/close 64549 (time 1758738220.98 total 791.39 last 73.40) total: 64768 open/close in 794.41 seconds: 81.53 ops/second - /unlink 708 (time 1758738237.94 total 10.01 last 70.71) - /unlink 1430 (time 1758738247.95 total 20.02 last 72.15) - /unlink 2288 (time 1758738257.96 total 30.03 last 85.72) - /unlink 3165 (time 1758738267.98 total 40.05 last 87.57) - /unlink 4055 (time 1758738277.98 total 50.05 last 88.94) - /unlink 4884 (time 1758738287.99 total 60.06 last 82.83) - /unlink 5711 (time 1758738298.00 total 70.06 last 82.67) - /unlink 6526 (time 1758738308.00 total 80.07 last 81.46) - /unlink 7374 (time 1758738318.01 total 90.08 last 84.70) - /unlink 8161 (time 1758738328.02 total 100.09 last 78.63) - /unlink 8942 (time 1758738338.02 total 110.09 last 78.09) - /unlink 9790 (time 1758738348.03 total 120.10 last 84.75) - /unlink 10000 (time 1758738350.78 total 122.85 last 76.34) - /unlink 10801 (time 1758738360.79 total 132.86 last 79.97) - /unlink 11653 (time 1758738370.80 total 142.87 last 85.11) - /unlink 12596 (time 1758738380.81 total 152.88 last 94.26) - /unlink 13485 (time 1758738390.82 total 162.88 last 88.85) - /unlink 14356 (time 1758738400.82 total 172.89 last 87.03) - /unlink 15246 (time 1758738410.83 total 182.90 last 88.95) - /unlink 16133 (time 1758738420.83 total 192.90 last 88.68) - /unlink 16942 (time 1758738430.85 total 202.92 last 80.75) - /unlink 17696 (time 1758738440.86 total 212.93 last 75.35) - /unlink 18512 (time 1758738450.86 total 222.93 last 81.58) - /unlink 19318 (time 1758738460.86 total 232.93 last 80.57) - /unlink 20000 (time 1758738469.38 total 241.45 last 80.07) - /unlink 20751 (time 1758738479.38 total 251.45 last 75.08) - /unlink 21493 (time 1758738489.39 total 261.46 last 74.17) - /unlink 22298 (time 1758738499.39 total 271.46 last 80.48) - /unlink 23087 (time 1758738509.40 total 281.47 last 78.84) - /unlink 23886 (time 1758738519.40 total 291.47 last 79.85) - /unlink 24759 (time 1758738529.41 total 301.48 last 87.27) - /unlink 25592 (time 1758738539.41 total 311.48 last 83.23) - /unlink 26426 (time 1758738549.42 total 321.49 last 83.35) - /unlink 27274 (time 1758738559.43 total 331.50 last 84.73) - /unlink 28136 (time 1758738569.44 total 341.51 last 86.09) - /unlink 28986 (time 1758738579.45 total 351.52 last 84.97) - /unlink 29833 (time 1758738589.45 total 361.52 last 84.68) - /unlink 30725 (time 1758738599.49 total 371.55 last 88.87) - /unlink 31568 (time 1758738609.49 total 381.56 last 84.27) - /unlink 32392 (time 1758738619.49 total 391.56 last 82.39) - /unlink 33161 (time 1758738629.51 total 401.58 last 76.79) - /unlink 33904 (time 1758738639.51 total 411.58 last 74.24) - /unlink 34644 (time 1758738649.52 total 421.59 last 73.93) - /unlink 35382 (time 1758738659.54 total 431.61 last 73.68) - /unlink 36148 (time 1758738669.54 total 441.61 last 76.59) - /unlink 36991 (time 1758738679.54 total 451.61 last 84.26) - /unlink 37913 (time 1758738689.56 total 461.63 last 92.09) - /unlink 38698 (time 1758738699.56 total 471.63 last 78.48) - /unlink 39432 (time 1758738709.56 total 481.63 last 73.39) - /unlink 40000 (time 1758738716.79 total 488.86 last 78.54) - /unlink 40737 (time 1758738726.80 total 498.87 last 73.66) - /unlink 41410 (time 1758738736.81 total 508.88 last 67.25) - /unlink 42177 (time 1758738746.81 total 518.88 last 76.68) - /unlink 42973 (time 1758738756.81 total 528.88 last 79.59) - /unlink 43723 (time 1758738766.82 total 538.89 last 74.95) - /unlink 44551 (time 1758738776.82 total 548.89 last 82.80) - /unlink 45358 (time 1758738786.82 total 558.89 last 80.64) - /unlink 46152 (time 1758738796.83 total 568.90 last 79.32) - /unlink 47008 (time 1758738806.84 total 578.91 last 85.55) - /unlink 47860 (time 1758738816.85 total 588.92 last 85.13) - /unlink 48615 (time 1758738826.86 total 598.93 last 75.39) - /unlink 49388 (time 1758738836.87 total 608.94 last 77.23) - /unlink 50000 (time 1758738843.92 total 615.99 last 86.82) - /unlink 50859 (time 1758738853.93 total 626.00 last 85.80) - /unlink 51670 (time 1758738863.94 total 636.01 last 81.06) - /unlink 52455 (time 1758738873.96 total 646.03 last 78.37) - /unlink 53234 (time 1758738883.96 total 656.03 last 77.90) - /unlink 53988 (time 1758738893.96 total 666.03 last 75.35) - /unlink 54827 (time 1758738903.97 total 676.04 last 83.86) - /unlink 55602 (time 1758738913.97 total 686.04 last 77.48) - /unlink 56403 (time 1758738923.98 total 696.05 last 80.06) - /unlink 57246 (time 1758738933.98 total 706.05 last 84.26) - /unlink 58082 (time 1758738943.98 total 716.05 last 83.58) - /unlink 58938 (time 1758738953.99 total 726.06 last 85.53) - /unlink 59779 (time 1758738964.00 total 736.07 last 84.03) - /unlink 60000 (time 1758738966.47 total 738.54 last 89.47) - /unlink 60872 (time 1758738976.47 total 748.54 last 87.15) - /unlink 61781 (time 1758738986.48 total 758.55 last 90.82) - /unlink 62586 (time 1758738996.49 total 768.56 last 80.46) - /unlink 63389 (time 1758739006.51 total 778.58 last 80.09) - /unlink 64147 (time 1758739016.52 total 788.59 last 75.72) total: 64768 /unlink in 796.77 seconds: 81.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 833 (time 1758739060.02 total 10.01 last 83.26) - open/close 1657 (time 1758739070.02 total 20.01 last 82.37) - open/close 2497 (time 1758739080.04 total 30.02 last 83.86) - open/close 3340 (time 1758739090.04 total 40.03 last 84.29) - open/close 4127 (time 1758739100.05 total 50.03 last 78.63) - open/close 4940 (time 1758739110.05 total 60.04 last 81.29) - open/close 5777 (time 1758739120.06 total 70.04 last 83.63) - open/close 6602 (time 1758739130.07 total 80.05 last 82.43) - open/close 7414 (time 1758739140.07 total 90.06 last 81.18) - open/close 8231 (time 1758739150.08 total 100.07 last 81.60) - open/close 9070 (time 1758739160.09 total 110.07 last 83.85) - open/close 9904 (time 1758739170.10 total 120.08 last 83.30) - open/close 10735 (time 1758739180.11 total 130.09 last 83.03) - open/close 11527 (time 1758739190.12 total 140.10 last 79.13) - open/close 12378 (time 1758739200.12 total 150.11 last 85.07) - open/close 13168 (time 1758739210.13 total 160.12 last 78.92) - open/close 13949 (time 1758739220.14 total 170.12 last 78.04) - open/close 14728 (time 1758739230.16 total 180.14 last 77.78) - open/close 15464 (time 1758739240.17 total 190.15 last 73.51) - open/close 16276 (time 1758739250.19 total 200.17 last 81.04) - open/close 17101 (time 1758739260.19 total 210.18 last 82.46) - open/close 17953 (time 1758739270.21 total 220.20 last 85.02) - open/close 18744 (time 1758739280.22 total 230.20 last 79.05) - open/close 19596 (time 1758739290.23 total 240.21 last 85.16) - open/close 20000 (time 1758739294.74 total 244.72 last 89.58) - open/close 20838 (time 1758739304.74 total 254.73 last 83.73) - open/close 21681 (time 1758739314.75 total 264.73 last 84.28) - open/close 22490 (time 1758739324.75 total 274.73 last 80.90) - open/close 23274 (time 1758739334.75 total 284.74 last 78.37) - open/close 24101 (time 1758739344.75 total 294.74 last 82.68) - open/close 24991 (time 1758739354.76 total 304.75 last 88.93) - open/close 25784 (time 1758739364.77 total 314.75 last 79.23) - open/close 26570 (time 1758739374.78 total 324.76 last 78.55) - open/close 27413 (time 1758739384.78 total 334.76 last 84.29) - open/close 28210 (time 1758739394.78 total 344.77 last 79.67) - open/close 29009 (time 1758739404.78 total 354.77 last 79.89) - open/close 29828 (time 1758739414.78 total 364.77 last 81.89) - open/close 30000 (time 1758739416.88 total 366.86 last 82.26) - open/close 30836 (time 1758739426.88 total 376.86 last 83.59) - open/close 31593 (time 1758739436.88 total 386.87 last 75.65) - open/close 32292 (time 1758739446.89 total 396.87 last 69.85) - open/close 32928 (time 1758739456.90 total 406.89 last 63.52) - open/close 33690 (time 1758739466.91 total 416.90 last 76.13) - open/close 34435 (time 1758739476.91 total 426.90 last 74.49) - open/close 35148 (time 1758739486.92 total 436.90 last 71.28) - open/close 35975 (time 1758739496.93 total 446.91 last 82.59) - open/close 36771 (time 1758739506.93 total 456.91 last 79.60) - open/close 37571 (time 1758739516.94 total 466.93 last 79.89) - open/close 38415 (time 1758739526.95 total 476.93 last 84.37) - open/close 39221 (time 1758739536.97 total 486.95 last 80.46) - open/close 40000 (time 1758739546.57 total 496.56 last 81.08) - open/close 40820 (time 1758739556.57 total 506.56 last 82.00) - open/close 41515 (time 1758739566.60 total 516.59 last 69.29) - open/close 42217 (time 1758739576.61 total 526.59 last 70.16) - open/close 42963 (time 1758739586.64 total 536.62 last 74.39) - open/close 43676 (time 1758739596.64 total 546.63 last 71.25) - open/close 44401 (time 1758739606.65 total 556.63 last 72.48) - open/close 45166 (time 1758739616.65 total 566.63 last 76.48) - open/close 45943 (time 1758739626.67 total 576.66 last 77.52) - open/close 46704 (time 1758739636.68 total 586.67 last 76.01) - open/close 47535 (time 1758739646.69 total 596.67 last 83.06) - open/close 48290 (time 1758739656.71 total 606.69 last 75.35) - open/close 49103 (time 1758739666.71 total 616.70 last 81.27) - open/close 49893 (time 1758739676.72 total 626.70 last 78.97) - open/close 50652 (time 1758739686.72 total 636.70 last 75.88) - open/close 51319 (time 1758739696.72 total 646.70 last 66.69) - open/close 52064 (time 1758739706.75 total 656.73 last 74.29) - open/close 52809 (time 1758739716.75 total 666.73 last 74.50) - open/close 53590 (time 1758739726.76 total 676.75 last 77.97) - open/close 54384 (time 1758739736.77 total 686.76 last 79.31) - open/close 55151 (time 1758739746.78 total 696.76 last 76.68) - open/close 55883 (time 1758739756.79 total 706.77 last 73.12) - open/close 56629 (time 1758739766.80 total 716.78 last 74.50) - open/close 57400 (time 1758739776.81 total 726.79 last 77.06) - open/close 58189 (time 1758739786.81 total 736.79 last 78.87) - open/close 58975 (time 1758739796.81 total 746.80 last 78.57) - open/close 59840 (time 1758739806.82 total 756.80 last 86.49) - open/close 60000 (time 1758739809.04 total 759.02 last 72.03) - open/close 60794 (time 1758739819.05 total 769.03 last 79.32) - open/close 61618 (time 1758739829.05 total 779.04 last 82.34) - open/close 62463 (time 1758739839.07 total 789.05 last 84.40) - open/close 63259 (time 1758739849.08 total 799.07 last 79.47) - open/close 64041 (time 1758739859.09 total 809.07 last 78.17) total: 64768 open/close in 817.69 seconds: 79.21 ops/second - /unlink 775 (time 1758739881.08 total 10.00 last 77.48) - /unlink 1537 (time 1758739891.08 total 20.00 last 76.20) - /unlink 2286 (time 1758739901.09 total 30.01 last 74.84) - /unlink 3060 (time 1758739911.10 total 40.02 last 77.34) - /unlink 3720 (time 1758739921.11 total 50.03 last 65.92) - /unlink 4550 (time 1758739931.12 total 60.04 last 82.89) - /unlink 5363 (time 1758739941.14 total 70.06 last 81.16) - /unlink 6169 (time 1758739951.15 total 80.07 last 80.52) - /unlink 6990 (time 1758739961.16 total 90.08 last 82.05) - /unlink 7792 (time 1758739971.17 total 100.09 last 80.07) - /unlink 8576 (time 1758739981.18 total 110.10 last 78.37) - /unlink 9419 (time 1758739991.18 total 120.10 last 84.28) - /unlink 10000 (time 1758739998.56 total 127.48 last 78.65) - /unlink 10785 (time 1758740008.57 total 137.49 last 78.48) - /unlink 11579 (time 1758740018.57 total 147.49 last 79.36) - /unlink 12356 (time 1758740028.58 total 157.50 last 77.61) - /unlink 13179 (time 1758740038.59 total 167.51 last 82.29) - /unlink 13973 (time 1758740048.59 total 177.52 last 79.33) - /unlink 14751 (time 1758740058.60 total 187.52 last 77.77) - /unlink 15587 (time 1758740068.62 total 197.54 last 83.46) - /unlink 16466 (time 1758740078.65 total 207.57 last 87.61) - /unlink 17318 (time 1758740088.65 total 217.57 last 85.17) - /unlink 18226 (time 1758740098.66 total 227.58 last 90.73) - /unlink 19086 (time 1758740108.67 total 237.59 last 85.95) - /unlink 20000 (time 1758740118.37 total 247.29 last 94.16) - /unlink 21030 (time 1758740128.38 total 257.30 last 102.95) - /unlink 22017 (time 1758740138.38 total 267.30 last 98.66) - /unlink 22934 (time 1758740148.39 total 277.31 last 91.66) - /unlink 23903 (time 1758740158.40 total 287.32 last 96.80) - /unlink 24947 (time 1758740168.40 total 297.32 last 104.35) - /unlink 26007 (time 1758740178.41 total 307.33 last 105.90) - /unlink 27006 (time 1758740188.42 total 317.34 last 99.80) - /unlink 27994 (time 1758740198.43 total 327.35 last 98.72) - /unlink 29016 (time 1758740208.43 total 337.35 last 102.17) - /unlink 30000 (time 1758740218.36 total 347.28 last 99.07) - /unlink 31011 (time 1758740228.37 total 357.29 last 101.05) - /unlink 31978 (time 1758740238.38 total 367.30 last 96.59) - /unlink 33081 (time 1758740248.38 total 377.30 last 110.25) - /unlink 34014 (time 1758740258.39 total 387.31 last 93.27) - /unlink 34928 (time 1758740268.39 total 397.31 last 91.36) - /unlink 35814 (time 1758740278.40 total 407.32 last 88.49) - /unlink 36725 (time 1758740288.41 total 417.33 last 91.03) - /unlink 37586 (time 1758740298.41 total 427.33 last 86.10) - /unlink 38512 (time 1758740308.42 total 437.34 last 92.55) - /unlink 39444 (time 1758740318.42 total 447.34 last 93.15) - /unlink 40000 (time 1758740324.93 total 453.85 last 85.49) - /unlink 40907 (time 1758740334.94 total 463.86 last 90.58) - /unlink 41870 (time 1758740344.96 total 473.88 last 96.15) - /unlink 42767 (time 1758740354.96 total 483.88 last 89.69) - /unlink 43771 (time 1758740364.96 total 493.88 last 100.35) - /unlink 44705 (time 1758740374.98 total 503.90 last 93.26) - /unlink 45619 (time 1758740384.98 total 513.90 last 91.38) - /unlink 46554 (time 1758740394.98 total 523.90 last 93.48) - /unlink 47497 (time 1758740404.98 total 533.91 last 94.27) - /unlink 48409 (time 1758740414.99 total 543.91 last 91.18) - /unlink 49529 (time 1758740424.99 total 553.91 last 111.95) - /unlink 50000 (time 1758740429.01 total 557.93 last 117.29) - /unlink 51010 (time 1758740439.01 total 567.93 last 100.93) - /unlink 52338 (time 1758740449.02 total 577.94 last 132.72) - /unlink 53582 (time 1758740459.03 total 587.95 last 124.32) - /unlink 54592 (time 1758740469.03 total 597.95 last 100.98) - /unlink 55670 (time 1758740479.04 total 607.96 last 107.69) - /unlink 56812 (time 1758740489.04 total 617.96 last 114.14) - /unlink 57861 (time 1758740499.05 total 627.97 last 104.86) - /unlink 58908 (time 1758740509.05 total 637.97 last 104.69) - /unlink 60000 (time 1758740517.41 total 646.33 last 130.59) - /unlink 61155 (time 1758740527.42 total 656.34 last 115.40) - /unlink 62394 (time 1758740537.42 total 666.34 last 123.88) - /unlink 63584 (time 1758740547.42 total 676.34 last 118.96) total: 64768 /unlink in 685.62 seconds: 94.47 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1319 (time 1758740589.49 total 10.00 last 131.85) - open/close 2844 (time 1758740599.49 total 20.01 last 152.40) - open/close 4094 (time 1758740609.51 total 30.03 last 124.80) - open/close 5334 (time 1758740619.51 total 40.03 last 123.94) - open/close 6562 (time 1758740629.52 total 50.03 last 122.77) - open/close 8375 (time 1758740639.52 total 60.04 last 181.28) - open/close 9918 (time 1758740649.52 total 70.04 last 154.27) - open/close 11469 (time 1758740659.53 total 80.04 last 154.99) - open/close 12654 (time 1758740669.53 total 90.05 last 118.48) - open/close 13928 (time 1758740679.53 total 100.05 last 127.32) - open/close 15249 (time 1758740689.53 total 110.05 last 132.10) - open/close 16618 (time 1758740699.54 total 120.05 last 136.86) - open/close 18350 (time 1758740709.54 total 130.06 last 173.09) - open/close 19825 (time 1758740719.55 total 140.07 last 147.36) - open/close 21284 (time 1758740729.55 total 150.07 last 145.87) - open/close 22659 (time 1758740739.56 total 160.07 last 137.47) - open/close 24150 (time 1758740749.56 total 170.08 last 149.08) - open/close 25922 (time 1758740759.56 total 180.08 last 177.09) - open/close 27753 (time 1758740769.57 total 190.08 last 183.06) - open/close 29915 (time 1758740779.57 total 200.09 last 216.19) - open/close 31437 (time 1758740789.57 total 210.09 last 152.18) - open/close 33042 (time 1758740799.58 total 220.09 last 160.38) - open/close 34922 (time 1758740809.58 total 230.10 last 187.90) - open/close 36983 (time 1758740819.58 total 240.10 last 206.04) - open/close 39132 (time 1758740829.59 total 250.11 last 214.82) - open/close 40000 (time 1758740834.38 total 254.89 last 181.29) - open/close 42116 (time 1758740844.38 total 264.90 last 211.56) - open/close 44496 (time 1758740854.38 total 274.90 last 237.89) - open/close 46705 (time 1758740864.38 total 284.90 last 220.89) - open/close 48595 (time 1758740874.38 total 294.90 last 188.98) - open/close 50000 (time 1758740882.69 total 303.21 last 169.18) - open/close 51924 (time 1758740892.69 total 313.21 last 192.36) - open/close 53861 (time 1758740902.69 total 323.21 last 193.61) - open/close 55859 (time 1758740912.70 total 333.21 last 199.78) - open/close 58009 (time 1758740922.70 total 343.22 last 214.91) - open/close 60000 (time 1758740932.39 total 352.90 last 205.53) - open/close 63094 (time 1758740942.39 total 362.91 last 309.36) total: 64768 open/close in 369.49 seconds: 175.29 ops/second - /unlink 4348 (time 1758740960.72 total 10.00 last 434.72) - /unlink 8265 (time 1758740970.72 total 20.00 last 391.63) - /unlink 10000 (time 1758740974.88 total 24.16 last 417.20) - /unlink 14283 (time 1758740984.88 total 34.16 last 428.26) - /unlink 19227 (time 1758740994.88 total 44.16 last 494.35) - /unlink 20000 (time 1758740997.03 total 46.31 last 360.52) - /unlink 24819 (time 1758741007.03 total 56.31 last 481.85) - /unlink 29343 (time 1758741017.03 total 66.31 last 452.24) - /unlink 33425 (time 1758741027.03 total 76.32 last 408.11) - /unlink 39632 (time 1758741037.04 total 86.32 last 620.68) - /unlink 44428 (time 1758741047.04 total 96.32 last 479.59) - /unlink 49968 (time 1758741057.04 total 106.32 last 553.80) - /unlink 53679 (time 1758741067.04 total 116.32 last 371.06) - /unlink 58082 (time 1758741077.04 total 126.32 last 440.23) - /unlink 60000 (time 1758741080.87 total 130.15 last 501.08) - /unlink 64393 (time 1758741090.87 total 140.15 last 439.25) total: 64768 /unlink in 141.05 seconds: 459.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2445 (time 1758741124.36 total 10.00 last 244.48) - open/close 4422 (time 1758741134.36 total 20.01 last 197.59) - open/close 6768 (time 1758741144.37 total 30.01 last 234.56) - open/close 9650 (time 1758741154.37 total 40.01 last 288.14) - open/close 12067 (time 1758741164.37 total 50.01 last 241.66) - open/close 14566 (time 1758741174.37 total 60.01 last 249.87) - open/close 16685 (time 1758741184.37 total 70.01 last 211.89) - open/close 18925 (time 1758741194.37 total 80.02 last 223.96) - open/close 20000 (time 1758741199.67 total 85.31 last 203.00) - open/close 22090 (time 1758741209.67 total 95.31 last 208.98) - open/close 24310 (time 1758741219.67 total 105.31 last 221.96) - open/close 26921 (time 1758741229.67 total 115.31 last 261.08) - open/close 29696 (time 1758741239.68 total 125.32 last 277.38) - open/close 32244 (time 1758741249.68 total 135.32 last 254.75) - open/close 34527 (time 1758741259.68 total 145.32 last 228.27) - open/close 36709 (time 1758741269.69 total 155.33 last 218.05) - open/close 39111 (time 1758741279.69 total 165.33 last 240.18) - open/close 40000 (time 1758741282.74 total 168.38 last 291.74) - open/close 42726 (time 1758741292.74 total 178.38 last 272.59) - open/close 45122 (time 1758741302.74 total 188.38 last 239.53) - open/close 47571 (time 1758741312.74 total 198.38 last 244.89) - open/close 49433 (time 1758741322.74 total 208.39 last 186.09) - open/close 50000 (time 1758741325.08 total 210.72 last 243.06) - open/close 52119 (time 1758741335.08 total 220.73 last 211.78) - open/close 53683 (time 1758741345.09 total 230.73 last 156.25) - open/close 55396 (time 1758741355.10 total 240.74 last 171.23) - open/close 57056 (time 1758741365.10 total 250.74 last 165.98) - open/close 58634 (time 1758741375.10 total 260.74 last 157.75) - open/close 60000 (time 1758741381.93 total 267.57 last 200.11) - open/close 62114 (time 1758741391.93 total 277.57 last 211.40) - open/close 64607 (time 1758741401.93 total 287.57 last 249.30) total: 64768 open/close in 288.03 seconds: 224.86 ops/second - /unlink 4470 (time 1758741413.95 total 10.00 last 446.90) - /unlink 8313 (time 1758741423.95 total 20.00 last 384.30) - /unlink 10000 (time 1758741428.62 total 24.67 last 361.34) - /unlink 14459 (time 1758741438.62 total 34.67 last 445.86) - /unlink 18392 (time 1758741448.62 total 44.67 last 393.28) - /unlink 20000 (time 1758741451.30 total 47.35 last 600.26) - /unlink 25176 (time 1758741461.30 total 57.35 last 517.58) - /unlink 30000 (time 1758741469.71 total 65.76 last 573.60) - /unlink 35893 (time 1758741479.71 total 75.76 last 589.26) - /unlink 40000 (time 1758741487.97 total 84.02 last 497.63) - /unlink 44471 (time 1758741497.97 total 94.02 last 447.02) - /unlink 49745 (time 1758741507.97 total 104.02 last 527.38) - /unlink 54742 (time 1758741517.97 total 114.02 last 499.62) - /unlink 59045 (time 1758741527.97 total 124.02 last 430.13) - /unlink 60000 (time 1758741530.39 total 126.44 last 394.73) - /unlink 64487 (time 1758741540.39 total 136.44 last 448.66) total: 64768 /unlink in 136.94 seconds: 472.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2241 (time 1758741567.94 total 10.00 last 224.07) - open/close 5357 (time 1758741577.94 total 20.00 last 311.59) - open/close 8939 (time 1758741587.94 total 30.00 last 358.19) - open/close 10000 (time 1758741591.90 total 33.96 last 267.92) - open/close 13589 (time 1758741601.91 total 43.96 last 358.86) - open/close 17313 (time 1758741611.91 total 53.97 last 372.29) - open/close 20000 (time 1758741618.73 total 60.79 last 394.03) - open/close 23059 (time 1758741628.73 total 70.79 last 305.86) - open/close 26960 (time 1758741638.73 total 80.79 last 390.06) - open/close 30000 (time 1758741647.71 total 89.77 last 338.37) - open/close 33292 (time 1758741657.72 total 99.78 last 329.07) - open/close 36700 (time 1758741667.72 total 109.78 last 340.73) - open/close 39290 (time 1758741677.73 total 119.78 last 258.86) - open/close 40000 (time 1758741680.83 total 122.89 last 228.54) - open/close 43017 (time 1758741690.83 total 132.89 last 301.69) - open/close 45743 (time 1758741700.83 total 142.89 last 272.56) - open/close 48496 (time 1758741710.83 total 152.89 last 275.29) - open/close 50000 (time 1758741716.61 total 158.67 last 260.45) - open/close 52842 (time 1758741726.61 total 168.67 last 284.18) - open/close 55551 (time 1758741736.61 total 178.67 last 270.83) - open/close 58358 (time 1758741746.61 total 188.67 last 280.63) - open/close 60000 (time 1758741751.21 total 193.27 last 357.25) - open/close 63645 (time 1758741761.21 total 203.27 last 364.47) total: 64768 open/close in 205.69 seconds: 314.89 ops/second - /unlink 5447 (time 1758741774.95 total 10.00 last 544.64) - /unlink 10000 (time 1758741782.43 total 17.48 last 608.67) - /unlink 15369 (time 1758741792.43 total 27.48 last 536.86) - /unlink 20000 (time 1758741800.44 total 35.49 last 578.25) - /unlink 23895 (time 1758741810.44 total 45.49 last 389.48) - /unlink 29055 (time 1758741820.45 total 55.50 last 515.69) - /unlink 33757 (time 1758741830.45 total 65.50 last 470.16) - /unlink 37688 (time 1758741840.45 total 75.50 last 393.04) - /unlink 40000 (time 1758741846.50 total 81.55 last 382.43) - /unlink 44093 (time 1758741856.50 total 91.55 last 409.26) - /unlink 48330 (time 1758741866.51 total 101.55 last 423.32) - /unlink 50000 (time 1758741869.92 total 104.97 last 488.70) - /unlink 55321 (time 1758741879.93 total 114.98 last 531.89) - /unlink 59287 (time 1758741889.93 total 124.98 last 396.57) - /unlink 63583 (time 1758741899.93 total 134.98 last 429.56) total: 64768 /unlink in 138.36 seconds: 468.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg421-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 PASS 106 (4590s) Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 oleg421-server: oleg421-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4751 sec ========== 15:26:05 (1758741965) === conf-sanity: start cleanup 15:26:06 (1758741966) === === conf-sanity: finish cleanup 15:26:07 (1758741967) ===