-----============= acceptance-small: conf-sanity ============----- Fri Jun 12 12:09:41 EDT 2026 oleg420-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg420-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg420-server oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg420-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg420-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server == conf-sanity test 154: expand .. on rename after MDT backup restore ========================================================== 12:13:52 (1781280832) SKIP: conf-sanity test_154 ldiskfs only test SKIP 154 (5s) SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a == conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 12:14:02 (1781280842) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg420-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg420-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 614 (time 1781281028.31 total 10.00 last 61.38) - open/close 1311 (time 1781281038.32 total 20.01 last 69.68) - open/close 1937 (time 1781281048.34 total 30.03 last 62.47) - open/close 2559 (time 1781281058.35 total 40.04 last 62.13) - open/close 3192 (time 1781281068.38 total 50.07 last 63.10) - open/close 3827 (time 1781281078.39 total 60.08 last 63.45) - open/close 4447 (time 1781281088.40 total 70.09 last 61.91) - open/close 5118 (time 1781281098.41 total 80.10 last 67.06) - open/close 5703 (time 1781281108.42 total 90.11 last 58.41) - open/close 6273 (time 1781281118.44 total 100.13 last 56.90) - open/close 6908 (time 1781281128.44 total 110.14 last 63.48) - open/close 7563 (time 1781281138.46 total 120.15 last 65.43) - open/close 8231 (time 1781281148.46 total 130.15 last 66.77) - open/close 8888 (time 1781281158.47 total 140.16 last 65.64) - open/close 9481 (time 1781281168.47 total 150.16 last 59.30) total: 10000 open/close in 159.22 seconds: 62.80 ops/second - unlinked 0 (time 1781281182 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 664 (time 1781281296.40 total 10.01 last 66.32) - open/close 1264 (time 1781281306.43 total 20.04 last 59.85) - open/close 1882 (time 1781281316.43 total 30.04 last 61.80) - open/close 2565 (time 1781281326.44 total 40.05 last 68.23) - open/close 3255 (time 1781281336.45 total 50.06 last 68.94) - open/close 3861 (time 1781281346.48 total 60.09 last 60.42) - open/close 4474 (time 1781281356.49 total 70.10 last 61.20) - open/close 5091 (time 1781281366.49 total 80.10 last 61.70) - open/close 5711 (time 1781281376.51 total 90.12 last 61.90) - open/close 6325 (time 1781281386.52 total 100.13 last 61.34) - open/close 6995 (time 1781281396.53 total 110.14 last 66.96) - open/close 7597 (time 1781281406.53 total 120.14 last 60.19) - open/close 8182 (time 1781281416.54 total 130.15 last 58.45) - open/close 8780 (time 1781281426.54 total 140.15 last 59.76) - open/close 9363 (time 1781281436.55 total 150.16 last 58.25) - open/close 9990 (time 1781281446.56 total 160.17 last 62.66) total: 10000 open/close in 160.30 seconds: 62.38 ops/second - unlinked 0 (time 1781281450 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 674 (time 1781281559.24 total 10.00 last 67.38) - open/close 1395 (time 1781281569.24 total 20.01 last 72.05) - open/close 2044 (time 1781281579.25 total 30.02 last 64.83) - open/close 2743 (time 1781281589.25 total 40.02 last 69.89) - open/close 3421 (time 1781281599.26 total 50.03 last 67.74) - open/close 4128 (time 1781281609.27 total 60.03 last 70.69) - open/close 4769 (time 1781281619.27 total 70.04 last 64.07) - open/close 5413 (time 1781281629.28 total 80.05 last 64.31) - open/close 5958 (time 1781281639.29 total 90.05 last 54.49) - open/close 6569 (time 1781281649.29 total 100.06 last 61.05) - open/close 7163 (time 1781281659.30 total 110.07 last 59.36) - open/close 7739 (time 1781281669.31 total 120.07 last 57.57) - open/close 8382 (time 1781281679.31 total 130.08 last 64.25) - open/close 9023 (time 1781281689.32 total 140.09 last 64.07) - open/close 9710 (time 1781281699.32 total 150.09 last 68.69) total: 10000 open/close in 154.18 seconds: 64.86 ops/second - unlinked 0 (time 1781281707 ; total 1 ; last 1) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 500 (time 1781281810.81 total 10.01 last 49.93) - open/close 1102 (time 1781281820.83 total 20.03 last 60.10) - open/close 1763 (time 1781281830.85 total 30.05 last 65.97) - open/close 2439 (time 1781281840.86 total 40.06 last 67.55) - open/close 3127 (time 1781281850.87 total 50.07 last 68.71) - open/close 3836 (time 1781281860.88 total 60.08 last 70.85) - open/close 4355 (time 1781281870.88 total 70.08 last 51.89) - open/close 4979 (time 1781281880.88 total 80.08 last 62.39) - open/close 5681 (time 1781281890.88 total 90.09 last 70.17) - open/close 6349 (time 1781281900.90 total 100.10 last 66.70) - open/close 7043 (time 1781281910.91 total 110.11 last 69.34) - open/close 7743 (time 1781281920.92 total 120.12 last 69.90) - open/close 8420 (time 1781281930.93 total 130.13 last 67.62) - open/close 9061 (time 1781281940.95 total 140.15 last 64.03) - open/close 9667 (time 1781281950.96 total 150.16 last 60.53) total: 10000 open/close in 155.88 seconds: 64.15 ops/second - unlinked 0 (time 1781281961 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 631 (time 1781282068.64 total 10.01 last 63.03) - open/close 1271 (time 1781282078.65 total 20.02 last 63.97) - open/close 1944 (time 1781282088.65 total 30.02 last 67.29) - open/close 2602 (time 1781282098.66 total 40.03 last 65.73) - open/close 3188 (time 1781282108.66 total 50.03 last 58.57) - open/close 3763 (time 1781282118.67 total 60.04 last 57.47) - open/close 4379 (time 1781282128.67 total 70.04 last 61.58) - open/close 5024 (time 1781282138.68 total 80.05 last 64.46) - open/close 5673 (time 1781282148.69 total 90.06 last 64.84) - open/close 6302 (time 1781282158.71 total 100.08 last 62.78) - open/close 7010 (time 1781282168.72 total 110.09 last 70.69) - open/close 7649 (time 1781282178.74 total 120.11 last 63.80) - open/close 8349 (time 1781282188.74 total 130.11 last 69.99) - open/close 9002 (time 1781282198.75 total 140.11 last 65.26) - open/close 9675 (time 1781282208.75 total 150.12 last 67.24) total: 10000 open/close in 154.68 seconds: 64.65 ops/second - unlinked 0 (time 1781282217 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 602 (time 1781282322.89 total 10.01 last 60.13) - open/close 1194 (time 1781282332.89 total 20.01 last 59.20) - open/close 1838 (time 1781282342.89 total 30.02 last 64.35) - open/close 2513 (time 1781282352.90 total 40.03 last 67.44) - open/close 3183 (time 1781282362.91 total 50.03 last 66.98) - open/close 3878 (time 1781282372.92 total 60.05 last 69.41) - open/close 4513 (time 1781282382.93 total 70.06 last 63.42) - open/close 5143 (time 1781282392.95 total 80.08 last 62.87) - open/close 5710 (time 1781282402.96 total 90.09 last 56.66) - open/close 6319 (time 1781282412.96 total 100.09 last 60.90) - open/close 7004 (time 1781282422.97 total 110.09 last 68.45) - open/close 7537 (time 1781282432.98 total 120.10 last 53.24) - open/close 8158 (time 1781282442.98 total 130.11 last 62.06) - open/close 8754 (time 1781282452.98 total 140.11 last 59.59) - open/close 9429 (time 1781282463.01 total 150.14 last 67.34) total: 10000 open/close in 159.30 seconds: 62.78 ops/second - unlinked 0 (time 1781282476 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 734 (time 1781282584.22 total 10.02 last 73.27) - open/close 1450 (time 1781282594.22 total 20.02 last 71.58) - open/close 2100 (time 1781282604.24 total 30.04 last 64.88) - open/close 2769 (time 1781282614.25 total 40.04 last 66.86) - open/close 3451 (time 1781282624.25 total 50.05 last 68.17) - open/close 4114 (time 1781282634.26 total 60.06 last 66.25) - open/close 4776 (time 1781282644.26 total 70.06 last 66.17) - open/close 5398 (time 1781282654.27 total 80.07 last 62.16) - open/close 6070 (time 1781282664.28 total 90.07 last 67.16) - open/close 6653 (time 1781282674.29 total 100.09 last 58.21) - open/close 7280 (time 1781282684.29 total 110.09 last 62.69) - open/close 7993 (time 1781282694.29 total 120.09 last 71.28) - open/close 8611 (time 1781282704.31 total 130.11 last 61.69) - open/close 9261 (time 1781282714.32 total 140.12 last 64.96) - open/close 9887 (time 1781282724.32 total 150.12 last 62.56) total: 10000 open/close in 151.72 seconds: 65.91 ops/second - unlinked 0 (time 1781282730 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 703 (time 1781282831.89 total 10.01 last 70.23) - open/close 1362 (time 1781282841.89 total 20.01 last 65.89) - open/close 2032 (time 1781282851.89 total 30.01 last 67.00) - open/close 2661 (time 1781282861.89 total 40.01 last 62.88) - open/close 3393 (time 1781282871.90 total 50.02 last 73.13) - open/close 4124 (time 1781282881.91 total 60.03 last 73.04) - open/close 4874 (time 1781282891.92 total 70.04 last 74.96) - open/close 5560 (time 1781282901.92 total 80.04 last 68.58) - open/close 6235 (time 1781282911.94 total 90.05 last 67.39) - open/close 6830 (time 1781282921.95 total 100.07 last 59.42) - open/close 7485 (time 1781282931.96 total 110.08 last 65.42) - open/close 8096 (time 1781282941.97 total 120.09 last 61.03) - open/close 8706 (time 1781282951.98 total 130.10 last 60.94) - open/close 9335 (time 1781282961.99 total 140.10 last 62.87) - open/close 9973 (time 1781282971.99 total 150.10 last 63.80) total: 10000 open/close in 150.52 seconds: 66.44 ops/second - unlinked 0 (time 1781282977 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 628 (time 1781283077.81 total 10.01 last 62.76) - open/close 1185 (time 1781283087.81 total 20.01 last 55.68) - open/close 1861 (time 1781283097.84 total 30.03 last 67.45) - open/close 2570 (time 1781283107.84 total 40.03 last 70.89) - open/close 3218 (time 1781283117.84 total 50.04 last 64.76) - open/close 3948 (time 1781283127.85 total 60.04 last 72.99) - open/close 4689 (time 1781283137.85 total 70.05 last 74.04) - open/close 5482 (time 1781283147.86 total 80.06 last 79.24) - open/close 6222 (time 1781283157.89 total 90.08 last 73.80) - open/close 6901 (time 1781283167.92 total 100.11 last 67.70) - open/close 7554 (time 1781283177.92 total 110.11 last 65.28) - open/close 8198 (time 1781283187.92 total 120.12 last 64.36) - open/close 8839 (time 1781283197.94 total 130.14 last 64.00) - open/close 9518 (time 1781283207.95 total 140.14 last 67.86) total: 10000 open/close in 147.23 seconds: 67.92 ops/second - unlinked 0 (time 1781283218 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 655 (time 1781283316.58 total 10.01 last 65.45) - open/close 1343 (time 1781283326.61 total 20.03 last 68.61) - open/close 1996 (time 1781283336.62 total 30.04 last 65.24) - open/close 2649 (time 1781283346.62 total 40.05 last 65.27) - open/close 3306 (time 1781283356.63 total 50.06 last 65.62) - open/close 3896 (time 1781283366.64 total 60.06 last 58.97) - open/close 4537 (time 1781283376.65 total 70.08 last 64.01) - open/close 5198 (time 1781283386.66 total 80.09 last 66.05) - open/close 5854 (time 1781283396.68 total 90.11 last 65.44) - open/close 6495 (time 1781283406.69 total 100.12 last 64.04) - open/close 7156 (time 1781283416.71 total 110.13 last 66.00) - open/close 7798 (time 1781283426.71 total 120.14 last 64.14) - open/close 8432 (time 1781283436.73 total 130.15 last 63.33) - open/close 9069 (time 1781283446.73 total 140.16 last 63.65) - open/close 9742 (time 1781283456.74 total 150.17 last 67.27) total: 10000 open/close in 154.01 seconds: 64.93 ops/second - unlinked 0 (time 1781283464 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 674 (time 1781283574.47 total 10.00 last 67.37) - open/close 1418 (time 1781283584.48 total 20.02 last 74.31) - open/close 2108 (time 1781283594.49 total 30.03 last 68.92) - open/close 2801 (time 1781283604.49 total 40.03 last 69.29) - open/close 3492 (time 1781283614.50 total 50.04 last 69.04) - open/close 4165 (time 1781283624.51 total 60.05 last 67.21) - open/close 4796 (time 1781283634.52 total 70.06 last 63.04) - open/close 5429 (time 1781283644.53 total 80.07 last 63.27) - open/close 6018 (time 1781283654.54 total 90.08 last 58.85) - open/close 6596 (time 1781283664.55 total 100.09 last 57.74) - open/close 7217 (time 1781283674.55 total 110.09 last 62.10) - open/close 7844 (time 1781283684.56 total 120.10 last 62.63) - open/close 8514 (time 1781283694.58 total 130.12 last 66.86) - open/close 9153 (time 1781283704.59 total 140.12 last 63.85) - open/close 9813 (time 1781283714.60 total 150.14 last 65.92) total: 10000 open/close in 152.90 seconds: 65.40 ops/second - unlinked 0 (time 1781283721 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4224 377344 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4831 s, 2.2 MB/s fail_loc=0x80000204 Failing ost1 on oleg420-server Stopping /mnt/lustre-ost1 (opts:) on oleg420-server reboot facets: ost1 Failover ost1 to oleg420-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1781283913.01 total 36.07 last 0.00) total: 100 open/close in 38.76 seconds: 2.58 ops/second umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg420-server: LNET ready to unload oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. PASS 122b (3121s) SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:06:16 (1781283976) start mds service on oleg420-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg420-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre oleg420-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (260s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F == conf-sanity test 124: check failover after replace_nids ========================================================== 13:10:38 (1781284238) SKIP: conf-sanity test_124 needs MDT failover setup SKIP 124 (4s) == conf-sanity test 125: check l_tunedisk only tunes OSTs and their slave devices ========================================================== 13:10:43 (1781284243) SKIP: conf-sanity test_125 ldiskfs only test SKIP 125 (4s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:10:49 (1781284249) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (3s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:10:52 (1781284252) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (4s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:10:58 (1781284258) Checking servers environments Checking clients oleg420-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg420-server: mount.lustre: according to /etc/mtab lustre-mdt2/mdt2 is already mounted on /mnt/lustre-mds2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17 Start of lustre-mdt2/mdt2 on mds2 failed 17 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg420-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17 Start of lustre-ost1/ost1 on ost1 failed 17 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab oleg420-server@tcp:/lustre is already mounted on /mnt/lustre Starting client oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre Started clients oleg420-client.virtnet: 192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f3284024800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f3284024800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server oleg420-server: oleg420-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg420-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg420-client: oleg420-server: ssh exited with exit code 19 checking for existing Lustre data: not found start mds service on oleg420-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 PASS 130 (95s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 13:12:33 (1781284353) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (3s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg420-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg420-server oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3881 sec ========== 13:14:25 (1781284465)