-----============= acceptance-small: conf-sanity ============----- Thu Oct 23 13:11:13 EDT 2025 oleg449-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg449-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg449-server oleg449-server: oleg449-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg449-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg449-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg449-server: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg449-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg449-server: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg449-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a == conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 13:13:34 (1761239614) Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) oleg449-server: oleg449-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg449-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg449-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock oleg449-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 972 (time 1761239726.56 total 10.01 last 97.13) - open/close 1926 (time 1761239736.56 total 20.01 last 95.39) - open/close 2832 (time 1761239746.57 total 30.01 last 90.54) - open/close 3704 (time 1761239756.57 total 40.02 last 87.16) - open/close 4626 (time 1761239766.59 total 50.03 last 92.08) - open/close 5620 (time 1761239776.59 total 60.04 last 99.35) - open/close 6556 (time 1761239786.59 total 70.04 last 93.59) - open/close 7542 (time 1761239796.60 total 80.05 last 98.52) - open/close 8511 (time 1761239806.60 total 90.05 last 96.89) - open/close 9458 (time 1761239816.61 total 100.06 last 94.63) total: 10000 open/close in 105.33 seconds: 94.94 ops/second - unlinked 0 (time 1761239824 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1306 (time 1761239888.42 total 10.00 last 130.58) - open/close 2528 (time 1761239898.42 total 20.00 last 122.17) - open/close 3832 (time 1761239908.43 total 30.01 last 130.35) - open/close 5147 (time 1761239918.43 total 40.02 last 131.39) - open/close 6650 (time 1761239928.44 total 50.02 last 150.29) - open/close 7659 (time 1761239938.44 total 60.02 last 100.89) - open/close 8615 (time 1761239948.44 total 70.02 last 95.59) - open/close 9509 (time 1761239958.44 total 80.02 last 89.36) total: 10000 open/close in 86.20 seconds: 116.01 ops/second - unlinked 0 (time 1761239968 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 749 (time 1761240036.88 total 10.01 last 74.84) - open/close 1543 (time 1761240046.89 total 20.01 last 79.35) - open/close 2289 (time 1761240056.89 total 30.01 last 74.60) - open/close 3053 (time 1761240066.90 total 40.02 last 76.32) - open/close 3914 (time 1761240076.91 total 50.03 last 86.02) - open/close 4708 (time 1761240086.91 total 60.04 last 79.38) - open/close 5609 (time 1761240096.92 total 70.04 last 90.02) - open/close 6390 (time 1761240106.92 total 80.04 last 78.10) - open/close 7153 (time 1761240116.92 total 90.05 last 76.30) - open/close 7879 (time 1761240126.92 total 100.05 last 72.60) - open/close 8608 (time 1761240136.93 total 110.06 last 72.81) - open/close 9337 (time 1761240146.93 total 120.06 last 72.89) total: 10000 open/close in 128.79 seconds: 77.65 ops/second - unlinked 0 (time 1761240158 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 783 (time 1761240235.80 total 10.00 last 78.27) - open/close 1596 (time 1761240245.81 total 20.01 last 81.23) - open/close 2347 (time 1761240255.83 total 30.03 last 74.97) - open/close 3116 (time 1761240265.83 total 40.03 last 76.86) - open/close 3857 (time 1761240275.85 total 50.05 last 73.99) - open/close 4689 (time 1761240285.86 total 60.06 last 83.11) - open/close 5476 (time 1761240295.86 total 70.06 last 78.69) - open/close 6293 (time 1761240305.86 total 80.06 last 81.68) - open/close 7085 (time 1761240315.87 total 90.07 last 79.18) - open/close 7899 (time 1761240325.88 total 100.08 last 81.30) - open/close 8761 (time 1761240335.88 total 110.08 last 86.19) - open/close 9599 (time 1761240345.88 total 120.08 last 83.77) total: 10000 open/close in 124.83 seconds: 80.11 ops/second - unlinked 0 (time 1761240353 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 736 (time 1761240428.39 total 10.01 last 73.56) - open/close 1478 (time 1761240438.39 total 20.01 last 74.15) - open/close 2214 (time 1761240448.42 total 30.03 last 73.44) - open/close 2985 (time 1761240458.43 total 40.05 last 77.00) - open/close 3732 (time 1761240468.43 total 50.05 last 74.68) - open/close 4414 (time 1761240478.44 total 60.06 last 68.15) - open/close 5143 (time 1761240488.44 total 70.06 last 72.88) - open/close 5845 (time 1761240498.45 total 80.07 last 70.15) - open/close 6563 (time 1761240508.47 total 90.09 last 71.66) - open/close 7359 (time 1761240518.47 total 100.09 last 79.57) - open/close 8139 (time 1761240528.48 total 110.10 last 77.95) - open/close 8842 (time 1761240538.50 total 120.12 last 70.14) - open/close 9561 (time 1761240548.50 total 130.12 last 71.89) total: 10000 open/close in 136.59 seconds: 73.21 ops/second - unlinked 0 (time 1761240558 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 560 (time 1761240650.84 total 10.00 last 55.97) - open/close 1361 (time 1761240660.84 total 20.01 last 80.06) - open/close 2148 (time 1761240670.85 total 30.02 last 78.66) - open/close 2984 (time 1761240680.86 total 40.02 last 83.52) - open/close 3724 (time 1761240690.86 total 50.03 last 73.99) - open/close 4581 (time 1761240700.86 total 60.03 last 85.68) - open/close 5397 (time 1761240710.86 total 70.03 last 81.58) - open/close 6141 (time 1761240720.87 total 80.04 last 74.34) - open/close 6964 (time 1761240730.88 total 90.05 last 82.22) - open/close 7778 (time 1761240740.89 total 100.06 last 81.31) - open/close 8613 (time 1761240750.90 total 110.07 last 83.44) - open/close 9469 (time 1761240760.91 total 120.07 last 85.54) total: 10000 open/close in 127.87 seconds: 78.20 ops/second - unlinked 0 (time 1761240772 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 779 (time 1761240859.33 total 10.01 last 77.83) - open/close 1618 (time 1761240869.33 total 20.02 last 83.83) - open/close 2453 (time 1761240879.34 total 30.02 last 83.47) - open/close 3238 (time 1761240889.34 total 40.02 last 78.49) - open/close 4091 (time 1761240899.36 total 50.04 last 85.14) - open/close 4938 (time 1761240909.37 total 60.05 last 84.64) - open/close 5808 (time 1761240919.37 total 70.05 last 86.99) - open/close 6522 (time 1761240929.37 total 80.05 last 71.39) - open/close 7289 (time 1761240939.37 total 90.05 last 76.68) - open/close 8163 (time 1761240949.38 total 100.06 last 87.32) - open/close 8966 (time 1761240959.39 total 110.07 last 80.24) - open/close 9798 (time 1761240969.40 total 120.08 last 83.13) total: 10000 open/close in 122.45 seconds: 81.67 ops/second - unlinked 0 (time 1761240975 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 869 (time 1761241057.57 total 10.01 last 86.82) - open/close 1737 (time 1761241067.57 total 20.01 last 86.79) - open/close 2656 (time 1761241077.59 total 30.03 last 91.75) - open/close 3488 (time 1761241087.59 total 40.03 last 83.19) - open/close 4315 (time 1761241097.59 total 50.03 last 82.67) - open/close 5171 (time 1761241107.60 total 60.04 last 85.53) - open/close 5997 (time 1761241117.61 total 70.05 last 82.53) - open/close 6851 (time 1761241127.61 total 80.05 last 85.40) - open/close 7713 (time 1761241137.61 total 90.05 last 86.19) - open/close 8611 (time 1761241147.61 total 100.05 last 89.79) - open/close 9399 (time 1761241157.62 total 110.06 last 78.76) total: 10000 open/close in 117.68 seconds: 84.98 ops/second - unlinked 0 (time 1761241168 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 856 (time 1761241251.07 total 10.01 last 85.55) - open/close 1727 (time 1761241261.08 total 20.01 last 87.06) - open/close 2560 (time 1761241271.08 total 30.02 last 83.23) - open/close 3263 (time 1761241281.09 total 40.03 last 70.25) - open/close 4058 (time 1761241291.09 total 50.03 last 79.48) - open/close 4963 (time 1761241301.10 total 60.03 last 90.49) - open/close 5903 (time 1761241311.10 total 70.03 last 93.96) - open/close 6811 (time 1761241321.11 total 80.04 last 90.75) - open/close 7692 (time 1761241331.11 total 90.05 last 88.02) - open/close 8630 (time 1761241341.12 total 100.06 last 93.72) - open/close 9531 (time 1761241351.12 total 110.06 last 90.10) total: 10000 open/close in 115.77 seconds: 86.38 ops/second - unlinked 0 (time 1761241360 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 947 (time 1761241433.25 total 10.01 last 94.65) - open/close 1889 (time 1761241443.27 total 20.02 last 94.06) - open/close 2741 (time 1761241453.27 total 30.02 last 85.19) - open/close 3567 (time 1761241463.27 total 40.02 last 82.57) - open/close 4516 (time 1761241473.28 total 50.03 last 94.83) - open/close 5425 (time 1761241483.28 total 60.03 last 90.88) - open/close 6378 (time 1761241493.29 total 70.04 last 95.22) - open/close 7227 (time 1761241503.30 total 80.05 last 84.83) - open/close 8092 (time 1761241513.31 total 90.06 last 86.39) - open/close 9011 (time 1761241523.32 total 100.07 last 91.82) - open/close 9963 (time 1761241533.32 total 110.08 last 95.14) total: 10000 open/close in 110.47 seconds: 90.52 ops/second - unlinked 0 (time 1761241536 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1086 (time 1761241610.75 total 10.01 last 108.47) - open/close 2016 (time 1761241620.76 total 20.01 last 92.98) - open/close 3008 (time 1761241630.76 total 30.02 last 99.19) - open/close 3902 (time 1761241640.77 total 40.02 last 89.33) - open/close 4833 (time 1761241650.77 total 50.03 last 93.06) - open/close 5749 (time 1761241660.78 total 60.03 last 91.53) - open/close 6733 (time 1761241670.78 total 70.04 last 98.40) - open/close 7753 (time 1761241680.78 total 80.04 last 101.95) - open/close 8712 (time 1761241690.78 total 90.04 last 95.89) - open/close 9721 (time 1761241700.79 total 100.05 last 100.85) total: 10000 open/close in 103.05 seconds: 97.04 ops/second - unlinked 0 (time 1761241706 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5120 376448 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225939 s, 4.6 MB/s fail_loc=0x80000204 Failing ost1 on oleg449-server Stopping /mnt/lustre-ost1 (opts:) on oleg449-server reboot facets: ost1 Failover ost1 to oleg449-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761241843.11 total 33.49 last 0.00) total: 100 open/close in 35.22 seconds: 2.84 ops/second umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local oleg449-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg449-server: LNET ready to unload oleg449-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. PASS 122b (2265s) SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:51:27 (1761241887) start mds service on oleg449-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions oleg449-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg449-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg449-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock oleg449-server@tcp:/lustre /mnt/lustre oleg449-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (151s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F == conf-sanity test 124: check failover after replace_nids ========================================================== 13:54:00 (1761242040) SKIP: conf-sanity test_124 needs MDT failover setup SKIP 124 (2s) == conf-sanity test 125: check l_tunedisk only tunes OSTs and their slave devices ========================================================== 13:54:03 (1761242043) SKIP: conf-sanity test_125 ldiskfs only test SKIP 125 (2s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:54:07 (1761242047) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (2s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:54:09 (1761242049) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (2s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:54:12 (1761242052) Checking servers environments Checking clients oleg449-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg449-server: mount.lustre: according to /etc/mtab lustre-mdt2/mdt2 is already mounted on /mnt/lustre-mds2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 17 Start of lustre-mdt2/mdt2 on mds2 failed 17 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg449-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 17 Start of lustre-ost1/ost1 on ost1 failed 17 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock oleg449-server@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab oleg449-server@tcp:/lustre is already mounted on /mnt/lustre Starting client oleg449-client.virtnet: -o user_xattr,flock oleg449-server@tcp:/lustre /mnt/lustre Started clients oleg449-client.virtnet: 192.168.204.149@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97d304e82800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97d304e82800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 3s: want 'procname_uid' got 'procname_uid' disable quota as required stop mds service on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server oleg449-server: oleg449-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg449-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg449-client: oleg449-server: ssh exited with exit code 19 checking for existing Lustre data: not found start mds service on oleg449-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0001 PASS 130 (61s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 13:55:14 (1761242114) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (3s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg449-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg449-server oleg449-server: oleg449-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: detected 4 online CPUs by sysfs oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg449-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2715 sec ========== 13:56:29 (1761242189)