-----============= acceptance-small: conf-sanity ============----- Thu Oct 23 09:55:38 EDT 2025 oleg441-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg441-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg441-server oleg441-server: oleg441-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: detected 4 online CPUs by sysfs oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg441-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg441-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg441-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg441-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a == conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:58:55 (1761227935) Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) oleg441-server: oleg441-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg441-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg441-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 879 (time 1761228061.13 total 10.01 last 87.81) - open/close 1884 (time 1761228071.13 total 20.02 last 100.45) - open/close 2750 (time 1761228081.15 total 30.03 last 86.50) - open/close 3685 (time 1761228091.16 total 40.04 last 93.40) - open/close 4470 (time 1761228101.16 total 50.04 last 78.49) - open/close 5322 (time 1761228111.16 total 60.04 last 85.18) - open/close 6226 (time 1761228121.16 total 70.04 last 90.39) - open/close 7038 (time 1761228131.17 total 80.05 last 81.16) - open/close 7811 (time 1761228141.18 total 90.06 last 77.22) - open/close 8628 (time 1761228151.19 total 100.07 last 81.61) - open/close 9519 (time 1761228161.20 total 110.08 last 89.03) total: 10000 open/close in 115.45 seconds: 86.61 ops/second - unlinked 0 (time 1761228169 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 948 (time 1761228252.39 total 10.00 last 94.79) - open/close 1869 (time 1761228262.42 total 20.02 last 91.88) - open/close 2723 (time 1761228272.43 total 30.04 last 85.26) - open/close 3362 (time 1761228282.44 total 40.05 last 63.87) - open/close 4080 (time 1761228292.46 total 50.06 last 71.67) - open/close 4900 (time 1761228302.46 total 60.07 last 81.97) - open/close 5693 (time 1761228312.46 total 70.07 last 79.29) - open/close 6509 (time 1761228322.46 total 80.07 last 81.58) - open/close 7401 (time 1761228332.48 total 90.08 last 89.09) - open/close 8266 (time 1761228342.48 total 100.09 last 86.45) - open/close 9186 (time 1761228352.49 total 110.10 last 91.90) total: 10000 open/close in 119.48 seconds: 83.69 ops/second - unlinked 0 (time 1761228364 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 782 (time 1761228448.58 total 10.00 last 78.16) - open/close 1575 (time 1761228458.59 total 20.01 last 79.29) - open/close 2490 (time 1761228468.59 total 30.01 last 91.44) - open/close 3397 (time 1761228478.60 total 40.02 last 90.60) - open/close 4223 (time 1761228488.60 total 50.02 last 82.59) - open/close 5041 (time 1761228498.61 total 60.03 last 81.73) - open/close 5812 (time 1761228508.62 total 70.04 last 77.03) - open/close 6564 (time 1761228518.63 total 80.05 last 75.17) - open/close 7355 (time 1761228528.63 total 90.05 last 79.06) - open/close 8205 (time 1761228538.64 total 100.05 last 84.97) - open/close 8986 (time 1761228548.65 total 110.07 last 78.00) - open/close 9795 (time 1761228558.66 total 120.08 last 80.83) total: 10000 open/close in 122.44 seconds: 81.67 ops/second - unlinked 0 (time 1761228564 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 907 (time 1761228648.79 total 10.00 last 90.69) - open/close 1807 (time 1761228658.79 total 20.00 last 89.96) - open/close 2748 (time 1761228668.80 total 30.01 last 94.03) - open/close 3577 (time 1761228678.82 total 40.03 last 82.76) - open/close 4589 (time 1761228688.82 total 50.03 last 101.17) - open/close 5573 (time 1761228698.83 total 60.04 last 98.33) - open/close 6413 (time 1761228708.84 total 70.05 last 83.91) - open/close 7231 (time 1761228718.85 total 80.06 last 81.74) - open/close 8052 (time 1761228728.85 total 90.06 last 82.08) - open/close 8778 (time 1761228738.87 total 100.08 last 72.45) - open/close 9572 (time 1761228748.87 total 110.08 last 79.37) total: 10000 open/close in 114.58 seconds: 87.28 ops/second - unlinked 0 (time 1761228756 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 899 (time 1761228837.08 total 10.00 last 89.90) - open/close 1815 (time 1761228847.09 total 20.01 last 91.51) - open/close 2620 (time 1761228857.10 total 30.02 last 80.45) - open/close 3407 (time 1761228867.10 total 40.02 last 78.69) - open/close 4143 (time 1761228877.11 total 50.03 last 73.52) - open/close 4908 (time 1761228887.11 total 60.03 last 76.49) - open/close 5721 (time 1761228897.11 total 70.03 last 81.28) - open/close 6509 (time 1761228907.13 total 80.05 last 78.65) - open/close 7295 (time 1761228917.13 total 90.06 last 78.56) - open/close 8132 (time 1761228927.14 total 100.06 last 83.68) - open/close 9007 (time 1761228937.14 total 110.07 last 87.44) - open/close 9911 (time 1761228947.15 total 120.07 last 90.39) total: 10000 open/close in 121.09 seconds: 82.58 ops/second - unlinked 0 (time 1761228951 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 867 (time 1761229033.31 total 10.01 last 86.59) - open/close 1639 (time 1761229043.32 total 20.03 last 77.09) - open/close 2487 (time 1761229053.33 total 30.03 last 84.79) - open/close 3281 (time 1761229063.33 total 40.03 last 79.38) - open/close 4014 (time 1761229073.34 total 50.04 last 73.23) - open/close 4799 (time 1761229083.35 total 60.05 last 78.40) - open/close 5594 (time 1761229093.36 total 70.06 last 79.45) - open/close 6675 (time 1761229103.36 total 80.06 last 108.05) - open/close 7628 (time 1761229113.37 total 90.08 last 95.18) - open/close 8472 (time 1761229123.39 total 100.09 last 84.25) - open/close 9281 (time 1761229133.39 total 110.09 last 80.89) total: 10000 open/close in 118.11 seconds: 84.67 ops/second - unlinked 0 (time 1761229144 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 789 (time 1761229226.28 total 10.00 last 78.88) - open/close 1598 (time 1761229236.28 total 20.01 last 80.85) - open/close 2406 (time 1761229246.30 total 30.02 last 80.69) - open/close 3232 (time 1761229256.30 total 40.02 last 82.59) - open/close 4116 (time 1761229266.30 total 50.03 last 88.37) - open/close 5029 (time 1761229276.30 total 60.03 last 91.28) - open/close 5950 (time 1761229286.32 total 70.05 last 91.94) - open/close 6837 (time 1761229296.33 total 80.05 last 88.63) - open/close 7711 (time 1761229306.34 total 90.06 last 87.34) - open/close 8376 (time 1761229316.35 total 100.08 last 66.41) - open/close 9020 (time 1761229326.36 total 110.09 last 64.33) - open/close 9781 (time 1761229336.36 total 120.09 last 76.09) total: 10000 open/close in 122.75 seconds: 81.47 ops/second - unlinked 0 (time 1761229342 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 850 (time 1761229420.08 total 10.00 last 84.96) - open/close 1669 (time 1761229430.08 total 20.01 last 81.85) - open/close 2432 (time 1761229440.10 total 30.03 last 76.16) - open/close 3223 (time 1761229450.10 total 40.03 last 79.10) - open/close 4034 (time 1761229460.10 total 50.03 last 81.09) - open/close 4783 (time 1761229470.12 total 60.05 last 74.74) - open/close 5488 (time 1761229480.14 total 70.06 last 70.41) - open/close 6158 (time 1761229490.16 total 80.09 last 66.84) - open/close 6911 (time 1761229500.16 total 90.09 last 75.27) - open/close 7656 (time 1761229510.17 total 100.10 last 74.44) - open/close 8548 (time 1761229520.18 total 110.11 last 89.11) - open/close 9332 (time 1761229530.18 total 120.11 last 78.39) total: 10000 open/close in 130.10 seconds: 76.86 ops/second - unlinked 0 (time 1761229543 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 788 (time 1761229629.82 total 10.00 last 78.79) - open/close 1432 (time 1761229639.83 total 20.01 last 64.35) - open/close 2207 (time 1761229649.83 total 30.01 last 77.47) - open/close 3017 (time 1761229659.83 total 40.01 last 81.00) - open/close 3823 (time 1761229669.83 total 50.01 last 80.60) - open/close 4672 (time 1761229679.83 total 60.02 last 84.87) - open/close 5494 (time 1761229689.85 total 70.03 last 82.07) - open/close 6319 (time 1761229699.86 total 80.04 last 82.45) - open/close 7170 (time 1761229709.86 total 90.04 last 85.08) - open/close 8066 (time 1761229719.86 total 100.04 last 89.57) - open/close 8918 (time 1761229729.86 total 110.05 last 85.18) - open/close 9757 (time 1761229739.87 total 120.05 last 83.86) total: 10000 open/close in 122.93 seconds: 81.35 ops/second - unlinked 0 (time 1761229746 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 808 (time 1761229828.68 total 10.01 last 80.71) - open/close 1437 (time 1761229838.69 total 20.03 last 62.81) - open/close 2173 (time 1761229848.70 total 30.03 last 73.57) - open/close 2988 (time 1761229858.71 total 40.04 last 81.42) - open/close 3754 (time 1761229868.72 total 50.05 last 76.49) - open/close 4596 (time 1761229878.72 total 60.06 last 84.18) - open/close 5460 (time 1761229888.73 total 70.06 last 86.35) - open/close 6198 (time 1761229898.74 total 80.08 last 73.71) - open/close 6890 (time 1761229908.75 total 90.08 last 69.16) - open/close 7679 (time 1761229918.75 total 100.09 last 78.86) - open/close 8505 (time 1761229928.76 total 110.09 last 82.54) - open/close 9369 (time 1761229938.77 total 120.11 last 86.29) total: 10000 open/close in 127.41 seconds: 78.49 ops/second - unlinked 0 (time 1761229949 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 669 (time 1761230039.75 total 10.01 last 66.81) - open/close 1336 (time 1761230049.77 total 20.03 last 66.57) - open/close 2122 (time 1761230059.78 total 30.04 last 78.52) - open/close 2994 (time 1761230069.80 total 40.06 last 87.08) - open/close 3830 (time 1761230079.80 total 50.06 last 83.59) - open/close 4684 (time 1761230089.82 total 60.08 last 85.22) - open/close 5510 (time 1761230099.82 total 70.08 last 82.58) - open/close 6344 (time 1761230109.83 total 80.09 last 83.35) - open/close 7161 (time 1761230119.84 total 90.10 last 81.57) - open/close 7967 (time 1761230129.85 total 100.11 last 80.56) - open/close 8794 (time 1761230139.86 total 110.12 last 82.65) - open/close 9603 (time 1761230149.87 total 120.13 last 80.78) total: 10000 open/close in 126.08 seconds: 79.32 ops/second - unlinked 0 (time 1761230160 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4224 377344 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268027 s, 3.9 MB/s fail_loc=0x80000204 Failing ost1 on oleg441-server Stopping /mnt/lustre-ost1 (opts:) on oleg441-server reboot facets: ost1 Failover ost1 to oleg441-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.57 seconds: 38.86 ops/second umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local oleg441-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg441-server: LNET ready to unload oleg441-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. PASS 122b (2399s) SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:39:08 (1761230348) start mds service on oleg441-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg441-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg441-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre oleg441-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (226s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F == conf-sanity test 124: check failover after replace_nids ========================================================== 10:42:56 (1761230576) SKIP: conf-sanity test_124 needs MDT failover setup SKIP 124 (3s) == conf-sanity test 125: check l_tunedisk only tunes OSTs and their slave devices ========================================================== 10:42:59 (1761230579) SKIP: conf-sanity test_125 ldiskfs only test SKIP 125 (4s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:43:05 (1761230585) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (3s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:43:08 (1761230588) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (3s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:43:13 (1761230593) Checking servers environments Checking clients oleg441-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: mount.lustre: according to /etc/mtab lustre-mdt2/mdt2 is already mounted on /mnt/lustre-mds2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 17 Start of lustre-mdt2/mdt2 on mds2 failed 17 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg441-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 17 Start of lustre-ost1/ost1 on ost1 failed 17 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab oleg441-server@tcp:/lustre is already mounted on /mnt/lustre Starting client oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre Started clients oleg441-client.virtnet: 192.168.204.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d606054a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d606054a800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server oleg441-server: oleg441-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg441-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg441-client: oleg441-server: ssh exited with exit code 19 checking for existing Lustre data: not found start mds service on oleg441-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0001 PASS 130 (89s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 10:44:42 (1761230682) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (3s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg441-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg441-server oleg441-server: oleg441-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3044 sec ========== 10:46:24 (1761230784)