-----============= acceptance-small: conf-sanity ============----- Wed Oct 22 04:22:21 EDT 2025 oleg124-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg124-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg124-server oleg124-server: oleg124-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg124-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg124-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg124-server: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg124-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a == conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 04:23:43 (1761121423) Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) oleg124-server: oleg124-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg124-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg124-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock oleg124-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 3568 (time 1761121480.96 total 10.00 last 356.67) - open/close 6497 (time 1761121490.96 total 20.00 last 292.86) - open/close 9802 (time 1761121500.97 total 30.01 last 330.41) total: 10000 open/close in 30.60 seconds: 326.75 ops/second - unlinked 0 (time 1761121503 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3036 (time 1761121532.77 total 10.00 last 303.58) - open/close 5613 (time 1761121542.77 total 20.00 last 257.70) - open/close 7906 (time 1761121552.77 total 30.00 last 229.29) total: 10000 open/close in 38.94 seconds: 256.78 ops/second - unlinked 0 (time 1761121563 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1203 (time 1761121602.83 total 10.01 last 120.22) - open/close 2381 (time 1761121612.84 total 20.01 last 117.71) - open/close 3600 (time 1761121622.84 total 30.02 last 121.86) - open/close 5247 (time 1761121632.86 total 40.03 last 164.45) - open/close 7096 (time 1761121642.87 total 50.04 last 184.72) - open/close 8631 (time 1761121652.87 total 60.05 last 153.45) total: 10000 open/close in 67.61 seconds: 147.90 ops/second - unlinked 0 (time 1761121662 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2374 (time 1761121699.18 total 10.00 last 237.35) - open/close 5015 (time 1761121709.18 total 20.00 last 264.09) - open/close 7337 (time 1761121719.19 total 30.01 last 232.12) - open/close 9748 (time 1761121729.19 total 40.01 last 241.07) total: 10000 open/close in 41.11 seconds: 243.26 ops/second - unlinked 0 (time 1761121732 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2517 (time 1761121769.10 total 10.00 last 251.69) - open/close 5292 (time 1761121779.10 total 20.00 last 277.43) - open/close 7765 (time 1761121789.10 total 30.00 last 247.25) - open/close 9826 (time 1761121799.11 total 40.01 last 205.96) total: 10000 open/close in 40.83 seconds: 244.93 ops/second - unlinked 0 (time 1761121801 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2128 (time 1761121839.74 total 10.00 last 212.79) - open/close 4413 (time 1761121849.75 total 20.00 last 228.44) - open/close 6346 (time 1761121859.75 total 30.01 last 193.19) - open/close 8732 (time 1761121869.75 total 40.01 last 238.53) total: 10000 open/close in 45.57 seconds: 219.43 ops/second - unlinked 0 (time 1761121877 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2173 (time 1761121909.33 total 10.01 last 217.19) - open/close 4719 (time 1761121919.33 total 20.01 last 254.53) - open/close 6436 (time 1761121929.33 total 30.01 last 171.66) - open/close 8918 (time 1761121939.34 total 40.02 last 248.08) total: 10000 open/close in 44.86 seconds: 222.92 ops/second - unlinked 0 (time 1761121946 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2342 (time 1761121983.64 total 10.00 last 234.15) - open/close 4463 (time 1761121993.64 total 20.01 last 211.97) - open/close 6609 (time 1761122003.64 total 30.01 last 214.58) - open/close 9186 (time 1761122013.65 total 40.01 last 257.62) total: 10000 open/close in 43.17 seconds: 231.65 ops/second - unlinked 0 (time 1761122018 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 805 (time 1761122113.85 total 10.01 last 80.41) - open/close 1670 (time 1761122123.86 total 20.02 last 86.45) - open/close 2674 (time 1761122133.86 total 30.02 last 100.37) - open/close 3637 (time 1761122143.86 total 40.02 last 96.29) - open/close 4458 (time 1761122153.87 total 50.03 last 82.01) - open/close 5358 (time 1761122163.87 total 60.04 last 89.96) - open/close 6121 (time 1761122173.88 total 70.04 last 76.23) - open/close 6991 (time 1761122183.89 total 80.05 last 86.99) - open/close 7833 (time 1761122193.89 total 90.05 last 84.17) - open/close 8724 (time 1761122203.89 total 100.05 last 89.09) - open/close 9727 (time 1761122213.89 total 110.05 last 100.26) total: 10000 open/close in 112.48 seconds: 88.90 ops/second - unlinked 0 (time 1761122218 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1018 (time 1761122293.01 total 10.00 last 101.79) - open/close 1886 (time 1761122303.02 total 20.01 last 86.72) - open/close 2715 (time 1761122313.02 total 30.01 last 82.86) - open/close 3529 (time 1761122323.03 total 40.02 last 81.34) - open/close 4305 (time 1761122333.04 total 50.03 last 77.54) - open/close 4836 (time 1761122343.09 total 60.08 last 52.84) - open/close 5241 (time 1761122353.10 total 70.09 last 40.46) - open/close 5664 (time 1761122363.12 total 80.11 last 42.23) - open/close 6034 (time 1761122373.14 total 90.13 last 36.93) - open/close 6447 (time 1761122383.15 total 100.14 last 41.25) - open/close 6855 (time 1761122393.17 total 110.17 last 40.69) - open/close 7412 (time 1761122403.20 total 120.19 last 55.57) - open/close 7989 (time 1761122413.25 total 130.25 last 57.38) - open/close 8587 (time 1761122423.26 total 140.25 last 59.74) - open/close 9210 (time 1761122433.26 total 150.26 last 62.29) - open/close 9851 (time 1761122443.28 total 160.27 last 64.02) total: 10000 open/close in 162.36 seconds: 61.59 ops/second - unlinked 0 (time 1761122449 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 630 (time 1761122556.46 total 10.00 last 62.99) - open/close 1268 (time 1761122566.46 total 20.01 last 63.76) - open/close 2038 (time 1761122576.46 total 30.01 last 77.00) - open/close 2743 (time 1761122586.48 total 40.02 last 70.41) - open/close 3389 (time 1761122596.48 total 50.03 last 64.57) - open/close 4056 (time 1761122606.48 total 60.03 last 66.68) - open/close 4740 (time 1761122616.48 total 70.03 last 68.40) - open/close 5411 (time 1761122626.49 total 80.03 last 67.08) - open/close 6032 (time 1761122636.51 total 90.06 last 61.95) - open/close 6647 (time 1761122646.52 total 100.07 last 61.42) - open/close 7276 (time 1761122656.54 total 110.08 last 62.83) - open/close 7930 (time 1761122666.54 total 120.09 last 65.38) - open/close 8542 (time 1761122676.56 total 130.11 last 61.08) - open/close 9130 (time 1761122686.57 total 140.12 last 58.72) - open/close 9743 (time 1761122696.58 total 150.13 last 61.24) total: 10000 open/close in 154.56 seconds: 64.70 ops/second - unlinked 0 (time 1761122705 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389467 s, 2.7 MB/s fail_loc=0x80000204 Failing ost1 on oleg124-server Stopping /mnt/lustre-ost1 (opts:) on oleg124-server reboot facets: ost1 Failover ost1 to oleg124-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761122872.73 total 35.96 last 0.00) total: 100 open/close in 38.83 seconds: 2.58 ops/second umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg124-server: LNET ready to unload oleg124-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. PASS 122b (1495s) SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:48:50 (1761122930) start mds service on oleg124-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg124-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock oleg124-server@tcp:/lustre /mnt/lustre oleg124-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (162s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F == conf-sanity test 124: check failover after replace_nids ========================================================== 04:51:33 (1761123093) SKIP: conf-sanity test_124 needs >= 2 MDTs SKIP 124 (2s) == conf-sanity test 125: check l_tunedisk only tunes OSTs and their slave devices ========================================================== 04:51:35 (1761123095) SKIP: conf-sanity test_125 ldiskfs only test SKIP 125 (2s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:51:38 (1761123098) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (3s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:51:41 (1761123101) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (2s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:51:44 (1761123104) SKIP: conf-sanity test_130 needs >= 2 MDTs SKIP 130 (3s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 04:51:47 (1761123107) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (3s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg124-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server oleg124-server: oleg124-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1831 sec ========== 04:52:53 (1761123173)