-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 08:51:35 EDT 2025 oleg306-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg306-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg306-server oleg306-server: oleg306-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg306-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a == conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 08:53:07 (1761310387) Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) oleg306-server: oleg306-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg306-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock oleg306-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 1505 (time 1761310459.86 total 10.01 last 150.42) - open/close 3677 (time 1761310469.87 total 20.01 last 217.09) - open/close 6095 (time 1761310479.87 total 30.01 last 241.79) - open/close 8967 (time 1761310489.87 total 40.01 last 287.11) total: 10000 open/close in 46.24 seconds: 216.25 ops/second - unlinked 0 (time 1761310497 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1095 (time 1761310553.33 total 10.00 last 109.46) - open/close 2099 (time 1761310563.33 total 20.01 last 100.36) - open/close 3097 (time 1761310573.34 total 30.02 last 99.67) - open/close 4135 (time 1761310583.35 total 40.03 last 103.71) - open/close 5281 (time 1761310593.36 total 50.04 last 114.51) - open/close 6411 (time 1761310603.36 total 60.04 last 112.94) - open/close 7677 (time 1761310613.37 total 70.05 last 126.47) - open/close 9075 (time 1761310623.38 total 80.06 last 139.71) total: 10000 open/close in 84.98 seconds: 117.67 ops/second - unlinked 0 (time 1761310630 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 2662 (time 1761310672.70 total 10.00 last 266.17) - open/close 5285 (time 1761310682.70 total 20.00 last 262.26) - open/close 7861 (time 1761310692.70 total 30.00 last 257.58) total: 10000 open/close in 39.53 seconds: 252.99 ops/second - unlinked 0 (time 1761310703 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2009 (time 1761310741.11 total 10.00 last 200.87) - open/close 4233 (time 1761310751.11 total 20.00 last 222.35) - open/close 6900 (time 1761310761.11 total 30.01 last 266.64) - open/close 9696 (time 1761310771.11 total 40.01 last 279.58) total: 10000 open/close in 41.30 seconds: 242.13 ops/second - unlinked 0 (time 1761310774 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 931 (time 1761310813.13 total 10.00 last 93.09) - open/close 1785 (time 1761310823.14 total 20.01 last 85.36) - open/close 2551 (time 1761310833.15 total 30.02 last 76.52) - open/close 3405 (time 1761310843.15 total 40.02 last 85.36) - open/close 4083 (time 1761310853.15 total 50.02 last 67.77) - open/close 4932 (time 1761310863.16 total 60.03 last 84.83) - open/close 5842 (time 1761310873.17 total 70.04 last 90.96) - open/close 6884 (time 1761310883.17 total 80.04 last 104.19) - open/close 7935 (time 1761310893.17 total 90.04 last 105.05) - open/close 8915 (time 1761310903.17 total 100.04 last 97.98) - open/close 9981 (time 1761310913.18 total 110.05 last 106.53) total: 10000 open/close in 110.18 seconds: 90.76 ops/second - unlinked 0 (time 1761310915 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 738 (time 1761310981.71 total 10.02 last 73.66) - open/close 1557 (time 1761310991.71 total 20.02 last 81.89) - open/close 2441 (time 1761311001.71 total 30.02 last 88.36) - open/close 3350 (time 1761311011.73 total 40.04 last 90.79) - open/close 4296 (time 1761311021.73 total 50.04 last 94.59) - open/close 5202 (time 1761311031.74 total 60.05 last 90.48) - open/close 6123 (time 1761311041.75 total 70.06 last 91.98) - open/close 7098 (time 1761311051.76 total 80.07 last 97.46) - open/close 8125 (time 1761311061.76 total 90.07 last 102.65) - open/close 9344 (time 1761311071.77 total 100.08 last 121.81) total: 10000 open/close in 106.71 seconds: 93.71 ops/second - unlinked 0 (time 1761311080 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1704 (time 1761311132.50 total 10.00 last 170.39) - open/close 3705 (time 1761311142.51 total 20.00 last 200.03) - open/close 5335 (time 1761311152.51 total 30.01 last 162.93) - open/close 6991 (time 1761311162.52 total 40.01 last 165.54) - open/close 8433 (time 1761311172.52 total 50.01 last 144.15) total: 10000 open/close in 59.61 seconds: 167.75 ops/second - unlinked 0 (time 1761311184 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1386 (time 1761311228.26 total 10.02 last 138.35) - open/close 2662 (time 1761311238.26 total 20.02 last 127.53) - open/close 3805 (time 1761311248.27 total 30.03 last 114.24) - open/close 5142 (time 1761311258.27 total 40.03 last 133.70) - open/close 6484 (time 1761311268.27 total 50.03 last 134.18) - open/close 8035 (time 1761311278.27 total 60.03 last 155.04) - open/close 9300 (time 1761311288.28 total 70.04 last 126.39) - open/close 9972 (time 1761311298.30 total 80.06 last 67.11) total: 10000 open/close in 80.64 seconds: 124.01 ops/second - unlinked 0 (time 1761311304 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 880 (time 1761311383.11 total 10.00 last 87.98) - open/close 1887 (time 1761311393.11 total 20.01 last 100.64) - open/close 2601 (time 1761311403.13 total 30.03 last 71.27) - open/close 3181 (time 1761311413.14 total 40.04 last 57.95) - open/close 3692 (time 1761311423.15 total 50.05 last 51.04) - open/close 4160 (time 1761311433.16 total 60.06 last 46.75) - open/close 4697 (time 1761311443.17 total 70.06 last 53.67) - open/close 5378 (time 1761311453.17 total 80.07 last 68.07) - open/close 6193 (time 1761311463.18 total 90.08 last 81.44) - open/close 6987 (time 1761311473.18 total 100.08 last 79.39) - open/close 7846 (time 1761311483.19 total 110.09 last 85.83) - open/close 8650 (time 1761311493.20 total 120.10 last 80.32) - open/close 9518 (time 1761311503.23 total 130.12 last 86.56) total: 10000 open/close in 135.90 seconds: 73.58 ops/second - unlinked 0 (time 1761311512 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 764 (time 1761311595.02 total 10.01 last 76.29) - open/close 1534 (time 1761311605.03 total 20.03 last 76.89) - open/close 2275 (time 1761311615.05 total 30.05 last 73.97) - open/close 2967 (time 1761311625.06 total 40.05 last 69.16) - open/close 3768 (time 1761311635.06 total 50.06 last 80.05) - open/close 4580 (time 1761311645.07 total 60.07 last 81.13) - open/close 5379 (time 1761311655.07 total 70.07 last 79.88) - open/close 6128 (time 1761311665.07 total 80.07 last 74.90) - open/close 6952 (time 1761311675.08 total 90.07 last 82.37) - open/close 7742 (time 1761311685.08 total 100.07 last 79.00) - open/close 8508 (time 1761311695.08 total 110.08 last 76.57) - open/close 9261 (time 1761311705.10 total 120.10 last 75.16) total: 10000 open/close in 129.04 seconds: 77.49 ops/second - unlinked 0 (time 1761311717 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 878 (time 1761311805.64 total 10.01 last 87.75) - open/close 1757 (time 1761311815.64 total 20.01 last 87.85) - open/close 2674 (time 1761311825.66 total 30.03 last 91.56) - open/close 3531 (time 1761311835.66 total 40.03 last 85.66) - open/close 4411 (time 1761311845.67 total 50.04 last 87.92) - open/close 5218 (time 1761311855.69 total 60.06 last 80.55) - open/close 6003 (time 1761311865.70 total 70.07 last 78.38) - open/close 6909 (time 1761311875.71 total 80.08 last 90.56) - open/close 7821 (time 1761311885.71 total 90.08 last 91.17) - open/close 8675 (time 1761311895.72 total 100.09 last 85.33) - open/close 9535 (time 1761311905.72 total 110.09 last 85.98) total: 10000 open/close in 115.85 seconds: 86.31 ops/second - unlinked 0 (time 1761311915 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4352 377216 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322737 s, 3.2 MB/s fail_loc=0x80000204 Failing ost1 on oleg306-server Stopping /mnt/lustre-ost1 (opts:) on oleg306-server reboot facets: ost1 Failover ost1 to oleg306-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.10 seconds: 47.64 ops/second umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg306-server: LNET ready to unload oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. PASS 122b (1686s) SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:21:23 (1761312083) start mds service on oleg306-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock oleg306-server@tcp:/lustre /mnt/lustre oleg306-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (154s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F == conf-sanity test 124: check failover after replace_nids ========================================================== 09:23:59 (1761312239) SKIP: conf-sanity test_124 needs >= 2 MDTs SKIP 124 (2s) == conf-sanity test 125: check l_tunedisk only tunes OSTs and their slave devices ========================================================== 09:24:01 (1761312241) SKIP: conf-sanity test_125 ldiskfs only test SKIP 125 (3s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:24:05 (1761312245) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (3s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:24:08 (1761312248) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (2s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:24:12 (1761312252) SKIP: conf-sanity test_130 needs >= 2 MDTs SKIP 130 (2s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 09:24:15 (1761312255) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (2s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg306-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server oleg306-server: oleg306-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2024 sec ========== 09:25:20 (1761312320)