-----============= acceptance-small: conf-sanity ============----- Tue Oct 14 10:25:04 EDT 2025 oleg433-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg433-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg433-server oleg433-server: oleg433-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg433-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg433-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a == conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 10:26:40 (1760452000) Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) oleg433-server: oleg433-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg433-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg433-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock oleg433-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 1695 (time 1760452060.46 total 10.00 last 169.42) - open/close 4030 (time 1760452070.46 total 20.00 last 233.50) - open/close 6531 (time 1760452080.46 total 30.01 last 250.08) - open/close 9291 (time 1760452090.47 total 40.01 last 275.77) total: 10000 open/close in 43.06 seconds: 232.22 ops/second - unlinked 0 (time 1760452095 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2391 (time 1760452133.75 total 10.00 last 239.04) - open/close 4717 (time 1760452143.75 total 20.00 last 232.57) - open/close 6809 (time 1760452153.76 total 30.01 last 209.17) - open/close 8574 (time 1760452163.76 total 40.01 last 176.47) total: 10000 open/close in 48.75 seconds: 205.13 ops/second - unlinked 0 (time 1760452174 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2757 (time 1760452213.07 total 10.00 last 275.68) - open/close 5165 (time 1760452223.08 total 20.01 last 240.67) - open/close 7346 (time 1760452233.08 total 30.01 last 218.04) - open/close 9132 (time 1760452243.09 total 40.01 last 178.53) total: 10000 open/close in 44.41 seconds: 225.18 ops/second - unlinked 0 (time 1760452249 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2608 (time 1760452287.35 total 10.00 last 260.79) - open/close 4765 (time 1760452297.35 total 20.00 last 215.68) - open/close 6569 (time 1760452307.35 total 30.00 last 180.39) - open/close 8527 (time 1760452317.35 total 40.00 last 195.76) total: 10000 open/close in 47.31 seconds: 211.37 ops/second - unlinked 0 (time 1760452326 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1892 (time 1760452365.81 total 10.00 last 189.18) - open/close 3538 (time 1760452375.81 total 20.01 last 164.53) - open/close 5483 (time 1760452385.81 total 30.01 last 194.43) - open/close 7502 (time 1760452395.82 total 40.01 last 201.84) - open/close 9588 (time 1760452405.82 total 50.01 last 208.58) total: 10000 open/close in 51.83 seconds: 192.93 ops/second - unlinked 0 (time 1760452409 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1863 (time 1760452447.06 total 10.00 last 186.23) - open/close 3833 (time 1760452457.06 total 20.01 last 196.97) - open/close 5858 (time 1760452467.07 total 30.01 last 202.46) - open/close 7802 (time 1760452477.07 total 40.01 last 194.31) - open/close 9837 (time 1760452487.07 total 50.02 last 203.44) total: 10000 open/close in 50.83 seconds: 196.75 ops/second - unlinked 0 (time 1760452489 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2192 (time 1760452528.05 total 10.00 last 219.20) - open/close 4213 (time 1760452538.05 total 20.00 last 202.10) - open/close 6847 (time 1760452548.06 total 30.01 last 263.24) - open/close 9571 (time 1760452558.06 total 40.01 last 272.35) total: 10000 open/close in 41.51 seconds: 240.89 ops/second - unlinked 0 (time 1760452561 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2431 (time 1760452597.30 total 10.00 last 243.08) - open/close 4847 (time 1760452607.31 total 20.00 last 241.55) - open/close 7405 (time 1760452617.31 total 30.00 last 255.75) - open/close 9600 (time 1760452627.31 total 40.01 last 219.46) total: 10000 open/close in 42.03 seconds: 237.92 ops/second - unlinked 0 (time 1760452631 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2246 (time 1760452664.44 total 10.00 last 224.51) - open/close 4415 (time 1760452674.44 total 20.01 last 216.84) - open/close 6606 (time 1760452684.44 total 30.01 last 219.10) - open/close 8711 (time 1760452694.44 total 40.01 last 210.43) total: 10000 open/close in 46.76 seconds: 213.84 ops/second - unlinked 0 (time 1760452703 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2251 (time 1760452739.67 total 10.00 last 225.08) - open/close 5157 (time 1760452749.67 total 20.00 last 290.52) - open/close 8195 (time 1760452759.68 total 30.01 last 303.73) total: 10000 open/close in 36.40 seconds: 274.76 ops/second - unlinked 0 (time 1760452767 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2090 (time 1760452801.16 total 10.00 last 208.96) - open/close 4598 (time 1760452811.17 total 20.01 last 250.47) - open/close 7124 (time 1760452821.17 total 30.02 last 252.60) - open/close 9566 (time 1760452831.18 total 40.02 last 244.11) total: 10000 open/close in 41.82 seconds: 239.12 ops/second - unlinked 0 (time 1760452834 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5376 376192 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190071 s, 5.5 MB/s fail_loc=0x80000204 Failing ost1 on oleg433-server Stopping /mnt/lustre-ost1 (opts:) on oleg433-server reboot facets: ost1 Failover ost1 to oleg433-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.66 seconds: 37.60 ops/second umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg433-server: LNET ready to unload oleg433-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. PASS 122b (903s) SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:41:48 (1760452908) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg433-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock oleg433-server@tcp:/lustre /mnt/lustre oleg433-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (75s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F == conf-sanity test 124: check failover after replace_nids ========================================================== 10:43:03 (1760452983) SKIP: conf-sanity test_124 needs >= 2 MDTs SKIP 124 (1s) == conf-sanity test 125: check l_tunedisk only tunes OSTs and their slave devices ========================================================== 10:43:05 (1760452985) SKIP: conf-sanity test_125 ldiskfs only test SKIP 125 (1s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:43:06 (1760452986) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (1s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:43:07 (1760452987) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (1s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:43:09 (1760452989) SKIP: conf-sanity test_130 needs >= 2 MDTs SKIP 130 (1s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 10:43:10 (1760452990) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (1s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg433-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server oleg433-server: oleg433-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1120 sec ========== 10:43:44 (1760453024)