-----============= acceptance-small: conf-sanity ============----- Tue Dec 2 22:44:10 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg159-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg159-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg159-server oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag SKIP: conf-sanity test_123ah skipping ALWAYS excluded test 123ah == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:47:00 (1764733620) start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre oleg159-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (154s) == conf-sanity test 123aj: check permanent TBF rules ===== 22:49:34 (1764733774) cleanup test 123aj oleg159-server: error: executing set_param: No such file or directory pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 ost.OSS.ost_io.nrs_policies=fifo PASS 123aj (35s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F SKIP: conf-sanity test_123G skipping ALWAYS excluded test 123G SKIP: conf-sanity test_123H skipping ALWAYS excluded test 123H == conf-sanity test 124: check failover after replace_nids ========================================================== 22:50:13 (1764733813) SKIP: conf-sanity test_124 needs >= 2 MDTs SKIP 124 (3s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:50:18 (1764733818) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (3s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:50:21 (1764733821) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (3s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:50:25 (1764733825) SKIP: conf-sanity test_130 needs >= 2 MDTs SKIP 130 (3s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 22:50:28 (1764733828) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (3s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 SKIP: conf-sanity test_133 skipping ALWAYS excluded test 133 == conf-sanity test 134: check_iam works without faults == 22:50:33 (1764733833) SKIP: conf-sanity test_134 ldiskfs only test SKIP 134 (3s) SKIP: conf-sanity test_135 skipping ALWAYS excluded test 135 SKIP: conf-sanity test_136 skipping ALWAYS excluded test 136 SKIP: conf-sanity test_137 skipping ALWAYS excluded test 137 == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:50:40 (1764733840) SKIP: conf-sanity test_140 needs >= 2 MDTs SKIP 140 (3s) SKIP: conf-sanity test_150 skipping ALWAYS excluded test 150 == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 22:50:44 (1764733844) SKIP: conf-sanity test_151a ldiskfs only test SKIP 151a (3s) == conf-sanity test 151b: -ENOSPC doesn't affect mount === 22:50:47 (1764733847) SKIP: conf-sanity test_151b ldiskfs only test SKIP 151b (3s) SKIP: conf-sanity test_152 skipping ALWAYS excluded test 152 SKIP: conf-sanity test_153a skipping ALWAYS excluded test 153a SKIP: conf-sanity test_153b skipping ALWAYS excluded test 153b SKIP: conf-sanity test_153c skipping ALWAYS excluded test 153c == conf-sanity test 154: expand .. on rename after MDT backup restore ========================================================== 22:50:56 (1764733856) SKIP: conf-sanity test_154 ldiskfs only test SKIP 154 (3s) SKIP: conf-sanity test_155 skipping ALWAYS excluded test 155 == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:51:00 (1764733860) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg159-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Checking servers environments Checking clients oleg159-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre Starting client oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre Started clients oleg159-client.virtnet: 192.168.201.159@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff916258c43000.idle_timeout=debug osc.lustre-OST0001-osc-ffff916258c43000.idle_timeout=debug disable quota as required (1) root_fid_export: 0x200000007:0x1:0x0 (1) root_fid_client: 0x200000007:0x1:0x0 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping client oleg159-client.virtnet /mnt/lustre opts: Starting client oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre/d156.conf-sanity /mnt/lustre Mount client oleg159-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.159@tcp:/lustre/d156.conf-sanity /mnt/lustre Started clients oleg159-client.virtnet: 192.168.201.159@tcp:/lustre/d156.conf-sanity on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) (2) root_fid_export: 0x200000401:0x1:0x0 (2) root_fid_client: 0x200000401:0x1:0x0 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping client oleg159-client.virtnet /mnt/lustre opts: PASS 156 (149s) == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 22:53:29 (1764734009) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg159-server Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mgc.MGC192.168.201.159@tcp.import= import: name: MGC192.168.201.159@tcp target: MGS state: FULL connect_flags: [ version, mgs_nidlist, barrier, adaptive_timeouts, full20, imp_recov, bulk_mbits, second_flags, reply_mbits, large_nid ] connect_data: flags: 0xa000011001002060 instance: 0 target_version: 2.17.0.0 import_flags: [ pingable, connect_tried ] connection: failover_nids: [ "0@lo", "192.168.252.160@tcp" ] nids_stats: "0@lo": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 26 } "192.168.252.160@tcp": { connects: 0, replied: 0, uptodate: unknown, sec_ago: never } current_connection: "0@lo" connection_attempts: 1 generation: 1 in-progress_invalidations: 0 idle: 1 sec umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 PASS 160 (280s) == conf-sanity test 161: test '-o mgsname' option ======== 22:58:09 (1764734289) start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre test-mgs-custom:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose 0 0 test-mgs-custom:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose) Add temp hostname to /etc/hosts: 192.168.201.159 test-mgs-auto Test hostname: test-mgs-auto@tcp Expected device: test-mgs-auto@tcp:/lustre Device in /proc/mounts: test-mgs-auto@tcp:/lustre test-mgs-auto@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose) umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 PASS 161 (68s) == conf-sanity test 200a: check CPU partitions =========== 22:59:17 (1764734357) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=4 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=4' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 0 1 : 1 2 : 2 3 : 3 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy PASS 200a (32s) == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 22:59:49 (1764734389) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="0[0-3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="0[0-3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:0[0-3] 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="0[1-2]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="0[1-2]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 1 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy PASS 200b (51s) == conf-sanity test 200c: set CPU pattern using NUMA node layout ========================================================== 23:00:40 (1764734440) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="0[0-3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="0[0-3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N 0[0]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N 0[0]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" cpu_npartitions=1 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N" cpu_npartitions=1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 0 1 2 3 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy PASS 200c (105s) == conf-sanity test 200d: set CPU pattern to exclude only CPU 1 ========================================================== 23:02:25 (1764734545) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' full_table: 0 : 0 1 1 : 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 cpu_pattern="X[1]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2 cpu_pattern="X[1]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with CPU 1 excluded: /sys/module/libcfs/parameters/cpu_npartitions:2 /sys/module/libcfs/parameters/cpu_pattern:X[1] 0 : 0 1 : 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 cpu_pattern="X[3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2 cpu_pattern="X[3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with last CPU excluded: /sys/module/libcfs/parameters/cpu_npartitions:2 /sys/module/libcfs/parameters/cpu_pattern:X[3] 0 : 0 1 1 : 2 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system full_table: 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N X[1]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N X[1]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with CPU 1 excluded: /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N X[1] 0 : 0 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N X[3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N X[3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with last CPU excluded: /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N X[3] 0 : 0 1 2 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy PASS 200d (130s) == conf-sanity test 200e: set CPU pattern using relative core exclusion ========================================================== 23:04:35 (1764734675) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system full_table: 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N C[0]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N C[0]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N C[0] table with npartitions=2: 0 : 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system full_table: 0 : 0 1 1 : 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg159-server stop mds service on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: LNET unconfigure error 22: Invalid argument oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="C[0]" cpu_npartitions=2 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="C[0]" cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system /sys/module/libcfs/parameters/cpu_npartitions:2 /sys/module/libcfs/parameters/cpu_pattern:C[0] table with npartitions=2: 0 : 1 1 : 3 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy PASS 200e (85s) SKIP: conf-sanity test_802a skipping ALWAYS excluded test 802a Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1360 sec ========== 23:06:52 (1764734812) === conf-sanity: start cleanup 23:06:53 (1764734813) === === conf-sanity: finish cleanup 23:06:55 (1764734815) ===