-----============= acceptance-small: conf-sanity ============----- Mon Aug 24 16:30:16 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg257-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION >= v2_15_63-53-g1a6ef725c2 (34683217 >= 34553653) for LU-16938, go 81 - see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683217 >= 34554143) for LU-18348, go 32b 32c - see CLIENT_VERSION >= v2_16_55-19-gf69f27cc35 (34683217 >= 34617107) for LU-17920, go 123ae 123ag excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg257-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg257-server oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server SKIP: conf-sanity test_121 skipping ALWAYS excluded test 121 SKIP: conf-sanity test_122a skipping ALWAYS excluded test 122a SKIP: conf-sanity test_123aa skipping ALWAYS excluded test 123aa SKIP: conf-sanity test_123ab skipping ALWAYS excluded test 123ab SKIP: conf-sanity test_123ac skipping ALWAYS excluded test 123ac SKIP: conf-sanity test_123ad skipping ALWAYS excluded test 123ad SKIP: conf-sanity test_123ae skipping ALWAYS excluded test 123ae SKIP: conf-sanity test_123af skipping ALWAYS excluded test 123af SKIP: conf-sanity test_123ag skipping ALWAYS excluded test 123ag SKIP: conf-sanity test_123ah skipping ALWAYS excluded test 123ah == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:33:25 (1787603605) start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre oleg257-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (189s) == conf-sanity test 123aj: check permanent TBF rules ===== 16:36:34 (1787603794) cleanup test 123aj oleg257-server: error: executing set_param: No such file or directory pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 ost.OSS.ost_io.nrs_policies=fifo PASS 123aj (45s) SKIP: conf-sanity test_123F skipping ALWAYS excluded test 123F SKIP: conf-sanity test_123G skipping ALWAYS excluded test 123G SKIP: conf-sanity test_123H skipping ALWAYS excluded test 123H == conf-sanity test 124: check failover after replace_nids ========================================================== 16:37:25 (1787603845) SKIP: conf-sanity test_124 needs >= 2 MDTs SKIP 124 (3s) SKIP: conf-sanity test_126 skipping ALWAYS excluded test 126 == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:37:30 (1787603850) SKIP: conf-sanity test_127 ldiskfs only test SKIP 127 (4s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:37:34 (1787603854) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (3s) SKIP: conf-sanity test_129 skipping ALWAYS excluded test 129 == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:37:40 (1787603860) SKIP: conf-sanity test_130 needs >= 2 MDTs SKIP 130 (4s) == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:37:44 (1787603864) SKIP: conf-sanity test_131 ldiskfs only test SKIP 131 (3s) SKIP: conf-sanity test_132 skipping ALWAYS excluded test 132 SKIP: conf-sanity test_133 skipping ALWAYS excluded test 133 == conf-sanity test 134: check_iam works without faults == 16:37:51 (1787603871) SKIP: conf-sanity test_134 ldiskfs only test SKIP 134 (3s) SKIP: conf-sanity test_135 skipping ALWAYS excluded test 135 SKIP: conf-sanity test_136 skipping ALWAYS excluded test 136 SKIP: conf-sanity test_137 skipping ALWAYS excluded test 137 == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:38:00 (1787603880) SKIP: conf-sanity test_140 needs >= 2 MDTs SKIP 140 (3s) SKIP: conf-sanity test_150 skipping ALWAYS excluded test 150 == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:38:05 (1787603885) SKIP: conf-sanity test_151a ldiskfs only test SKIP 151a (4s) == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:38:09 (1787603889) SKIP: conf-sanity test_151b ldiskfs only test SKIP 151b (3s) SKIP: conf-sanity test_152 skipping ALWAYS excluded test 152 SKIP: conf-sanity test_153a skipping ALWAYS excluded test 153a SKIP: conf-sanity test_153b skipping ALWAYS excluded test 153b SKIP: conf-sanity test_153c skipping ALWAYS excluded test 153c == conf-sanity test 154: expand .. on rename after MDT backup restore ========================================================== 16:38:19 (1787603899) SKIP: conf-sanity test_154 ldiskfs only test SKIP 154 (3s) SKIP: conf-sanity test_155 skipping ALWAYS excluded test 155 == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:38:24 (1787603904) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg257-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Checking servers environments Checking clients oleg257-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre Starting client oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre Started clients oleg257-client.virtnet: 192.168.202.157@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d9850e0d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d9850e0d000.idle_timeout=debug disable quota as required (1) root_fid_export: 0x200000007:0x1:0x0 (1) root_fid_client: 0x200000007:0x1:0x0 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping client oleg257-client.virtnet /mnt/lustre opts: Starting client oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre/d156.conf-sanity /mnt/lustre Mount client oleg257-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.157@tcp:/lustre/d156.conf-sanity /mnt/lustre Started clients oleg257-client.virtnet: 192.168.202.157@tcp:/lustre/d156.conf-sanity on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) (2) root_fid_export: 0x200000401:0x1:0x0 (2) root_fid_client: 0x200000401:0x1:0x0 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping client oleg257-client.virtnet /mnt/lustre opts: PASS 156 (176s) == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:41:21 (1787604081) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg257-server start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre allow_register=1 allow_register=0 Permanent disk data: Device: lustre-mdt1_2/mdt1_2 Target: fs4348:MDT0001 Index: 1 Lustre FS: fs4348 Mount type: zfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.157@tcp:192.168.202.157@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.202.157@tcp:192.168.202.157@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=97 lustre:index=1 lustre:fsname=fs4348 lustre:svname=fs4348:MDT0001 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg257-server: mount.lustre: mount lustre-mdt1_2/mdt1_2 at /mnt/lustre-fs2mds failed: Permission denied pdsh@oleg257-client: oleg257-server: ssh exited with exit code 13 Start of lustre-mdt1_2/mdt1_2 on fs2mds failed 13 allow_register=1 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started fs4348-MDT0001 mdt.fs4348-MDT0001.recovery_status=status: INACTIVE Stopping /mnt/lustre-fs2mds (opts:-f) on oleg257-server allow_register=0 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started fs4348-MDT0001 mdt.fs4348-MDT0001.recovery_status=status: INACTIVE Stopping /mnt/lustre-fs2mds (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 affected facets: mgs allow_register=0 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started fs4348-MDT0001 mdt.fs4348-MDT0001.recovery_status=status: INACTIVE Stopping /mnt/lustre-fs2mds (opts:-f) on oleg257-server allow_register functionality test completed successfully Cleaning up test 157a... allow_register=1 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. PASS 157a (221s) == conf-sanity test 157b: verify allow_register (block new OSTs, allow existing) ========================================================== 16:45:02 (1787604302) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: fs12914:OST0000 Index: 0 Lustre FS: fs12914 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.157@tcp:192.168.202.157@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.157@tcp:192.168.202.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=fs12914 lustre:svname=fs12914:OST0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-fs12914-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started fs12914-OST0000 obdfilter.fs12914-OST0000.recovery_status=status: INACTIVE allow_register=0 Stopping /mnt/lustre-fs2ost (opts:-f) on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-fs12914-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started fs12914-OST0000 obdfilter.fs12914-OST0000.recovery_status=status: INACTIVE Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: fs12914:OST0001 Index: 1 Lustre FS: fs12914 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.157@tcp:192.168.202.157@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost2_2 /tmp/lustre-ost2_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.157@tcp:192.168.202.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1 lustre:fsname=fs12914 lustre:svname=fs12914:OST0001 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs3ost: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost oleg257-server: mount.lustre: mount lustre-ost2_2/ost2_2 at /mnt/lustre-fs3ost failed: Permission denied pdsh@oleg257-client: oleg257-server: ssh exited with exit code 13 oleg257-server: error: set_param: param_path 'seq/cli-fs12914:OST0001-super/width': No such file or directory oleg257-server: error: set_param: setting 'seq/cli-fs12914:OST0001-super/width'='65536': No such file or directory pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 Start of lustre-ost2_2/ost2_2 on fs3ost failed 13 allow_register=1 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs3ost: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost seq.cli-fs12914-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started fs12914-OST0001 obdfilter.fs12914-OST0000.recovery_status=status: INACTIVE obdfilter.fs12914-OST0001.recovery_status=status: INACTIVE allow_register=0 Stopping /mnt/lustre-fs2ost (opts:-f) on oleg257-server Stopping /mnt/lustre-fs3ost (opts:-f) on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-fs12914-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started fs12914-OST0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs3ost: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost seq.cli-fs12914-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started fs12914-OST0001 Passed: Existing targets can re-register, new targets blocked Cleaning up test 157b... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg257-server Stopping /mnt/lustre-fs3ost (opts:-f) on oleg257-server allow_register=1 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. PASS 157b (257s) == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 16:49:19 (1787604559) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mgc.MGC192.168.202.157@tcp.import= import: name: MGC192.168.202.157@tcp target: MGS state: FULL connect_flags: [ version, mgs_nidlist, barrier, adaptive_timeouts, full20, imp_recov, bulk_mbits, second_flags, reply_mbits, large_nid ] connect_data: flags: 0xa000011001002060 instance: 0 target_version: 2.17.57.81 import_flags: [ IMPF_PINGABLE, IMPF_CONNECT_TRIED ] connection: failover_nids: [ "0@lo", "192.168.252.160@tcp" ] nids_stats: "0@lo": { connects: 1, replied: 1, uptodate: uptodate, sec_ago: 47 } "192.168.252.160@tcp": { connects: 0, replied: 0, uptodate: unknown, sec_ago: never } current_connection: "0@lo" connection_attempts: 1 generation: 1 in-progress_invalidations: 0 idle: 1 sec umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 PASS 160 (360s) == conf-sanity test 161: test '-o mgsname' option ======== 16:55:20 (1787604920) start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre test-mgs-custom:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose 0 0 test-mgs-custom:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose) 192.168.202.157 oleg257-server.virtnet Hostname oleg257-server.virtnet is already resolvable Test hostname: oleg257-server.virtnet@tcp Expected device: oleg257-server.virtnet@tcp:/lustre Device in /proc/mounts: oleg257-server.virtnet@tcp:/lustre oleg257-server.virtnet@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,nouser_xattr,verbose) umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. PASS 161 (98s) == conf-sanity test 162: replace nids with -o noclient === 16:56:58 (1787605018) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov -o nosvc,noclient lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all Start lustre-mdt1/mdt1 without service Started lustre-MDT0000 set NIDs with failover replace MDS nid stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec setup single mount lustre success umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server PASS 162 (378s) == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 17:03:16 (1787605396) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre after initial mount: expected_clients=1 expected_clients=1 mount lustre on /mnt/lustre/d164.conf-sanity.1..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre/d164.conf-sanity.1 mount lustre on /mnt/lustre/d164.conf-sanity.2..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre/d164.conf-sanity.2 mount lustre on /mnt/lustre/d164.conf-sanity.3..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre/d164.conf-sanity.3 mount lustre on /mnt/lustre/d164.conf-sanity.4..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre/d164.conf-sanity.4 mount lustre on /mnt/lustre/d164.conf-sanity.5..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre/d164.conf-sanity.5 expected_clients after mounting 5 additional clients: 6 umount lustre on /mnt/lustre/d164.conf-sanity.1..... Stopping client oleg257-client.virtnet /mnt/lustre/d164.conf-sanity.1 (opts:) umount lustre on /mnt/lustre/d164.conf-sanity.2..... Stopping client oleg257-client.virtnet /mnt/lustre/d164.conf-sanity.2 (opts:) umount lustre on /mnt/lustre/d164.conf-sanity.3..... Stopping client oleg257-client.virtnet /mnt/lustre/d164.conf-sanity.3 (opts:) umount lustre on /mnt/lustre/d164.conf-sanity.4..... Stopping client oleg257-client.virtnet /mnt/lustre/d164.conf-sanity.4 (opts:) umount lustre on /mnt/lustre/d164.conf-sanity.5..... Stopping client oleg257-client.virtnet /mnt/lustre/d164.conf-sanity.5 (opts:) expected_clients after unmount: 6 expected_clients=1 umount lustre on /mnt/lustre/d164.conf-sanity.1..... umount lustre on /mnt/lustre/d164.conf-sanity.2..... umount lustre on /mnt/lustre/d164.conf-sanity.3..... umount lustre on /mnt/lustre/d164.conf-sanity.4..... umount lustre on /mnt/lustre/d164.conf-sanity.5..... umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. PASS 164 (175s) == conf-sanity test 200a: check CPU partitions =========== 17:06:11 (1787605571) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=4 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=4' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 0 1 : 1 2 : 2 3 : 3 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 200a (41s) == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 17:06:52 (1787605612) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="0[0-3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="0[0-3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:0[0-3] 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="0[1-2]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="0[1-2]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 1 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 200b (79s) == conf-sanity test 200c: set CPU pattern using NUMA node layout ========================================================== 17:08:11 (1787605691) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="0[0-3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="0[0-3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N 0[0]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N 0[0]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" cpu_npartitions=1 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N" cpu_npartitions=1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system 0 : 0 1 2 3 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 200c (140s) == conf-sanity test 200d: set CPU pattern to exclude only CPU 1 ========================================================== 17:10:31 (1787605831) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' full_table: 0 : 0 1 1 : 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 cpu_pattern="X[1]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2 cpu_pattern="X[1]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with CPU 1 excluded: /sys/module/libcfs/parameters/cpu_npartitions:2 /sys/module/libcfs/parameters/cpu_pattern:X[1] 0 : 0 1 : 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 cpu_pattern="X[3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2 cpu_pattern="X[3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with last CPU excluded: /sys/module/libcfs/parameters/cpu_npartitions:2 /sys/module/libcfs/parameters/cpu_pattern:X[3] 0 : 0 1 1 : 2 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system full_table: 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N X[1]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N X[1]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with CPU 1 excluded: /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N X[1] 0 : 0 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N X[3]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N X[3]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system table with last CPU excluded: /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N X[3] 0 : 0 1 2 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) LNET unconfigure error 22: Invalid argument PASS 200d (196s) == conf-sanity test 200e: set CPU pattern using relative core exclusion ========================================================== 17:13:47 (1787606027) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) LNET unconfigure error 22: Invalid argument /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system full_table: 0 : 0 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="N C[0]" libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="N C[0]"' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system /sys/module/libcfs/parameters/cpu_npartitions:0 /sys/module/libcfs/parameters/cpu_pattern:N C[0] table with npartitions=2: 0 : 1 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_npartitions=2 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system full_table: 0 : 0 1 1 : 2 3 umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server stop mds service on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: oleg257-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS=cpu_pattern="C[0]" cpu_npartitions=2 libcfs will create CPU partition based on online CPUs ../libcfs/libcfs/libcfs options: 'cpu_pattern="C[0]" cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system /sys/module/libcfs/parameters/cpu_npartitions:2 /sys/module/libcfs/parameters/cpu_pattern:C[0] table with npartitions=2: 0 : 1 1 : 3 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 200e (137s) == conf-sanity test 250: verify lcfg_erase removes filesystem parameters ========================================================== 17:16:04 (1787606164) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: tst250fs-mdt1/mdt1 Format ost1: tst250fs-ost1/ost1 Format ost2: tst250fs-ost2/ost2 Start mds1: mount -t lustre -o localrecov tst250fs-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on tst250fs-mdt1/mdt1 Started tst250fs-MDT0000 Start ost1: mount -t lustre -o localrecov tst250fs-ost1/ost1 /mnt/lustre-ost1 seq.cli-tst250fs-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on tst250fs-ost1/ost1 Started tst250fs-OST0000 Start ost2: mount -t lustre -o localrecov tst250fs-ost2/ost2 /mnt/lustre-ost2 seq.cli-tst250fs-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on tst250fs-ost2/ost2 Started tst250fs-OST0001 config_log: tst250fs-OST0000 config_log: tst250fs-client config_log: tst250fs-OST0001 config_log: tst250fs-MDT0000 Found parameters: - { index: 2, event: set_param, device: general, parameter: debug_raw_pointers, value: Y } - { index: 5, event: set_param, device: tst250fs, parameter: osc.tst250fs-*.max_rpcs_in_flight, value: 16 } - { index: 8, event: set_param, device: tst250fs, parameter: mdt.tst250fs-*.enable_pin_gid, value: -1 } - { index: 11, event: set_param, device: tst250fs, parameter: llite.tst250fs-*.max_read_ahead_mb, value: 32 } - { index: 14, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 512 } lcfg_erase output: lcfg_erase: erasing config log and params for fsname 'tst250fs' - cancelled param 'osc.tst250fs-*.max_rpcs_in_flight' - cancelled param 'mdt.tst250fs-*.enable_pin_gid' - cancelled param 'llite.tst250fs-*.max_read_ahead_mb' lcfg_erase: erased 3 parameters matching fsname 'tst250fs' remaining tst250fs params: remaining params: - { index: 2, event: set_param, device: general, parameter: debug_raw_pointers, value: Y } - { index: 14, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 512 } Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg257-server Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 PASS 250 (250s) SKIP: conf-sanity test_802a skipping ALWAYS excluded test 802a Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3057 sec ========== 17:21:14 (1787606474) === conf-sanity: start cleanup 17:21:15 (1787606475) === === conf-sanity: finish cleanup 17:21:18 (1787606478) ===