-----============= acceptance-small: conf-sanity ============----- Tue Jul  4 07:50:48 EDT 2023
excepting tests: 32newtarball 110
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg237-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server
oleg237-server: oleg237-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg237-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg237-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server

== conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:51:41 (1688471501)
Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f)
oleg237-server: oleg237-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg237-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg237-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg237-client.virtnet:  -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
fail_loc=0x142
start ost2 service on oleg237-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg237-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server
umount lustre on /mnt/lustre.....
Stopping client oleg237-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
unloading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing unload_modules_local
modules unloaded.
oleg237-server: 
oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg237-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg237-server: 
oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg237-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg237-server: 
oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg237-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2
oleg237-server: oleg237-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg237-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg237-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
PASS 51 (117s)

== conf-sanity test 64: check lfs df --lazy ============== 07:53:38 (1688471618)
start mds service on oleg237-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg237-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg237-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg237-client.virtnet:  -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre
stop ost2 service on oleg237-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server
/home/green/git/lustre-release/lustre/utils/lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       382976        3072      377856   1% /mnt/lustre[OST:0]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

umount lustre on /mnt/lustre.....
Stopping client oleg237-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
unloading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing unload_modules_local
modules unloaded.
oleg237-server: 
oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg237-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg237-server: 
oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg237-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg237-server: 
oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg237-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2
oleg237-server: oleg237-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg237-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg237-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
PASS 64 (91s)

== conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 07:55:09 (1688471709)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
oleg237-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
oleg237-server: 
oleg237-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg237-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22
oleg237-server: 
oleg237-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode
oleg237-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22
oleg237-server: 
oleg237-server: mkfs.lustre FATAL: Must specify --mgsnode
oleg237-server: mkfs.lustre: exiting with 22 (Invalid argument)
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 22
Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f)
oleg237-server: oleg237-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg237-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg237-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
PASS 79 (40s)

== conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 07:55:49 (1688471749)
start mds service on oleg237-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg237-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg237-client.virtnet:  -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
error: mount options too long
umount lustre on /mnt/lustre.....
Stopping client oleg237-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
unloading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2
pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2
PASS 98 (23s)

== conf-sanity test 102: obdclass module cleanup upon error ========================================================== 07:56:12 (1688471772)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg237-server
stop mds service on oleg237-server
unloading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing unload_modules_local
modules unloaded.
++ uname -r
+ do_facet mds1 ls -ali /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre
+ local verbose
+ local quiet
+ [[ mds1 == \-\-\v\e\r\b\o\s\e ]]
+ [[ mds1 == \-\-\q\u\i\e\t ]]
+ [[ mds1 == \-\q ]]
+ local facet=mds1
+ shift
++ facet_active_host mds1
+++ facet_active mds1
+++ local facet=mds1
+++ local activevar=mds1active
+++ '[' -f /tmp/mds1active ']'
+++ active=
+++ '[' -z '' ']'
+++ echo -n mds1
++ facet_host mds1
++ local facet=mds1
++ local varname
++ '[' mds1 == client ']'
++ varname=mds1_HOST
++ '[' -z '' ']'
++ '[' mds == ost ']'
++ '[' mds == mdt -o mds == mds -o mds == mgs ']'
++ local temp
++ '[' mds1 == mgsfailover ']'
++ temp=oleg237-server
++ eval export mds1_HOST=oleg237-server
+++ export mds1_HOST=oleg237-server
+++ mds1_HOST=oleg237-server
++ echo -n oleg237-server
+ local host=oleg237-server
+ '[' -z oleg237-server ']'
+ do_node oleg237-server ls -ali /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre
+ local verbose
+ local quiet
+ [[ oleg237-server == \-\-\v\e\r\b\o\s\e ]]
+ [[ oleg237-server == \-\-\q\u\i\e\t ]]
+ [[ oleg237-server == \-\q ]]
+ local HOST=oleg237-server
+ shift
+ local 'myPDSH=pdsh -S -Rssh -w'
+ '[' oleg237-server = oleg237-client.virtnet ']'
+ '[' -z 'pdsh -S -Rssh -w' -o 'pdsh -S -Rssh -w' = no_dsh ']'
+ false
+ [[ pdsh -S -Rssh -w == \r\s\h ]]
+ [[ pdsh -S -Rssh -w == *pdsh* ]]
+ [[ pdsh -S -Rssh -w != *-S* ]]
+ [[ -n '' ]]
+ sed 's/^oleg237-server: //'
+ pdsh -S -Rssh -w oleg237-server '(PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin;		cd /root; LUSTRE="/home/green/git/lustre-release/lustre" bash -c "ls -ali /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre")'
oleg237-server: ls: cannot access /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre: No such file or directory
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2
+ return 2
+ do_facet mds1 depmod -a
+ local verbose
+ local quiet
+ [[ mds1 == \-\-\v\e\r\b\o\s\e ]]
+ [[ mds1 == \-\-\q\u\i\e\t ]]
+ [[ mds1 == \-\q ]]
+ local facet=mds1
+ shift
++ facet_active_host mds1
+++ facet_active mds1
+++ local facet=mds1
+++ local activevar=mds1active
+++ '[' -f /tmp/mds1active ']'
+++ active=
+++ '[' -z '' ']'
+++ echo -n mds1
++ facet_host mds1
++ local facet=mds1
++ local varname
++ '[' mds1 == client ']'
++ varname=mds1_HOST
++ '[' -z '' ']'
++ '[' mds == ost ']'
++ '[' mds == mdt -o mds == mds -o mds == mgs ']'
++ local temp
++ '[' mds1 == mgsfailover ']'
++ temp=oleg237-server
++ eval export mds1_HOST=oleg237-server
+++ export mds1_HOST=oleg237-server
+++ mds1_HOST=oleg237-server
++ echo -n oleg237-server
+ local host=oleg237-server
+ '[' -z oleg237-server ']'
+ do_node oleg237-server depmod -a
+ local verbose
+ local quiet
+ [[ oleg237-server == \-\-\v\e\r\b\o\s\e ]]
+ [[ oleg237-server == \-\-\q\u\i\e\t ]]
+ [[ oleg237-server == \-\q ]]
+ local HOST=oleg237-server
+ shift
+ local 'myPDSH=pdsh -S -Rssh -w'
+ '[' oleg237-server = oleg237-client.virtnet ']'
+ '[' -z 'pdsh -S -Rssh -w' -o 'pdsh -S -Rssh -w' = no_dsh ']'
+ false
+ [[ pdsh -S -Rssh -w == \r\s\h ]]
+ [[ pdsh -S -Rssh -w == *pdsh* ]]
+ [[ pdsh -S -Rssh -w != *-S* ]]
+ [[ -n '' ]]
+ sed 's/^oleg237-server: //'
+ pdsh -S -Rssh -w oleg237-server '(PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin;		cd /root; LUSTRE="/home/green/git/lustre-release/lustre" bash -c "depmod -a")'
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.dep.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.dep.bin.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.alias.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.alias.bin.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.softdep.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.symbols.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.symbols.bin.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.builtin.bin.tmp, 1101, 644): Read-only file system
oleg237-server: depmod: ERROR: openat(/lib/modules/3.10.0-7.9-debug, modules.devname.tmp, 1101, 644): Read-only file system
+ return 0
++ uname -r
+ do_facet mds1 ls -ali /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre
+ local verbose
+ local quiet
+ [[ mds1 == \-\-\v\e\r\b\o\s\e ]]
+ [[ mds1 == \-\-\q\u\i\e\t ]]
+ [[ mds1 == \-\q ]]
+ local facet=mds1
+ shift
++ facet_active_host mds1
+++ facet_active mds1
+++ local facet=mds1
+++ local activevar=mds1active
+++ '[' -f /tmp/mds1active ']'
+++ active=
+++ '[' -z '' ']'
+++ echo -n mds1
++ facet_host mds1
++ local facet=mds1
++ local varname
++ '[' mds1 == client ']'
++ varname=mds1_HOST
++ '[' -z '' ']'
++ '[' mds == ost ']'
++ '[' mds == mdt -o mds == mds -o mds == mgs ']'
++ local temp
++ '[' mds1 == mgsfailover ']'
++ temp=oleg237-server
++ eval export mds1_HOST=oleg237-server
+++ export mds1_HOST=oleg237-server
+++ mds1_HOST=oleg237-server
++ echo -n oleg237-server
+ local host=oleg237-server
+ '[' -z oleg237-server ']'
+ do_node oleg237-server ls -ali /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre
+ local verbose
+ local quiet
+ [[ oleg237-server == \-\-\v\e\r\b\o\s\e ]]
+ [[ oleg237-server == \-\-\q\u\i\e\t ]]
+ [[ oleg237-server == \-\q ]]
+ local HOST=oleg237-server
+ shift
+ local 'myPDSH=pdsh -S -Rssh -w'
+ '[' oleg237-server = oleg237-client.virtnet ']'
+ '[' -z 'pdsh -S -Rssh -w' -o 'pdsh -S -Rssh -w' = no_dsh ']'
+ false
+ [[ pdsh -S -Rssh -w == \r\s\h ]]
+ [[ pdsh -S -Rssh -w == *pdsh* ]]
+ [[ pdsh -S -Rssh -w != *-S* ]]
+ [[ -n '' ]]
+ pdsh -S -Rssh -w oleg237-server '(PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin;		cd /root; LUSTRE="/home/green/git/lustre-release/lustre" bash -c "ls -ali /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre")'
+ sed 's/^oleg237-server: //'
oleg237-server: ls: cannot access /usr/lib/modules/3.10.0-7.9-debug/extra/kernel/net/lustre: No such file or directory
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2
+ return 2
+ do_facet mds1 modprobe libcfs
+ local verbose
+ local quiet
+ [[ mds1 == \-\-\v\e\r\b\o\s\e ]]
+ [[ mds1 == \-\-\q\u\i\e\t ]]
+ [[ mds1 == \-\q ]]
+ local facet=mds1
+ shift
++ facet_active_host mds1
+++ facet_active mds1
+++ local facet=mds1
+++ local activevar=mds1active
+++ '[' -f /tmp/mds1active ']'
+++ active=
+++ '[' -z '' ']'
+++ echo -n mds1
++ facet_host mds1
++ local facet=mds1
++ local varname
++ '[' mds1 == client ']'
++ varname=mds1_HOST
++ '[' -z '' ']'
++ '[' mds == ost ']'
++ '[' mds == mdt -o mds == mds -o mds == mgs ']'
++ local temp
++ '[' mds1 == mgsfailover ']'
++ temp=oleg237-server
++ eval export mds1_HOST=oleg237-server
+++ export mds1_HOST=oleg237-server
+++ mds1_HOST=oleg237-server
++ echo -n oleg237-server
+ local host=oleg237-server
+ '[' -z oleg237-server ']'
+ do_node oleg237-server modprobe libcfs
+ local verbose
+ local quiet
+ [[ oleg237-server == \-\-\v\e\r\b\o\s\e ]]
+ [[ oleg237-server == \-\-\q\u\i\e\t ]]
+ [[ oleg237-server == \-\q ]]
+ local HOST=oleg237-server
+ shift
+ local 'myPDSH=pdsh -S -Rssh -w'
+ '[' oleg237-server = oleg237-client.virtnet ']'
+ '[' -z 'pdsh -S -Rssh -w' -o 'pdsh -S -Rssh -w' = no_dsh ']'
+ false
+ [[ pdsh -S -Rssh -w == \r\s\h ]]
+ [[ pdsh -S -Rssh -w == *pdsh* ]]
+ [[ pdsh -S -Rssh -w != *-S* ]]
+ [[ -n '' ]]
+ pdsh -S -Rssh -w oleg237-server '(PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin;		cd /root; LUSTRE="/home/green/git/lustre-release/lustre" bash -c "modprobe libcfs")'
+ sed 's/^oleg237-server: //'
oleg237-server: modprobe: FATAL: Module libcfs not found.
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
+ return 1
+ error 'libcfs not loaded'
+ report_error 'libcfs not loaded'
+ local TYPE=FAIL
+ local dump=true
+ '[' 'xlibcfs not loaded' = xfalse ']'
+ log ' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ echo ' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
 conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded 
+ load_module ../libcfs/libcfs/libcfs
+ local module=../libcfs/libcfs/libcfs
+ shift
+ local ext=.ko
++ basename ../libcfs/libcfs/libcfs .ko
+ local base=libcfs
+ local path
+ local -A module_is_loaded_aa
+ local optvar
+ local mod
++ lsmod
++ awk '{ print $1; }'
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ for mod in '$(lsmod | awk '\''{ print $1; }'\'')'
+ module_is_loaded_aa[${mod//-/_}]=true
+ module_is_loaded libcfs
+ false
+ [[ -f /home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko ]]
+ path=/home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko
+ [[ -n /home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko ]]
++ modinfo --field=depends /home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko
++ tr , ' '
+ '[' 0 -eq 0 ']'
++ basename ../libcfs/libcfs/libcfs
++ tr a-z A-Z
+ optvar=MODOPTS_LIBCFS
+ eval set -- '$MODOPTS_LIBCFS'
++ set --
+ '[' 0 -eq 0 -a -n '' ']'
+ '[' 0 -gt 0 ']'
+ [[ -n /home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko ]]
+ lustre_insmod /home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko
+ local module=/home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko
+ shift
+ local args=
+ local msg
+ local rc=0
+ false
++ insmod /home/green/git/lustre-release/lustre/../libcfs/libcfs/libcfs.ko
+ msg=
+ return 0
+ local 'MSG= conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ MSG=' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+++ nodes_list
+++ local nodes=oleg237-client.virtnet
+++ local nodes_sort
+++ local i
+++ '[' -n oleg237-client.virtnet ']'
+++ nodes=oleg237-client.virtnet
+++ '[' 'pdsh -S -Rssh -w' -a 'pdsh -S -Rssh -w' '!=' no_dsh ']'
+++++ get_facets
+++++ local 'types=OST MDS MGS'
+++++ local list=
+++++ for entry in '$types'
++++++ echo OST
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=ost
++++++ echo OST
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=OST
+++++ case $type in
+++++ local count=OSTCOUNT
+++++ (( i=1 ))
+++++ (( i<=2 ))
+++++ list=' ost1'
+++++ (( i++ ))
+++++ (( i<=2 ))
+++++ list=' ost1 ost2'
+++++ (( i++ ))
+++++ (( i<=2 ))
+++++ for entry in '$types'
++++++ echo MDS
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=mds
++++++ echo MDS
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=MDS
+++++ case $type in
+++++ local count=MDSCOUNT
+++++ (( i=1 ))
+++++ (( i<=1 ))
+++++ list=' ost1 ost2 mds1'
+++++ (( i++ ))
+++++ (( i<=1 ))
+++++ for entry in '$types'
++++++ echo MGS
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=mgs
++++++ echo MGS
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=MGS
+++++ case $type in
+++++ list=' ost1 ost2 mds1 mgs'
++++++ comma_list ost1 ost2 mds1 mgs
++++++ tr ' ' ,
+++++++ tr -s ' ' '\n'
+++++++ sort -b -u
++++++ echo mds1 mgs ost1 ost2
+++++ echo mds1,mgs,ost1,ost2
++++ facets_nodes mds1,mgs,ost1,ost2
++++ local facets=mds1,mgs,ost1,ost2
++++ local facet
++++ local nodes
++++ local nodes_sort
++++ local i
++++ for facet in '${facets//,/ }'
+++++ facet_active_host mds1
++++++ facet_active mds1
++++++ local facet=mds1
++++++ local activevar=mds1active
++++++ '[' -f /tmp/mds1active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n mds1
+++++ facet_host mds1
+++++ local facet=mds1
+++++ local varname
+++++ '[' mds1 == client ']'
+++++ varname=mds1_HOST
+++++ '[' -z '' ']'
+++++ '[' mds == ost ']'
+++++ '[' mds == mdt -o mds == mds -o mds == mgs ']'
+++++ local temp
+++++ '[' mds1 == mgsfailover ']'
+++++ temp=oleg237-server
+++++ eval export mds1_HOST=oleg237-server
++++++ export mds1_HOST=oleg237-server
++++++ mds1_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host mgs
++++++ facet_active mgs
++++++ local facet=mgs
++++++ local activevar=mgsactive
++++++ '[' -f /tmp/mgsactive ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n mgs
+++++ facet_host mgs
+++++ local facet=mgs
+++++ local varname
+++++ '[' mgs == client ']'
+++++ varname=mgs_HOST
+++++ '[' -z oleg237-server ']'
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host ost1
++++++ facet_active ost1
++++++ local facet=ost1
++++++ local activevar=ost1active
++++++ '[' -f /tmp/ost1active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n ost1
+++++ facet_host ost1
+++++ local facet=ost1
+++++ local varname
+++++ '[' ost1 == client ']'
+++++ varname=ost1_HOST
+++++ '[' -z '' ']'
+++++ '[' ost == ost ']'
+++++ local fh=ost1_HOST
+++++ eval export ost1_HOST=
++++++ export ost1_HOST=
++++++ ost1_HOST=
+++++ '[' -z '' ']'
+++++ eval export ost1_HOST=oleg237-server
++++++ export ost1_HOST=oleg237-server
++++++ ost1_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host ost2
++++++ facet_active ost2
++++++ local facet=ost2
++++++ local activevar=ost2active
++++++ '[' -f /tmp/ost2active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n ost2
+++++ facet_host ost2
+++++ local facet=ost2
+++++ local varname
+++++ '[' ost2 == client ']'
+++++ varname=ost2_HOST
+++++ '[' -z '' ']'
+++++ '[' ost == ost ']'
+++++ local fh=ost2_HOST
+++++ eval export ost2_HOST=
++++++ export ost2_HOST=
++++++ ost2_HOST=
+++++ '[' -z '' ']'
+++++ eval export ost2_HOST=oleg237-server
++++++ export ost2_HOST=oleg237-server
++++++ ost2_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server oleg237-server oleg237-server'
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ sort -u
++++ nodes_sort=oleg237-server
++++ echo -n oleg237-server
+++ nodes='oleg237-client.virtnet oleg237-server'
++++ for i in '$nodes'
++++ echo oleg237-client.virtnet
++++ for i in '$nodes'
++++ echo oleg237-server
++++ sort -u
+++ nodes_sort='oleg237-client.virtnet
oleg237-server'
+++ echo -n oleg237-client.virtnet oleg237-server
++ comma_list oleg237-client.virtnet oleg237-server
++ tr ' ' ,
+++ tr -s ' ' '\n'
+++ sort -b -u
++ echo oleg237-client.virtnet oleg237-server
+ do_nodes oleg237-client.virtnet,oleg237-server /home/green/git/lustre-release/lustre/utils/lctl mark ' conf-sanity test_102: @@@@@@ FAIL: libcfs not loaded '
+ true
+ print_stack_trace 2
+ local skip=2
+ echo '  Trace dump:'
  Trace dump:
+ (( i=2 ))
+ (( i < 8  ))
+ local src=/home/green/git/lustre-release/lustre/tests/test-framework.sh
+ local lineno=6639
+ local funcname=error
+ echo '  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6639:error()'
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6639:error()
+ (( i++  ))
+ (( i < 8  ))
+ local src=/home/green/git/lustre-release/lustre/tests/conf-sanity.sh
+ local lineno=8120
+ local funcname=test_102
+ echo '  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:8120:test_102()'
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:8120:test_102()
+ (( i++  ))
+ (( i < 8  ))
+ local src=/home/green/git/lustre-release/lustre/tests/test-framework.sh
+ local lineno=6979
+ local funcname=run_one
+ echo '  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6979:run_one()'
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6979:run_one()
+ (( i++  ))
+ (( i < 8  ))
+ local src=/home/green/git/lustre-release/lustre/tests/test-framework.sh
+ local lineno=7028
+ local funcname=run_one_logged
+ echo '  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7028:run_one_logged()'
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7028:run_one_logged()
+ (( i++  ))
+ (( i < 8  ))
+ local src=/home/green/git/lustre-release/lustre/tests/test-framework.sh
+ local lineno=6850
+ local funcname=run_test
+ echo '  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6850:run_test()'
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6850:run_test()
+ (( i++  ))
+ (( i < 8  ))
+ local src=/home/green/git/lustre-release/lustre/tests/conf-sanity.sh
+ local lineno=8138
+ local funcname=main
+ echo '  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:8138:main()'
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:8138:main()
+ (( i++  ))
+ (( i < 8  ))
+ mkdir -p /tmp/testlogs/
+ true
+++ nodes_list
+++ local nodes=oleg237-client.virtnet
+++ local nodes_sort
+++ local i
+++ '[' -n oleg237-client.virtnet ']'
+++ nodes=oleg237-client.virtnet
+++ '[' 'pdsh -S -Rssh -w' -a 'pdsh -S -Rssh -w' '!=' no_dsh ']'
+++++ get_facets
+++++ local 'types=OST MDS MGS'
+++++ local list=
+++++ for entry in '$types'
++++++ echo OST
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=ost
++++++ echo OST
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=OST
+++++ case $type in
+++++ local count=OSTCOUNT
+++++ (( i=1 ))
+++++ (( i<=2 ))
+++++ list=' ost1'
+++++ (( i++ ))
+++++ (( i<=2 ))
+++++ list=' ost1 ost2'
+++++ (( i++ ))
+++++ (( i<=2 ))
+++++ for entry in '$types'
++++++ echo MDS
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=mds
++++++ echo MDS
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=MDS
+++++ case $type in
+++++ local count=MDSCOUNT
+++++ (( i=1 ))
+++++ (( i<=1 ))
+++++ list=' ost1 ost2 mds1'
+++++ (( i++ ))
+++++ (( i<=1 ))
+++++ for entry in '$types'
++++++ echo MGS
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=mgs
++++++ echo MGS
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=MGS
+++++ case $type in
+++++ list=' ost1 ost2 mds1 mgs'
++++++ comma_list ost1 ost2 mds1 mgs
++++++ tr ' ' ,
+++++++ sort -b -u
+++++++ tr -s ' ' '\n'
++++++ echo mds1 mgs ost1 ost2
+++++ echo mds1,mgs,ost1,ost2
++++ facets_nodes mds1,mgs,ost1,ost2
++++ local facets=mds1,mgs,ost1,ost2
++++ local facet
++++ local nodes
++++ local nodes_sort
++++ local i
++++ for facet in '${facets//,/ }'
+++++ facet_active_host mds1
++++++ facet_active mds1
++++++ local facet=mds1
++++++ local activevar=mds1active
++++++ '[' -f /tmp/mds1active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n mds1
+++++ facet_host mds1
+++++ local facet=mds1
+++++ local varname
+++++ '[' mds1 == client ']'
+++++ varname=mds1_HOST
+++++ '[' -z '' ']'
+++++ '[' mds == ost ']'
+++++ '[' mds == mdt -o mds == mds -o mds == mgs ']'
+++++ local temp
+++++ '[' mds1 == mgsfailover ']'
+++++ temp=oleg237-server
+++++ eval export mds1_HOST=oleg237-server
++++++ export mds1_HOST=oleg237-server
++++++ mds1_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host mgs
++++++ facet_active mgs
++++++ local facet=mgs
++++++ local activevar=mgsactive
++++++ '[' -f /tmp/mgsactive ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n mgs
+++++ facet_host mgs
+++++ local facet=mgs
+++++ local varname
+++++ '[' mgs == client ']'
+++++ varname=mgs_HOST
+++++ '[' -z oleg237-server ']'
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host ost1
++++++ facet_active ost1
++++++ local facet=ost1
++++++ local activevar=ost1active
++++++ '[' -f /tmp/ost1active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n ost1
+++++ facet_host ost1
+++++ local facet=ost1
+++++ local varname
+++++ '[' ost1 == client ']'
+++++ varname=ost1_HOST
+++++ '[' -z '' ']'
+++++ '[' ost == ost ']'
+++++ local fh=ost1_HOST
+++++ eval export ost1_HOST=
++++++ export ost1_HOST=
++++++ ost1_HOST=
+++++ '[' -z '' ']'
+++++ eval export ost1_HOST=oleg237-server
++++++ export ost1_HOST=oleg237-server
++++++ ost1_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host ost2
++++++ facet_active ost2
++++++ local facet=ost2
++++++ local activevar=ost2active
++++++ '[' -f /tmp/ost2active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n ost2
+++++ facet_host ost2
+++++ local facet=ost2
+++++ local varname
+++++ '[' ost2 == client ']'
+++++ varname=ost2_HOST
+++++ '[' -z '' ']'
+++++ '[' ost == ost ']'
+++++ local fh=ost2_HOST
+++++ eval export ost2_HOST=
++++++ export ost2_HOST=
++++++ ost2_HOST=
+++++ '[' -z '' ']'
+++++ eval export ost2_HOST=oleg237-server
++++++ export ost2_HOST=oleg237-server
++++++ ost2_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server oleg237-server oleg237-server'
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ sort -u
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
++++ nodes_sort=oleg237-server
++++ echo -n oleg237-server
+++ nodes='oleg237-client.virtnet oleg237-server'
++++ for i in '$nodes'
++++ echo oleg237-client.virtnet
++++ for i in '$nodes'
++++ echo oleg237-server
++++ sort -u
+++ nodes_sort='oleg237-client.virtnet
oleg237-server'
+++ echo -n oleg237-client.virtnet oleg237-server
++ comma_list oleg237-client.virtnet oleg237-server
++ tr ' ' ,
+++ tr -s ' ' '\n'
+++ sort -b -u
++ echo oleg237-client.virtnet oleg237-server
+ gather_logs oleg237-client.virtnet,oleg237-server
+ local list=oleg237-client.virtnet,oleg237-server
++ date +%s
+ local ts=1688471782
+ local docp=true
+ [[ ! -f /tmp/testlogs//results.yml ]]
+ '[' -f /tmp/testlogs//shared ']'
+ prefix=/tmp/testlogs//conf-sanity.test_102
+ suffix=1688471782.log
+ echo 'Dumping lctl log to /tmp/testlogs//conf-sanity.test_102.*.1688471782.log'
Dumping lctl log to /tmp/testlogs//conf-sanity.test_102.*.1688471782.log
+ '[' -n '' -o 'pdsh -S -Rssh -w' == no_dsh ']'
+ do_nodesv oleg237-client.virtnet,oleg237-server '/home/green/git/lustre-release/lustre/utils/lctl dk > /tmp/testlogs//conf-sanity.test_102.debug_log.\$(hostname -s).1688471782.log;
		dmesg > /tmp/testlogs//conf-sanity.test_102.dmesg.\$(hostname -s).1688471782.log'
+ do_nodes --verbose oleg237-client.virtnet,oleg237-server '/home/green/git/lustre-release/lustre/utils/lctl dk > /tmp/testlogs//conf-sanity.test_102.debug_log.\$(hostname -s).1688471782.log;
		dmesg > /tmp/testlogs//conf-sanity.test_102.dmesg.\$(hostname -s).1688471782.log'
+ local verbose
+ local quiet
+ [[ --verbose == \-\-\v\e\r\b\o\s\e ]]
+ verbose=--verbose
+ shift
+ [[ oleg237-client.virtnet,oleg237-server == \-\-\q\u\i\e\t ]]
+ [[ oleg237-client.virtnet,oleg237-server == \-\q ]]
+ local rnodes=oleg237-client.virtnet,oleg237-server
+ shift
+ single_local_node oleg237-client.virtnet,oleg237-server
+ '[' oleg237-client.virtnet,oleg237-server = oleg237-client.virtnet ']'
+ local 'myPDSH=pdsh -S -Rssh -w'
+ '[' -z 'pdsh -S -Rssh -w' -o 'pdsh -S -Rssh -w' = no_dsh -o 'pdsh -S -Rssh -w' = rsh ']'
++ get_node_count 'oleg237-client.virtnet oleg237-server'
++ local 'nodes=oleg237-client.virtnet oleg237-server'
++ echo oleg237-client.virtnet oleg237-server
++ wc -w
+ export FANOUT=2
+ FANOUT=2
+ false
+ [[ -n --verbose ]]
++ get_env_vars
++ local var
++ local value
+++ get_facets
+++ local 'types=OST MDS MGS'
+++ local list=
+++ for entry in '$types'
++++ echo OST
++++ tr '[:upper:]' '[:lower:]'
+++ local name=ost
++++ echo OST
++++ tr '[:lower:]' '[:upper:]'
+++ local type=OST
+++ case $type in
+++ local count=OSTCOUNT
+++ (( i=1 ))
+++ (( i<=2 ))
+++ list=' ost1'
+++ (( i++ ))
+++ (( i<=2 ))
+++ list=' ost1 ost2'
+++ (( i++ ))
+++ (( i<=2 ))
+++ for entry in '$types'
++++ echo MDS
++++ tr '[:upper:]' '[:lower:]'
+++ local name=mds
++++ echo MDS
++++ tr '[:lower:]' '[:upper:]'
+++ local type=MDS
+++ case $type in
+++ local count=MDSCOUNT
+++ (( i=1 ))
+++ (( i<=1 ))
+++ list=' ost1 ost2 mds1'
+++ (( i++ ))
+++ (( i<=1 ))
+++ for entry in '$types'
++++ echo MGS
++++ tr '[:upper:]' '[:lower:]'
+++ local name=mgs
++++ echo MGS
++++ tr '[:lower:]' '[:upper:]'
+++ local type=MGS
+++ case $type in
+++ list=' ost1 ost2 mds1 mgs'
++++ comma_list ost1 ost2 mds1 mgs
++++ tr ' ' ,
+++++ tr -s ' ' '\n'
+++++ sort -b -u
++++ echo mds1 mgs ost1 ost2
+++ echo mds1,mgs,ost1,ost2
++ local facets=mds1,mgs,ost1,ost2
++ local facet
++ for var in '${!MODOPTS_*}'
++ value=lbug_on_grant_miscount=1
++ echo -n ' MODOPTS_PTLRPC="lbug_on_grant_miscount=1"'
++ for facet in '${facets//,/ }'
++ var=mds1_FSTYPE
++ '[' -n zfs ']'
++ echo -n ' mds1_FSTYPE=zfs'
++ for facet in '${facets//,/ }'
++ var=mgs_FSTYPE
++ '[' -n '' ']'
++ for facet in '${facets//,/ }'
++ var=ost1_FSTYPE
++ '[' -n zfs ']'
++ echo -n ' ost1_FSTYPE=zfs'
++ for facet in '${facets//,/ }'
++ var=ost2_FSTYPE
++ '[' -n '' ']'
++ for var in MGSFSTYPE MDSFSTYPE OSTFSTYPE
++ '[' -n '' ']'
++ for var in MGSFSTYPE MDSFSTYPE OSTFSTYPE
++ '[' -n '' ']'
++ for var in MGSFSTYPE MDSFSTYPE OSTFSTYPE
++ '[' -n '' ']'
++ for var in VERBOSE
++ '[' -n false ']'
++ echo -n ' VERBOSE=false'
++ '[' -n zfs ']'
++ echo -n ' FSTYPE=zfs'
++ for var in LNETLND NETTYPE
++ '[' -n socklnd/ksocklnd ']'
++ echo -n ' LNETLND=socklnd/ksocklnd'
++ for var in LNETLND NETTYPE
++ '[' -n tcp ']'
++ echo -n ' NETTYPE=tcp'
+ pdsh -S -Rssh -w oleg237-client.virtnet,oleg237-server '(PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin; cd /root; LUSTRE="/home/green/git/lustre-release/lustre"  MODOPTS_PTLRPC="lbug_on_grant_miscount=1" mds1_FSTYPE=zfs ost1_FSTYPE=zfs VERBOSE=false FSTYPE=zfs LNETLND=socklnd/ksocklnd NETTYPE=tcp bash -c "/home/green/git/lustre-release/lustre/utils/lctl dk > /tmp/testlogs//conf-sanity.test_102.debug_log.\$(hostname -s).1688471782.log;
		dmesg > /tmp/testlogs//conf-sanity.test_102.dmesg.\$(hostname -s).1688471782.log")'
oleg237-server: invalid parameter 'dump_kernel'
oleg237-server: open(dump_kernel) failed: No such file or directory
+ return 0
+ '[' false = true ']'
+ '[' false = true ']'
+ '[' '!' -f /tmp/testlogs//shared ']'
++ exclude_items_from_list oleg237-client.virtnet,oleg237-server oleg237-client.virtnet
++ local list=oleg237-client.virtnet,oleg237-server
++ local excluded=oleg237-client.virtnet
++ local item
++ list='oleg237-client.virtnet oleg237-server'
++ for item in '${excluded//,/ }'
+++ echo ' oleg237-client.virtnet oleg237-server '
+++ sed -re 's/\s+oleg237-client.virtnet\s+/ /g'
++ list=' oleg237-server '
+++ comma_list oleg237-server
+++ tr ' ' ,
++++ tr -s ' ' '\n'
++++ sort -b -u
+++ echo oleg237-server
++ echo oleg237-server
+ local remote_nodes=oleg237-server
+ for node in '${remote_nodes//,/ }'
+ debugrestore
+ rsync -az -e ssh 'oleg237-server:/tmp/testlogs//conf-sanity.test_102.*.1688471782.log' /tmp/testlogs/
+ '[' -n '' ']'
+ true
+ DEBUGSAVE=
+ '[' -n '' ']'
+ true
+ DEBUGSAVE_SERVER=
+ '[' /tmp/conf-sanity.log ']'
+ echo 'conf-sanity: FAIL: test_102 libcfs not loaded'
+ '[' -z 'libcfs not loaded' ']'
++ echo FAIL
++ grep '^IGNORE'
+ [[ -n '' ]]
+ echo 'libcfs not loaded'
+ reset_fail_loc
+++ nodes_list
+++ local nodes=oleg237-client.virtnet
+++ local nodes_sort
+++ local i
+++ '[' -n oleg237-client.virtnet ']'
+++ nodes=oleg237-client.virtnet
+++ '[' 'pdsh -S -Rssh -w' -a 'pdsh -S -Rssh -w' '!=' no_dsh ']'
+++++ get_facets
+++++ local 'types=OST MDS MGS'
+++++ local list=
+++++ for entry in '$types'
++++++ echo OST
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=ost
++++++ echo OST
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=OST
+++++ case $type in
+++++ local count=OSTCOUNT
+++++ (( i=1 ))
+++++ (( i<=2 ))
+++++ list=' ost1'
+++++ (( i++ ))
+++++ (( i<=2 ))
+++++ list=' ost1 ost2'
+++++ (( i++ ))
+++++ (( i<=2 ))
+++++ for entry in '$types'
++++++ echo MDS
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=mds
++++++ tr '[:lower:]' '[:upper:]'
++++++ echo MDS
+++++ local type=MDS
+++++ case $type in
+++++ local count=MDSCOUNT
+++++ (( i=1 ))
+++++ (( i<=1 ))
+++++ list=' ost1 ost2 mds1'
+++++ (( i++ ))
+++++ (( i<=1 ))
+++++ for entry in '$types'
++++++ echo MGS
++++++ tr '[:upper:]' '[:lower:]'
+++++ local name=mgs
++++++ echo MGS
++++++ tr '[:lower:]' '[:upper:]'
+++++ local type=MGS
+++++ case $type in
+++++ list=' ost1 ost2 mds1 mgs'
++++++ comma_list ost1 ost2 mds1 mgs
++++++ tr ' ' ,
+++++++ tr -s ' ' '\n'
+++++++ sort -b -u
++++++ echo mds1 mgs ost1 ost2
+++++ echo mds1,mgs,ost1,ost2
++++ facets_nodes mds1,mgs,ost1,ost2
++++ local facets=mds1,mgs,ost1,ost2
++++ local facet
++++ local nodes
++++ local nodes_sort
++++ local i
++++ for facet in '${facets//,/ }'
+++++ facet_active_host mds1
++++++ facet_active mds1
++++++ local facet=mds1
++++++ local activevar=mds1active
++++++ '[' -f /tmp/mds1active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n mds1
+++++ facet_host mds1
+++++ local facet=mds1
+++++ local varname
+++++ '[' mds1 == client ']'
+++++ varname=mds1_HOST
+++++ '[' -z '' ']'
+++++ '[' mds == ost ']'
+++++ '[' mds == mdt -o mds == mds -o mds == mgs ']'
+++++ local temp
+++++ '[' mds1 == mgsfailover ']'
+++++ temp=oleg237-server
+++++ eval export mds1_HOST=oleg237-server
++++++ export mds1_HOST=oleg237-server
++++++ mds1_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host mgs
++++++ facet_active mgs
++++++ local facet=mgs
++++++ local activevar=mgsactive
++++++ '[' -f /tmp/mgsactive ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n mgs
+++++ facet_host mgs
+++++ local facet=mgs
+++++ local varname
+++++ '[' mgs == client ']'
+++++ varname=mgs_HOST
+++++ '[' -z oleg237-server ']'
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host ost1
++++++ facet_active ost1
++++++ local facet=ost1
++++++ local activevar=ost1active
++++++ '[' -f /tmp/ost1active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n ost1
+++++ facet_host ost1
+++++ local facet=ost1
+++++ local varname
+++++ '[' ost1 == client ']'
+++++ varname=ost1_HOST
+++++ '[' -z '' ']'
+++++ '[' ost == ost ']'
+++++ local fh=ost1_HOST
+++++ eval export ost1_HOST=
++++++ export ost1_HOST=
++++++ ost1_HOST=
+++++ '[' -z '' ']'
+++++ eval export ost1_HOST=oleg237-server
++++++ export ost1_HOST=oleg237-server
++++++ ost1_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server oleg237-server'
++++ for facet in '${facets//,/ }'
+++++ facet_active_host ost2
++++++ facet_active ost2
++++++ local facet=ost2
++++++ local activevar=ost2active
++++++ '[' -f /tmp/ost2active ']'
++++++ active=
++++++ '[' -z '' ']'
++++++ echo -n ost2
+++++ facet_host ost2
+++++ local facet=ost2
+++++ local varname
+++++ '[' ost2 == client ']'
+++++ varname=ost2_HOST
+++++ '[' -z '' ']'
+++++ '[' ost == ost ']'
+++++ local fh=ost2_HOST
+++++ eval export ost2_HOST=
++++++ export ost2_HOST=
++++++ ost2_HOST=
+++++ '[' -z '' ']'
+++++ eval export ost2_HOST=oleg237-server
++++++ export ost2_HOST=oleg237-server
++++++ ost2_HOST=oleg237-server
+++++ echo -n oleg237-server
++++ nodes=' oleg237-server oleg237-server oleg237-server oleg237-server'
+++++ for i in '$nodes'
+++++ sort -u
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
+++++ for i in '$nodes'
+++++ echo oleg237-server
++++ nodes_sort=oleg237-server
++++ echo -n oleg237-server
+++ nodes='oleg237-client.virtnet oleg237-server'
++++ sort -u
++++ for i in '$nodes'
++++ echo oleg237-client.virtnet
++++ for i in '$nodes'
++++ echo oleg237-server
+++ nodes_sort='oleg237-client.virtnet
oleg237-server'
+++ echo -n oleg237-client.virtnet oleg237-server
++ comma_list oleg237-client.virtnet oleg237-server
++ tr ' ' ,
+++ tr -s ' ' '\n'
+++ sort -b -u
++ echo oleg237-client.virtnet oleg237-server
+ do_nodes --quiet oleg237-client.virtnet,oleg237-server 'lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null'
+ local verbose
+ local quiet
+ [[ --quiet == \-\-\v\e\r\b\o\s\e ]]
+ [[ --quiet == \-\-\q\u\i\e\t ]]
+ quiet=--quiet
+ shift
+ local rnodes=oleg237-client.virtnet,oleg237-server
+ shift
+ single_local_node oleg237-client.virtnet,oleg237-server
+ '[' oleg237-client.virtnet,oleg237-server = oleg237-client.virtnet ']'
+ local 'myPDSH=pdsh -S -Rssh -w'
+ '[' -z 'pdsh -S -Rssh -w' -o 'pdsh -S -Rssh -w' = no_dsh -o 'pdsh -S -Rssh -w' = rsh ']'
++ get_node_count 'oleg237-client.virtnet oleg237-server'
++ local 'nodes=oleg237-client.virtnet oleg237-server'
++ echo oleg237-client.virtnet oleg237-server
++ wc -w
+ export FANOUT=2
+ FANOUT=2
+ false
+ [[ -n '' ]]
+ [[ pdsh -S -Rssh -w = *-N* ]]
+ sed -re 's/^[^:]*: //g'
++ get_env_vars
++ local var
++ local value
+++ get_facets
+++ local 'types=OST MDS MGS'
+++ local list=
+++ for entry in '$types'
++++ echo OST
++++ tr '[:upper:]' '[:lower:]'
+++ local name=ost
++++ echo OST
++++ tr '[:lower:]' '[:upper:]'
+++ local type=OST
+++ case $type in
+++ local count=OSTCOUNT
+++ (( i=1 ))
+++ (( i<=2 ))
+++ list=' ost1'
+++ (( i++ ))
+++ (( i<=2 ))
+++ list=' ost1 ost2'
+++ (( i++ ))
+++ (( i<=2 ))
+++ for entry in '$types'
++++ echo MDS
++++ tr '[:upper:]' '[:lower:]'
+++ local name=mds
++++ echo MDS
++++ tr '[:lower:]' '[:upper:]'
+++ local type=MDS
+++ case $type in
+++ local count=MDSCOUNT
+++ (( i=1 ))
+++ (( i<=1 ))
+++ list=' ost1 ost2 mds1'
+++ (( i++ ))
+++ (( i<=1 ))
+++ for entry in '$types'
++++ echo MGS
++++ tr '[:upper:]' '[:lower:]'
+++ local name=mgs
++++ echo MGS
++++ tr '[:lower:]' '[:upper:]'
+++ local type=MGS
+++ case $type in
+++ list=' ost1 ost2 mds1 mgs'
++++ comma_list ost1 ost2 mds1 mgs
++++ tr ' ' ,
+++++ tr -s ' ' '\n'
+++++ sort -b -u
++++ echo mds1 mgs ost1 ost2
+++ echo mds1,mgs,ost1,ost2
++ local facets=mds1,mgs,ost1,ost2
++ local facet
++ for var in '${!MODOPTS_*}'
++ value=lbug_on_grant_miscount=1
++ echo -n ' MODOPTS_PTLRPC="lbug_on_grant_miscount=1"'
++ for facet in '${facets//,/ }'
++ var=mds1_FSTYPE
++ '[' -n zfs ']'
++ echo -n ' mds1_FSTYPE=zfs'
++ for facet in '${facets//,/ }'
++ var=mgs_FSTYPE
++ '[' -n '' ']'
++ for facet in '${facets//,/ }'
++ var=ost1_FSTYPE
++ '[' -n zfs ']'
++ echo -n ' ost1_FSTYPE=zfs'
++ for facet in '${facets//,/ }'
++ var=ost2_FSTYPE
++ '[' -n '' ']'
++ for var in MGSFSTYPE MDSFSTYPE OSTFSTYPE
++ '[' -n '' ']'
++ for var in MGSFSTYPE MDSFSTYPE OSTFSTYPE
++ '[' -n '' ']'
++ for var in MGSFSTYPE MDSFSTYPE OSTFSTYPE
++ '[' -n '' ']'
++ for var in VERBOSE
++ '[' -n false ']'
++ echo -n ' VERBOSE=false'
++ '[' -n zfs ']'
++ echo -n ' FSTYPE=zfs'
++ for var in LNETLND NETTYPE
++ '[' -n socklnd/ksocklnd ']'
++ echo -n ' LNETLND=socklnd/ksocklnd'
++ for var in LNETLND NETTYPE
++ '[' -n tcp ']'
++ echo -n ' NETTYPE=tcp'
+ pdsh -S -Rssh -w oleg237-client.virtnet,oleg237-server '(PATH=$PATH:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin; cd /root; LUSTRE="/home/green/git/lustre-release/lustre"  MODOPTS_PTLRPC="lbug_on_grant_miscount=1" mds1_FSTYPE=zfs ost1_FSTYPE=zfs VERBOSE=false FSTYPE=zfs LNETLND=socklnd/ksocklnd NETTYPE=tcp bash -c "lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null")'
rsync: chown "/tmp/testlogs/.conf-sanity.test_102.debug_log.oleg237-server.1688471782.log.mvtUJZ" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.conf-sanity.test_102.dmesg.oleg237-server.1688471782.log.LsHIRi" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2
+ return 2
+ true
+ exit 1
FAIL 102 (11s)

== conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:56:25 (1688471785)
Checking servers environments
Checking clients oleg237-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
mount lustre  on /mnt/lustre.....
Starting client: oleg237-client.virtnet:  -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre
Starting client oleg237-client.virtnet:  -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre
Started clients oleg237-client.virtnet: 
192.168.201.137@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800c933b800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800c933b800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 5s: want 'procname_uid' got 'procname_uid'
disable quota as required
- { index: 8, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 467 }
- { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=467 }
- { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID }
- { index: 3, event: setup, device: lustre-clilov, UUID:  }
- { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID }
- { index: 7, event: setup, device: lustre-clilmv, UUID:  }
- { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv }
- { index: 13, event: add_uuid, nid: 192.168.201.137@tcp(0x20000c0a8c989), node: 192.168.201.137@tcp }
- { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.201.137@tcp }
- { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID }
- { index: 22, event: add_uuid, nid: 192.168.201.137@tcp(0x20000c0a8c989), node: 192.168.201.137@tcp }
- { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 24, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.137@tcp }
- { index: 25, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }
- { index: 31, event: add_uuid, nid: 192.168.201.137@tcp(0x20000c0a8c989), node: 192.168.201.137@tcp }
- { index: 32, event: attach, device: lustre-OST0001-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 33, event: setup, device: lustre-OST0001-osc, UUID: lustre-OST0001_UUID, node: 192.168.201.137@tcp }
- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0001_UUID, index: 1, gen: 1 }
- { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 }
- { index: 40, event: conf_param, parameter: sys.jobid_var=procname_uid, value: procname_uid }
- { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=256 }
- { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=467 }
- { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID }
- { index: 3, event: setup, device: lustre-clilov, UUID:  }
- { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID }
- { index: 7, event: setup, device: lustre-clilmv, UUID:  }
- { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv }
- { index: 13, event: add_uuid, nid: 192.168.201.137@tcp(0x20000c0a8c989), node: 192.168.201.137@tcp }
- { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID }
- { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.201.137@tcp }
- { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID }
- { index: 22, event: add_uuid, nid: 192.168.201.137@tcp(0x20000c0a8c989), node: 192.168.201.137@tcp }
- { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 24, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.137@tcp }
- { index: 25, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }
- { index: 31, event: add_uuid, nid: 192.168.201.137@tcp(0x20000c0a8c989), node: 192.168.201.137@tcp }
- { index: 32, event: attach, device: lustre-OST0001-osc, type: osc, UUID: lustre-clilov_UUID }
- { index: 33, event: setup, device: lustre-OST0001-osc, UUID: lustre-OST0001_UUID, node: 192.168.201.137@tcp }
- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0001_UUID, index: 1, gen: 1 }
- { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 }
- { index: 40, event: conf_param, parameter: sys.jobid_var=procname_uid, value: procname_uid }
- { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=256 }
PASS 123ae (37s)

== conf-sanity test 134: check_iam works without faults == 07:57:02 (1688471822)

 SKIP: conf-sanity test_134 ldiskfs only test
SKIP 134 (1s)
Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg237-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server
oleg237-server: oleg237-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 409 sec =========== 07:57:37 (1688471857)
conf-sanity: FAIL: test_102 libcfs not loaded
=== conf-sanity: start cleanup 07:57:37 (1688471857) ===
=== conf-sanity: finish cleanup 07:57:37 (1688471857) ===