-----============= acceptance-small: conf-sanity ============----- Fri Jun 12 12:12:49 EDT 2026
oleg202-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory
excepting tests: 55 63 67 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg202-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server

== conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:17:10 (1781281030)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1

   Permanent disk data:
Target:     test-33a:MDT0000
Index:      0
Lustre FS:  test-33a
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=test-33a
  lustre:svname=test-33a:MDT0000

   Permanent disk data:
Target:     test-33a:OST1fff
Index:      8191
Lustre FS:  test-33a
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters:  sys.timeout=20 mgsnode=192.168.202.102@tcp:192.168.202.102@tcp autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.202.102@tcp:192.168.202.102@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=8191
  lustre:fsname=test-33a
  lustre:svname=test-33a:OST1fff
Starting fs2mds: -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started test-33a-MDT0000
Starting fs2ost: -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started test-33a-OST1fff
ok.
/mnt/lustre2/hosts
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 8191
	obdidx		 objid		 objid		 group
	  8191	             2	          0x2	             0

Stopping /mnt/lustre-fs2ost (opts:-f) on oleg202-server
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg202-server
stop ost1 service on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
umount /mnt/lustre2 ...
umount: /mnt/lustre2: not mounted.
stopping fs2mds ...
stopping fs2ost ...
PASS 33a (130s)

== conf-sanity test 33b: Drop cancel during umount ======= 12:19:20 (1781281160)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260203 s, 4.0 MB/s
fail_loc=0x80000304
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 33b (162s)

== conf-sanity test 34a: umount with opened file should be fail ========================================================== 12:22:02 (1781281322)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
multiop /mnt/lustre/file vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.20776
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: target is busy.
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 34a (165s)

== conf-sanity test 34b: force umount with failed mds should be normal ========================================================== 12:24:47 (1781281487)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
manual umount lustre on /mnt/lustre....
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 34b (168s)

== conf-sanity test 34c: force umount with failed ost should be normal ========================================================== 12:27:35 (1781281655)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
manual umount lustre on /mnt/lustre....
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 34c (188s)

== conf-sanity test 35a: Reconnect to the last active server first ========================================================== 12:30:43 (1781281843)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2026-06-1212h32m48s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (32354) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 191 lines, 191 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 35a (436s)

== conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 12:38:00 (1781282280)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
debug=ha
conf-sanity.sh test_35b 2026-06-1212h39m18s
Set up a fake failnode for the MDS
at_max=0
at_max=0
Injecting EBUSY on MDS
fail_loc=0x80000136
mdc.lustre-MDT0000-mdc-ffff8c7945f57000.stats=clear
mdc.lustre-MDT0001-mdc-ffff8c7945f57000.stats=clear
Creating a test file and stat it
  File: /mnt/lustre/d35b.conf-sanity/f35b.conf-sanity
  Size: 0         	Blocks: 1          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115238826934274  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-12 12:40:14.000000000 -0400
Modify: 2026-06-12 12:40:14.000000000 -0400
Change: 2026-06-12 12:40:14.000000000 -0400
 Birth: 2026-06-12 12:40:14.000000000 -0400
Stop injecting EBUSY on MDS
fail_loc=0
done
at_max=600
at_max=600
Debug log: 74 lines, 74 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 35b (396s)

== conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 12:44:37 (1781282677)

 SKIP: conf-sanity test_36 remote OST
SKIP 36 (4s)

== conf-sanity test 37: verify set tunables works for symlink device ========================================================== 12:44:41 (1781282681)

 SKIP: conf-sanity test_37 ldiskfs only test
SKIP 37 (4s)

== conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:44:46 (1781282686)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
copying 10 files to /mnt/lustre/d38.conf-sanity
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35
osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=43
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
delete lov_objid file on MDS
000000                   34
000008
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=66
osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
000000                   66
000008
oleg202-server: 8+0 records in
oleg202-server: 8+0 records out
oleg202-server: 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000613572 s, 6.7 MB/s
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=99
osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
000000                   98
000008
files compared the same
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 38 (316s)

== conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ========================================================== 12:50:02 (1781283002)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug +malloc all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug +malloc all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug +malloc all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 39 (137s)

== conf-sanity test 40: race during service thread startup ========================================================== 12:52:19 (1781283139)
start ost1 service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
fail_loc=0x80000706
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 40 (231s)

== conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================================== 12:56:10 (1781283370)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov -o nosvc -n  lustre-mdt1/mdt1 /mnt/lustre-mds1
nomtab: 1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
Start lustre-mdt1/mdt1 without service
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
start mds service on oleg202-server
Starting mds1: -o localrecov -o nomgs,force  lustre-mdt1/mdt1 /mnt/lustre-mds1
force: 1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
PASS 41a (126s)

== conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:58:16 (1781283496)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov -o nosvc -n  lustre-mdt1/mdt1 /mnt/lustre-mds1
nomtab: 1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
Start lustre-mdt1/mdt1 without service
Commit the device label on lustre-mdt1/mdt1
Started lustre:MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg202-server
Starting mds1: -o localrecov -o nomgs,force  lustre-mdt1/mdt1 /mnt/lustre-mds1
force: 1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
PASS 41b (236s)

== conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:02:12 (1781283732)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
fail_loc=0x80000716
oleg202-server: mount.lustre: mount lustre-mdt1/mdt1 at /mnt/lustre-mds1 failed: Operation already in progress
oleg202-server: The target service is already running. (lustre-mdt1/mdt1)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 114
fail_loc=0x0
1st MDT start succeed
2nd MDT start failed with 114
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-server: oleg202-server.virtnet: executing lsmod
fail_loc=0x80000716
oleg202-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Operation already in progress
oleg202-server: The target service is already running. (lustre-ost1/ost1)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 114
fail_loc=0x0
1st OST start succeed
2nd OST start failed with 114
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 41c (377s)

== conf-sanity test 42: allow client/server mount/unmount with invalid config param ========================================================== 13:08:29 (1781284109)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 42 (286s)

== conf-sanity test 43a: check root_squash and nosquash_nids ========================================================== 13:13:15 (1781284395)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
Setting lustre.mdt.root_squash from 0:0 to 0:0
Setting lustre.mdt.nosquash_nids from NONE to NONE
Setting lustre.mdt.root_squash from 0:0 to 500:500
Waiting 90s for '500:500'
Updated after 6s: want '500:500' got '500:500'
/mnt/lustre/f43a.conf-sanity-userfile: owner uid 500 (-rw-------): root read permission is granted - ok
/mnt/lustre/f43a.conf-sanity-userfile: owner uid 500 (-rw-------): root write permission is granted - ok
/mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is denied - ok
/mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is denied - ok
/mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is denied - ok
/mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is denied - ok
/mnt/lustre/f43a.conf-sanity-user1file: owner uid 501 (-rw-------): root read permission is denied - ok
/mnt/lustre/f43a.conf-sanity-user1file: owner uid 501 (-rw-------): root write permission is denied - ok
/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 3777: 83945 Terminated              runas -u $ID1 tail -f $DIR/$tfile-user1file > /dev/null 2>&1
Setting lustre-MDTall.mdt.nosquash_nids from NONE to 2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp
Waiting 90s for '2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp'
Updated after 2s: want '2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp' got '2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp'
/mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root read permission is granted - ok
/mnt/lustre/f43a.conf-sanity-rootfile: owner uid 0 (-rw-------): root write permission is granted - ok
/mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root unlink permission is granted - ok
/mnt/lustre/d43a.conf-sanity-rootdir: owner uid 0 (drwx------): root create permission is granted - ok
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 43a (169s)

== conf-sanity test 43b: parse nosquash_nids with commas in expr_list ========================================================== 13:16:04 (1781284564)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
modprobe: FATAL: Module obdclass not found in directory /lib/modules/4.18.0rh8.10-debug
modprobe: FATAL: Module ptlrpc not found in directory /lib/modules/4.18.0rh8.10-debug
modprobe: FATAL: Module lnet not found in directory /lib/modules/4.18.0rh8.10-debug
modprobe: FATAL: Module lmv not found in directory /lib/modules/4.18.0rh8.10-debug
modprobe: FATAL: Module mdc not found in directory /lib/modules/4.18.0rh8.10-debug
modprobe: FATAL: Module lov not found in directory /lib/modules/4.18.0rh8.10-debug
insmod: ERROR: could not insert module /home/green/git/lustre-release/lustre/llite/lustre.ko: Unknown symbol in module
IOC_LIBCFS_GET_NI error 22: Invalid argument

   Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS:  test1234
Mount type: zfs
Flags:      0x64
              (MGS first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.root_squash=500:500 mdt.nosquash_nids=[,,]@tcp
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:sys.timeout=20
  lustre:mdt.root_squash=500:500
  lustre:mdt.nosquash_nids=[,,]@tcp
  lustre:version=1
  lustre:flags=100
  lustre:index=65535
  lustre:fsname=test1234
  lustre:svname=MGS
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting fs2mds: -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started MGS
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg202-server
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 43b (99s)

== conf-sanity test 44: mounted client proc entry exists ========================================================== 13:17:44 (1781284664)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 44 (137s)

 SKIP: conf-sanity test_45 skipping SLOW test 45

== conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 13:20:04 (1781284804)
Testing with 2 OSTs
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c79474d4800.ost_server_uuid 40
osc.lustre-OST0001-osc-ffff8c79474d4800.ost_server_uuid in IDLE state after 0 sec
mount lustre  on /mnt/lustre2.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre2
/mnt/lustre2
stripe_count:  -1 stripe_size:   1048576 pattern:       raid0 stripe_offset: -1

/mnt/lustre2/widestripe
lmm_stripe_count:  2
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	            34	         0x22	             0
	     1	             2	          0x2	             0

  File: /mnt/lustre/widestripe
  Size: 3         	Blocks: 2          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115238826934273  Links: 1
Access: (0674/-rw-rwxr--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-12 13:25:02.000000000 -0400
Modify: 2026-06-12 13:25:03.000000000 -0400
Change: 2026-06-12 13:25:04.000000000 -0400
 Birth: 2026-06-12 13:25:03.000000000 -0400
umount lustre on /mnt/lustre2.....
Stopping client oleg202-client.virtnet /mnt/lustre2 (opts:)
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
stop ost1 service on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 46a (588s)

== conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 13:29:52 (1781285392)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
ldlm.namespaces.lustre-MDT0000-mdc-ffff8c79478e3800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8c79478e3800.lru_size=100
ldlm.namespaces.lustre-OST0000-osc-ffff8c79478e3800.lru_size=100
ldlm.namespaces.lustre-MDT0000-mdc-ffff8c79478e3800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8c79478e3800.lru_size=100
Failing ost1 on oleg202-server
Stopping /mnt/lustre-ost1 (opts:) on oleg202-server
reboot facets: ost1
Failover ost1 to oleg202-server
mount facets: ost1
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
Failing mds1 on oleg202-server
Stopping /mnt/lustre-mds1 (opts:) on oleg202-server
reboot facets: mds1
Failover mds1 to oleg202-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
ldlm.namespaces.lustre-MDT0000-mdc-ffff8c79478e3800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff8c79478e3800.lru_size=100
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 47 (288s)

== conf-sanity test 48: too many acls on file ============ 13:34:40 (1781285680)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
/mnt/lustre
stripe_count:  -1 stripe_size:   1048576 pattern:       raid0 stripe_offset: -1

/mnt/lustre/widestripe
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	            99	         0x63	             0

It is expected to hold at least 4500 ACL entries
  File: /mnt/lustre/widestripe
  Size: 3         	Blocks: 1          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115272381366274  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-12 13:36:12.000000000 -0400
Modify: 2026-06-12 13:36:12.000000000 -0400
Change: 2026-06-12 13:46:26.000000000 -0400
 Birth: 2026-06-12 13:36:12.000000000 -0400
getfacl: Removing leading '/' from absolute path names
Failing mds1 on oleg202-server
Stopping /mnt/lustre-mds1 (opts:) on oleg202-server
reboot facets: mds1
Failover mds1 to oleg202-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg202-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 48 (972s)

== conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:50:52 (1781286652)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
check ldlm_timout...
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 49a (200s)

== conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:54:12 (1781286852)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 49b (215s)

== conf-sanity test 50a: lazystatfs all servers available ========================================================== 13:57:47 (1781287067)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
llite.lustre-ffff8c7945521000.lazystatfs=1
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7877
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3200      378368   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID       382976        3072      377856   1% /mnt/lustre[OST:0]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4126: kill: (132844) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 50a (154s)

== conf-sanity test 50b: lazystatfs all servers down ===== 14:00:21 (1781287221)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
llite.lustre-ffff8c7945cba000.lazystatfs=1
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
oleg202-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8c7945cba000.ost_server_uuid 40
  0 UP mgc MGC192.168.202.102@tcp 212a646f-ea28-437c-8e7c-14e3d5d0495e 4
  1 UP lov lustre-clilov-ffff8c7945cba000 41964848-f8ed-4a39-9045-13aa5b72dde6 3
  2 UP lmv lustre-clilmv-ffff8c7945cba000 41964848-f8ed-4a39-9045-13aa5b72dde6 4
  3 UP mdc lustre-MDT0000-mdc-ffff8c7945cba000 41964848-f8ed-4a39-9045-13aa5b72dde6 4
  4 UP mdc lustre-MDT0001-mdc-ffff8c7945cba000 41964848-f8ed-4a39-9045-13aa5b72dde6 4
  5 UP osc lustre-OST0000-osc-ffff8c7945cba000 41964848-f8ed-4a39-9045-13aa5b72dde6 4
OSCs should all be DISCONN
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7877
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3200      378368   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]

filesystem_summary:       767232        6272      756864   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4126: kill: (136974) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 50b (156s)

== conf-sanity test 50c: lazystatfs one server down ====== 14:02:58 (1781287378)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
llite.lustre-ffff8c7944376000.lazystatfs=1
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
oleg202-server: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7877
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       382976        3072      377856   1% /mnt/lustre[OST:1]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4126: kill: (139835) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 50c (351s)

== conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:08:48 (1781287728)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
llite.lustre-ffff8c7947526800.lazystatfs=1
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
multiop /mnt/lustre vf_
TMPPIPE=/tmp/multiop_open_wait_pipe.7877
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]
lustre-OST0001_UUID       382976        3072      377856   1% /mnt/lustre[OST:1]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

/home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4126: kill: (146651) - No such process
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 50d (347s)

== conf-sanity test 50e: normal statfs all servers down == 14:14:36 (1781288076)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
oleg202-server: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
llite.lustre-ffff8c79458e5000.lazystatfs=0
multiop /mnt/lustre v_f
TMPPIPE=/tmp/multiop_open_wait_pipe.7877
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 50e (353s)

== conf-sanity test 50f: normal statfs one server in down ========================================================== 14:20:29 (1781288429)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
oleg202-server: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
llite.lustre-ffff8c794711e000.lazystatfs=0
multiop /mnt/lustre v_f
TMPPIPE=/tmp/multiop_open_wait_pipe.7877
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 50f (447s)

== conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:27:56 (1781288876)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c794643a000.ost_server_uuid 40
osc.lustre-OST0001-osc-ffff8c794643a000.ost_server_uuid in IDLE state after 0 sec
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
Filesystem                  1K-blocks  Used Available Use% Mounted on
192.168.202.102@tcp:/lustre    382976  3072    377856   1% /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 50g (383s)

== conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:34:19 (1781289259)
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.102@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.102@tcp osc.active=0 autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.102@tcp
  lustre:osc.active=0
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
Setting lustre-OST0000.osc.active from 0 to 1
Waiting 90s for '1'
Updated after 6s: want '1' got '1'
create a file after OST1 is activated
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 1.14784 s, 9.1 MB/s
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 50h (167s)

== conf-sanity test 50i: activate deactivated MDT ======== 14:37:06 (1781289426)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1
              (MDT )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity

   Permanent disk data:
Target:     lustre-MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x41
              (MDT update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdc.active=0
Writing lustre-mdt2/mdt2 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:mdc.active=0
  lustre:version=1
  lustre:flags=65
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0001
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
No device found for name MGS: Invalid argument
This command must be run on the MGS.
error: conf_param: No such device
Setting lustre-MDT0001.mdc.active from 0 to 1
Waiting 90s for '1'
Updated after 6s: want '1' got '1'
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40
oleg202-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40
oleg202-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec
total: 1 open/close in 0.16 seconds: 6.40 ops/second
Setting lustre-MDT0001.mdc.active from 1 to 0
Waiting 90s for '0'
Updated after 9s: want '0' got '0'
check osp.lustre-MDT0001-osp-MDT0000.active
target updated after 0 sec (got 0)
lfs mkdir: dirstripe error on '/mnt/lustre/d50i.conf-sanity/2': No such device
lfs setdirstripe: cannot create dir '/mnt/lustre/d50i.conf-sanity/2': No such device
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop ost1 service on oleg202-server
PASS 50i (203s)

== conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:40:29 (1781289629)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
fail_loc=0x142
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 51 (511s)

== conf-sanity test 52: check recovering objects from lost+found ========================================================== 14:49:00 (1781290140)

 SKIP: conf-sanity test_52 ldiskfs only test
SKIP 52 (5s)

== conf-sanity test 53a: check OSS thread count params === 14:49:05 (1781290145)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
checking (($tstarted && $tmin && $tmax)) (((3 && 3 && 116)))...
checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((3 >= 3 && 3 <= 116 )))...
ost.OSS.ost.threads_min=4
ost.OSS.ost.threads_max=115
checking (($tmin2 == ($tmin + $nthrs) &&
			    $tmax2 == ($tmax - $nthrs))) (((4 == (3 + 1) &&
			    115 == (116 - 1))))...
ost.OSS.ost.threads_max=4
checking (($tmin2 == $tmin && $tmax2 == $tmin)) (((4 == 4 && 4 == 4)))...
oleg202-server: error: set_param: setting /sys/fs/lustre/ost/OSS/ost/threads_max=3: Numerical result out of range
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 34
checking (($tmin2 <= $tmax2)) (((4 <= 4)))...
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
setmodopts: MODOPTS_OST= oss_num_threads=16
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ost/ost options: 'oss_num_threads=16'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg202-server: ost/ost options: 'oss_num_threads=16'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((16 >= 16 && 16 <= 16 )))...
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 53a (443s)

== conf-sanity test 53b: check MDS thread count params === 14:56:28 (1781290588)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
checking (($tstarted && $tmin && $tmax)) (((4 && 3 && 116)))...
checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((4 >= 3 && 4 <= 116 )))...
mds.MDS.mdt..threads_min=4
mds.MDS.mdt..threads_max=115
checking (($tmin2 == ($tmin + $nthrs) &&
			    $tmax2 == ($tmax - $nthrs))) (((4 == (3 + 1) &&
			    115 == (116 - 1))))...
mds.MDS.mdt..threads_max=4
checking (($tmin2 == $tmin && $tmax2 == $tmin)) (((4 == 4 && 4 == 4)))...
oleg202-server: error: set_param: setting /sys/fs/lustre/mds/MDS/mdt//threads_max=3: Numerical result out of range
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 34
checking (($tmin2 <= $tmax2)) (((4 <= 4)))...
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
setmodopts: MODOPTS_MDT= mds_num_threads=16
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mds_num_threads=16'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg202-server: mdt/mdt options: 'mds_num_threads=16'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
checking (($tstarted >= $tmin && $tstarted <= $tmax )) (((16 >= 16 && 16 <= 16 )))...
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 53b (449s)

== conf-sanity test 54a: test llverdev and partial verify of device ========================================================== 15:03:57 (1781291037)

 SKIP: conf-sanity test_54a ldiskfs only test
SKIP 54a (5s)

== conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:04:02 (1781291042)

 SKIP: conf-sanity test_54b ldiskfs only test
SKIP 54b (3s)

 SKIP: conf-sanity test_55 skipping excluded test 55

== conf-sanity test 56a: check big OST indexes and out-of-index-order start ========================================================== 15:04:07 (1781291047)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2

   Permanent disk data:
Target:     lustre:OST2710
Index:      10000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=10000
  lustre:fsname=lustre
  lustre:svname=lustre:OST2710

   Permanent disk data:
Target:     lustre:OST03e8
Index:      1000
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost2 /dev/vdf
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost2/ost2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost2/ost2 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=1000
  lustre:fsname=lustre
  lustre:svname=lustre:OST03e8
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST2710
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST03e8
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
ok
OBDS:
1000: lustre-OST03e8_UUID ACTIVE
10000: lustre-OST2710_UUID ACTIVE
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0179024 s, 229 kB/s
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:)
Stopping client oleg202-client.virtnet /mnt/lustre opts:
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 56a (472s)

== conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:11:59 (1781291519)

 SKIP: conf-sanity test_56b needs >= 3 MDTs
SKIP 56b (4s)

== conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:12:03 (1781291523)
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.102@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.102@tcp failover.node=192.168.202.102@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.102@tcp
  lustre:failover.node=192.168.202.102@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot assign requested address
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 99
Start of lustre-ost1/ost1 on ost1 failed 99
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 57a (323s)

== conf-sanity test 57b: initial registration from servicenode should not fail ========================================================== 15:17:26 (1781291846)
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.102@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x1042
              (OST update no_primnode )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.102@tcp failover.node=192.168.202.102@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.102@tcp
  lustre:failover.node=192.168.202.102@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=4162
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 57b (368s)

== conf-sanity test 58: missing llog files must not prevent MDT from mounting ========================================================== 15:23:34 (1781292214)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
total: 100 open/close in 1.91 seconds: 52.29 ops/second
 - unlinked 0 (time 1781292316 ; total 0 ; last 0)
total: 100 unlinks in 1 seconds: 100.000000 unlinks/second
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
fail_loc=0
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 58 (236s)

== conf-sanity test 59: writeconf mount option =========== 15:27:30 (1781292450)
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
original ost count: 1 (expect > 0)
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
after mdt writeconf count: 0 (expect 0)
OST start without writeconf should fail:
oleg202-server: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: No such file or directory
oleg202-server: Is the MGS specification correct?
oleg202-server: Is the filesystem name correct?
oleg202-server: If upgrading, is the copied client log valid? (see upgrade docs)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
OST start with writeconf should succeed:
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
after ost writeconf count: 1 (expect 1)
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
after ost2 writeconf count: 2 (expect 2)
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 59 (461s)

== conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 15:35:11 (1781292911)

 SKIP: conf-sanity test_60 ldiskfs only test
SKIP 60 (3s)

== conf-sanity test 61: large xattr ====================== 15:35:14 (1781292914)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity
shrink value of trusted.big on /mnt/lustre/f61.conf-sanity
grow value of trusted.big on /mnt/lustre/f61.conf-sanity
check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS
Failing mds1 on oleg202-server
Stopping /mnt/lustre-mds1 (opts:) on oleg202-server
reboot facets: mds1
Failover mds1 to oleg202-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 61 (180s)

== conf-sanity test 62: start with disabled journal ====== 15:38:15 (1781293095)

 SKIP: conf-sanity test_62 ldiskfs only test
SKIP 62 (5s)

 SKIP: conf-sanity test_63 skipping excluded test 63

== conf-sanity test 64: check lfs df --lazy ============== 15:38:21 (1781293101)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start ost2 service on oleg202-server
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
stop ost2 service on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/utils/lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID       383616        3328      378240   1% /mnt/lustre[MDT:0]
lustre-MDT0001_UUID       383616        3072      378496   1% /mnt/lustre[MDT:1]
lustre-OST0000_UUID       382976        3072      377856   1% /mnt/lustre[OST:0]

filesystem_summary:       382976        3072      377856   1% /mnt/lustre

umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:-f)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
oleg202-server: 
oleg202-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg202-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 19
checking for existing Lustre data: not found
tunefs failed, reformatting instead
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 64 (442s)

== conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 15:45:43 (1781293543)

 SKIP: conf-sanity test_65 ldiskfs only test
SKIP 65 (3s)

== conf-sanity test 66: replace nids ===================== 15:45:46 (1781293546)
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
Setting lustre-OST0000.osc.active from 1 to 0
Waiting 90s for '0'
Updated after 8s: want '0' got '0'
replace_nids should fail if MDS, OSTs and clients are UP
oleg202-server: error: replace_nids: Operation now in progress
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 115
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
replace_nids should fail if MDS and OSTs are UP
oleg202-server: error: replace_nids: Operation now in progress
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 115
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
replace_nids should fail if MDS is UP
oleg202-server: error: replace_nids: Operation now in progress
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 115
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
start mds service on oleg202-server
Starting mds1: -o localrecov -o nosvc  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
Start lustre-mdt1/mdt1 without service
Started lustre-MDT0000
command should accept two parameters
oleg202-server: replace primary NIDs for a device
oleg202-server: usage: replace_nids <device> <nid1>[,nid2,nid3:nid4,nid5:nid6]
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 4
correct device name should be passed
oleg202-server: error: replace_nids: No such device or address
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 6
wrong nids list should not destroy the system
oleg202-server: replace primary NIDs for a device
oleg202-server: usage: replace_nids <device> <nid1>[,nid2,nid3:nid4,nid5:nid6]
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 4
oleg202-server: replace primary NIDs for a device
oleg202-server: usage: replace_nids <device> <nid1>[,nid2,nid3:nid4,nid5:nid6]
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 4
replace OST nid
command should accept two parameters
oleg202-server: replace primary NIDs for a device
oleg202-server: usage: replace_nids <device> <nid1>[,nid2,nid3:nid4,nid5:nid6]
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 4
wrong nids list should not destroy the system
oleg202-server: replace primary NIDs for a device
oleg202-server: usage: replace_nids <device> <nid1>[,nid2,nid3:nid4,nid5:nid6]
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 4
set NIDs with failover
replace MDS nid
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Setting lustre-OST0000.osc.active from 0 to 1
Waiting 90s for '1'
Waiting 80s for '1'
Updated after 25s: want '1' got '1'
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
setup single mount lustre success
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 66 (518s)

 SKIP: conf-sanity test_67 skipping excluded test 67

== conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:54:27 (1781294067)
umount lustre on /mnt/lustre.....
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
seq.ctl-lustre-MDT0000.fldb=[0x2c0000400-12884902912):0:mdt
seq.srv-lustre-MDT0000.space=clear
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
seq.srv-lustre-MDT0000.space=[0x300000400 - 0x340000400]:0:mdt
[0x200000bd1:0x1:0x0]
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 68 (130s)

 SKIP: conf-sanity test_69 skipping SLOW test 69

== conf-sanity test 70a: start MDT0, then OST, then MDT1 ========================================================== 15:56:41 (1781294201)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 70a (149s)

== conf-sanity test 70b: start OST, MDT1, MDT0 =========== 15:59:10 (1781294350)
start ost1 service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 70b (256s)

== conf-sanity test 70c: stop MDT0, mkdir fail, create remote dir fail ========================================================== 16:03:26 (1781294606)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
deactivate lustre-MDT0000-mdc-ffff8c79509b7800
mkdir: cannot create directory '/mnt/lustre/d70c.conf-sanity': Cannot send after transport endpoint shutdown
lfs mkdir: cannot resolve path '/mnt/lustre/d70c.conf-sanity/remote_dir': Cannot send after transport endpoint shutdown (108)
lfs mkdir: '/mnt/lustre/d70c.conf-sanity/remote_dir' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108)
lfs setdirstripe: cannot create dir '/mnt/lustre/d70c.conf-sanity/remote_dir': Cannot send after transport endpoint shutdown
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
lsof: status error on /mnt/lustre: Cannot send after transport endpoint shutdown
lsof 4.93.2
 latest revision: https://github.com/lsof-org/lsof
 latest FAQ: https://github.com/lsof-org/lsof/blob/master/00FAQ
 latest (non-formatted) man page: https://github.com/lsof-org/lsof/blob/master/Lsof.8
 usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-E] [+|-e s] [+|-f[gG]]
 [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
 [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Use the ``-h'' option to get more help information.
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
PASS 70c (141s)

== conf-sanity test 70d: stop MDT1, mkdir succeed, create remote dir fail ========================================================== 16:05:47 (1781294747)
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
deactivate lustre-MDT0001-mdc-ffff8c794a45c800
lfs mkdir: dirstripe error on '/mnt/lustre/d70d.conf-sanity/remote_dir': No such device
lfs setdirstripe: cannot create dir '/mnt/lustre/d70d.conf-sanity/remote_dir': No such device
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 70d (156s)

== conf-sanity test 70e: Sync-on-Cancel will be enabled by default on DNE ========================================================== 16:08:23 (1781294903)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:MDT0000
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid

   Permanent disk data:
Target:     lustre:MDT0001
Index:      1
Lustre FS:  lustre
Mount type: zfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt2 /dev/vdd
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt2/mdt2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt2/mdt2 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=97
  lustre:index=1
  lustre:fsname=lustre
  lustre:svname=lustre:MDT0001
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-server: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40
oleg202-server: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40
oleg202-server: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
umount lustre on /mnt/lustre.....
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
LNET ready to unload
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
unloading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy
oleg202-server: LNET ready to unload
oleg202-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2
PASS 70e (318s)

== conf-sanity test 71a: start MDT0 OST0, MDT1, OST1 ===== 16:13:41 (1781295221)

 SKIP: conf-sanity test_71a needs separate MGS/MDT
SKIP 71a (5s)

== conf-sanity test 71b: start MDT1, OST0, MDT0, OST1 ==== 16:13:47 (1781295227)

 SKIP: conf-sanity test_71b needs separate MGS/MDT
SKIP 71b (5s)

== conf-sanity test 71c: start OST0, OST1, MDT1, MDT0 ==== 16:13:51 (1781295231)

 SKIP: conf-sanity test_71c needs separate MGS/MDT
SKIP 71c (3s)

== conf-sanity test 71d: start OST0, MDT1, MDT0, OST1 ==== 16:13:54 (1781295234)

 SKIP: conf-sanity test_71d needs separate MGS/MDT
SKIP 71d (4s)

== conf-sanity test 71e: start OST0, MDT1, OST1, MDT0 ==== 16:13:58 (1781295238)

 SKIP: conf-sanity test_71e needs separate MGS/MDT
SKIP 71e (4s)

== conf-sanity test 72: test fast symlink with extents flag enabled ========================================================== 16:14:02 (1781295242)

 SKIP: conf-sanity test_72 ldiskfs only test
SKIP 72 (5s)

== conf-sanity test 73: failnode to update from mountdata properly ========================================================== 16:14:07 (1781295247)
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x2
              (OST )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp autodegrade=on sys.timeout=20 failover.node=1.2.3.4@tcp autodegrade=on
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:failover.node=1.2.3.4@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start mds service on oleg202-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre
       failover_nids: [ 192.168.202.102@tcp, 1.2.3.4@tcp ]
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 73 (150s)

== conf-sanity test 75: The order of --index should be irrelevant ========================================================== 16:16:37 (1781295397)

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000

   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x45
              (MDT MGS update )
Persistent mount opts: 
Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1 /dev/vdc
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1/mdt1
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1/mdt1 properties
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=69
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-MDT0000

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.102@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.102@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg202-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
start mds service on oleg202-server
Starting mds2: -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-mdt2/mdt2
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg202-server: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40
oleg202-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
PASS 75 (243s)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format mds2: lustre-mdt2/mdt2
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
== conf-sanity test complete, duration 14956 sec ========= 16:22:09 (1781295729)