-----============= acceptance-small: conf-sanity ============----- Tue Sep  8 07:19:49 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg412-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex
- see CLIENT_VERSION >= v2_15_63-53-g1a6ef725c2 (34683240 >= 34553653) for LU-16938, go 81
- see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683240 >= 34554143) for LU-18348, go 32b 32c
- see CLIENT_VERSION >= v2_16_55-19-gf69f27cc35 (34683240 >= 34617107) for LU-17920, go 123ae 123ag
excepting tests: 55 63 67 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a
skipping tests SLOW=no: 45 69 106 111 114
Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg412-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg412-server
oleg412-server: oleg412-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg412-server: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg412-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server

== conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:23:10 (1788866590)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1

   Permanent disk data:
Device:     lustre-mdt1_2/mdt1_2
Target:     test-33a:MDT0000
Index:      0
Lustre FS:  test-33a
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.204.112@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=test-33a
  lustre:svname=test-33a:MDT0000

   Permanent disk data:
Device:     lustre-ost1_2/ost1_2
Target:     test-33a:OST1fff
Index:      8191
Lustre FS:  test-33a
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp:192.168.204.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=192.168.204.112@tcp:192.168.204.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=8191
  lustre:fsname=test-33a
  lustre:svname=test-33a:OST1fff
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started test-33a-MDT0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1875: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-test-33a-OST1fff-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started test-33a-OST1fff
ok.
/mnt/lustre2/hosts
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 8191
	obdidx		 objid		 objid		 group
	  8191	             2	          0x2	   0x240000400

config_log: test-33a-MDT0000
cancel catalog test-33a-MDT0000 log_idx 20: done
cancel catalog test-33a-MDT0000 log_idx 19: done
cancel catalog test-33a-MDT0000 log_idx 18: done
del_ost: cancelled 3 catalog entries
config_log: test-33a-client
cancel catalog test-33a-client log_idx 25: done
cancel catalog test-33a-client log_idx 24: done
cancel catalog test-33a-client log_idx 23: done
del_ost: cancelled 3 catalog entries
ok.
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg412-server
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg412-server
stop ost1 service on oleg412-server
stop mds service on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
umount /mnt/lustre2 ...
umount: /mnt/lustre2: not mounted.
stopping fs2mds ...
stopping fs2ost ...
PASS 33a (133s)

== conf-sanity test 33b: Drop cancel during umount ======= 07:25:23 (1788866723)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150799 s, 7.0 MB/s
fail_loc=0x80000304
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
umount lustre on /mnt/lustre.....
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
PASS 33b (122s)

== conf-sanity test 33c: Mount ost with a large index number ========================================================== 07:27:25 (1788866845)
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt

   Permanent disk data:
Device:     lustre-mdt1_2/mdt1_2
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-mdt1_2/mdt1_2
  xattr=sa
  dnodesize=auto
  compression=off
Writing lustre-mdt1_2/mdt1_2 properties
  lustre:mgsnode=192.168.204.112@tcp
  lustre:sys.timeout=20
  lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity
  lustre:version=1
  lustre:flags=101
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:MDT0000

   Permanent disk data:
Device:     lustre-ost1_2/ost1_2
Target:     lustre:OST07c6
Index:      1990
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.112@tcp:192.168.204.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1_2/ost1_2
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1_2/ost1_2 properties
  lustre:mgsnode=192.168.204.112@tcp:192.168.204.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=1990
  lustre:fsname=lustre
  lustre:svname=lustre:OST07c6
Start fs2mds: mount -t lustre -o localrecov  lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1_2/mdt1_2
Started lustre-MDT0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1875: OST_INDICES: bad array subscript
Start fs2ost: mount -t lustre -o localrecov  lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost
seq.cli-lustre-OST07c6-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST07c6
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
Creating new pool qpool1
oleg412-server: Pool lustre.qpool1 created
Waiting 90s for ''
Adding targets to pool
oleg412-server: OST lustre-OST07c6_UUID added to pool lustre.qpool1
Waiting 90s for 'lustre-OST07c6_UUID '
Updated after 7s: want 'lustre-OST07c6_UUID ' got 'lustre-OST07c6_UUID '
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
20+0 records in
19+0 records out
19922944 bytes (20 MB, 19 MiB) copied, 2.04163 s, 9.8 MB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
1+0 records in
0+0 records out
0 bytes copied, 0.135223 s, 0.0 kB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
20+0 records in
19+0 records out
19922944 bytes (20 MB, 19 MiB) copied, 3.14305 s, 6.3 MB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
1+0 records in
0+0 records out
0 bytes copied, 0.115562 s, 0.0 kB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
20+0 records in
19+0 records out
19922944 bytes (20 MB, 19 MiB) copied, 2.42394 s, 8.2 MB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
1+0 records in
0+0 records out
0 bytes copied, 0.353515 s, 0.0 kB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
20+0 records in
19+0 records out
19922944 bytes (20 MB, 19 MiB) copied, 3.43893 s, 5.8 MB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
1+0 records in
0+0 records out
0 bytes copied, 0.178714 s, 0.0 kB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
20+0 records in
19+0 records out
19922944 bytes (20 MB, 19 MiB) copied, 3.20307 s, 6.2 MB/s
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [of=/mnt/lustre/d33c.conf-sanity/f1] [bs=1M] [count=30] [oflag=direct]
dd: error writing '/mnt/lustre/d33c.conf-sanity/f1': Disk quota exceeded
1+0 records in
0+0 records out
0 bytes copied, 0.227328 s, 0.0 kB/s
Destroy the created pools: qpool1
lustre.qpool1
oleg412-server: OST lustre-OST07c6_UUID removed from pool lustre.qpool1
oleg412-server: Pool lustre.qpool1 destroyed
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-fs2ost (opts:-f) on oleg412-server
Stopping /mnt/lustre-fs2mds (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
PASS 33c (216s)

== conf-sanity test 33d: Don't panic when enable project quota ========================================================== 07:31:01 (1788867061)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
lfs setquota: warning: inode hardlimit '1024' smaller than minimum qunit size
See 'lfs help setquota' or Lustre manual for details
osd-zfs.lustre-MDT0000.quota_slave.enabled=p
osd-zfs.lustre-OST0000.quota_slave.enabled=p
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
PASS 33d (109s)

== conf-sanity test 34a: umount with opened file should be fail ========================================================== 07:32:50 (1788867170)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
multiop /mnt/lustre/file vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.25125
manual umount lustre on /mnt/lustre....
umount: /mnt/lustre: target is busy.
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
PASS 34a (113s)

== conf-sanity test 34b: force umount with failed mds should be normal ========================================================== 07:34:43 (1788867283)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
manual umount lustre on /mnt/lustre....
umount lustre on /mnt/lustre.....
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
PASS 34b (142s)

== conf-sanity test 34c: force umount with failed ost should be normal ========================================================== 07:37:05 (1788867425)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
manual umount lustre on /mnt/lustre....
umount lustre on /mnt/lustre.....
stop ost1 service on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
PASS 34c (113s)

== conf-sanity test 35a: Reconnect to the last active server first ========================================================== 07:38:58 (1788867538)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2026-09-08 7h40m20s
Stopping the MDT: lustre-MDT0000
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
Restarting the MDT: lustre-MDT0000
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
Wait for df (34939) ... 
done
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout
Debug log: 79 lines, 79 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
oleg412-server: 
oleg412-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg412-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 19
oleg412-server: 
oleg412-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg412-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 19
oleg412-server: 
oleg412-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg412-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 19
tunefs.lustre failed, reformatting instead
Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
oleg412-server: oleg412-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg412-server: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg412-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
PASS 35a (298s)

== conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 07:43:56 (1788867836)
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
debug=ha
conf-sanity.sh test_35b 2026-09-08 7h44m35s
Set up a fake failnode for the MDS
at_max=0
at_max=0
Injecting EBUSY on MDS
fail_loc=0x80000136
mdc.lustre-MDT0000-mdc-ffff8f2b0645e000.stats=clear
Creating a test file and stat it
  File: /mnt/lustre/d35b.conf-sanity/f35b.conf-sanity
  Size: 0         	Blocks: 1          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115238826934274  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-09-08 07:45:27.000000000 -0400
Modify: 2026-09-08 07:45:27.000000000 -0400
Change: 2026-09-08 07:45:27.000000000 -0400
 Birth: 2026-09-08 07:45:27.000000000 -0400
Stop injecting EBUSY on MDS
fail_loc=0
done
at_max=600
at_max=600
Debug log: 44 lines, 44 kept, 0 dropped, 0 bad.
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
oleg412-server: 
oleg412-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre
oleg412-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 19
oleg412-server: 
oleg412-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre
oleg412-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 19
oleg412-server: 
oleg412-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre
oleg412-server: tunefs.lustre: exiting with 19 (No such device)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 19
tunefs.lustre failed, reformatting instead
Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
oleg412-server: oleg412-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg412-server: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg412-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
PASS 35b (270s)

== conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 07:48:27 (1788868107)

 SKIP: conf-sanity test_36 remote OST
SKIP 36 (3s)

== conf-sanity test 37: verify set tunables works for symlink device ========================================================== 07:48:30 (1788868110)

 SKIP: conf-sanity test_37 ldiskfs only test
SKIP 37 (3s)

== conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:48:33 (1788868113)
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
copying 10 files to /mnt/lustre/d38.conf-sanity
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
delete lov_objid file on MDS
000000                   43
000008
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
000000                    2
000008
oleg412-server: 8+0 records in
oleg412-server: 8+0 records out
oleg412-server: 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000944364 s, 4.3 MB/s
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
000000                   34
000008
files compared the same
umount lustre on /mnt/lustre.....
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
PASS 38 (197s)

== conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ========================================================== 07:51:51 (1788868311)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug +malloc all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug +malloc all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
PASS 39 (104s)

== conf-sanity test 40: race during service thread startup ========================================================== 07:53:35 (1788868415)
start ost1 service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
fail_loc=0x80000706
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
umount lustre on /mnt/lustre.....
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2
PASS 40 (139s)

== conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================================== 07:55:54 (1788868554)
start mds service on oleg412-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov -o nosvc -n  lustre-mdt1/mdt1 /mnt/lustre-mds1
nomtab: 1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-OST0000
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov -o nomgs,force  lustre-mdt1/mdt1 /mnt/lustre-mds1
force: 1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Started lustre-MDT0000
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
blah blah
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
PASS 41a (107s)

== conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:57:41 (1788868661)
Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:)
Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f)
oleg412-server: oleg412-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov -o nosvc -n  lustre-mdt1/mdt1 /mnt/lustre-mds1
nomtab: 1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
Start lustre-mdt1/mdt1 without service
Commit the device label on lustre-mdt1/mdt1