-----============= acceptance-small: sanity-scrub ============----- Wed May 8 18:02:07 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: Force libcfs to create 2 CPU partitions client=34553605 MDS=34553605 OSS=34553605 Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:) Stopping client oleg342-client.virtnet /mnt/lustre opts: Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg342-server unloading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local modules unloaded. === sanity-scrub: start setup 18:02:26 (1715205746) === Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 oleg342-server: oleg342-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: Force libcfs to create 2 CPU partitions oleg342-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg342-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Checking servers environments Checking clients oleg342-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre Starting client oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre Started clients oleg342-client.virtnet: 192.168.203.142@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b432a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b432a800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 4s: want 'procname_uid' got 'procname_uid' disable quota as required === sanity-scrub: finish setup 18:03:12 (1715205792) === debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 18:03:13 (1715205793) preparing... Wed May 8 18:03:14 EDT 2024 creating 0 files on mds1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899459 s, 117 MB/s prepared Wed May 8 18:03:15 EDT 2024. stop mds1 starting MDTs without disabling OI scrub oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre PASS 0 (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 18:03:26 (1715205806) preparing... Wed May 8 18:03:26 EDT 2024 creating 0 files on mds1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114686 s, 91.4 MB/s prepared Wed May 8 18:03:27 EDT 2024. stop mds1 start mds1 without disabling OI scrub: -o localrecov -o user_xattr oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre fail_loc=0x193 fail_loc=0 192.168.203.142@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop mds1 start mds1 with disabling OI scrub: -o localrecov -o user_xattr,noscrub oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 PASS 1a (15s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 18:03:42 (1715205822) Checking servers environments Checking clients oleg342-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg342-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg342-server: mount.lustre: according to /etc/mtab lustre-ost2/ost2 is already mounted on /mnt/lustre-ost2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 17 seq.cli-lustre-OST0001-super.width=65536 Start of lustre-ost2/ost2 on ost2 failed 17 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre Starting client oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre Started clients oleg342-client.virtnet: 192.168.203.142@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b4302000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b4302000.idle_timeout=debug disable quota as required preparing... Wed May 8 18:03:57 EDT 2024 creating 0 files on mds1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00755474 s, 139 MB/s prepared Wed May 8 18:03:58 EDT 2024. fail_loc=0x198 fail_loc=0 stop mds1 start MDTs without disabling OI scrub oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre PASS 1b (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 18:04:09 (1715205849) SKIP: sanity-scrub test_1c ldiskfs special test SKIP 1c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 18:04:11 (1715205851) SKIP: sanity-scrub test_2 ldiskfs special test SKIP 2 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 18:04:14 (1715205854) preparing... Wed May 8 18:04:14 EDT 2024 creating 0 files on mds1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109486 s, 95.8 MB/s prepared Wed May 8 18:04:15 EDT 2024. fail_loc=0x193 fail_loc=0 stop mds1 starting MDTs with OI scrub disabled oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre PASS 4a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 18:04:33 (1715205873) SKIP: sanity-scrub test_4b ldiskfs special test SKIP 4b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 18:04:35 (1715205875) SKIP: sanity-scrub test_4c ldiskfs special test SKIP 4c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 18:04:38 (1715205878) SKIP: sanity-scrub test_4d ldiskfs only test SKIP 4d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 4e: FID reuse can be fixed ========== 18:04:40 (1715205880) SKIP: sanity-scrub test_4e ldiskfs only test SKIP 4e (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-scrub test 5: OI scrub state machine =========== 18:04:42 (1715205882)