-----============= acceptance-small: conf-sanity ============----- Thu Aug 22 01:28:20 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server == conf-sanity test 45: long unlink handling in ptlrpcd == 01:29:48 (1724304588) start mds service on oleg408-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg408-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 45 (152s) == conf-sanity test 69: replace an OST with the same index ========================================================== 01:32:21 (1724304741) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 5793 (time 1724304779.35 total 10.00 last 579.29) - open/close 9837 (time 1724304789.35 total 20.00 last 404.30) total: 10000 open/close in 20.48 seconds: 488.34 ops/second - unlinked 0 (time 1724304790 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 6071 (time 1724304814.76 total 10.00 last 607.04) total: 10000 open/close in 16.24 seconds: 615.95 ops/second - unlinked 0 (time 1724304821 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 6204 (time 1724304842.38 total 10.00 last 620.39) total: 10000 open/close in 16.03 seconds: 623.93 ops/second - unlinked 0 (time 1724304849 ; total 0 ; last 0) total: 10000 unlinks in 8 seconds: 1250.000000 unlinks/second - open/close 6350 (time 1724304868.55 total 10.00 last 634.98) - open/close 8843 (time 1724304878.56 total 20.01 last 249.16) total: 10000 open/close in 26.57 seconds: 376.35 ops/second - unlinked 0 (time 1724304886 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4535 (time 1724304911.25 total 10.00 last 453.47) - open/close 7001 (time 1724304921.25 total 20.00 last 246.52) total: 10000 open/close in 29.20 seconds: 342.49 ops/second - unlinked 0 (time 1724304931 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4784 (time 1724304954.84 total 10.00 last 478.37) - open/close 9072 (time 1724304964.84 total 20.00 last 428.80) total: 10000 open/close in 21.62 seconds: 462.46 ops/second - unlinked 0 (time 1724304967 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 5952 (time 1724304988.81 total 10.00 last 595.18) - open/close 9652 (time 1724304998.81 total 20.00 last 369.84) total: 10000 open/close in 21.41 seconds: 467.18 ops/second - unlinked 0 (time 1724305001 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3377 (time 1724305026.36 total 10.00 last 337.64) - open/close 6395 (time 1724305036.36 total 20.00 last 301.76) - open/close 8514 (time 1724305046.36 total 30.00 last 211.87) - open/close 9752 (time 1724305056.37 total 40.02 last 123.63) total: 10000 open/close in 42.09 seconds: 237.58 ops/second - unlinked 0 (time 1724305060 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1846 (time 1724305097.86 total 10.01 last 184.46) - open/close 3698 (time 1724305107.87 total 20.01 last 185.08) - open/close 5831 (time 1724305117.87 total 30.01 last 213.28) - open/close 7970 (time 1724305127.87 total 40.02 last 213.82) - open/close 9796 (time 1724305137.87 total 50.02 last 182.56) total: 10000 open/close in 50.96 seconds: 196.24 ops/second - unlinked 0 (time 1724305140 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1344 (time 1724305183.02 total 10.00 last 134.39) - open/close 2841 (time 1724305193.02 total 20.01 last 149.63) - open/close 4240 (time 1724305203.03 total 30.01 last 139.88) - open/close 5794 (time 1724305213.03 total 40.01 last 155.34) - open/close 7357 (time 1724305223.03 total 50.01 last 156.28) - open/close 8979 (time 1724305233.03 total 60.01 last 162.19) total: 10000 open/close in 66.22 seconds: 151.01 ops/second - unlinked 0 (time 1724305240 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg408-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre On OST0, 17564 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 69 (580s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 01:42:01 (1724305321) oleg408-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity oleg408-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity umount lustre on /mnt/lustre..... stop ost1 service on oleg408-server stop mds service on oleg408-server stop mds service on oleg408-server LNET unconfigure error 22: Invalid argument unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: LNET unconfigure error 22: Invalid argument modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O large_dir -i 1048576 -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=192.168.204.108@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=131072 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre Starting client oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre Started clients oleg408-client.virtnet: 192.168.204.108@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) mount lustre on /mnt/lustre2..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 Starting client oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 Started clients oleg408-client.virtnet: 192.168.204.108@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2280828 1696 2159132 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1388 126828 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1388 126828 2% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 2280 272 2008 12% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 302 49698 1% /mnt/lustre[OST:0] filesystem_summary: 2280 272 2008 12% /mnt/lustre creating 60000 hardlinks to oleg408-client.virtnet-0 creating 60000 hardlinks to oleg408-client.virtnet-1 waiting for PIDs 26847 26861 to complete - link 1595 (time 1724305366.79 total 10.00 last 159.48) - link 2185 (time 1724305376.81 total 20.03 last 58.86) - link 2678 (time 1724305386.83 total 30.04 last 49.21) - link 3183 (time 1724305396.84 total 40.06 last 50.43) - link 3830 (time 1724305406.85 total 50.06 last 64.66) - link 4458 (time 1724305416.86 total 60.07 last 62.77) - link 5349 (time 1724305426.86 total 70.07 last 89.07) - link 6051 (time 1724305436.87 total 80.08 last 70.13) - link 6675 (time 1724305446.88 total 90.09 last 62.33) - link 7403 (time 1724305456.89 total 100.10 last 72.75) - link 8084 (time 1724305466.90 total 110.11 last 68.04) - link 9029 (time 1724305476.90 total 120.11 last 94.48) - link 9819 (time 1724305486.90 total 130.11 last 78.99) - link 10000 (time 1724305489.38 total 132.59 last 72.98) - link 10651 (time 1724305499.39 total 142.60 last 65.04) - link 11598 (time 1724305509.40 total 152.61 last 94.61) - link 12502 (time 1724305519.41 total 162.62 last 90.31) - link 13248 (time 1724305529.42 total 172.63 last 74.55) - link 14058 (time 1724305539.43 total 182.64 last 80.90) - link 14847 (time 1724305549.46 total 192.68 last 78.62) - link 15551 (time 1724305559.49 total 202.70 last 70.24) - link 16246 (time 1724305569.50 total 212.71 last 69.43) - link 16962 (time 1724305579.51 total 222.72 last 71.50) - link 17649 (time 1724305589.53 total 232.74 last 68.56) - link 18381 (time 1724305599.55 total 242.76 last 73.06) - link 19290 (time 1724305609.55 total 252.77 last 90.89) - link 20000 (time 1724305616.74 total 259.95 last 98.81) - link 21018 (time 1724305626.74 total 269.95 last 101.77) - link 21824 (time 1724305636.75 total 279.96 last 80.52) - link 22606 (time 1724305646.75 total 289.96 last 78.19) - link 23510 (time 1724305656.76 total 299.97 last 90.30) - link 24531 (time 1724305666.77 total 309.99 last 101.97) - link 25301 (time 1724305676.80 total 320.01 last 76.83) - link 26139 (time 1724305686.80 total 330.01 last 83.76) - link 26974 (time 1724305696.81 total 340.02 last 83.43) - link 27758 (time 1724305706.81 total 350.02 last 78.39) - link 28442 (time 1724305716.81 total 360.02 last 68.40) - link 29065 (time 1724305726.83 total 370.04 last 62.21) - link 29721 (time 1724305736.83 total 380.05 last 65.54) - link 30000 (time 1724305741.17 total 384.38 last 64.38) - link 30606 (time 1724305751.18 total 394.39 last 60.52) - link 31240 (time 1724305761.21 total 404.42 last 63.22) - link 31836 (time 1724305771.22 total 414.43 last 59.55) - link 32436 (time 1724305781.22 total 424.43 last 59.99) - link 33041 (time 1724305791.24 total 434.45 last 60.40) - link 33622 (time 1724305801.25 total 444.46 last 58.04) - link 34311 (time 1724305811.25 total 454.46 last 68.87) - link 35014 (time 1724305821.26 total 464.47 last 70.24) - link 35863 (time 1724305831.28 total 474.49 last 84.75) - link 36628 (time 1724305841.28 total 484.49 last 76.50) - link 37390 (time 1724305851.28 total 494.49 last 76.18) - link 38374 (time 1724305861.29 total 504.50 last 98.33) - link 39153 (time 1724305871.30 total 514.52 last 77.79) - link 39886 (time 1724305881.32 total 524.53 last 73.21) - link 40515 (time 1724305891.33 total 534.54 last 62.84) - link 41189 (time 1724305901.33 total 544.54 last 67.40) - link 41939 (time 1724305911.33 total 554.54 last 74.99) - link 42605 (time 1724305921.33 total 564.54 last 66.60) - link 43305 (time 1724305931.34 total 574.55 last 69.93) - link 43952 (time 1724305941.34 total 584.55 last 64.69) - link 44699 (time 1724305951.34 total 594.55 last 74.70) - link 45389 (time 1724305961.35 total 604.56 last 68.96) - link 46019 (time 1724305971.36 total 614.57 last 62.91) - link 46735 (time 1724305981.37 total 624.58 last 71.56) - link 47392 (time 1724305991.37 total 634.59 last 65.66) - link 48021 (time 1724306001.38 total 644.60 last 62.83) - link 48691 (time 1724306011.39 total 654.60 last 66.95) - link 49438 (time 1724306021.40 total 664.61 last 74.65) - link 50000 (time 1724306028.41 total 671.63 last 80.09) - link 50722 (time 1724306038.42 total 681.63 last 72.19) - link 51436 (time 1724306048.44 total 691.65 last 71.22) - link 52007 (time 1724306058.45 total 701.66 last 57.06) - link 52676 (time 1724306068.45 total 711.67 last 66.87) - link 53326 (time 1724306078.45 total 721.67 last 65.00) - link 54193 (time 1724306088.46 total 731.67 last 86.63) - link 55083 (time 1724306098.47 total 741.69 last 88.89) - link 55919 (time 1724306108.49 total 751.70 last 83.47) - link 56650 (time 1724306118.49 total 761.71 last 73.07) - link 57342 (time 1724306128.50 total 771.71 last 69.17) - link 58010 (time 1724306138.50 total 781.71 last 66.79) - link 58629 (time 1724306148.50 total 791.72 last 61.88) - link 59201 (time 1724306158.51 total 801.73 last 57.14) - link 59899 (time 1724306168.53 total 811.74 last 69.71) total: 60000 link in 813.37 seconds: 73.77 ops/second - link 1587 (time 1724305366.94 total 10.00 last 158.63) - link 2168 (time 1724305376.95 total 20.02 last 58.02) - link 2655 (time 1724305386.97 total 30.04 last 48.58) - link 3162 (time 1724305396.98 total 40.05 last 50.68) - link 3801 (time 1724305406.98 total 50.05 last 63.88) - link 4441 (time 1724305416.99 total 60.06 last 63.94) - link 5322 (time 1724305427.00 total 70.07 last 88.05) - link 6022 (time 1724305437.01 total 80.08 last 69.92) - link 6660 (time 1724305447.01 total 90.08 last 63.76) - link 7382 (time 1724305457.02 total 100.09 last 72.16) - link 8058 (time 1724305467.03 total 110.10 last 67.53) - link 9006 (time 1724305477.03 total 120.10 last 94.79) - link 9795 (time 1724305487.03 total 130.10 last 78.90) - link 10000 (time 1724305489.93 total 133.00 last 70.66) - link 10665 (time 1724305499.93 total 143.00 last 66.50) - link 11618 (time 1724305509.93 total 153.00 last 95.29) - link 12508 (time 1724305519.93 total 163.00 last 88.99) - link 13256 (time 1724305529.94 total 173.01 last 74.75) - link 14061 (time 1724305539.95 total 183.02 last 80.41) - link 14868 (time 1724305549.97 total 193.03 last 80.58) - link 15562 (time 1724305559.98 total 203.05 last 69.33) - link 16269 (time 1724305569.98 total 213.05 last 70.66) - link 16973 (time 1724305579.99 total 223.06 last 70.33) - link 17664 (time 1724305590.01 total 233.08 last 68.99) - link 18395 (time 1724305600.01 total 243.08 last 73.09) - link 19309 (time 1724305610.02 total 253.08 last 91.35) - link 20000 (time 1724305617.03 total 260.10 last 98.46) - link 21014 (time 1724305627.04 total 270.11 last 101.34) - link 21819 (time 1724305637.06 total 280.13 last 80.30) - link 22615 (time 1724305647.06 total 290.13 last 79.60) - link 23526 (time 1724305657.07 total 300.14 last 91.02) - link 24542 (time 1724305667.08 total 310.15 last 101.54) - link 25314 (time 1724305677.09 total 320.16 last 77.11) - link 26142 (time 1724305687.10 total 330.16 last 82.75) - link 26986 (time 1724305697.10 total 340.17 last 84.35) - link 27755 (time 1724305707.11 total 350.18 last 76.81) - link 28440 (time 1724305717.12 total 360.19 last 68.45) - link 29056 (time 1724305727.13 total 370.20 last 61.53) - link 29711 (time 1724305737.14 total 380.21 last 65.45) - link 30000 (time 1724305741.65 total 384.72 last 64.08) - link 30613 (time 1724305751.66 total 394.73 last 61.23) - link 31246 (time 1724305761.66 total 404.73 last 63.28) - link 31849 (time 1724305771.68 total 414.74 last 60.23) - link 32447 (time 1724305781.69 total 424.75 last 59.74) - link 33035 (time 1724305791.69 total 434.76 last 58.77) - link 33613 (time 1724305801.69 total 444.76 last 57.80) - link 34307 (time 1724305811.72 total 454.78 last 69.24) - link 34992 (time 1724305821.72 total 464.79 last 68.49) - link 35833 (time 1724305831.72 total 474.79 last 84.04) - link 36605 (time 1724305841.73 total 484.80 last 77.15) - link 37391 (time 1724305851.74 total 494.81 last 78.55) - link 38346 (time 1724305861.74 total 504.81 last 95.48) - link 39135 (time 1724305871.75 total 514.82 last 78.82) - link 39852 (time 1724305881.75 total 524.82 last 71.70) - link 40000 (time 1724305884.22 total 527.29 last 59.89) - link 40656 (time 1724305894.23 total 537.29 last 65.57) - link 41338 (time 1724305904.25 total 547.31 last 68.06) - link 42071 (time 1724305914.26 total 557.33 last 73.22) - link 42733 (time 1724305924.27 total 567.34 last 66.13) - link 43416 (time 1724305934.27 total 577.34 last 68.28) - link 44067 (time 1724305944.28 total 587.34 last 65.07) - link 44844 (time 1724305954.28 total 597.35 last 77.66) - link 45481 (time 1724305964.30 total 607.36 last 63.60) - link 46158 (time 1724305974.31 total 617.38 last 67.60) - link 46847 (time 1724305984.32 total 627.38 last 68.87) - link 47503 (time 1724305994.33 total 637.39 last 65.53) - link 48139 (time 1724306004.34 total 647.41 last 63.53) - link 48813 (time 1724306014.35 total 657.41 last 67.35) - link 49593 (time 1724306024.35 total 667.42 last 77.99) - link 50000 (time 1724306029.45 total 672.52 last 79.80) - link 50721 (time 1724306039.46 total 682.52 last 72.04) - link 51421 (time 1724306049.46 total 692.53 last 69.97) - link 51995 (time 1724306059.46 total 702.53 last 57.39) - link 52683 (time 1724306069.46 total 712.53 last 68.79) - link 53330 (time 1724306079.48 total 722.54 last 64.62) - link 54214 (time 1724306089.48 total 732.55 last 88.35) - link 55083 (time 1724306099.49 total 742.56 last 86.81) - link 55908 (time 1724306109.50 total 752.57 last 82.41) - link 56634 (time 1724306119.51 total 762.58 last 72.55) - link 57331 (time 1724306129.53 total 772.60 last 69.56) - link 57988 (time 1724306139.54 total 782.61 last 65.63) - link 58606 (time 1724306149.56 total 792.63 last 61.68) - link 59200 (time 1724306159.57 total 802.64 last 59.33) - link 59888 (time 1724306169.58 total 812.65 last 68.75) total: 60000 link in 814.29 seconds: 73.68 ops/second estimate 34298s left after 120000 files / 815s umount lustre on /mnt/lustre2..... Stopping client oleg408-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. ETA 34298s after 120000 files / 815s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg408-server: e2fsck 1.46.2.wc5 (26-Mar-2022) oleg408-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 19) [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 180 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 181 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 182 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 183 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 184 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 185 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 190 badness 0 to 2 for 10084 [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] Pass 1: Memory used: 380k/908k (123k/258k), time: 0.03/ 0.02/ 0.00 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 71.11MB/s [Thread 0] Scanned group range [0, 19), inodes 281 Pass 2: Checking directory structure Pass 2: Memory used: 380k/0k (84k/297k), time: 0.50/ 0.31/ 0.08 Pass 2: I/O read: 49MB, write: 0MB, rate: 98.73MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 380k/0k (85k/296k), time: 0.55/ 0.34/ 0.08 Paoleg408-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51490816, 266) != expected (51499008, 266) oleg408-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51490816, 266) != expected (51499008, 266) oleg408-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51490816, 266) != expected (51499008, 266) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 4 ss 3: Memory used: 380k/0k (83k/298k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 380k/0k (69k/312k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 5586.59MB/s Pass 5: Checking group summary information Pass 5: Memory used: 380k/0k (67k/313k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 513.08MB/s Update quota info for quota type 0? no Update quota info for quota type 1? no Update quota info for quota type 2? no lustre-MDT0000: ********** WARNING: Filesystem still has errors ********** 276 inodes used (12.11%, out of 2280) 3 non-contiguous files (1.1%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 37909 blocks used (6.32%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 380k/0k (67k/314k), time: 0.55/ 0.34/ 0.08 I/O read: 49MB, write: 0MB, rate: 88.53MB/s pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (914s) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1755 sec ========== 01:57:36 (1724306256) === conf-sanity: start cleanup 01:57:37 (1724306257) === === conf-sanity: finish cleanup 01:57:37 (1724306257) ===