-----============= acceptance-small: conf-sanity ============----- Tue Oct 1 02:28:24 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg113-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg113-server oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg113-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg113-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server == conf-sanity test 45: long unlink handling in ptlrpcd == 02:30:18 (1727764218) start mds service on oleg113-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg113-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg113-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg113-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server unloading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 45 (182s) == conf-sanity test 69: replace an OST with the same index ========================================================== 02:33:20 (1727764400) start mds service on oleg113-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg113-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1542 (time 1727764467.11 total 10.01 last 154.12) - open/close 2799 (time 1727764477.12 total 20.01 last 125.58) - open/close 4218 (time 1727764487.12 total 30.02 last 141.87) - open/close 5570 (time 1727764497.13 total 40.02 last 135.12) - open/close 6846 (time 1727764507.14 total 50.03 last 127.52) - open/close 8347 (time 1727764517.14 total 60.03 last 150.03) - open/close 9752 (time 1727764527.15 total 70.04 last 140.42) total: 10000 open/close in 71.96 seconds: 138.96 ops/second - unlinked 0 (time 1727764530 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1395 (time 1727764582.49 total 10.00 last 139.47) - open/close 2722 (time 1727764592.50 total 20.01 last 132.61) - open/close 4095 (time 1727764602.50 total 30.01 last 137.27) - open/close 5379 (time 1727764612.50 total 40.02 last 128.33) - open/close 6618 (time 1727764622.50 total 50.02 last 123.88) - open/close 7921 (time 1727764632.51 total 60.02 last 130.23) - open/close 9251 (time 1727764642.51 total 70.03 last 132.96) total: 10000 open/close in 76.24 seconds: 131.17 ops/second - unlinked 0 (time 1727764650 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1398 (time 1727764704.71 total 10.00 last 139.77) - open/close 2556 (time 1727764714.72 total 20.01 last 115.74) - open/close 3820 (time 1727764724.72 total 30.01 last 126.37) - open/close 5160 (time 1727764734.72 total 40.01 last 133.93) - open/close 6598 (time 1727764744.73 total 50.02 last 143.79) - open/close 7818 (time 1727764754.74 total 60.03 last 121.87) - open/close 9116 (time 1727764764.74 total 70.03 last 129.71) total: 10000 open/close in 76.45 seconds: 130.81 ops/second - unlinked 0 (time 1727764772 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1280 (time 1727764827.18 total 10.01 last 127.92) - open/close 2626 (time 1727764837.18 total 20.01 last 134.56) - open/close 4073 (time 1727764847.19 total 30.01 last 144.62) - open/close 5336 (time 1727764857.19 total 40.02 last 126.27) - open/close 6727 (time 1727764867.19 total 50.02 last 139.08) - open/close 8146 (time 1727764877.19 total 60.02 last 141.88) - open/close 9353 (time 1727764887.20 total 70.03 last 120.56) total: 10000 open/close in 74.81 seconds: 133.67 ops/second - unlinked 0 (time 1727764894 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1425 (time 1727764948.16 total 10.01 last 142.43) - open/close 2709 (time 1727764958.16 total 20.01 last 128.35) - open/close 4057 (time 1727764968.17 total 30.02 last 134.71) - open/close 5541 (time 1727764978.17 total 40.02 last 148.36) - open/close 6854 (time 1727764988.17 total 50.02 last 131.26) - open/close 8249 (time 1727764998.18 total 60.02 last 139.47) - open/close 9766 (time 1727765008.18 total 70.02 last 151.68) total: 10000 open/close in 71.62 seconds: 139.62 ops/second - unlinked 0 (time 1727765011 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1363 (time 1727765066.38 total 10.01 last 136.23) - open/close 2806 (time 1727765076.38 total 20.01 last 144.25) - open/close 4120 (time 1727765086.38 total 30.01 last 131.38) - open/close 5453 (time 1727765096.38 total 40.01 last 133.30) - open/close 6751 (time 1727765106.38 total 50.01 last 129.80) - open/close 8032 (time 1727765116.39 total 60.02 last 128.02) - open/close 9367 (time 1727765126.39 total 70.02 last 133.45) total: 10000 open/close in 74.36 seconds: 134.48 ops/second - unlinked 0 (time 1727765132 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1450 (time 1727765184.40 total 10.00 last 144.94) - open/close 3006 (time 1727765194.41 total 20.01 last 155.55) - open/close 4287 (time 1727765204.41 total 30.01 last 128.10) - open/close 5525 (time 1727765214.41 total 40.01 last 123.73) - open/close 6936 (time 1727765224.41 total 50.02 last 141.08) - open/close 8214 (time 1727765234.42 total 60.02 last 127.73) - open/close 9460 (time 1727765244.42 total 70.02 last 124.56) total: 10000 open/close in 74.19 seconds: 134.78 ops/second - unlinked 0 (time 1727765250 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1135 (time 1727765307.06 total 10.00 last 113.45) - open/close 2299 (time 1727765317.07 total 20.02 last 116.24) - open/close 3796 (time 1727765327.07 total 30.02 last 149.65) - open/close 5524 (time 1727765337.08 total 40.03 last 172.66) - open/close 6896 (time 1727765347.08 total 50.03 last 137.19) - open/close 8411 (time 1727765357.08 total 60.03 last 151.49) - open/close 9689 (time 1727765367.09 total 70.03 last 127.76) total: 10000 open/close in 72.10 seconds: 138.69 ops/second - unlinked 0 (time 1727765370 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1451 (time 1727765412.68 total 10.00 last 145.04) - open/close 2973 (time 1727765422.68 total 20.00 last 152.20) - open/close 4606 (time 1727765432.68 total 30.01 last 163.24) - open/close 6226 (time 1727765442.70 total 40.02 last 161.77) - open/close 7725 (time 1727765452.71 total 50.03 last 149.71) - open/close 9130 (time 1727765462.71 total 60.04 last 140.44) total: 10000 open/close in 65.58 seconds: 152.49 ops/second - unlinked 0 (time 1727765469 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1496 (time 1727765519.28 total 10.00 last 149.57) - open/close 2933 (time 1727765529.28 total 20.00 last 143.67) - open/close 4521 (time 1727765539.29 total 30.01 last 158.76) - open/close 6190 (time 1727765549.29 total 40.01 last 166.84) - open/close 7750 (time 1727765559.29 total 50.01 last 155.96) - open/close 9252 (time 1727765569.30 total 60.02 last 150.07) total: 10000 open/close in 65.11 seconds: 153.59 ops/second - unlinked 0 (time 1727765575 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre On OST0, 14589 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server unloading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 PASS 69 (1264s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 02:54:25 (1727765665) oleg113-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity oleg113-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity umount lustre on /mnt/lustre..... stop ost1 service on oleg113-server stop mds service on oleg113-server stop mds service on oleg113-server LNET unconfigure error 22: Invalid argument unloading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local oleg113-server: LNET unconfigure error 22: Invalid argument modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O large_dir -i 1048576 -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=192.168.201.113@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=131072 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Starting client oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre Started clients oleg113-client.virtnet: 192.168.201.113@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) mount lustre on /mnt/lustre2..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Starting client oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Started clients oleg113-client.virtnet: 192.168.201.113@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2280828 1696 2159132 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1388 126828 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1388 126828 2% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 2280 272 2008 12% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 302 49698 1% /mnt/lustre[OST:0] filesystem_summary: 2280 272 2008 12% /mnt/lustre creating 60000 hardlinks to oleg113-client.virtnet-0 creating 60000 hardlinks to oleg113-client.virtnet-1 waiting for PIDs 27475 27489 to complete - link 742 (time 1727765720.20 total 10.02 last 74.06) - link 1485 (time 1727765730.21 total 20.02 last 74.29) - link 2150 (time 1727765740.21 total 30.03 last 66.45) - link 2872 (time 1727765750.23 total 40.04 last 72.12) - link 3517 (time 1727765760.23 total 50.04 last 64.47) - link 4125 (time 1727765770.25 total 60.06 last 60.70) - link 4746 (time 1727765780.25 total 70.06 last 62.10) - link 5452 (time 1727765790.26 total 80.08 last 70.50) - link 6162 (time 1727765800.28 total 90.09 last 70.89) - link 6745 (time 1727765810.29 total 100.10 last 58.23) - link 7387 (time 1727765820.30 total 110.12 last 64.12) - link 7980 (time 1727765830.30 total 120.12 last 59.29) - link 8592 (time 1727765840.31 total 130.12 last 61.17) - link 9351 (time 1727765850.32 total 140.13 last 75.82) - link 9968 (time 1727765860.32 total 150.14 last 61.69) - link 10668 (time 1727765870.33 total 160.14 last 69.97) - link 11457 (time 1727765880.33 total 170.14 last 78.89) - link 12172 (time 1727765890.33 total 180.15 last 71.44) - link 12829 (time 1727765900.35 total 190.16 last 65.60) - link 13551 (time 1727765910.36 total 200.17 last 72.14) - link 14327 (time 1727765920.37 total 210.18 last 77.50) - link 15132 (time 1727765930.38 total 220.19 last 80.42) - link 15950 (time 1727765940.39 total 230.20 last 81.75) - link 16669 (time 1727765950.40 total 240.21 last 71.83) - link 17521 (time 1727765960.40 total 250.21 last 85.18) - link 18281 (time 1727765970.41 total 260.23 last 75.89) - link 18946 (time 1727765980.43 total 270.24 last 66.40) - link 19653 (time 1727765990.44 total 280.25 last 70.64) - link 20000 (time 1727765995.01 total 284.82 last 75.86) - link 20829 (time 1727766005.01 total 294.82 last 82.89) - link 21607 (time 1727766015.02 total 304.84 last 77.70) - link 22302 (time 1727766025.02 total 314.84 last 69.49) - link 22990 (time 1727766035.03 total 324.85 last 68.75) - link 23710 (time 1727766045.04 total 334.85 last 71.95) - link 24441 (time 1727766055.05 total 344.86 last 73.02) - link 25129 (time 1727766065.06 total 354.87 last 68.76) - link 25886 (time 1727766075.07 total 364.88 last 75.61) - link 26734 (time 1727766085.07 total 374.88 last 84.78) - link 27726 (time 1727766095.08 total 384.89 last 99.14) - link 28627 (time 1727766105.08 total 394.90 last 90.05) - link 29422 (time 1727766115.09 total 404.91 last 79.43) - link 30000 (time 1727766122.95 total 412.76 last 73.59) - link 30828 (time 1727766132.97 total 422.78 last 82.62) - link 31645 (time 1727766142.98 total 432.79 last 81.63) - link 32384 (time 1727766152.99 total 442.80 last 73.82) - link 33173 (time 1727766163.00 total 452.82 last 78.77) - link 33893 (time 1727766173.02 total 462.83 last 71.87) - link 34528 (time 1727766183.03 total 472.84 last 63.44) - link 35117 (time 1727766193.04 total 482.86 last 58.82) - link 35773 (time 1727766203.05 total 492.87 last 65.54) - link 36467 (time 1727766213.07 total 502.89 last 69.25) - link 37106 (time 1727766223.07 total 512.89 last 63.89) - link 37704 (time 1727766233.08 total 522.90 last 59.74) - link 38307 (time 1727766243.09 total 532.90 last 60.27) - link 38926 (time 1727766253.10 total 542.91 last 61.85) - link 39501 (time 1727766263.10 total 552.92 last 57.48) - link 40000 (time 1727766270.38 total 560.19 last 68.58) - link 40635 (time 1727766280.38 total 570.19 last 63.49) - link 41296 (time 1727766290.39 total 580.21 last 66.01) - link 41850 (time 1727766300.40 total 590.22 last 55.34) - link 42425 (time 1727766310.41 total 600.22 last 57.46) - link 42976 (time 1727766320.42 total 610.24 last 55.03) - link 43485 (time 1727766330.44 total 620.25 last 50.83) - link 44025 (time 1727766340.46 total 630.27 last 53.88) - link 44506 (time 1727766350.47 total 640.29 last 48.03) - link 45030 (time 1727766360.48 total 650.29 last 52.36) - link 45509 (time 1727766370.50 total 660.32 last 47.78) - link 45942 (time 1727766380.51 total 670.33 last 43.26) - link 46355 (time 1727766390.53 total 680.34 last 41.25) - link 46875 (time 1727766400.54 total 690.35 last 51.95) - link 47527 (time 1727766410.56 total 700.37 last 65.07) - link 48128 (time 1727766420.56 total 710.37 last 60.07) - link 48728 (time 1727766430.56 total 720.37 last 60.00) - link 49276 (time 1727766440.57 total 730.39 last 54.74) - link 49994 (time 1727766450.57 total 740.39 last 71.78) - link 50573 (time 1727766460.58 total 750.40 last 57.86) - link 51122 (time 1727766470.59 total 760.40 last 54.87) - link 51696 (time 1727766480.61 total 770.42 last 57.29) - link 52255 (time 1727766490.61 total 780.43 last 55.87) - link 52792 (time 1727766500.62 total 790.43 last 53.68) - link 53371 (time 1727766510.62 total 800.44 last 57.86) - link 53972 (time 1727766520.63 total 810.45 last 60.05) - link 54554 (time 1727766530.64 total 820.46 last 58.13) - link 55132 (time 1727766540.65 total 830.46 last 57.77) - link 55700 (time 1727766550.65 total 840.47 last 56.79) - link 56309 (time 1727766560.66 total 850.48 last 60.82) - link 56859 (time 1727766570.67 total 860.49 last 54.95) - link 57502 (time 1727766580.68 total 870.49 last 64.24) - link 58164 (time 1727766590.69 total 880.51 last 66.13) - link 58739 (time 1727766600.71 total 890.52 last 57.42) - link 59292 (time 1727766610.72 total 900.53 last 55.22) - link 59876 (time 1727766620.74 total 910.55 last 58.32) total: 60000 link in 912.47 seconds: 65.76 ops/second - link 736 (time 1727765720.19 total 10.01 last 73.56) - link 1470 (time 1727765730.19 total 20.01 last 73.39) - link 2133 (time 1727765740.19 total 30.01 last 66.26) - link 2857 (time 1727765750.21 total 40.03 last 72.29) - link 3504 (time 1727765760.21 total 50.03 last 64.67) - link 4116 (time 1727765770.22 total 60.04 last 61.16) - link 4739 (time 1727765780.24 total 70.06 last 62.19) - link 5444 (time 1727765790.24 total 80.06 last 70.48) - link 6161 (time 1727765800.25 total 90.07 last 71.60) - link 6747 (time 1727765810.27 total 100.09 last 58.52) - link 7372 (time 1727765820.28 total 110.09 last 62.45) - link 7967 (time 1727765830.30 total 120.12 last 59.34) - link 8586 (time 1727765840.31 total 130.13 last 61.87) - link 9333 (time 1727765850.32 total 140.14 last 74.61) - link 9946 (time 1727765860.35 total 150.17 last 61.12) - link 10640 (time 1727765870.36 total 160.18 last 69.32) - link 11412 (time 1727765880.36 total 170.18 last 77.20) - link 12122 (time 1727765890.38 total 180.20 last 70.85) - link 12780 (time 1727765900.39 total 190.21 last 65.79) - link 13508 (time 1727765910.40 total 200.22 last 72.72) - link 14293 (time 1727765920.40 total 210.22 last 78.49) - link 15098 (time 1727765930.40 total 220.22 last 80.50) - link 15924 (time 1727765940.40 total 230.22 last 82.57) - link 16636 (time 1727765950.41 total 240.23 last 71.14) - link 17487 (time 1727765960.42 total 250.24 last 84.99) - link 18247 (time 1727765970.45 total 260.27 last 75.83) - link 18924 (time 1727765980.46 total 270.28 last 67.59) - link 19637 (time 1727765990.47 total 280.29 last 71.23) - link 20000 (time 1727765995.28 total 285.10 last 75.50) - link 20817 (time 1727766005.29 total 295.11 last 81.63) - link 21590 (time 1727766015.30 total 305.12 last 77.21) - link 22284 (time 1727766025.31 total 315.13 last 69.31) - link 22963 (time 1727766035.32 total 325.14 last 67.85) - link 23683 (time 1727766045.33 total 335.15 last 71.93) - link 24412 (time 1727766055.33 total 345.15 last 72.89) - link 25099 (time 1727766065.34 total 355.16 last 68.65) - link 25852 (time 1727766075.34 total 365.16 last 75.26) - link 26703 (time 1727766085.35 total 375.17 last 85.06) - link 27716 (time 1727766095.35 total 385.17 last 101.29) - link 28608 (time 1727766105.36 total 395.18 last 89.15) - link 29414 (time 1727766115.36 total 405.18 last 80.58) - link 30000 (time 1727766123.16 total 412.98 last 75.08) - link 30814 (time 1727766133.16 total 422.98 last 81.39) - link 31624 (time 1727766143.18 total 433.00 last 80.90) - link 32371 (time 1727766153.19 total 443.01 last 74.59) - link 33148 (time 1727766163.20 total 453.02 last 77.66) - link 33866 (time 1727766173.20 total 463.02 last 71.78) - link 34509 (time 1727766183.21 total 473.03 last 64.26) - link 35101 (time 1727766193.21 total 483.03 last 59.16) - link 35763 (time 1727766203.23 total 493.05 last 66.10) - link 36454 (time 1727766213.25 total 503.07 last 68.99) - link 37088 (time 1727766223.26 total 513.08 last 63.30) - link 37678 (time 1727766233.27 total 523.09 last 58.96) - link 38288 (time 1727766243.28 total 533.10 last 60.95) - link 38900 (time 1727766253.28 total 543.10 last 61.18) - link 39473 (time 1727766263.28 total 553.10 last 57.27) - link 40000 (time 1727766271.02 total 560.84 last 68.09) - link 40640 (time 1727766281.03 total 570.85 last 63.94) - link 41288 (time 1727766291.06 total 580.88 last 64.64) - link 41844 (time 1727766301.07 total 590.89 last 55.53) - link 42424 (time 1727766311.08 total 600.90 last 57.94) - link 42977 (time 1727766321.09 total 610.91 last 55.27) - link 43502 (time 1727766331.11 total 620.93 last 52.40) - link 44039 (time 1727766341.12 total 630.93 last 53.65) - link 44515 (time 1727766351.12 total 640.94 last 47.60) - link 45058 (time 1727766361.13 total 650.95 last 54.24) - link 45521 (time 1727766371.13 total 660.95 last 46.27) - link 45952 (time 1727766381.15 total 670.97 last 43.02) - link 46363 (time 1727766391.17 total 680.99 last 41.04) - link 46883 (time 1727766401.17 total 690.99 last 51.96) - link 47531 (time 1727766411.19 total 701.01 last 64.73) - link 48151 (time 1727766421.20 total 711.02 last 61.94) - link 48752 (time 1727766431.21 total 721.03 last 60.01) - link 49312 (time 1727766441.21 total 731.03 last 55.99) - link 50000 (time 1727766450.72 total 740.54 last 72.35) - link 50585 (time 1727766460.73 total 750.55 last 58.45) - link 51120 (time 1727766470.74 total 760.56 last 53.48) - link 51700 (time 1727766480.75 total 770.57 last 57.90) - link 52257 (time 1727766490.76 total 780.58 last 55.69) - link 52788 (time 1727766500.77 total 790.59 last 53.01) - link 53369 (time 1727766510.79 total 800.61 last 58.02) - link 53972 (time 1727766520.80 total 810.62 last 60.19) - link 54551 (time 1727766530.81 total 820.63 last 57.84) - link 55131 (time 1727766540.82 total 830.64 last 57.97) - link 55697 (time 1727766550.82 total 840.64 last 56.57) - link 56305 (time 1727766560.84 total 850.66 last 60.71) - link 56863 (time 1727766570.86 total 860.68 last 55.71) - link 57512 (time 1727766580.87 total 870.69 last 64.83) - link 58169 (time 1727766590.88 total 880.70 last 65.60) - link 58738 (time 1727766600.90 total 890.72 last 56.80) - link 59288 (time 1727766610.91 total 900.73 last 54.95) - link 59865 (time 1727766620.92 total 910.74 last 57.64) total: 60000 link in 912.74 seconds: 65.74 ops/second estimate 38507s left after 120000 files / 913s umount lustre on /mnt/lustre2..... Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. ETA 38507s after 120000 files / 913s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg113-server: e2fsck 1.46.2.wc5 (26-Mar-2022) oleg113-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 19) [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10oleg113-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51384320, 266) != expected (51392512, 266) oleg113-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51384320, 266) != expected (51392512, 266) oleg113-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51384320, 266) != expected (51392512, 266) 084 [Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 180 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 181 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 188 badness 0 to 2 for 10084 [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] Pass 1: Memory used: 380k/908k (124k/257k), time: 0.04/ 0.01/ 0.00 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 51.92MB/s [Thread 0] Scanned group range [0, 19), inodes 281 Pass 2: Checking directory structure Pass 2: Memory used: 380k/0k (85k/296k), time: 0.71/ 0.39/ 0.05 Pass 2: I/O read: 49MB, write: 0MB, rate: 69.04MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 380k/0k (85k/296k), time: 0.78/ 0.40/ 0.06 Papdsh@oleg113-client: oleg113-server: ssh exited with exit code 4 ss 3: Memory used: 380k/0k (83k/298k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 380k/0k (70k/311k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 358.68MB/s Pass 5: Checking group summary information Pass 5: Memory used: 380k/0k (68k/313k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 242.48MB/s Update quota info for quota type 0? no Update quota info for quota type 1? no Update quota info for quota type 2? no lustre-MDT0000: ********** WARNING: Filesystem still has errors ********** 276 inodes used (12.11%, out of 2280) 3 non-contiguous files (1.1%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 37883 blocks used (6.31%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 380k/0k (67k/314k), time: 0.79/ 0.41/ 0.06 I/O read: 49MB, write: 0MB, rate: 62.11MB/s pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (1038s) Stopping clients: oleg113-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg113-client.virtnet /mnt/lustre2 (opts:-f) oleg113-server: oleg113-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2629 sec ========== 03:12:13 (1727766733) === conf-sanity: start cleanup 03:12:14 (1727766734) === === conf-sanity: finish cleanup 03:12:14 (1727766734) ===