-----============= acceptance-small: conf-sanity ============----- Wed May 29 15:44:22 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg134-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg134-server oleg134-server: oleg134-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg134-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg134-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:46:01 (1717011961) start mds service on oleg134-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg134-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg134-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg134-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 PASS 45 (186s) == conf-sanity test 69: replace an OST with the same index ========================================================== 15:49:07 (1717012147) start mds service on oleg134-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg134-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1649 (time 1717012202.99 total 10.01 last 164.82) - open/close 3127 (time 1717012213.00 total 20.01 last 147.70) - open/close 4671 (time 1717012223.00 total 30.02 last 154.35) - open/close 6128 (time 1717012233.01 total 40.02 last 145.67) - open/close 7539 (time 1717012243.01 total 50.02 last 141.09) - open/close 9078 (time 1717012253.01 total 60.02 last 153.83) total: 10000 open/close in 66.23 seconds: 150.99 ops/second - unlinked 0 (time 1717012260 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1767 (time 1717012306.39 total 10.00 last 176.63) - open/close 3592 (time 1717012316.39 total 20.00 last 182.49) - open/close 5143 (time 1717012326.39 total 30.01 last 155.06) - open/close 6907 (time 1717012336.40 total 40.01 last 176.35) - open/close 8397 (time 1717012346.40 total 50.01 last 149.00) - open/close 9826 (time 1717012356.40 total 60.01 last 142.87) total: 10000 open/close in 61.26 seconds: 163.24 ops/second - unlinked 0 (time 1717012359 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1465 (time 1717012407.19 total 10.00 last 146.47) - open/close 2990 (time 1717012417.20 total 20.01 last 152.42) - open/close 4960 (time 1717012427.20 total 30.01 last 196.97) - open/close 6430 (time 1717012437.20 total 40.01 last 146.97) - open/close 8090 (time 1717012447.21 total 50.01 last 165.95) - open/close 9707 (time 1717012457.21 total 60.02 last 161.64) total: 10000 open/close in 61.94 seconds: 161.46 ops/second - unlinked 0 (time 1717012460 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1523 (time 1717012507.01 total 10.01 last 152.18) - open/close 3050 (time 1717012517.01 total 20.01 last 152.69) - open/close 4515 (time 1717012527.01 total 30.01 last 146.48) - open/close 5973 (time 1717012537.02 total 40.02 last 145.69) - open/close 7358 (time 1717012547.03 total 50.02 last 138.44) - open/close 8740 (time 1717012557.03 total 60.02 last 138.20) - open/close 9990 (time 1717012567.04 total 70.03 last 124.86) total: 10000 open/close in 70.11 seconds: 142.64 ops/second - unlinked 0 (time 1717012569 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1346 (time 1717012619.70 total 10.01 last 134.42) - open/close 2695 (time 1717012629.70 total 20.02 last 134.86) - open/close 3971 (time 1717012639.71 total 30.02 last 127.59) - open/close 5169 (time 1717012649.71 total 40.02 last 119.73) - open/close 6499 (time 1717012659.71 total 50.03 last 132.96) - open/close 7814 (time 1717012669.72 total 60.04 last 131.37) - open/close 8991 (time 1717012679.73 total 70.04 last 117.66) total: 10000 open/close in 77.56 seconds: 128.93 ops/second - unlinked 0 (time 1717012689 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1401 (time 1717012740.07 total 10.00 last 140.08) - open/close 2923 (time 1717012750.07 total 20.00 last 152.16) - open/close 4371 (time 1717012760.07 total 30.01 last 144.78) - open/close 5898 (time 1717012770.07 total 40.01 last 152.67) - open/close 7421 (time 1717012780.07 total 50.01 last 152.28) - open/close 8836 (time 1717012790.08 total 60.02 last 141.36) total: 10000 open/close in 68.19 seconds: 146.65 ops/second - unlinked 0 (time 1717012799 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1366 (time 1717012851.13 total 10.00 last 136.57) - open/close 2719 (time 1717012861.13 total 20.01 last 135.20) - open/close 4083 (time 1717012871.14 total 30.01 last 136.33) - open/close 5536 (time 1717012881.14 total 40.01 last 145.30) - open/close 6901 (time 1717012891.15 total 50.02 last 136.37) - open/close 8224 (time 1717012901.15 total 60.02 last 132.29) - open/close 9643 (time 1717012911.16 total 70.03 last 141.78) total: 10000 open/close in 72.42 seconds: 138.08 ops/second - unlinked 0 (time 1717012915 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1433 (time 1717012965.05 total 10.00 last 143.29) - open/close 2858 (time 1717012975.06 total 20.00 last 142.45) - open/close 4323 (time 1717012985.06 total 30.01 last 146.45) - open/close 5881 (time 1717012995.06 total 40.01 last 155.77) - open/close 7385 (time 1717013005.07 total 50.02 last 150.29) - open/close 8829 (time 1717013015.07 total 60.02 last 144.35) total: 10000 open/close in 68.75 seconds: 145.46 ops/second - unlinked 0 (time 1717013025 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1424 (time 1717013077.54 total 10.01 last 142.29) - open/close 2992 (time 1717013087.55 total 20.01 last 156.71) - open/close 4450 (time 1717013097.55 total 30.02 last 145.76) - open/close 6005 (time 1717013107.55 total 40.02 last 155.46) - open/close 7681 (time 1717013117.56 total 50.02 last 167.58) - open/close 9296 (time 1717013127.56 total 60.02 last 161.43) total: 10000 open/close in 64.19 seconds: 155.79 ops/second - unlinked 0 (time 1717013132 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1604 (time 1717013176.42 total 10.00 last 160.34) - open/close 3200 (time 1717013186.42 total 20.00 last 159.59) - open/close 4891 (time 1717013196.42 total 30.01 last 169.01) - open/close 6448 (time 1717013206.43 total 40.02 last 155.58) - open/close 8148 (time 1717013216.43 total 50.02 last 169.97) - open/close 9961 (time 1717013226.44 total 60.02 last 181.25) total: 10000 open/close in 60.34 seconds: 165.71 ops/second - unlinked 0 (time 1717013228 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 PASS 69 (1179s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:08:46 (1717013326) oleg134-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity oleg134-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity umount lustre on /mnt/lustre..... stop ost1 service on oleg134-server stop mds service on oleg134-server stop mds service on oleg134-server LNET unconfigure error 22: Invalid argument unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: LNET unconfigure error 22: Invalid argument modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O large_dir -i 1048576 -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg134-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=131072 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre Starting client oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre Started clients oleg134-client.virtnet: 192.168.201.134@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) mount lustre on /mnt/lustre2..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre2 Starting client oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre2 Started clients oleg134-client.virtnet: 192.168.201.134@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2280828 1700 2159128 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1388 126828 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1388 126828 2% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 2280 273 2007 12% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 302 49698 1% /mnt/lustre[OST:0] filesystem_summary: 2280 273 2007 12% /mnt/lustre creating 60000 hardlinks to oleg134-client.virtnet-0 creating 60000 hardlinks to oleg134-client.virtnet-1 waiting for PIDs 27227 27241 to complete - link 726 (time 1717013387.07 total 10.00 last 72.58) - link 1452 (time 1717013397.07 total 20.01 last 72.57) - link 2128 (time 1717013407.09 total 30.03 last 67.46) - link 2797 (time 1717013417.09 total 40.03 last 66.90) - link 3362 (time 1717013427.10 total 50.04 last 56.44) - link 3970 (time 1717013437.11 total 60.04 last 60.77) - link 4581 (time 1717013447.12 total 70.05 last 61.05) - link 5104 (time 1717013457.13 total 80.06 last 52.25) - link 5656 (time 1717013467.14 total 90.08 last 55.12) - link 6338 (time 1717013477.15 total 100.08 last 68.15) - link 6974 (time 1717013487.15 total 110.09 last 63.59) - link 7519 (time 1717013497.16 total 120.09 last 54.48) - link 8186 (time 1717013507.17 total 130.10 last 66.65) - link 8925 (time 1717013517.18 total 140.11 last 73.81) - link 9631 (time 1717013527.18 total 150.12 last 70.56) - link 10000 (time 1717013533.67 total 156.60 last 56.89) - link 10603 (time 1717013543.67 total 166.61 last 60.26) - link 11197 (time 1717013553.69 total 176.62 last 59.32) - link 11798 (time 1717013563.70 total 186.63 last 60.03) - link 12415 (time 1717013573.71 total 196.64 last 61.64) - link 13167 (time 1717013583.72 total 206.65 last 75.16) - link 13913 (time 1717013593.72 total 216.65 last 74.57) - link 14514 (time 1717013603.72 total 226.65 last 60.09) - link 15171 (time 1717013613.74 total 236.67 last 65.60) - link 15758 (time 1717013623.74 total 246.68 last 58.67) - link 16495 (time 1717013633.76 total 256.69 last 73.56) - link 17213 (time 1717013643.77 total 266.70 last 71.75) - link 17934 (time 1717013653.78 total 276.71 last 72.04) - link 18750 (time 1717013663.78 total 286.71 last 81.57) - link 19481 (time 1717013673.79 total 296.72 last 73.05) - link 20000 (time 1717013682.45 total 305.39 last 59.90) - link 20596 (time 1717013692.46 total 315.39 last 59.56) - link 21245 (time 1717013702.47 total 325.41 last 64.81) - link 21938 (time 1717013712.48 total 335.41 last 69.25) - link 22726 (time 1717013722.48 total 345.42 last 78.76) - link 23569 (time 1717013732.49 total 355.43 last 84.22) - link 24354 (time 1717013742.50 total 365.43 last 78.45) - link 25198 (time 1717013752.50 total 375.43 last 84.38) - link 26072 (time 1717013762.51 total 385.44 last 87.34) - link 26865 (time 1717013772.52 total 395.45 last 79.24) - link 27677 (time 1717013782.52 total 405.45 last 81.16) - link 28508 (time 1717013792.52 total 415.46 last 83.08) - link 29283 (time 1717013802.53 total 425.46 last 77.47) - link 30000 (time 1717013810.87 total 433.80 last 85.93) - link 30729 (time 1717013820.88 total 443.81 last 72.82) - link 31483 (time 1717013830.88 total 453.81 last 75.40) - link 32410 (time 1717013840.88 total 463.82 last 92.69) - link 33334 (time 1717013850.89 total 473.82 last 92.37) - link 34069 (time 1717013860.90 total 483.83 last 73.38) - link 34640 (time 1717013870.91 total 493.84 last 57.07) - link 35266 (time 1717013880.91 total 503.84 last 62.59) - link 35895 (time 1717013890.92 total 513.86 last 62.82) - link 36637 (time 1717013900.93 total 523.87 last 74.12) - link 37474 (time 1717013910.94 total 533.87 last 83.66) - link 38187 (time 1717013920.94 total 543.87 last 71.27) - link 38809 (time 1717013930.96 total 553.89 last 62.11) - link 39417 (time 1717013940.97 total 563.90 last 60.71) - link 40000 (time 1717013950.19 total 573.12 last 63.24) - link 40646 (time 1717013960.20 total 583.13 last 64.55) - link 41332 (time 1717013970.21 total 593.14 last 68.54) - link 42074 (time 1717013980.22 total 603.15 last 74.12) - link 42667 (time 1717013990.23 total 613.16 last 59.25) - link 43234 (time 1717014000.24 total 623.17 last 56.62) - link 43798 (time 1717014010.24 total 633.18 last 56.38) - link 44519 (time 1717014020.25 total 643.19 last 72.04) - link 45205 (time 1717014030.26 total 653.19 last 68.54) - link 45915 (time 1717014040.26 total 663.19 last 71.00) - link 46730 (time 1717014050.28 total 673.21 last 81.38) - link 47523 (time 1717014060.28 total 683.21 last 79.26) - link 48307 (time 1717014070.29 total 693.22 last 78.33) - link 49077 (time 1717014080.30 total 703.23 last 76.96) - link 49823 (time 1717014090.30 total 713.23 last 74.55) - link 50000 (time 1717014092.88 total 715.81 last 68.65) - link 50806 (time 1717014102.89 total 725.83 last 80.48) - link 51486 (time 1717014112.91 total 735.84 last 67.92) - link 52148 (time 1717014122.91 total 745.85 last 66.15) - link 52811 (time 1717014132.92 total 755.85 last 66.26) - link 53565 (time 1717014142.93 total 765.86 last 75.34) - link 54281 (time 1717014152.93 total 775.86 last 71.58) - link 54902 (time 1717014162.93 total 785.87 last 62.08) - link 55514 (time 1717014172.94 total 795.88 last 61.14) - link 56093 (time 1717014182.94 total 805.88 last 57.90) - link 56732 (time 1717014192.95 total 815.88 last 63.88) - link 57354 (time 1717014202.95 total 825.88 last 62.17) - link 57933 (time 1717014212.96 total 835.89 last 57.85) - link 58585 (time 1717014222.97 total 845.90 last 65.14) - link 59276 (time 1717014232.99 total 855.92 last 68.98) - link 59972 (time 1717014243.00 total 865.93 last 69.54) total: 60000 link in 866.45 seconds: 69.25 ops/second - link 716 (time 1717013386.97 total 10.01 last 71.54) - link 1445 (time 1717013396.97 total 20.01 last 72.87) - link 2126 (time 1717013406.98 total 30.02 last 68.02) - link 2796 (time 1717013416.99 total 40.03 last 66.98) - link 3362 (time 1717013427.00 total 50.04 last 56.54) - link 3966 (time 1717013437.00 total 60.04 last 60.37) - link 4580 (time 1717013447.01 total 70.05 last 61.35) - link 5103 (time 1717013457.02 total 80.06 last 52.25) - link 5664 (time 1717013467.03 total 90.06 last 56.07) - link 6342 (time 1717013477.03 total 100.07 last 67.77) - link 6989 (time 1717013487.04 total 110.08 last 64.65) - link 7531 (time 1717013497.05 total 120.08 last 54.15) - link 8190 (time 1717013507.05 total 130.09 last 65.85) - link 8925 (time 1717013517.06 total 140.10 last 73.47) - link 9637 (time 1717013527.06 total 150.10 last 71.17) - link 10000 (time 1717013533.40 total 156.44 last 57.30) - link 10602 (time 1717013543.40 total 166.44 last 60.16) - link 11202 (time 1717013553.41 total 176.45 last 59.94) - link 11794 (time 1717013563.42 total 186.46 last 59.18) - link 12402 (time 1717013573.44 total 196.48 last 60.63) - link 13146 (time 1717013583.45 total 206.49 last 74.34) - link 13898 (time 1717013593.46 total 216.50 last 75.17) - link 14495 (time 1717013603.46 total 226.50 last 59.68) - link 15156 (time 1717013613.46 total 236.50 last 66.08) - link 15741 (time 1717013623.48 total 246.51 last 58.41) - link 16482 (time 1717013633.49 total 256.53 last 73.96) - link 17207 (time 1717013643.51 total 266.55 last 72.38) - link 17915 (time 1717013653.52 total 276.56 last 70.74) - link 18728 (time 1717013663.52 total 286.56 last 81.30) - link 19456 (time 1717013673.52 total 296.56 last 72.76) - link 20000 (time 1717013682.48 total 305.52 last 60.76) - link 20601 (time 1717013692.48 total 315.52 last 60.09) - link 21246 (time 1717013702.49 total 325.53 last 64.43) - link 21948 (time 1717013712.50 total 335.54 last 70.14) - link 22735 (time 1717013722.51 total 345.55 last 78.60) - link 23576 (time 1717013732.53 total 355.57 last 83.96) - link 24363 (time 1717013742.53 total 365.57 last 78.70) - link 25206 (time 1717013752.53 total 375.57 last 84.28) - link 26076 (time 1717013762.54 total 385.58 last 86.89) - link 26878 (time 1717013772.55 total 395.59 last 80.17) - link 27703 (time 1717013782.55 total 405.59 last 82.44) - link 28532 (time 1717013792.56 total 415.60 last 82.83) - link 29298 (time 1717013802.57 total 425.61 last 76.51) - link 30000 (time 1717013810.65 total 433.68 last 86.97) - link 30727 (time 1717013820.65 total 443.69 last 72.63) - link 31492 (time 1717013830.67 total 453.71 last 76.41) - link 32417 (time 1717013840.67 total 463.71 last 92.49) - link 33346 (time 1717013850.68 total 473.72 last 92.79) - link 34093 (time 1717013860.68 total 483.72 last 74.66) - link 34671 (time 1717013870.69 total 493.73 last 57.79) - link 35301 (time 1717013880.69 total 503.73 last 62.97) - link 35924 (time 1717013890.70 total 513.74 last 62.24) - link 36664 (time 1717013900.71 total 523.75 last 73.94) - link 37499 (time 1717013910.71 total 533.75 last 83.47) - link 38224 (time 1717013920.72 total 543.76 last 72.44) - link 38848 (time 1717013930.73 total 553.77 last 62.33) - link 39448 (time 1717013940.74 total 563.78 last 59.96) - link 40000 (time 1717013949.54 total 572.58 last 62.70) - link 40637 (time 1717013959.56 total 582.60 last 63.57) - link 41317 (time 1717013969.56 total 592.60 last 68.00) - link 42062 (time 1717013979.57 total 602.61 last 74.48) - link 42652 (time 1717013989.58 total 612.62 last 58.91) - link 43225 (time 1717013999.59 total 622.63 last 57.26) - link 43800 (time 1717014009.59 total 632.63 last 57.49) - link 44505 (time 1717014019.59 total 642.63 last 70.50) - link 45207 (time 1717014029.60 total 652.64 last 70.13) - link 45896 (time 1717014039.61 total 662.65 last 68.85) - link 46707 (time 1717014049.62 total 672.65 last 81.04) - link 47505 (time 1717014059.64 total 682.68 last 79.59) - link 48264 (time 1717014069.65 total 692.69 last 75.86) - link 49038 (time 1717014079.65 total 702.69 last 77.38) - link 49791 (time 1717014089.66 total 712.70 last 75.21) - link 50000 (time 1717014092.62 total 715.66 last 70.73) - link 50819 (time 1717014102.63 total 725.67 last 81.82) - link 51479 (time 1717014112.63 total 735.67 last 65.98) - link 52154 (time 1717014122.64 total 745.68 last 67.40) - link 52821 (time 1717014132.66 total 755.70 last 66.59) - link 53561 (time 1717014142.66 total 765.70 last 73.99) - link 54281 (time 1717014152.66 total 775.70 last 72.00) - link 54899 (time 1717014162.67 total 785.71 last 61.77) - link 55516 (time 1717014172.67 total 795.71 last 61.66) - link 56091 (time 1717014182.69 total 805.73 last 57.41) - link 56720 (time 1717014192.70 total 815.74 last 62.82) - link 57344 (time 1717014202.70 total 825.74 last 62.39) - link 57918 (time 1717014212.72 total 835.75 last 57.33) - link 58562 (time 1717014222.72 total 845.76 last 64.37) - link 59259 (time 1717014232.73 total 855.77 last 69.65) - link 59949 (time 1717014242.75 total 865.78 last 68.88) total: 60000 link in 866.71 seconds: 69.23 ops/second estimate 36431s left after 120000 files / 867s umount lustre on /mnt/lustre2..... Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. ETA 36431s after 120000 files / 867s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg134-server: e2fsck 1.46.2.wc5 (26-Mar-2022) oleg134-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 19) [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2564: increase inode 128 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 129 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 130 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 131 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 136 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 180 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 181 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 185 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 187 badness 0 to 2 for 10084 [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] Pass 1: Memory used: 380k/908k (124k/257k), time: 0.02/ 0.00/ 0.01 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 87.82MB/s [Thread 0] Scanned group range [0, 19), inodes 281 Pass 2: Checking directory structure Pass 2: Memory used: 380k/0k (85k/296k), time: 0.58/ 0.34/ 0.06 Pass 2: I/O read: 49MB, write: 0MB, rate: 84.80MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 380k/0k (85k/296k), time: 0.62/ 0.35/ 0.07 Paoleg134-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51564544, 266) != expected (51572736, 266) oleg134-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51564544, 266) != expected (51572736, 266) oleg134-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51564544, 266) != expected (51572736, 266) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 4 ss 3: Memory used: 380k/0k (83k/298k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 380k/0k (70k/311k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 6849.31MB/s Pass 5: Checking group summary information Pass 5: Memory used: 380k/0k (68k/313k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 464.90MB/s Update quota info for quota type 0? no Update quota info for quota type 1? no Update quota info for quota type 2? no lustre-MDT0000: ********** WARNING: Filesystem still has errors ********** 276 inodes used (12.11%, out of 2280) 3 non-contiguous files (1.1%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 37927 blocks used (6.32%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 380k/0k (67k/314k), time: 0.63/ 0.35/ 0.07 I/O read: 49MB, write: 0MB, rate: 78.28MB/s pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 oleg134-server: oleg134-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (984s) Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) oleg134-server: oleg134-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2472 sec ========== 16:25:34 (1717014334) === conf-sanity: start cleanup 16:25:35 (1717014335) === === conf-sanity: finish cleanup 16:25:35 (1717014335) ===