-----============= acceptance-small: conf-sanity ============----- Tue Feb 27 12:09:41 EST 2024 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg143-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg143-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg143-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg143-server oleg143-server: oleg143-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg143-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 45: long unlink handling in ptlrpcd == 12:11:03 (1709053863) start mds service on oleg143-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg143-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 PASS 45 (153s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg143-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg143-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg143-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 oleg143-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg143-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 69: replace an OST with the same index ========================================================== 12:13:39 (1709054019) start mds service on oleg143-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg143-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1425 (time 1709054076.19 total 10.00 last 142.50) - open/close 3090 (time 1709054086.19 total 20.00 last 166.44) - open/close 4802 (time 1709054096.19 total 30.00 last 171.19) - open/close 5449 (time 1709054113.88 total 47.69 last 36.59) - open/close 6163 (time 1709054123.88 total 57.69 last 71.37) - open/close 7260 (time 1709054133.89 total 67.70 last 109.59) - open/close 8365 (time 1709054143.90 total 77.71 last 110.46) - open/close 9175 (time 1709054153.91 total 87.72 last 80.90) - open/close 9770 (time 1709054163.91 total 97.72 last 59.47) total: 10000 open/close in 101.48 seconds: 98.54 ops/second - unlinked 0 (time 1709054169 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 943 (time 1709054245.83 total 10.01 last 94.24) - open/close 1970 (time 1709054255.84 total 20.02 last 102.57) - open/close 2885 (time 1709054265.84 total 30.02 last 91.46) - open/close 3697 (time 1709054275.84 total 40.03 last 81.19) - open/close 4367 (time 1709054285.86 total 50.04 last 66.88) - open/close 4961 (time 1709054295.87 total 60.05 last 59.36) - open/close 5640 (time 1709054305.89 total 70.07 last 67.74) - open/close 5801 (time 1709054319.73 total 83.91 last 11.63) - open/close 6456 (time 1709054329.74 total 93.93 last 65.41) - open/close 7082 (time 1709054339.75 total 103.93 last 62.59) - open/close 7669 (time 1709054349.75 total 113.93 last 58.69) - open/close 8265 (time 1709054359.77 total 123.95 last 59.49) - open/close 8833 (time 1709054369.78 total 133.96 last 56.75) - open/close 9444 (time 1709054379.80 total 143.98 last 60.94) total: 10000 open/close in 153.18 seconds: 65.28 ops/second - unlinked 0 (time 1709054391 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 1057 (time 1709054506.51 total 10.00 last 105.68) - open/close 2017 (time 1709054516.52 total 20.01 last 95.88) - open/close 2854 (time 1709054526.52 total 30.02 last 83.68) - open/close 3689 (time 1709054536.53 total 40.02 last 83.45) - open/close 4474 (time 1709054546.54 total 50.03 last 78.42) - open/close 5151 (time 1709054556.55 total 60.04 last 67.63) - open/close 5435 (time 1709054571.76 total 75.25 last 18.67) - open/close 6158 (time 1709054581.76 total 85.26 last 72.26) - open/close 6769 (time 1709054591.76 total 95.26 last 61.09) - open/close 7358 (time 1709054601.78 total 105.28 last 58.78) - open/close 7932 (time 1709054611.78 total 115.28 last 57.40) - open/close 8531 (time 1709054621.79 total 125.29 last 59.85) - open/close 9140 (time 1709054631.80 total 135.29 last 60.87) - open/close 9654 (time 1709054641.80 total 145.30 last 51.37) total: 10000 open/close in 151.99 seconds: 65.79 ops/second - unlinked 0 (time 1709054650 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 986 (time 1709054771.14 total 10.01 last 98.54) - open/close 2042 (time 1709054781.15 total 20.02 last 105.50) - open/close 3117 (time 1709054791.15 total 30.02 last 107.44) - open/close 4051 (time 1709054801.16 total 40.03 last 93.36) - open/close 4894 (time 1709054811.16 total 50.03 last 84.24) - open/close 5687 (time 1709054821.16 total 60.03 last 79.30) - open/close 6450 (time 1709054831.16 total 70.03 last 76.29) - open/close 7192 (time 1709054841.17 total 80.04 last 74.14) - open/close 7878 (time 1709054851.18 total 90.05 last 68.54) - open/close 8479 (time 1709054861.19 total 100.06 last 60.05) - open/close 9074 (time 1709054871.19 total 110.06 last 59.47) - open/close 9583 (time 1709054881.20 total 120.07 last 50.88) total: 10000 open/close in 126.70 seconds: 78.92 ops/second - unlinked 0 (time 1709054889 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1410 (time 1709054973.91 total 10.00 last 141.00) - open/close 2429 (time 1709054983.92 total 20.01 last 101.77) - open/close 3296 (time 1709054993.93 total 30.02 last 86.64) - open/close 4065 (time 1709055003.93 total 40.02 last 76.86) - open/close 4810 (time 1709055013.94 total 50.03 last 74.43) - open/close 5528 (time 1709055023.95 total 60.05 last 71.70) - open/close 6215 (time 1709055033.98 total 70.07 last 68.51) - open/close 6904 (time 1709055043.98 total 80.08 last 68.89) - open/close 7491 (time 1709055053.99 total 90.08 last 58.67) - open/close 8108 (time 1709055063.99 total 100.08 last 61.68) - open/close 8795 (time 1709055074.00 total 110.09 last 68.67) - open/close 9394 (time 1709055084.01 total 120.10 last 59.82) - open/close 9954 (time 1709055094.01 total 130.10 last 55.99) total: 10000 open/close in 130.82 seconds: 76.44 ops/second - unlinked 0 (time 1709055096 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1264 (time 1709055174.77 total 10.01 last 126.22) - open/close 2280 (time 1709055184.78 total 20.02 last 101.56) - open/close 3229 (time 1709055194.78 total 30.02 last 94.88) - open/close 4139 (time 1709055204.78 total 40.02 last 90.99) - open/close 4436 (time 1709055219.37 total 54.61 last 20.36) - open/close 5211 (time 1709055229.37 total 64.62 last 77.45) - open/close 6015 (time 1709055239.38 total 74.62 last 80.33) - open/close 6781 (time 1709055249.38 total 84.63 last 76.57) - open/close 7459 (time 1709055259.40 total 94.64 last 67.70) - open/close 8086 (time 1709055269.42 total 104.66 last 62.60) - open/close 8700 (time 1709055279.43 total 114.67 last 61.31) - open/close 9218 (time 1709055289.48 total 124.72 last 51.56) - open/close 9728 (time 1709055299.48 total 134.73 last 50.97) total: 10000 open/close in 140.04 seconds: 71.41 ops/second - unlinked 0 (time 1709055306 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 1070 (time 1709055422.98 total 10.00 last 106.98) - open/close 2095 (time 1709055432.99 total 20.01 last 102.39) - open/close 2964 (time 1709055442.99 total 30.02 last 86.84) - open/close 3937 (time 1709055453.00 total 40.03 last 97.23) - open/close 4756 (time 1709055463.01 total 50.03 last 81.84) - open/close 5008 (time 1709055477.75 total 64.77 last 17.10) - open/close 5845 (time 1709055487.75 total 74.77 last 83.68) - open/close 6753 (time 1709055497.75 total 84.77 last 90.78) - open/close 7557 (time 1709055507.75 total 94.78 last 80.37) - open/close 8380 (time 1709055517.75 total 104.78 last 82.29) - open/close 9237 (time 1709055527.76 total 114.79 last 85.63) - open/close 9971 (time 1709055537.77 total 124.79 last 73.35) total: 10000 open/close in 125.28 seconds: 79.82 ops/second - unlinked 0 (time 1709055539 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1476 (time 1709055611.95 total 10.00 last 147.53) - open/close 2837 (time 1709055621.96 total 20.02 last 135.94) - open/close 3286 (time 1709055637.27 total 35.32 last 29.34) - open/close 4297 (time 1709055647.27 total 45.32 last 101.05) - open/close 5208 (time 1709055657.28 total 55.33 last 91.00) - open/close 6346 (time 1709055667.28 total 65.34 last 113.78) - open/close 7282 (time 1709055677.29 total 75.35 last 93.50) - open/close 8116 (time 1709055687.30 total 85.35 last 83.37) - open/close 8780 (time 1709055697.30 total 95.35 last 66.39) - open/close 9554 (time 1709055707.30 total 105.35 last 77.39) total: 10000 open/close in 109.22 seconds: 91.56 ops/second - unlinked 0 (time 1709055712 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1652 (time 1709055778.22 total 10.01 last 165.06) - open/close 2874 (time 1709055788.22 total 20.01 last 122.14) - open/close 3897 (time 1709055798.22 total 30.02 last 102.29) - open/close 4869 (time 1709055808.23 total 40.02 last 97.14) - open/close 5863 (time 1709055818.24 total 50.03 last 99.34) - open/close 6695 (time 1709055828.24 total 60.03 last 83.20) - open/close 7528 (time 1709055838.27 total 70.06 last 83.02) - open/close 8283 (time 1709055848.27 total 80.07 last 75.46) - open/close 8416 (time 1709055860.80 total 92.59 last 10.62) - open/close 9202 (time 1709055870.80 total 102.59 last 78.58) - open/close 9923 (time 1709055880.81 total 112.60 last 72.01) total: 10000 open/close in 113.73 seconds: 87.93 ops/second - unlinked 0 (time 1709055883 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1310 (time 1709055974.73 total 10.00 last 130.98) - open/close 2451 (time 1709055984.73 total 20.01 last 114.03) - open/close 3510 (time 1709055994.74 total 30.01 last 105.86) - open/close 4576 (time 1709056004.74 total 40.02 last 106.52) - open/close 5546 (time 1709056014.76 total 50.03 last 96.90) - open/close 6492 (time 1709056024.76 total 60.03 last 94.59) - open/close 7354 (time 1709056034.76 total 70.04 last 86.15) - open/close 8313 (time 1709056044.77 total 80.04 last 95.83) - open/close 9098 (time 1709056054.77 total 90.05 last 78.47) - open/close 9890 (time 1709056064.78 total 100.05 last 79.16) total: 10000 open/close in 101.67 seconds: 98.36 ops/second - unlinked 0 (time 1709056067 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg143-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 PASS 69 (2163s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg143-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg143-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg143-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 oleg143-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg143-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 12:49:46 (1709056186) oleg143-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity oleg143-server: debugfs 1.46.2.wc5 (26-Mar-2022) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity umount lustre on /mnt/lustre..... stop ost1 service on oleg143-server stop mds service on oleg143-server stop mds service on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O large_dir -i 1048576 -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -i 1048576 -b 4096 -J size=93 -I 1024 -q -O large_dir,uninit_bg,^extents,dirdata,dir_nlink,quota,project,huge_file,ea_inode,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init,packed_meta_blocks -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg143-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -b 4096 -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O large_dir,uninit_bg,extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=131072 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre Starting client oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre Started clients oleg143-client.virtnet: 192.168.201.143@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) mount lustre on /mnt/lustre2..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre2 Starting client oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre2 Started clients oleg143-client.virtnet: 192.168.201.143@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2280828 1696 2159132 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1388 126828 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1388 126828 2% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 2280 272 2008 12% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 302 49698 1% /mnt/lustre[OST:0] filesystem_summary: 2280 272 2008 12% /mnt/lustre creating 60000 hardlinks to oleg143-client.virtnet-0 creating 60000 hardlinks to oleg143-client.virtnet-1 waiting for PIDs 27894 27908 to complete - link 885 (time 1709056238.95 total 10.01 last 88.40) - link 1955 (time 1709056248.95 total 20.02 last 106.94) - link 2901 (time 1709056258.96 total 30.02 last 94.56) - link 3788 (time 1709056268.96 total 40.03 last 88.63) - link 4638 (time 1709056278.97 total 50.04 last 84.92) - link 5607 (time 1709056288.98 total 60.04 last 96.85) - link 6511 (time 1709056298.99 total 70.05 last 90.31) - link 7537 (time 1709056308.99 total 80.06 last 102.56) - link 8492 (time 1709056318.99 total 90.06 last 95.50) - link 9436 (time 1709056328.99 total 100.06 last 94.39) - link 10000 (time 1709056334.45 total 105.51 last 103.44) - link 10938 (time 1709056344.45 total 115.52 last 93.75) - link 11816 (time 1709056354.46 total 125.52 last 87.73) - link 12745 (time 1709056364.46 total 135.53 last 92.87) - link 13683 (time 1709056374.47 total 145.53 last 93.76) - link 14699 (time 1709056384.47 total 155.54 last 101.54) - link 15909 (time 1709056394.47 total 165.54 last 120.99) - link 16833 (time 1709056404.47 total 175.54 last 92.39) - link 17776 (time 1709056414.48 total 185.54 last 94.28) - link 18871 (time 1709056424.48 total 195.54 last 109.50) - link 19948 (time 1709056434.48 total 205.54 last 107.67) - link 20915 (time 1709056444.49 total 215.56 last 96.57) - link 21920 (time 1709056454.50 total 225.56 last 100.43) - link 22820 (time 1709056464.50 total 235.57 last 89.95) - link 23748 (time 1709056474.51 total 245.58 last 92.73) - link 24586 (time 1709056484.51 total 255.58 last 83.79) - link 25718 (time 1709056494.52 total 265.58 last 113.14) - link 26805 (time 1709056504.53 total 275.60 last 108.54) - link 27762 (time 1709056514.54 total 285.61 last 95.63) - link 28818 (time 1709056524.55 total 295.62 last 105.49) - link 29710 (time 1709056534.55 total 305.62 last 89.17) - link 30000 (time 1709056537.65 total 308.71 last 93.74) - link 30871 (time 1709056547.65 total 318.71 last 87.09) - link 31903 (time 1709056557.66 total 328.73 last 103.07) - link 32804 (time 1709056567.66 total 338.73 last 90.07) - link 33767 (time 1709056577.67 total 348.74 last 96.21) - link 34704 (time 1709056587.68 total 358.74 last 93.66) - link 35631 (time 1709056597.68 total 368.75 last 92.65) - link 36611 (time 1709056607.69 total 378.75 last 97.98) - link 37441 (time 1709056617.70 total 388.76 last 82.90) - link 38288 (time 1709056627.70 total 398.77 last 84.65) - link 39173 (time 1709056637.71 total 408.77 last 88.48) - link 40000 (time 1709056647.12 total 418.19 last 87.81) - link 40998 (time 1709056657.14 total 428.20 last 99.69) - link 42002 (time 1709056667.17 total 438.23 last 100.07) - link 42785 (time 1709056677.17 total 448.24 last 78.25) - link 43596 (time 1709056687.23 total 458.29 last 80.69) - link 44432 (time 1709056697.23 total 468.30 last 83.56) - link 45215 (time 1709056707.26 total 478.32 last 78.10) - link 45979 (time 1709056717.26 total 488.33 last 76.37) - link 46942 (time 1709056727.26 total 498.33 last 96.26) - link 47758 (time 1709056737.27 total 508.33 last 81.57) - link 48665 (time 1709056747.28 total 518.35 last 90.59) - link 49538 (time 1709056757.29 total 528.36 last 87.19) - link 50000 (time 1709056762.55 total 533.62 last 87.85) - link 50974 (time 1709056772.56 total 543.62 last 97.35) - link 51817 (time 1709056782.56 total 553.62 last 84.28) - link 52579 (time 1709056792.57 total 563.63 last 76.14) - link 53443 (time 1709056802.57 total 573.64 last 86.36) - link 54371 (time 1709056812.60 total 583.67 last 92.51) - link 55256 (time 1709056822.61 total 593.68 last 88.42) - link 56165 (time 1709056832.62 total 603.68 last 90.86) - link 57018 (time 1709056842.63 total 613.69 last 85.22) - link 58009 (time 1709056852.63 total 623.69 last 99.07) - link 58884 (time 1709056862.63 total 633.70 last 87.48) - link 59666 (time 1709056872.64 total 643.70 last 78.15) total: 60000 link in 648.81 seconds: 92.48 ops/second - link 889 (time 1709056239.10 total 10.00 last 88.86) - link 1975 (time 1709056249.10 total 20.01 last 108.58) - link 2919 (time 1709056259.11 total 30.02 last 94.30) - link 3813 (time 1709056269.11 total 40.02 last 89.36) - link 4661 (time 1709056279.12 total 50.03 last 84.72) - link 5632 (time 1709056289.13 total 60.04 last 97.03) - link 6540 (time 1709056299.13 total 70.04 last 90.78) - link 7585 (time 1709056309.13 total 80.04 last 104.48) - link 8543 (time 1709056319.14 total 90.05 last 95.76) - link 9491 (time 1709056329.15 total 100.06 last 94.69) - link 10000 (time 1709056334.02 total 104.93 last 104.43) - link 10938 (time 1709056344.03 total 114.94 last 93.75) - link 11815 (time 1709056354.03 total 124.94 last 87.69) - link 12730 (time 1709056364.03 total 134.94 last 91.47) - link 13677 (time 1709056374.04 total 144.95 last 94.61) - link 14681 (time 1709056384.05 total 154.96 last 100.33) - link 15891 (time 1709056394.05 total 164.96 last 120.99) - link 16823 (time 1709056404.06 total 174.97 last 93.12) - link 17752 (time 1709056414.07 total 184.97 last 92.83) - link 18847 (time 1709056424.07 total 194.98 last 109.42) - link 19932 (time 1709056434.08 total 204.99 last 108.41) - link 20900 (time 1709056444.09 total 215.00 last 96.75) - link 21897 (time 1709056454.09 total 225.00 last 99.65) - link 22796 (time 1709056464.11 total 235.01 last 89.78) - link 23724 (time 1709056474.11 total 245.02 last 92.71) - link 24556 (time 1709056484.12 total 255.03 last 83.17) - link 25673 (time 1709056494.12 total 265.03 last 111.69) - link 26782 (time 1709056504.13 total 275.04 last 110.80) - link 27716 (time 1709056514.14 total 285.05 last 93.31) - link 28765 (time 1709056524.14 total 295.05 last 104.87) - link 29660 (time 1709056534.15 total 305.05 last 89.46) - link 30000 (time 1709056537.77 total 308.68 last 93.86) - link 30869 (time 1709056547.77 total 318.68 last 86.90) - link 31899 (time 1709056557.78 total 328.69 last 102.92) - link 32805 (time 1709056567.79 total 338.70 last 90.50) - link 33765 (time 1709056577.79 total 348.70 last 95.99) - link 34708 (time 1709056587.80 total 358.71 last 94.15) - link 35643 (time 1709056597.81 total 368.72 last 93.44) - link 36612 (time 1709056607.81 total 378.72 last 96.89) - link 37445 (time 1709056617.82 total 388.73 last 83.24) - link 38292 (time 1709056627.82 total 398.73 last 84.68) - link 39164 (time 1709056637.84 total 408.75 last 87.05) - link 40000 (time 1709056647.35 total 418.26 last 87.91) - link 40995 (time 1709056657.35 total 428.26 last 99.48) - link 41992 (time 1709056667.37 total 438.28 last 99.54) - link 42773 (time 1709056677.37 total 448.28 last 78.08) - link 43573 (time 1709056687.38 total 458.29 last 79.93) - link 44413 (time 1709056697.38 total 468.29 last 83.95) - link 45184 (time 1709056707.40 total 478.31 last 76.96) - link 45960 (time 1709056717.41 total 488.32 last 77.53) - link 46919 (time 1709056727.42 total 498.32 last 95.88) - link 47731 (time 1709056737.42 total 508.33 last 81.18) - link 48648 (time 1709056747.42 total 518.33 last 91.70) - link 49502 (time 1709056757.42 total 528.33 last 85.39) - link 50000 (time 1709056763.04 total 533.95 last 88.55) - link 50966 (time 1709056773.05 total 543.96 last 96.50) - link 51803 (time 1709056783.06 total 553.97 last 83.62) - link 52582 (time 1709056793.07 total 563.98 last 77.84) - link 53442 (time 1709056803.07 total 573.98 last 85.99) - link 54370 (time 1709056813.07 total 583.98 last 92.79) - link 55254 (time 1709056823.08 total 593.99 last 88.37) - link 56150 (time 1709056833.08 total 603.99 last 89.54) - link 57016 (time 1709056843.08 total 613.99 last 86.59) - link 57998 (time 1709056853.10 total 624.01 last 98.08) - link 58867 (time 1709056863.10 total 634.01 last 86.87) - link 59634 (time 1709056873.11 total 644.01 last 76.66) total: 60000 link in 649.20 seconds: 92.42 ops/second estimate 27292s left after 120000 files / 650s umount lustre on /mnt/lustre2..... Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. ETA 27292s after 120000 files / 650s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg143-server: e2fsck 1.46.2.wc5 (26-Mar-2022) oleg143-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 19) [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2564: increase inode 85 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 86 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 87 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 88 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 89 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 90 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 91 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 92 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 93 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 94 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 95 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 96 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 97 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 98 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 99 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 100 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 101 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 102 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 103 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 104 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 105 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 106 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 107 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 108 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 109 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 110 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 111 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 112 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 113 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 114 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 115 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 119 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 120 badness 0 to 2 for 10084 [Thread 0] group 1 finished [Thread 0] e2fsck_pass1_run:2564: increase inode 132 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 133 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 134 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 135 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 137 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 138 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 139 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 140 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 141 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 142 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 143 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 144 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 145 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 146 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 147 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 148 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 149 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 150 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 151 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 153 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 154 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 155 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 156 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 157 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 158 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 159 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 160 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 161 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 162 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 163 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 164 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 165 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 166 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 167 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 168 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 169 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 170 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 171 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 172 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 173 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 174 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 175 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 176 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 177 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 178 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 179 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 180 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 181 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 182 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 183 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 184 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 185 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2564: increase inode 191 badness 0 to 2 for 10084 [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] Pass 1: Memory used: 380k/908k (124k/257k), time: 0.02/ 0.01/ 0.00 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 119.18MB/s [Thread 0] Scanned group range [0, 19), inodes 281 Pass 2: Checking directory structure Pass 2: Memory used: 380k/0k (85k/296k), time: 0.39/ 0.31/ 0.06 Pass 2: I/O read: 49MB, write: 0MB, rate: 126.89MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 380k/0k (85k/296k), time: 0.42/ 0.32/ 0.07 oleg143-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51601408, 266) != expected (51609600, 266) oleg143-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51601408, 266) != expected (51609600, 266) oleg143-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (51601408, 266) != expected (51609600, 266) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 4 Pass 3: Memory used: 380k/0k (83k/298k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 380k/0k (69k/312k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 5952.38MB/s Pass 5: Checking group summary information Pass 5: Memory used: 380k/0k (68k/313k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 768.64MB/s Update quota info for quota type 0? no Update quota info for quota type 1? no Update quota info for quota type 2? no lustre-MDT0000: ********** WARNING: Filesystem still has errors ********** 276 inodes used (12.11%, out of 2280) 3 non-contiguous files (1.1%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 37936 blocks used (6.32%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 380k/0k (67k/314k), time: 0.42/ 0.32/ 0.07 I/O read: 49MB, write: 0MB, rate: 117.08MB/s pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 Stopping clients: oleg143-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg143-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 oleg143-server: oleg143-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (746s) debug_raw_pointers=0 debug_raw_pointers=0 Stopping clients: oleg143-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg143-client.virtnet /mnt/lustre2 (opts:-f) oleg143-server: oleg143-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3170 sec ========== 13:02:32 (1709056952) === conf-sanity: start cleanup 13:02:32 (1709056952) === === conf-sanity: finish cleanup 13:02:32 (1709056952) ===