== conf-sanity test 69: replace an OST with the same index ========================================================== 15:49:07 (1717012147) start mds service on oleg134-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg134-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1649 (time 1717012202.99 total 10.01 last 164.82) - open/close 3127 (time 1717012213.00 total 20.01 last 147.70) - open/close 4671 (time 1717012223.00 total 30.02 last 154.35) - open/close 6128 (time 1717012233.01 total 40.02 last 145.67) - open/close 7539 (time 1717012243.01 total 50.02 last 141.09) - open/close 9078 (time 1717012253.01 total 60.02 last 153.83) total: 10000 open/close in 66.23 seconds: 150.99 ops/second - unlinked 0 (time 1717012260 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1767 (time 1717012306.39 total 10.00 last 176.63) - open/close 3592 (time 1717012316.39 total 20.00 last 182.49) - open/close 5143 (time 1717012326.39 total 30.01 last 155.06) - open/close 6907 (time 1717012336.40 total 40.01 last 176.35) - open/close 8397 (time 1717012346.40 total 50.01 last 149.00) - open/close 9826 (time 1717012356.40 total 60.01 last 142.87) total: 10000 open/close in 61.26 seconds: 163.24 ops/second - unlinked 0 (time 1717012359 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1465 (time 1717012407.19 total 10.00 last 146.47) - open/close 2990 (time 1717012417.20 total 20.01 last 152.42) - open/close 4960 (time 1717012427.20 total 30.01 last 196.97) - open/close 6430 (time 1717012437.20 total 40.01 last 146.97) - open/close 8090 (time 1717012447.21 total 50.01 last 165.95) - open/close 9707 (time 1717012457.21 total 60.02 last 161.64) total: 10000 open/close in 61.94 seconds: 161.46 ops/second - unlinked 0 (time 1717012460 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1523 (time 1717012507.01 total 10.01 last 152.18) - open/close 3050 (time 1717012517.01 total 20.01 last 152.69) - open/close 4515 (time 1717012527.01 total 30.01 last 146.48) - open/close 5973 (time 1717012537.02 total 40.02 last 145.69) - open/close 7358 (time 1717012547.03 total 50.02 last 138.44) - open/close 8740 (time 1717012557.03 total 60.02 last 138.20) - open/close 9990 (time 1717012567.04 total 70.03 last 124.86) total: 10000 open/close in 70.11 seconds: 142.64 ops/second - unlinked 0 (time 1717012569 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1346 (time 1717012619.70 total 10.01 last 134.42) - open/close 2695 (time 1717012629.70 total 20.02 last 134.86) - open/close 3971 (time 1717012639.71 total 30.02 last 127.59) - open/close 5169 (time 1717012649.71 total 40.02 last 119.73) - open/close 6499 (time 1717012659.71 total 50.03 last 132.96) - open/close 7814 (time 1717012669.72 total 60.04 last 131.37) - open/close 8991 (time 1717012679.73 total 70.04 last 117.66) total: 10000 open/close in 77.56 seconds: 128.93 ops/second - unlinked 0 (time 1717012689 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1401 (time 1717012740.07 total 10.00 last 140.08) - open/close 2923 (time 1717012750.07 total 20.00 last 152.16) - open/close 4371 (time 1717012760.07 total 30.01 last 144.78) - open/close 5898 (time 1717012770.07 total 40.01 last 152.67) - open/close 7421 (time 1717012780.07 total 50.01 last 152.28) - open/close 8836 (time 1717012790.08 total 60.02 last 141.36) total: 10000 open/close in 68.19 seconds: 146.65 ops/second - unlinked 0 (time 1717012799 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1366 (time 1717012851.13 total 10.00 last 136.57) - open/close 2719 (time 1717012861.13 total 20.01 last 135.20) - open/close 4083 (time 1717012871.14 total 30.01 last 136.33) - open/close 5536 (time 1717012881.14 total 40.01 last 145.30) - open/close 6901 (time 1717012891.15 total 50.02 last 136.37) - open/close 8224 (time 1717012901.15 total 60.02 last 132.29) - open/close 9643 (time 1717012911.16 total 70.03 last 141.78) total: 10000 open/close in 72.42 seconds: 138.08 ops/second - unlinked 0 (time 1717012915 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1433 (time 1717012965.05 total 10.00 last 143.29) - open/close 2858 (time 1717012975.06 total 20.00 last 142.45) - open/close 4323 (time 1717012985.06 total 30.01 last 146.45) - open/close 5881 (time 1717012995.06 total 40.01 last 155.77) - open/close 7385 (time 1717013005.07 total 50.02 last 150.29) - open/close 8829 (time 1717013015.07 total 60.02 last 144.35) total: 10000 open/close in 68.75 seconds: 145.46 ops/second - unlinked 0 (time 1717013025 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1424 (time 1717013077.54 total 10.01 last 142.29) - open/close 2992 (time 1717013087.55 total 20.01 last 156.71) - open/close 4450 (time 1717013097.55 total 30.02 last 145.76) - open/close 6005 (time 1717013107.55 total 40.02 last 155.46) - open/close 7681 (time 1717013117.56 total 50.02 last 167.58) - open/close 9296 (time 1717013127.56 total 60.02 last 161.43) total: 10000 open/close in 64.19 seconds: 155.79 ops/second - unlinked 0 (time 1717013132 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1604 (time 1717013176.42 total 10.00 last 160.34) - open/close 3200 (time 1717013186.42 total 20.00 last 159.59) - open/close 4891 (time 1717013196.42 total 30.01 last 169.01) - open/close 6448 (time 1717013206.43 total 40.02 last 155.58) - open/close 8148 (time 1717013216.43 total 50.02 last 169.97) - open/close 9961 (time 1717013226.44 total 60.02 last 181.25) total: 10000 open/close in 60.34 seconds: 165.71 ops/second - unlinked 0 (time 1717013228 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg134-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2