== conf-sanity test 69: replace an OST with the same index ========================================================== 02:33:20 (1727764400) start mds service on oleg113-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg113-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1542 (time 1727764467.11 total 10.01 last 154.12) - open/close 2799 (time 1727764477.12 total 20.01 last 125.58) - open/close 4218 (time 1727764487.12 total 30.02 last 141.87) - open/close 5570 (time 1727764497.13 total 40.02 last 135.12) - open/close 6846 (time 1727764507.14 total 50.03 last 127.52) - open/close 8347 (time 1727764517.14 total 60.03 last 150.03) - open/close 9752 (time 1727764527.15 total 70.04 last 140.42) total: 10000 open/close in 71.96 seconds: 138.96 ops/second - unlinked 0 (time 1727764530 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1395 (time 1727764582.49 total 10.00 last 139.47) - open/close 2722 (time 1727764592.50 total 20.01 last 132.61) - open/close 4095 (time 1727764602.50 total 30.01 last 137.27) - open/close 5379 (time 1727764612.50 total 40.02 last 128.33) - open/close 6618 (time 1727764622.50 total 50.02 last 123.88) - open/close 7921 (time 1727764632.51 total 60.02 last 130.23) - open/close 9251 (time 1727764642.51 total 70.03 last 132.96) total: 10000 open/close in 76.24 seconds: 131.17 ops/second - unlinked 0 (time 1727764650 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1398 (time 1727764704.71 total 10.00 last 139.77) - open/close 2556 (time 1727764714.72 total 20.01 last 115.74) - open/close 3820 (time 1727764724.72 total 30.01 last 126.37) - open/close 5160 (time 1727764734.72 total 40.01 last 133.93) - open/close 6598 (time 1727764744.73 total 50.02 last 143.79) - open/close 7818 (time 1727764754.74 total 60.03 last 121.87) - open/close 9116 (time 1727764764.74 total 70.03 last 129.71) total: 10000 open/close in 76.45 seconds: 130.81 ops/second - unlinked 0 (time 1727764772 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1280 (time 1727764827.18 total 10.01 last 127.92) - open/close 2626 (time 1727764837.18 total 20.01 last 134.56) - open/close 4073 (time 1727764847.19 total 30.01 last 144.62) - open/close 5336 (time 1727764857.19 total 40.02 last 126.27) - open/close 6727 (time 1727764867.19 total 50.02 last 139.08) - open/close 8146 (time 1727764877.19 total 60.02 last 141.88) - open/close 9353 (time 1727764887.20 total 70.03 last 120.56) total: 10000 open/close in 74.81 seconds: 133.67 ops/second - unlinked 0 (time 1727764894 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1425 (time 1727764948.16 total 10.01 last 142.43) - open/close 2709 (time 1727764958.16 total 20.01 last 128.35) - open/close 4057 (time 1727764968.17 total 30.02 last 134.71) - open/close 5541 (time 1727764978.17 total 40.02 last 148.36) - open/close 6854 (time 1727764988.17 total 50.02 last 131.26) - open/close 8249 (time 1727764998.18 total 60.02 last 139.47) - open/close 9766 (time 1727765008.18 total 70.02 last 151.68) total: 10000 open/close in 71.62 seconds: 139.62 ops/second - unlinked 0 (time 1727765011 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1363 (time 1727765066.38 total 10.01 last 136.23) - open/close 2806 (time 1727765076.38 total 20.01 last 144.25) - open/close 4120 (time 1727765086.38 total 30.01 last 131.38) - open/close 5453 (time 1727765096.38 total 40.01 last 133.30) - open/close 6751 (time 1727765106.38 total 50.01 last 129.80) - open/close 8032 (time 1727765116.39 total 60.02 last 128.02) - open/close 9367 (time 1727765126.39 total 70.02 last 133.45) total: 10000 open/close in 74.36 seconds: 134.48 ops/second - unlinked 0 (time 1727765132 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1450 (time 1727765184.40 total 10.00 last 144.94) - open/close 3006 (time 1727765194.41 total 20.01 last 155.55) - open/close 4287 (time 1727765204.41 total 30.01 last 128.10) - open/close 5525 (time 1727765214.41 total 40.01 last 123.73) - open/close 6936 (time 1727765224.41 total 50.02 last 141.08) - open/close 8214 (time 1727765234.42 total 60.02 last 127.73) - open/close 9460 (time 1727765244.42 total 70.02 last 124.56) total: 10000 open/close in 74.19 seconds: 134.78 ops/second - unlinked 0 (time 1727765250 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1135 (time 1727765307.06 total 10.00 last 113.45) - open/close 2299 (time 1727765317.07 total 20.02 last 116.24) - open/close 3796 (time 1727765327.07 total 30.02 last 149.65) - open/close 5524 (time 1727765337.08 total 40.03 last 172.66) - open/close 6896 (time 1727765347.08 total 50.03 last 137.19) - open/close 8411 (time 1727765357.08 total 60.03 last 151.49) - open/close 9689 (time 1727765367.09 total 70.03 last 127.76) total: 10000 open/close in 72.10 seconds: 138.69 ops/second - unlinked 0 (time 1727765370 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1451 (time 1727765412.68 total 10.00 last 145.04) - open/close 2973 (time 1727765422.68 total 20.00 last 152.20) - open/close 4606 (time 1727765432.68 total 30.01 last 163.24) - open/close 6226 (time 1727765442.70 total 40.02 last 161.77) - open/close 7725 (time 1727765452.71 total 50.03 last 149.71) - open/close 9130 (time 1727765462.71 total 60.04 last 140.44) total: 10000 open/close in 65.58 seconds: 152.49 ops/second - unlinked 0 (time 1727765469 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1496 (time 1727765519.28 total 10.00 last 149.57) - open/close 2933 (time 1727765529.28 total 20.00 last 143.67) - open/close 4521 (time 1727765539.29 total 30.01 last 158.76) - open/close 6190 (time 1727765549.29 total 40.01 last 166.84) - open/close 7750 (time 1727765559.29 total 50.01 last 155.96) - open/close 9252 (time 1727765569.30 total 60.02 last 150.07) total: 10000 open/close in 65.11 seconds: 153.59 ops/second - unlinked 0 (time 1727765575 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre On OST0, 14589 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server unloading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2