== conf-sanity test 69: replace an OST with the same index ========================================================== 15:06:33 (1746126393) start mds service on oleg454-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: detected 4 online CPUs by sysfs oleg454-server: MODOPTS_LIBCFS= oleg454-server: Force libcfs to create 2 CPU partitions oleg454-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg454-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg454-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg454-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49536 inodes available. Want 99872. rc=0 - open/close 1067 (time 1746126460.74 total 10.01 last 106.55) - open/close 2183 (time 1746126470.75 total 20.02 last 111.53) - open/close 3218 (time 1746126480.75 total 30.02 last 103.49) - open/close 4325 (time 1746126490.75 total 40.03 last 110.65) - open/close 5394 (time 1746126500.76 total 50.04 last 106.78) - open/close 6551 (time 1746126510.78 total 60.05 last 115.53) - open/close 7744 (time 1746126520.79 total 70.06 last 119.16) - open/close 8817 (time 1746126530.79 total 80.06 last 107.28) - open/close 9867 (time 1746126540.79 total 90.07 last 104.97) total: 10000 open/close in 91.45 seconds: 109.35 ops/second - unlinked 0 (time 1746126544 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1099 (time 1746126612.97 total 10.01 last 109.75) - open/close 2157 (time 1746126622.98 total 20.02 last 105.71) - open/close 3195 (time 1746126632.99 total 30.04 last 103.65) - open/close 4310 (time 1746126643.00 total 40.04 last 111.46) - open/close 5512 (time 1746126653.00 total 50.04 last 120.17) - open/close 6776 (time 1746126663.00 total 60.05 last 126.35) - open/close 7930 (time 1746126673.01 total 70.05 last 115.39) - open/close 8997 (time 1746126683.01 total 80.05 last 106.70) total: 10000 open/close in 87.61 seconds: 114.15 ops/second - unlinked 0 (time 1746126692 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1462 (time 1746126744.00 total 10.00 last 146.13) - open/close 2853 (time 1746126754.00 total 20.01 last 139.05) - open/close 4290 (time 1746126764.00 total 30.01 last 143.70) - open/close 5974 (time 1746126774.01 total 40.02 last 168.29) - open/close 7865 (time 1746126784.01 total 50.02 last 189.06) - open/close 9635 (time 1746126794.02 total 60.02 last 176.91) total: 10000 open/close in 62.88 seconds: 159.02 ops/second - unlinked 0 (time 1746126798 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1375 (time 1746126845.34 total 10.01 last 137.43) - open/close 3054 (time 1746126855.34 total 20.01 last 167.89) - open/close 5052 (time 1746126865.35 total 30.01 last 199.74) - open/close 6837 (time 1746126875.35 total 40.01 last 178.43) - open/close 8249 (time 1746126885.35 total 50.01 last 141.17) - open/close 9680 (time 1746126895.36 total 60.02 last 143.00) total: 10000 open/close in 62.37 seconds: 160.33 ops/second - unlinked 0 (time 1746126899 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 2332 (time 1746126946.80 total 10.01 last 233.07) - open/close 4216 (time 1746126956.80 total 20.01 last 188.37) - open/close 5815 (time 1746126966.80 total 30.01 last 159.90) - open/close 7286 (time 1746126976.81 total 40.01 last 147.02) - open/close 8899 (time 1746126986.81 total 50.02 last 161.22) total: 10000 open/close in 57.08 seconds: 175.20 ops/second - unlinked 0 (time 1746126995 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1704 (time 1746127038.94 total 10.00 last 170.37) - open/close 3485 (time 1746127048.95 total 20.00 last 178.07) - open/close 5206 (time 1746127058.95 total 30.00 last 172.09) - open/close 7094 (time 1746127068.95 total 40.00 last 188.78) - open/close 9137 (time 1746127078.95 total 50.01 last 204.27) total: 10000 open/close in 54.17 seconds: 184.59 ops/second - unlinked 0 (time 1746127084 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1951 (time 1746127122.56 total 10.00 last 195.06) - open/close 3949 (time 1746127132.57 total 20.01 last 199.67) - open/close 5892 (time 1746127142.57 total 30.01 last 194.29) - open/close 7865 (time 1746127152.57 total 40.01 last 197.21) total: 10000 open/close in 49.73 seconds: 201.09 ops/second - unlinked 0 (time 1746127163 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1470 (time 1746127203.52 total 10.00 last 146.95) - open/close 3202 (time 1746127213.52 total 20.01 last 173.10) - open/close 4521 (time 1746127223.52 total 30.01 last 131.89) - open/close 6103 (time 1746127233.52 total 40.01 last 158.20) - open/close 7696 (time 1746127243.52 total 50.01 last 159.29) - open/close 9858 (time 1746127253.52 total 60.01 last 216.19) total: 10000 open/close in 60.64 seconds: 164.92 ops/second - unlinked 0 (time 1746127255 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1745 (time 1746127295.30 total 10.00 last 174.42) - open/close 3912 (time 1746127305.30 total 20.01 last 216.64) - open/close 5541 (time 1746127315.31 total 30.02 last 162.69) - open/close 6594 (time 1746127325.31 total 40.02 last 105.30) - open/close 8153 (time 1746127335.31 total 50.02 last 155.88) total: 10000 open/close in 59.98 seconds: 166.73 ops/second - unlinked 0 (time 1746127346 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1809 (time 1746127387.20 total 10.00 last 180.87) - open/close 3346 (time 1746127397.20 total 20.00 last 153.68) - open/close 4805 (time 1746127407.20 total 30.01 last 145.86) - open/close 6822 (time 1746127417.20 total 40.01 last 201.63) - open/close 8772 (time 1746127427.21 total 50.01 last 194.89) total: 10000 open/close in 58.49 seconds: 170.96 ops/second - unlinked 0 (time 1746127437 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.154@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg454-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg454-server: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg454-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg454-server: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg454-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre On OST0, 18869 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg454-server unloading modules via unload_modules_local on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2