== conf-sanity test 69: replace an OST with the same index ========================================================== 22:23:25 (1746066205) start mds service on oleg203-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg203-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1491 (time 1746066260.93 total 10.00 last 149.08) - open/close 2768 (time 1746066270.94 total 20.01 last 127.63) - open/close 4101 (time 1746066280.95 total 30.01 last 133.20) - open/close 5373 (time 1746066290.95 total 40.02 last 127.18) - open/close 6770 (time 1746066300.96 total 50.02 last 139.58) - open/close 8409 (time 1746066310.96 total 60.02 last 163.89) - open/close 9870 (time 1746066320.96 total 70.03 last 146.07) total: 10000 open/close in 70.87 seconds: 141.10 ops/second - unlinked 0 (time 1746066323 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1624 (time 1746066369.21 total 10.00 last 162.36) - open/close 3090 (time 1746066379.21 total 20.00 last 146.57) - open/close 4377 (time 1746066389.21 total 30.01 last 128.64) - open/close 5531 (time 1746066399.21 total 40.01 last 115.40) - open/close 6603 (time 1746066409.21 total 50.01 last 107.20) - open/close 7918 (time 1746066419.21 total 60.01 last 131.49) - open/close 9317 (time 1746066429.22 total 70.01 last 139.87) total: 10000 open/close in 74.74 seconds: 133.79 ops/second - unlinked 0 (time 1746066435 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1216 (time 1746066490.45 total 10.01 last 121.44) - open/close 2504 (time 1746066500.45 total 20.02 last 128.73) - open/close 3787 (time 1746066510.45 total 30.02 last 128.29) - open/close 5000 (time 1746066520.46 total 40.03 last 121.18) - open/close 6433 (time 1746066530.47 total 50.03 last 143.28) - open/close 7771 (time 1746066540.47 total 60.04 last 133.70) - open/close 8886 (time 1746066550.48 total 70.04 last 111.42) - open/close 9872 (time 1746066560.49 total 80.06 last 98.49) total: 10000 open/close in 81.52 seconds: 122.67 ops/second - unlinked 0 (time 1746066564 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1082 (time 1746066636.25 total 10.01 last 108.07) - open/close 2199 (time 1746066646.26 total 20.02 last 111.63) - open/close 3380 (time 1746066656.27 total 30.03 last 118.01) - open/close 4373 (time 1746066666.27 total 40.03 last 99.27) - open/close 5447 (time 1746066676.27 total 50.03 last 107.37) - open/close 6642 (time 1746066686.28 total 60.04 last 119.44) - open/close 8018 (time 1746066696.28 total 70.04 last 137.53) - open/close 9332 (time 1746066706.29 total 80.04 last 131.35) total: 10000 open/close in 84.97 seconds: 117.69 ops/second - unlinked 0 (time 1746066712 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1392 (time 1746066769.37 total 10.00 last 139.18) - open/close 2704 (time 1746066779.38 total 20.01 last 131.07) - open/close 4036 (time 1746066789.38 total 30.01 last 133.18) - open/close 5321 (time 1746066799.39 total 40.02 last 128.46) - open/close 6555 (time 1746066809.39 total 50.03 last 123.28) - open/close 7983 (time 1746066819.40 total 60.03 last 142.75) - open/close 9217 (time 1746066829.41 total 70.04 last 123.30) total: 10000 open/close in 76.30 seconds: 131.07 ops/second - unlinked 0 (time 1746066837 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1142 (time 1746066895.78 total 10.01 last 114.11) - open/close 2385 (time 1746066905.79 total 20.01 last 124.26) - open/close 3719 (time 1746066915.79 total 30.01 last 133.40) - open/close 5485 (time 1746066925.79 total 40.01 last 176.55) - open/close 6861 (time 1746066935.79 total 50.02 last 137.55) - open/close 8271 (time 1746066945.80 total 60.03 last 140.87) - open/close 9594 (time 1746066955.81 total 70.03 last 132.25) total: 10000 open/close in 73.06 seconds: 136.88 ops/second - unlinked 0 (time 1746066960 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1484 (time 1746067020.91 total 10.00 last 148.34) - open/close 2757 (time 1746067030.92 total 20.01 last 127.17) - open/close 4194 (time 1746067040.92 total 30.02 last 143.67) - open/close 5526 (time 1746067050.93 total 40.03 last 133.01) - open/close 6889 (time 1746067060.93 total 50.03 last 136.27) - open/close 8247 (time 1746067070.94 total 60.04 last 135.69) - open/close 9695 (time 1746067080.95 total 70.04 last 144.75) total: 10000 open/close in 71.83 seconds: 139.22 ops/second - unlinked 0 (time 1746067084 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1479 (time 1746067126.23 total 10.00 last 147.84) - open/close 3136 (time 1746067136.23 total 20.01 last 165.58) - open/close 5266 (time 1746067146.23 total 30.01 last 212.98) - open/close 6899 (time 1746067156.24 total 40.02 last 163.24) - open/close 8110 (time 1746067166.24 total 50.02 last 121.03) - open/close 9337 (time 1746067176.25 total 60.02 last 122.69) total: 10000 open/close in 64.79 seconds: 154.35 ops/second - unlinked 0 (time 1746067182 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1452 (time 1746067236.04 total 10.00 last 145.17) - open/close 2867 (time 1746067246.05 total 20.01 last 141.37) - open/close 4369 (time 1746067256.06 total 30.02 last 150.12) - open/close 5976 (time 1746067266.06 total 40.02 last 160.62) - open/close 7277 (time 1746067276.07 total 50.03 last 130.05) - open/close 8563 (time 1746067286.07 total 60.03 last 128.53) total: 10000 open/close in 69.72 seconds: 143.43 ops/second - unlinked 0 (time 1746067297 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1895 (time 1746067347.52 total 10.00 last 189.45) - open/close 3834 (time 1746067357.52 total 20.00 last 193.88) - open/close 5689 (time 1746067367.52 total 30.01 last 185.46) - open/close 7051 (time 1746067377.53 total 40.01 last 136.19) - open/close 8413 (time 1746067387.53 total 50.02 last 136.07) - open/close 9818 (time 1746067397.54 total 60.02 last 140.46) total: 10000 open/close in 61.29 seconds: 163.15 ops/second - unlinked 0 (time 1746067400 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg203-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre On OST0, 15261 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2