== conf-sanity test 69: replace an OST with the same index ========================================================== 21:59:44 (1746064784) start mds service on oleg436-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions oleg436-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg436-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg436-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg436-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1885 (time 1746064840.58 total 10.00 last 188.48) - open/close 4073 (time 1746064850.58 total 20.01 last 218.62) - open/close 5402 (time 1746064860.58 total 30.01 last 132.90) - open/close 6943 (time 1746064870.59 total 40.02 last 154.00) - open/close 8485 (time 1746064880.60 total 50.02 last 154.12) total: 10000 open/close in 59.03 seconds: 169.42 ops/second - unlinked 0 (time 1746064890 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1812 (time 1746064935.12 total 10.01 last 181.03) - open/close 4159 (time 1746064945.12 total 20.01 last 234.67) - open/close 6000 (time 1746064955.12 total 30.01 last 184.09) - open/close 8151 (time 1746064965.13 total 40.01 last 215.04) total: 10000 open/close in 48.43 seconds: 206.49 ops/second - unlinked 0 (time 1746064974 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1465 (time 1746065021.85 total 10.00 last 146.48) - open/close 3212 (time 1746065031.85 total 20.00 last 174.68) - open/close 4961 (time 1746065041.86 total 30.01 last 174.84) - open/close 6721 (time 1746065051.86 total 40.01 last 175.95) - open/close 8533 (time 1746065061.87 total 50.02 last 181.08) total: 10000 open/close in 58.52 seconds: 170.90 ops/second - unlinked 0 (time 1746065071 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1600 (time 1746065119.15 total 10.00 last 159.98) - open/close 3424 (time 1746065129.16 total 20.00 last 182.34) - open/close 5265 (time 1746065139.16 total 30.01 last 184.09) - open/close 6822 (time 1746065149.16 total 40.01 last 155.66) - open/close 8375 (time 1746065159.16 total 50.01 last 155.28) - open/close 9864 (time 1746065169.17 total 60.01 last 148.82) total: 10000 open/close in 61.04 seconds: 163.84 ops/second - unlinked 0 (time 1746065171 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1808 (time 1746065223.57 total 10.01 last 180.69) - open/close 3321 (time 1746065233.57 total 20.01 last 151.24) - open/close 4967 (time 1746065243.58 total 30.01 last 164.54) - open/close 6914 (time 1746065253.58 total 40.02 last 194.65) - open/close 8918 (time 1746065263.58 total 50.02 last 200.32) total: 10000 open/close in 56.84 seconds: 175.94 ops/second - unlinked 0 (time 1746065271 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1887 (time 1746065317.07 total 10.01 last 188.58) - open/close 3613 (time 1746065327.07 total 20.01 last 172.52) - open/close 5657 (time 1746065337.08 total 30.01 last 204.37) - open/close 7731 (time 1746065347.08 total 40.01 last 207.36) - open/close 9706 (time 1746065357.08 total 50.02 last 197.45) total: 10000 open/close in 51.49 seconds: 194.21 ops/second - unlinked 0 (time 1746065359 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1509 (time 1746065401.19 total 10.00 last 150.90) - open/close 3392 (time 1746065411.20 total 20.01 last 188.20) - open/close 4877 (time 1746065421.20 total 30.01 last 148.48) - open/close 6385 (time 1746065431.20 total 40.01 last 150.78) - open/close 7620 (time 1746065441.21 total 50.02 last 123.38) - open/close 8944 (time 1746065451.21 total 60.02 last 132.37) total: 10000 open/close in 67.65 seconds: 147.81 ops/second - unlinked 0 (time 1746065460 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1577 (time 1746065515.53 total 10.00 last 157.69) - open/close 3346 (time 1746065525.53 total 20.01 last 176.80) - open/close 5006 (time 1746065535.54 total 30.01 last 165.89) - open/close 6587 (time 1746065545.54 total 40.02 last 158.05) - open/close 8474 (time 1746065555.54 total 50.02 last 188.69) total: 10000 open/close in 58.10 seconds: 172.11 ops/second - unlinked 0 (time 1746065565 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2326 (time 1746065598.26 total 10.00 last 232.58) - open/close 4370 (time 1746065608.27 total 20.00 last 204.35) - open/close 6380 (time 1746065618.27 total 30.01 last 200.90) - open/close 8297 (time 1746065628.27 total 40.01 last 191.70) total: 10000 open/close in 46.54 seconds: 214.87 ops/second - unlinked 0 (time 1746065635 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 1678 (time 1746065669.83 total 10.00 last 167.78) - open/close 3161 (time 1746065679.83 total 20.01 last 148.22) - open/close 4550 (time 1746065689.84 total 30.02 last 138.79) - open/close 5961 (time 1746065699.86 total 40.03 last 140.82) - open/close 7084 (time 1746065709.86 total 50.04 last 112.30) - open/close 8295 (time 1746065719.86 total 60.04 last 121.08) - open/close 9509 (time 1746065729.87 total 70.04 last 121.38) total: 10000 open/close in 73.90 seconds: 135.32 ops/second - unlinked 0 (time 1746065735 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.136@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg436-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg436-server: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg436-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec oleg436-server: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg436-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg436-server unloading modules via unload_modules_local on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2