== conf-sanity test 69: replace an OST with the same index ========================================================== 16:57:40 (1746133060) start mds service on oleg323-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg323-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1122 (time 1746133116.66 total 10.01 last 112.14) - open/close 2410 (time 1746133126.66 total 20.01 last 128.79) - open/close 3751 (time 1746133136.66 total 30.01 last 134.05) - open/close 4861 (time 1746133146.66 total 40.01 last 110.99) - open/close 6041 (time 1746133156.67 total 50.02 last 117.89) - open/close 7226 (time 1746133166.68 total 60.02 last 118.48) - open/close 8365 (time 1746133176.68 total 70.02 last 113.87) - open/close 9611 (time 1746133186.68 total 80.03 last 124.58) total: 10000 open/close in 83.69 seconds: 119.48 ops/second - unlinked 0 (time 1746133192 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1267 (time 1746133248.78 total 10.00 last 126.70) - open/close 2602 (time 1746133258.78 total 20.00 last 133.46) - open/close 3950 (time 1746133268.78 total 30.01 last 134.72) - open/close 5157 (time 1746133278.79 total 40.01 last 120.68) - open/close 6323 (time 1746133288.79 total 50.01 last 116.59) - open/close 7361 (time 1746133298.80 total 60.02 last 103.71) - open/close 8473 (time 1746133308.80 total 70.02 last 111.18) - open/close 9632 (time 1746133318.80 total 80.03 last 115.85) total: 10000 open/close in 83.23 seconds: 120.15 ops/second - unlinked 0 (time 1746133323 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1663 (time 1746133383.66 total 10.00 last 166.27) - open/close 3124 (time 1746133393.67 total 20.01 last 146.02) - open/close 4336 (time 1746133403.67 total 30.01 last 121.19) - open/close 5744 (time 1746133413.67 total 40.01 last 140.80) - open/close 7156 (time 1746133423.68 total 50.02 last 141.07) - open/close 8480 (time 1746133433.68 total 60.02 last 132.38) - open/close 9756 (time 1746133443.68 total 70.02 last 127.57) total: 10000 open/close in 72.01 seconds: 138.87 ops/second - unlinked 0 (time 1746133447 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1196 (time 1746133511.79 total 10.00 last 119.54) - open/close 2544 (time 1746133521.79 total 20.01 last 134.79) - open/close 3828 (time 1746133531.79 total 30.01 last 128.36) - open/close 4881 (time 1746133541.79 total 40.01 last 105.26) - open/close 6318 (time 1746133551.80 total 50.01 last 143.69) - open/close 7760 (time 1746133561.80 total 60.02 last 144.16) - open/close 9014 (time 1746133571.80 total 70.02 last 125.34) total: 10000 open/close in 77.80 seconds: 128.53 ops/second - unlinked 0 (time 1746133581 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1318 (time 1746133633.86 total 10.00 last 131.78) - open/close 2502 (time 1746133643.87 total 20.01 last 118.30) - open/close 3709 (time 1746133653.87 total 30.01 last 120.68) - open/close 4984 (time 1746133663.87 total 40.01 last 127.49) - open/close 6408 (time 1746133673.87 total 50.01 last 142.39) - open/close 7659 (time 1746133683.87 total 60.02 last 125.04) - open/close 9024 (time 1746133693.88 total 70.02 last 136.44) total: 10000 open/close in 77.65 seconds: 128.79 ops/second - unlinked 0 (time 1746133703 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1238 (time 1746133761.08 total 10.00 last 123.76) - open/close 2697 (time 1746133771.08 total 20.01 last 145.86) - open/close 4041 (time 1746133781.09 total 30.01 last 134.31) - open/close 5240 (time 1746133791.10 total 40.02 last 119.80) - open/close 6666 (time 1746133801.10 total 50.03 last 142.53) - open/close 7773 (time 1746133811.12 total 60.04 last 110.53) - open/close 8883 (time 1746133821.13 total 70.05 last 110.87) total: 10000 open/close in 79.58 seconds: 125.65 ops/second - unlinked 0 (time 1746133832 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1218 (time 1746133893.47 total 10.01 last 121.66) - open/close 2467 (time 1746133903.48 total 20.02 last 124.82) - open/close 3759 (time 1746133913.48 total 30.02 last 129.17) - open/close 5043 (time 1746133923.49 total 40.02 last 128.34) - open/close 6356 (time 1746133933.49 total 50.03 last 131.21) - open/close 7650 (time 1746133943.50 total 60.03 last 129.35) - open/close 8750 (time 1746133953.51 total 70.05 last 109.86) - open/close 9766 (time 1746133963.52 total 80.06 last 101.49) total: 10000 open/close in 82.35 seconds: 121.43 ops/second - unlinked 0 (time 1746133967 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 991 (time 1746134039.08 total 10.01 last 99.03) - open/close 1996 (time 1746134049.08 total 20.01 last 100.49) - open/close 3030 (time 1746134059.08 total 30.01 last 103.36) - open/close 4056 (time 1746134069.09 total 40.02 last 102.47) - open/close 5101 (time 1746134079.09 total 50.02 last 104.50) - open/close 6055 (time 1746134089.09 total 60.03 last 95.39) - open/close 7114 (time 1746134099.10 total 70.03 last 105.85) - open/close 8136 (time 1746134109.11 total 80.04 last 102.10) - open/close 9158 (time 1746134119.12 total 90.05 last 102.09) total: 10000 open/close in 97.80 seconds: 102.25 ops/second - unlinked 0 (time 1746134129 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1189 (time 1746134195.21 total 10.00 last 118.86) - open/close 2399 (time 1746134205.21 total 20.01 last 120.96) - open/close 3571 (time 1746134215.22 total 30.01 last 117.14) - open/close 4723 (time 1746134225.23 total 40.02 last 115.07) - open/close 5752 (time 1746134235.23 total 50.02 last 102.89) - open/close 6973 (time 1746134245.23 total 60.02 last 122.09) - open/close 8204 (time 1746134255.24 total 70.03 last 123.03) - open/close 9314 (time 1746134265.24 total 80.04 last 110.93) total: 10000 open/close in 85.32 seconds: 117.21 ops/second - unlinked 0 (time 1746134272 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1173 (time 1746134330.63 total 10.02 last 117.12) - open/close 2449 (time 1746134340.63 total 20.02 last 127.55) - open/close 3760 (time 1746134350.63 total 30.02 last 131.09) - open/close 4920 (time 1746134360.64 total 40.02 last 115.96) - open/close 6053 (time 1746134370.64 total 50.03 last 113.26) - open/close 7265 (time 1746134380.64 total 60.03 last 121.18) - open/close 8433 (time 1746134390.65 total 70.03 last 116.76) - open/close 9605 (time 1746134400.65 total 80.03 last 117.18) total: 10000 open/close in 83.73 seconds: 119.42 ops/second - unlinked 0 (time 1746134406 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg323-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre On OST0, 17085 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2