== conf-sanity test 69: replace an OST with the same index ========================================================== 18:31:46 (1746138706) start mds service on oleg310-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg310-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg310-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1094 (time 1746138761.61 total 10.01 last 109.34) - open/close 2288 (time 1746138771.62 total 20.02 last 119.28) - open/close 3441 (time 1746138781.62 total 30.02 last 115.26) - open/close 4690 (time 1746138791.63 total 40.03 last 124.79) - open/close 5681 (time 1746138801.63 total 50.03 last 99.06) - open/close 6632 (time 1746138811.65 total 60.05 last 94.96) - open/close 7780 (time 1746138821.66 total 70.05 last 114.73) - open/close 8830 (time 1746138831.66 total 80.06 last 104.93) total: 10000 open/close in 89.73 seconds: 111.45 ops/second - unlinked 0 (time 1746138843 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1254 (time 1746138900.17 total 10.00 last 125.36) - open/close 2394 (time 1746138910.17 total 20.01 last 113.96) - open/close 3535 (time 1746138920.18 total 30.02 last 114.00) - open/close 4606 (time 1746138930.19 total 40.02 last 107.05) - open/close 5813 (time 1746138940.19 total 50.03 last 120.64) - open/close 6851 (time 1746138950.19 total 60.03 last 103.80) - open/close 7951 (time 1746138960.20 total 70.03 last 109.97) - open/close 8912 (time 1746138970.20 total 80.04 last 96.03) total: 10000 open/close in 89.67 seconds: 111.52 ops/second - unlinked 0 (time 1746138981 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1150 (time 1746139045.07 total 10.00 last 114.97) - open/close 2336 (time 1746139055.08 total 20.01 last 118.51) - open/close 3478 (time 1746139065.08 total 30.01 last 114.20) - open/close 4609 (time 1746139075.08 total 40.02 last 113.04) - open/close 5784 (time 1746139085.08 total 50.02 last 117.50) - open/close 6915 (time 1746139095.08 total 60.02 last 113.07) - open/close 8045 (time 1746139105.09 total 70.02 last 112.95) - open/close 9174 (time 1746139115.09 total 80.03 last 112.87) total: 10000 open/close in 86.62 seconds: 115.45 ops/second - unlinked 0 (time 1746139123 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1309 (time 1746139180.01 total 10.01 last 130.82) - open/close 2508 (time 1746139190.03 total 20.03 last 119.66) - open/close 3801 (time 1746139200.03 total 30.03 last 129.27) - open/close 5031 (time 1746139210.03 total 40.03 last 122.98) - open/close 6257 (time 1746139220.04 total 50.03 last 122.55) - open/close 7426 (time 1746139230.04 total 60.04 last 116.83) - open/close 8683 (time 1746139240.05 total 70.04 last 125.65) - open/close 9878 (time 1746139250.06 total 80.06 last 119.37) total: 10000 open/close in 80.91 seconds: 123.60 ops/second - unlinked 0 (time 1746139252 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1262 (time 1746139308.22 total 10.00 last 126.16) - open/close 2460 (time 1746139318.22 total 20.00 last 119.79) - open/close 3658 (time 1746139328.22 total 30.00 last 119.79) - open/close 5045 (time 1746139338.22 total 40.01 last 138.62) - open/close 6597 (time 1746139348.23 total 50.01 last 155.15) - open/close 7920 (time 1746139358.24 total 60.02 last 132.19) - open/close 9196 (time 1746139368.24 total 70.03 last 127.55) total: 10000 open/close in 76.65 seconds: 130.47 ops/second - unlinked 0 (time 1746139376 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1281 (time 1746139427.08 total 10.00 last 128.06) - open/close 2717 (time 1746139437.09 total 20.01 last 143.55) - open/close 4209 (time 1746139447.09 total 30.01 last 149.17) - open/close 5731 (time 1746139457.09 total 40.01 last 152.18) - open/close 7125 (time 1746139467.09 total 50.01 last 139.38) - open/close 8341 (time 1746139477.10 total 60.02 last 121.52) - open/close 9855 (time 1746139487.10 total 70.02 last 151.40) total: 10000 open/close in 71.24 seconds: 140.37 ops/second - unlinked 0 (time 1746139489 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1475 (time 1746139538.35 total 10.00 last 147.50) - open/close 2953 (time 1746139548.35 total 20.00 last 147.77) - open/close 4516 (time 1746139558.35 total 30.00 last 156.27) - open/close 6307 (time 1746139568.36 total 40.01 last 179.04) - open/close 8130 (time 1746139578.36 total 50.02 last 182.15) - open/close 9568 (time 1746139588.37 total 60.02 last 143.76) total: 10000 open/close in 62.64 seconds: 159.63 ops/second - unlinked 0 (time 1746139592 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1041 (time 1746139649.02 total 10.01 last 104.03) - open/close 2177 (time 1746139659.02 total 20.01 last 113.55) - open/close 3380 (time 1746139669.02 total 30.01 last 120.28) - open/close 4668 (time 1746139679.03 total 40.02 last 128.77) - open/close 5798 (time 1746139689.03 total 50.02 last 112.99) - open/close 6953 (time 1746139699.03 total 60.02 last 115.45) - open/close 8052 (time 1746139709.04 total 70.03 last 109.84) - open/close 9135 (time 1746139719.04 total 80.03 last 108.24) total: 10000 open/close in 88.14 seconds: 113.46 ops/second - unlinked 0 (time 1746139728 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1102 (time 1746139785.32 total 10.00 last 110.16) - open/close 2177 (time 1746139795.33 total 20.01 last 107.47) - open/close 3398 (time 1746139805.33 total 30.01 last 122.07) - open/close 4635 (time 1746139815.33 total 40.01 last 123.68) - open/close 5919 (time 1746139825.33 total 50.01 last 128.40) - open/close 7135 (time 1746139835.33 total 60.02 last 121.53) - open/close 8797 (time 1746139845.34 total 70.02 last 166.19) total: 10000 open/close in 79.66 seconds: 125.53 ops/second - unlinked 0 (time 1746139856 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1264 (time 1746139904.30 total 10.01 last 126.29) - open/close 2758 (time 1746139914.30 total 20.01 last 149.37) - open/close 4404 (time 1746139924.30 total 30.01 last 164.53) - open/close 5724 (time 1746139934.31 total 40.02 last 131.96) - open/close 7064 (time 1746139944.31 total 50.02 last 133.97) - open/close 8226 (time 1746139954.31 total 60.02 last 116.15) - open/close 9632 (time 1746139964.32 total 70.03 last 140.51) total: 10000 open/close in 73.20 seconds: 136.61 ops/second - unlinked 0 (time 1746139969 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg310-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg310-server: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg310-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg310-server: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg310-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2