== conf-sanity test 69: replace an OST with the same index ========================================================== 16:03:02 (1710100982) start mds service on oleg355-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg355-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg355-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock oleg355-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1451 (time 1710101017.55 total 10.01 last 145.02) - open/close 2702 (time 1710101027.56 total 20.01 last 125.05) - open/close 3943 (time 1710101037.56 total 30.01 last 124.10) - open/close 4478 (time 1710101053.05 total 45.50 last 34.53) - open/close 5492 (time 1710101063.06 total 55.51 last 101.35) - open/close 6509 (time 1710101073.06 total 65.51 last 101.68) - open/close 7557 (time 1710101083.06 total 75.52 last 104.73) - open/close 8414 (time 1710101093.07 total 85.52 last 85.69) - open/close 9205 (time 1710101103.07 total 95.52 last 79.09) - open/close 9981 (time 1710101113.07 total 105.52 last 77.59) total: 10000 open/close in 105.72 seconds: 94.59 ops/second - unlinked 0 (time 1710101115 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1312 (time 1710101193.53 total 10.00 last 131.17) - open/close 2467 (time 1710101203.54 total 20.01 last 115.39) - open/close 3632 (time 1710101213.55 total 30.02 last 116.42) - open/close 4697 (time 1710101223.55 total 40.02 last 106.49) - open/close 5715 (time 1710101233.55 total 50.02 last 101.75) - open/close 6702 (time 1710101243.56 total 60.03 last 98.66) - open/close 7372 (time 1710101261.84 total 78.31 last 36.65) - open/close 8247 (time 1710101271.85 total 88.32 last 87.43) - open/close 9143 (time 1710101281.86 total 98.34 last 89.45) - open/close 9952 (time 1710101291.87 total 108.34 last 80.84) total: 10000 open/close in 108.90 seconds: 91.83 ops/second - unlinked 0 (time 1710101294 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1341 (time 1710101368.79 total 10.01 last 134.03) - open/close 2602 (time 1710101378.80 total 20.01 last 126.06) - open/close 3799 (time 1710101388.82 total 30.03 last 119.41) - open/close 4868 (time 1710101398.83 total 40.04 last 106.84) - open/close 5900 (time 1710101408.83 total 50.04 last 103.16) - open/close 6871 (time 1710101418.83 total 60.04 last 97.08) - open/close 7813 (time 1710101428.83 total 70.05 last 94.18) - open/close 8691 (time 1710101438.84 total 80.05 last 87.75) - open/close 9577 (time 1710101448.84 total 90.05 last 88.58) total: 10000 open/close in 95.08 seconds: 105.18 ops/second - unlinked 0 (time 1710101455 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1369 (time 1710101531.72 total 10.01 last 136.83) - open/close 2627 (time 1710101541.72 total 20.01 last 125.72) - open/close 3747 (time 1710101551.73 total 30.02 last 111.89) - open/close 4643 (time 1710101570.99 total 49.28 last 46.52) - open/close 5025 (time 1710101585.66 total 63.95 last 26.05) - open/close 6051 (time 1710101595.67 total 73.95 last 102.54) - open/close 7022 (time 1710101605.67 total 83.95 last 97.10) - open/close 7911 (time 1710101615.71 total 93.99 last 88.57) - open/close 8793 (time 1710101625.72 total 104.00 last 88.11) - open/close 9674 (time 1710101635.72 total 114.01 last 88.07) total: 10000 open/close in 117.55 seconds: 85.07 ops/second - unlinked 0 (time 1710101641 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1382 (time 1710101716.27 total 10.00 last 138.16) - open/close 2690 (time 1710101726.28 total 20.01 last 130.75) - open/close 3897 (time 1710101736.28 total 30.01 last 120.64) - open/close 5004 (time 1710101746.29 total 40.02 last 110.61) - open/close 6106 (time 1710101756.30 total 50.03 last 110.12) - open/close 7119 (time 1710101766.30 total 60.03 last 101.30) - open/close 8058 (time 1710101776.30 total 70.03 last 93.88) - open/close 8939 (time 1710101786.31 total 80.04 last 87.97) - open/close 9771 (time 1710101796.33 total 90.06 last 83.08) total: 10000 open/close in 92.80 seconds: 107.76 ops/second - unlinked 0 (time 1710101800 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1354 (time 1710101873.48 total 10.01 last 135.30) - open/close 2525 (time 1710101883.48 total 20.01 last 117.10) - open/close 3663 (time 1710101893.49 total 30.01 last 113.72) - open/close 4676 (time 1710101903.53 total 40.05 last 100.94) - open/close 5654 (time 1710101913.53 total 50.05 last 97.78) - open/close 6577 (time 1710101923.53 total 60.06 last 92.25) - open/close 7342 (time 1710101943.26 total 79.78 last 38.79) - open/close 8322 (time 1710101953.26 total 89.78 last 98.00) - open/close 9216 (time 1710101963.26 total 99.79 last 89.35) total: 10000 open/close in 108.83 seconds: 91.88 ops/second - unlinked 0 (time 1710101973 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 676 (time 1710102051.82 total 15.85 last 42.64) - open/close 1955 (time 1710102061.82 total 25.86 last 127.83) - open/close 3087 (time 1710102071.82 total 35.86 last 113.20) - open/close 4256 (time 1710102081.83 total 45.86 last 116.84) - open/close 5266 (time 1710102091.84 total 55.88 last 100.85) - open/close 6308 (time 1710102101.85 total 65.88 last 104.16) - open/close 7314 (time 1710102111.85 total 75.89 last 100.53) - open/close 8264 (time 1710102121.85 total 85.89 last 94.99) - open/close 9210 (time 1710102131.86 total 95.89 last 94.59) total: 10000 open/close in 105.08 seconds: 95.17 ops/second - unlinked 0 (time 1710102142 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1563 (time 1710102209.56 total 10.00 last 156.27) - open/close 3019 (time 1710102219.56 total 20.01 last 145.55) - open/close 4292 (time 1710102229.56 total 30.01 last 127.28) - open/close 5555 (time 1710102239.57 total 40.02 last 126.14) - open/close 6691 (time 1710102249.58 total 50.03 last 113.52) - open/close 7737 (time 1710102259.59 total 60.04 last 104.49) - open/close 8708 (time 1710102269.60 total 70.04 last 97.07) - open/close 9668 (time 1710102279.60 total 80.05 last 95.94) total: 10000 open/close in 83.64 seconds: 119.56 ops/second - unlinked 0 (time 1710102284 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1673 (time 1710102348.51 total 10.00 last 167.22) - open/close 3150 (time 1710102358.51 total 20.01 last 147.65) - open/close 4545 (time 1710102368.51 total 30.01 last 139.46) - open/close 5829 (time 1710102378.51 total 40.01 last 128.39) - open/close 7036 (time 1710102388.53 total 50.02 last 120.56) - open/close 8145 (time 1710102398.53 total 60.03 last 110.83) - open/close 9207 (time 1710102408.53 total 70.03 last 106.19) total: 10000 open/close in 77.78 seconds: 128.57 ops/second - unlinked 0 (time 1710102417 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1781 (time 1710102480.68 total 10.00 last 178.07) - open/close 3476 (time 1710102490.69 total 20.01 last 169.41) - open/close 5102 (time 1710102500.69 total 30.01 last 162.60) - open/close 6626 (time 1710102510.69 total 40.01 last 152.32) - open/close 8028 (time 1710102520.69 total 50.02 last 140.15) - open/close 9327 (time 1710102530.70 total 60.02 last 129.85) total: 10000 open/close in 65.49 seconds: 152.70 ops/second - unlinked 0 (time 1710102537 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.155@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg355-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg355-server: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg355-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec oleg355-server: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg355-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock oleg355-server@tcp:/lustre /mnt/lustre On OST0, 17658 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server unloading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2