== conf-sanity test 69: replace an OST with the same index ========================================================== 16:31:16 (1786566676) start mds service on oleg410-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg410-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg410-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 905 (time 1786566768.21 total 10.01 last 90.41) - open/close 1773 (time 1786566778.22 total 20.02 last 86.68) - open/close 2651 (time 1786566788.23 total 30.03 last 87.75) - open/close 3616 (time 1786566798.23 total 40.03 last 96.49) - open/close 4548 (time 1786566808.23 total 50.03 last 93.18) - open/close 5503 (time 1786566818.23 total 60.03 last 95.48) - open/close 6426 (time 1786566828.23 total 70.04 last 92.27) - open/close 7378 (time 1786566838.24 total 80.04 last 95.17) - open/close 8365 (time 1786566848.25 total 90.05 last 98.56) - open/close 9419 (time 1786566858.26 total 100.07 last 105.29) total: 10000 open/close in 106.07 seconds: 94.28 ops/second - unlinked 0 (time 1786566866 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 973 (time 1786566934.49 total 10.00 last 97.27) - open/close 1989 (time 1786566944.49 total 20.01 last 101.56) - open/close 2914 (time 1786566954.49 total 30.01 last 92.45) - open/close 3837 (time 1786566964.50 total 40.01 last 92.30) - open/close 4516 (time 1786566974.50 total 50.02 last 67.88) - open/close 5037 (time 1786566984.51 total 60.02 last 52.06) - open/close 5463 (time 1786566994.51 total 70.03 last 42.59) - open/close 5978 (time 1786567004.51 total 80.03 last 51.48) - open/close 6520 (time 1786567014.51 total 90.03 last 54.19) - open/close 7268 (time 1786567024.52 total 100.03 last 74.77) - open/close 7791 (time 1786567034.53 total 110.05 last 52.22) - open/close 8315 (time 1786567044.54 total 120.06 last 52.35) - open/close 8992 (time 1786567054.55 total 130.07 last 67.65) - open/close 9689 (time 1786567064.57 total 140.08 last 69.59) total: 10000 open/close in 144.60 seconds: 69.16 ops/second - unlinked 0 (time 1786567074 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 786 (time 1786567158.52 total 10.02 last 78.46) - open/close 1511 (time 1786567168.52 total 20.02 last 72.50) - open/close 2156 (time 1786567178.52 total 30.02 last 64.48) - open/close 2864 (time 1786567188.52 total 40.03 last 70.77) - open/close 3553 (time 1786567198.53 total 50.03 last 68.86) - open/close 4313 (time 1786567208.55 total 60.05 last 75.84) - open/close 5119 (time 1786567218.55 total 70.05 last 80.58) - open/close 5916 (time 1786567228.56 total 80.06 last 79.66) - open/close 6685 (time 1786567238.57 total 90.07 last 76.82) - open/close 7493 (time 1786567248.57 total 100.07 last 80.80) - open/close 8230 (time 1786567258.58 total 110.08 last 73.62) - open/close 8890 (time 1786567268.59 total 120.09 last 65.96) - open/close 9638 (time 1786567278.61 total 130.11 last 74.60) total: 10000 open/close in 135.19 seconds: 73.97 ops/second - unlinked 0 (time 1786567289 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 832 (time 1786567372.96 total 10.00 last 83.18) - open/close 1682 (time 1786567382.97 total 20.01 last 84.89) - open/close 2440 (time 1786567392.98 total 30.02 last 75.79) - open/close 3284 (time 1786567402.98 total 40.02 last 84.34) - open/close 4154 (time 1786567412.99 total 50.03 last 86.92) - open/close 4948 (time 1786567423.00 total 60.04 last 79.37) - open/close 5818 (time 1786567433.00 total 70.04 last 86.93) - open/close 6565 (time 1786567443.01 total 80.05 last 74.64) - open/close 7382 (time 1786567453.01 total 90.05 last 81.68) - open/close 8174 (time 1786567463.02 total 100.06 last 79.12) - open/close 8963 (time 1786567473.03 total 110.07 last 78.88) - open/close 9647 (time 1786567483.04 total 120.08 last 68.33) total: 10000 open/close in 124.76 seconds: 80.15 ops/second - unlinked 0 (time 1786567494 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 767 (time 1786567583.41 total 10.00 last 76.69) - open/close 1470 (time 1786567593.42 total 20.01 last 70.28) - open/close 2221 (time 1786567603.42 total 30.01 last 75.08) - open/close 2915 (time 1786567613.42 total 40.01 last 69.37) - open/close 3685 (time 1786567623.42 total 50.01 last 76.99) - open/close 4463 (time 1786567633.42 total 60.01 last 77.80) - open/close 5219 (time 1786567643.43 total 70.02 last 75.53) - open/close 6026 (time 1786567653.44 total 80.03 last 80.64) - open/close 6758 (time 1786567663.45 total 90.04 last 73.16) - open/close 7546 (time 1786567673.46 total 100.05 last 78.72) - open/close 8356 (time 1786567683.47 total 110.06 last 80.89) - open/close 9184 (time 1786567693.47 total 120.06 last 82.78) total: 10000 open/close in 129.28 seconds: 77.35 ops/second - unlinked 0 (time 1786567707 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 855 (time 1786567795.28 total 10.00 last 85.50) - open/close 1733 (time 1786567805.28 total 20.00 last 87.78) - open/close 2512 (time 1786567815.29 total 30.01 last 77.87) - open/close 3304 (time 1786567825.29 total 40.01 last 79.16) - open/close 4126 (time 1786567835.30 total 50.02 last 82.15) - open/close 4892 (time 1786567845.30 total 60.02 last 76.59) - open/close 5536 (time 1786567855.31 total 70.03 last 64.33) - open/close 6233 (time 1786567865.32 total 80.04 last 69.62) - open/close 6939 (time 1786567875.32 total 90.04 last 70.59) - open/close 7605 (time 1786567885.33 total 100.05 last 66.56) - open/close 8406 (time 1786567895.33 total 110.05 last 80.07) - open/close 9146 (time 1786567905.34 total 120.06 last 73.97) - open/close 9923 (time 1786567915.36 total 130.08 last 77.52) total: 10000 open/close in 130.86 seconds: 76.42 ops/second - unlinked 0 (time 1786567921 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 805 (time 1786568005.30 total 10.01 last 80.41) - open/close 1579 (time 1786568015.30 total 20.01 last 77.38) - open/close 2383 (time 1786568025.31 total 30.02 last 80.33) - open/close 3171 (time 1786568035.31 total 40.03 last 78.77) - open/close 3964 (time 1786568045.32 total 50.03 last 79.26) - open/close 4779 (time 1786568055.33 total 60.04 last 81.41) - open/close 5597 (time 1786568065.33 total 70.05 last 81.79) - open/close 6384 (time 1786568075.34 total 80.05 last 78.67) - open/close 7208 (time 1786568085.34 total 90.06 last 82.33) - open/close 8012 (time 1786568095.36 total 100.07 last 80.29) - open/close 8803 (time 1786568105.37 total 110.08 last 79.04) - open/close 9695 (time 1786568115.37 total 120.08 last 89.17) total: 10000 open/close in 123.84 seconds: 80.75 ops/second - unlinked 0 (time 1786568123 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 812 (time 1786568202.36 total 10.01 last 81.13) - open/close 1649 (time 1786568212.36 total 20.01 last 83.69) - open/close 2557 (time 1786568222.36 total 30.01 last 90.76) - open/close 3408 (time 1786568232.37 total 40.02 last 85.05) - open/close 4362 (time 1786568242.38 total 50.03 last 95.29) - open/close 5253 (time 1786568252.39 total 60.04 last 89.01) - open/close 6097 (time 1786568262.40 total 70.05 last 84.36) - open/close 7002 (time 1786568272.41 total 80.06 last 90.38) - open/close 7931 (time 1786568282.41 total 90.06 last 92.89) - open/close 8949 (time 1786568292.42 total 100.07 last 101.67) - open/close 9880 (time 1786568302.43 total 110.08 last 93.09) total: 10000 open/close in 111.45 seconds: 89.72 ops/second - unlinked 0 (time 1786568307 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 937 (time 1786568384.00 total 10.00 last 93.69) - open/close 1843 (time 1786568394.00 total 20.01 last 90.52) - open/close 2762 (time 1786568404.01 total 30.02 last 91.85) - open/close 3603 (time 1786568414.03 total 40.03 last 83.97) - open/close 4433 (time 1786568424.04 total 50.04 last 82.89) - open/close 5325 (time 1786568434.05 total 60.06 last 89.08) - open/close 6233 (time 1786568444.05 total 70.06 last 90.78) - open/close 7080 (time 1786568454.06 total 80.07 last 84.64) - open/close 7953 (time 1786568464.06 total 90.07 last 87.28) - open/close 8833 (time 1786568474.07 total 100.08 last 87.93) - open/close 9770 (time 1786568484.08 total 110.08 last 93.67) total: 10000 open/close in 112.51 seconds: 88.88 ops/second - unlinked 0 (time 1786568489 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 826 (time 1786568569.72 total 10.01 last 82.50) - open/close 1727 (time 1786568579.72 total 20.02 last 90.05) - open/close 2644 (time 1786568589.73 total 30.03 last 91.61) - open/close 3540 (time 1786568599.73 total 40.03 last 89.58) - open/close 4426 (time 1786568609.75 total 50.04 last 88.50) - open/close 5282 (time 1786568619.75 total 60.05 last 85.55) - open/close 6176 (time 1786568629.76 total 70.06 last 89.32) - open/close 7012 (time 1786568639.77 total 80.07 last 83.49) - open/close 7843 (time 1786568649.81 total 90.10 last 82.82) - open/close 8667 (time 1786568659.81 total 100.11 last 82.36) - open/close 9491 (time 1786568669.81 total 110.11 last 82.38) total: 10000 open/close in 117.56 seconds: 85.06 ops/second - unlinked 0 (time 1786568682 ; total 1 ; last 1) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg410-server unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2