== conf-sanity test 69: replace an OST with the same index ========================================================== 15:02:04 (1777575724) start mds service on oleg407-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg407-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg407-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 871 (time 1777575834.03 total 10.00 last 87.06) - open/close 1718 (time 1777575844.04 total 20.01 last 84.64) - open/close 2510 (time 1777575854.05 total 30.02 last 79.10) - open/close 3310 (time 1777575864.06 total 40.03 last 79.93) - open/close 4038 (time 1777575874.06 total 50.03 last 72.80) - open/close 4726 (time 1777575884.07 total 60.04 last 68.73) - open/close 5473 (time 1777575894.09 total 70.06 last 74.56) - open/close 6239 (time 1777575904.09 total 80.06 last 76.59) - open/close 6923 (time 1777575914.09 total 90.07 last 68.38) - open/close 7724 (time 1777575924.10 total 100.08 last 79.98) - open/close 8461 (time 1777575934.11 total 110.08 last 73.69) - open/close 9113 (time 1777575944.11 total 120.09 last 65.16) - open/close 9842 (time 1777575954.11 total 130.09 last 72.89) total: 10000 open/close in 132.29 seconds: 75.59 ops/second - unlinked 0 (time 1777575960 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 679 (time 1777576050.72 total 10.01 last 67.85) - open/close 1468 (time 1777576060.73 total 20.02 last 78.82) - open/close 2177 (time 1777576070.74 total 30.03 last 70.82) - open/close 2878 (time 1777576080.76 total 40.05 last 69.96) - open/close 3420 (time 1777576090.77 total 50.06 last 54.16) - open/close 4052 (time 1777576100.78 total 60.07 last 63.11) - open/close 4719 (time 1777576110.79 total 70.08 last 66.62) - open/close 5511 (time 1777576120.79 total 80.08 last 79.19) - open/close 6301 (time 1777576130.81 total 90.10 last 78.89) - open/close 7092 (time 1777576140.81 total 100.10 last 79.06) - open/close 7844 (time 1777576150.82 total 110.11 last 75.18) - open/close 8644 (time 1777576160.82 total 120.11 last 80.00) - open/close 9403 (time 1777576170.83 total 130.12 last 75.80) total: 10000 open/close in 137.74 seconds: 72.60 ops/second - unlinked 0 (time 1777576182 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 772 (time 1777576271.03 total 10.00 last 77.17) - open/close 1532 (time 1777576281.04 total 20.01 last 75.95) - open/close 2285 (time 1777576291.04 total 30.01 last 75.28) - open/close 3048 (time 1777576301.04 total 40.02 last 76.28) - open/close 3770 (time 1777576311.06 total 50.03 last 72.07) - open/close 4463 (time 1777576321.08 total 60.06 last 69.15) - open/close 5164 (time 1777576331.09 total 70.07 last 70.00) - open/close 5964 (time 1777576341.09 total 80.07 last 80.00) - open/close 6742 (time 1777576351.10 total 90.08 last 77.73) - open/close 7526 (time 1777576361.12 total 100.10 last 78.26) - open/close 8255 (time 1777576371.13 total 110.11 last 72.84) - open/close 9030 (time 1777576381.14 total 120.12 last 77.40) - open/close 9782 (time 1777576391.15 total 130.12 last 75.18) total: 10000 open/close in 132.96 seconds: 75.21 ops/second - unlinked 0 (time 1777576399 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 746 (time 1777576489.54 total 10.02 last 74.46) - open/close 1467 (time 1777576499.54 total 20.02 last 72.08) - open/close 2210 (time 1777576509.54 total 30.02 last 74.29) - open/close 3015 (time 1777576519.54 total 40.02 last 80.50) - open/close 3703 (time 1777576529.56 total 50.04 last 68.66) - open/close 4310 (time 1777576539.57 total 60.05 last 60.66) - open/close 5109 (time 1777576549.57 total 70.05 last 79.88) - open/close 5875 (time 1777576559.58 total 80.07 last 76.50) - open/close 6604 (time 1777576569.60 total 90.08 last 72.78) - open/close 7381 (time 1777576579.61 total 100.09 last 77.65) - open/close 8148 (time 1777576589.61 total 110.09 last 76.70) - open/close 8898 (time 1777576599.62 total 120.10 last 74.92) - open/close 9675 (time 1777576609.62 total 130.11 last 77.65) total: 10000 open/close in 134.00 seconds: 74.63 ops/second - unlinked 0 (time 1777576617 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 837 (time 1777576711.61 total 10.01 last 83.61) - open/close 1586 (time 1777576721.61 total 20.01 last 74.88) - open/close 2257 (time 1777576731.62 total 30.02 last 67.06) - open/close 2985 (time 1777576741.62 total 40.03 last 72.76) - open/close 3602 (time 1777576751.63 total 50.03 last 61.68) - open/close 4393 (time 1777576761.63 total 60.03 last 79.08) - open/close 5186 (time 1777576771.64 total 70.05 last 79.19) - open/close 5928 (time 1777576781.65 total 80.05 last 74.18) - open/close 6562 (time 1777576791.65 total 90.05 last 63.40) - open/close 7406 (time 1777576801.65 total 100.06 last 84.33) - open/close 8033 (time 1777576811.67 total 110.08 last 62.58) - open/close 8779 (time 1777576821.69 total 120.09 last 74.51) - open/close 9505 (time 1777576831.70 total 130.10 last 72.51) total: 10000 open/close in 136.95 seconds: 73.02 ops/second - unlinked 0 (time 1777576843 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 618 (time 1777576935.00 total 10.02 last 61.70) - open/close 1336 (time 1777576945.02 total 20.03 last 71.69) - open/close 2109 (time 1777576955.03 total 30.04 last 77.21) - open/close 2867 (time 1777576965.04 total 40.06 last 75.70) - open/close 3615 (time 1777576975.06 total 50.08 last 74.62) - open/close 4395 (time 1777576985.07 total 60.08 last 77.99) - open/close 5125 (time 1777576995.07 total 70.09 last 72.97) - open/close 5896 (time 1777577005.08 total 80.09 last 77.05) - open/close 6542 (time 1777577015.08 total 90.10 last 64.56) - open/close 7107 (time 1777577025.09 total 100.11 last 56.46) - open/close 7731 (time 1777577035.10 total 110.11 last 62.34) - open/close 8411 (time 1777577045.11 total 120.13 last 67.93) - open/close 9201 (time 1777577055.13 total 130.15 last 78.84) - open/close 9943 (time 1777577065.14 total 140.16 last 74.12) total: 10000 open/close in 141.27 seconds: 70.79 ops/second - unlinked 0 (time 1777577070 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 733 (time 1777577159.98 total 10.01 last 73.22) - open/close 1509 (time 1777577169.98 total 20.02 last 77.56) - open/close 2260 (time 1777577179.99 total 30.02 last 75.04) - open/close 3036 (time 1777577189.99 total 40.03 last 77.59) - open/close 3849 (time 1777577199.99 total 50.03 last 81.27) - open/close 4659 (time 1777577210.00 total 60.03 last 80.98) - open/close 5391 (time 1777577220.01 total 70.04 last 73.14) - open/close 6175 (time 1777577230.01 total 80.05 last 78.33) - open/close 6864 (time 1777577240.02 total 90.05 last 68.86) - open/close 7702 (time 1777577250.02 total 100.06 last 83.78) - open/close 8468 (time 1777577260.02 total 110.06 last 76.59) - open/close 9272 (time 1777577270.03 total 120.06 last 80.34) total: 10000 open/close in 129.75 seconds: 77.07 ops/second - unlinked 0 (time 1777577284 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 731 (time 1777577376.71 total 10.00 last 73.07) - open/close 1355 (time 1777577386.72 total 20.02 last 62.33) - open/close 1932 (time 1777577396.72 total 30.02 last 57.68) - open/close 2517 (time 1777577406.74 total 40.04 last 58.37) - open/close 3129 (time 1777577416.76 total 50.06 last 61.11) - open/close 3898 (time 1777577426.77 total 60.07 last 76.79) - open/close 4636 (time 1777577436.78 total 70.07 last 73.77) - open/close 5420 (time 1777577446.80 total 80.10 last 78.22) - open/close 6182 (time 1777577456.81 total 90.10 last 76.16) - open/close 6967 (time 1777577466.81 total 100.10 last 78.48) - open/close 7750 (time 1777577476.84 total 110.13 last 78.07) - open/close 8574 (time 1777577486.84 total 120.14 last 82.35) - open/close 9336 (time 1777577496.85 total 130.15 last 76.16) total: 10000 open/close in 139.32 seconds: 71.77 ops/second - unlinked 0 (time 1777577510 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 807 (time 1777577598.98 total 10.00 last 80.68) - open/close 1615 (time 1777577608.99 total 20.01 last 80.77) - open/close 2411 (time 1777577618.99 total 30.01 last 79.58) - open/close 3182 (time 1777577629.00 total 40.02 last 77.03) - open/close 3890 (time 1777577639.00 total 50.02 last 70.78) - open/close 4598 (time 1777577649.01 total 60.02 last 70.77) - open/close 5320 (time 1777577659.01 total 70.03 last 72.14) - open/close 6058 (time 1777577669.01 total 80.03 last 73.80) - open/close 6834 (time 1777577679.02 total 90.04 last 77.55) - open/close 7493 (time 1777577689.04 total 100.06 last 65.78) - open/close 8262 (time 1777577699.05 total 110.06 last 76.85) - open/close 9069 (time 1777577709.05 total 120.06 last 80.70) - open/close 9859 (time 1777577719.05 total 130.07 last 78.93) total: 10000 open/close in 131.78 seconds: 75.88 ops/second - unlinked 0 (time 1777577725 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 820 (time 1777577815.90 total 10.00 last 81.98) - open/close 1597 (time 1777577825.90 total 20.00 last 77.69) - open/close 2426 (time 1777577835.91 total 30.01 last 82.89) - open/close 3205 (time 1777577845.93 total 40.03 last 77.73) - open/close 3885 (time 1777577855.94 total 50.04 last 67.92) - open/close 4741 (time 1777577865.95 total 60.05 last 85.51) - open/close 5404 (time 1777577875.96 total 70.05 last 66.26) - open/close 6241 (time 1777577885.96 total 80.06 last 83.66) - open/close 7011 (time 1777577895.97 total 90.07 last 76.95) - open/close 7742 (time 1777577905.97 total 100.07 last 73.09) - open/close 8548 (time 1777577915.98 total 110.08 last 80.52) - open/close 9374 (time 1777577925.98 total 120.08 last 82.56) total: 10000 open/close in 127.74 seconds: 78.29 ops/second - unlinked 0 (time 1777577937 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.107@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.107@tcp sys.timeout=20 start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: modules unloaded. pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2