== conf-sanity test 69: replace an OST with the same index ========================================================== 10:32:41 (1780324361) start mds service on oleg346-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg346-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg346-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1290 (time 1780324430.29 total 10.01 last 128.88) - open/close 2587 (time 1780324440.30 total 20.01 last 129.62) - open/close 3782 (time 1780324450.30 total 30.02 last 119.44) - open/close 4957 (time 1780324460.31 total 40.03 last 117.42) - open/close 6081 (time 1780324470.31 total 50.03 last 112.34) - open/close 7221 (time 1780324480.32 total 60.04 last 113.94) - open/close 8315 (time 1780324490.33 total 70.04 last 109.33) - open/close 9408 (time 1780324500.33 total 80.05 last 109.29) total: 10000 open/close in 85.11 seconds: 117.49 ops/second - unlinked 0 (time 1780324507 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1164 (time 1780324567.66 total 10.00 last 116.38) - open/close 2295 (time 1780324577.66 total 20.01 last 113.00) - open/close 3451 (time 1780324587.67 total 30.01 last 115.57) - open/close 4623 (time 1780324597.67 total 40.01 last 117.19) - open/close 5777 (time 1780324607.67 total 50.02 last 115.37) - open/close 6851 (time 1780324617.68 total 60.02 last 107.35) - open/close 8055 (time 1780324627.68 total 70.03 last 120.34) - open/close 9259 (time 1780324637.69 total 80.03 last 120.31) total: 10000 open/close in 86.12 seconds: 116.11 ops/second - unlinked 0 (time 1780324645 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1311 (time 1780324700.94 total 10.00 last 131.04) - open/close 2724 (time 1780324710.95 total 20.01 last 141.26) - open/close 4009 (time 1780324720.95 total 30.01 last 128.49) - open/close 5327 (time 1780324730.95 total 40.02 last 131.70) - open/close 6717 (time 1780324740.96 total 50.02 last 138.96) - open/close 8002 (time 1780324750.96 total 60.02 last 128.43) - open/close 9348 (time 1780324760.97 total 70.03 last 134.57) total: 10000 open/close in 75.74 seconds: 132.02 ops/second - unlinked 0 (time 1780324768 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1330 (time 1780324822.49 total 10.01 last 132.91) - open/close 2594 (time 1780324832.49 total 20.01 last 126.39) - open/close 3840 (time 1780324842.49 total 30.01 last 124.53) - open/close 4976 (time 1780324852.50 total 40.02 last 113.50) - open/close 6351 (time 1780324862.51 total 50.03 last 137.45) - open/close 7739 (time 1780324872.51 total 60.03 last 138.71) - open/close 9116 (time 1780324882.52 total 70.04 last 137.66) total: 10000 open/close in 76.03 seconds: 131.52 ops/second - unlinked 0 (time 1780324890 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1613 (time 1780324935.51 total 10.00 last 161.30) - open/close 2931 (time 1780324945.51 total 20.00 last 131.74) - open/close 4281 (time 1780324955.52 total 30.01 last 134.92) - open/close 5740 (time 1780324965.53 total 40.02 last 145.82) - open/close 7193 (time 1780324975.53 total 50.02 last 145.25) - open/close 8535 (time 1780324985.53 total 60.02 last 134.16) - open/close 9859 (time 1780324995.54 total 70.03 last 132.32) total: 10000 open/close in 71.06 seconds: 140.73 ops/second - unlinked 0 (time 1780324998 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1624 (time 1780325050.36 total 10.00 last 162.37) - open/close 3311 (time 1780325060.36 total 20.00 last 168.66) - open/close 4828 (time 1780325070.37 total 30.01 last 151.65) - open/close 6445 (time 1780325080.37 total 40.01 last 161.67) - open/close 8265 (time 1780325090.38 total 50.02 last 181.89) total: 10000 open/close in 59.98 seconds: 166.72 ops/second - unlinked 0 (time 1780325102 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1831 (time 1780325148.35 total 10.00 last 183.02) - open/close 3770 (time 1780325158.35 total 20.01 last 193.87) - open/close 6227 (time 1780325168.36 total 30.01 last 245.55) - open/close 8977 (time 1780325178.36 total 40.01 last 274.99) total: 10000 open/close in 44.38 seconds: 225.30 ops/second - unlinked 0 (time 1780325184 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2410 (time 1780325220.29 total 10.00 last 240.92) - open/close 5057 (time 1780325230.29 total 20.00 last 264.67) - open/close 7451 (time 1780325240.29 total 30.00 last 239.39) - open/close 9927 (time 1780325250.29 total 40.00 last 247.60) total: 10000 open/close in 40.21 seconds: 248.70 ops/second - unlinked 0 (time 1780325252 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3181 (time 1780325281.64 total 10.00 last 318.08) - open/close 6459 (time 1780325291.64 total 20.00 last 327.71) total: 10000 open/close in 29.97 seconds: 333.67 ops/second - unlinked 0 (time 1780325303 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3435 (time 1780325330.87 total 10.00 last 343.42) - open/close 6433 (time 1780325340.87 total 20.00 last 299.79) total: 10000 open/close in 28.93 seconds: 345.69 ops/second - unlinked 0 (time 1780325351 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg346-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre On OST0, 16246 used inodes rc=0 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg346-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg346-server unloading modules via unload_modules_local on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2