== conf-sanity test 69: replace an OST with the same index ========================================================== 09:41:28 (1782481288) start mds service on oleg235-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg235-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg235-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg235-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 997 (time 1782481389.71 total 10.01 last 99.58) - open/close 2000 (time 1782481399.72 total 20.02 last 100.17) - open/close 3163 (time 1782481409.73 total 30.03 last 116.26) - open/close 4129 (time 1782481419.73 total 40.04 last 96.53) - open/close 5155 (time 1782481429.76 total 50.06 last 102.36) - open/close 6183 (time 1782481439.77 total 60.07 last 102.70) - open/close 7164 (time 1782481449.78 total 70.08 last 98.01) - open/close 8125 (time 1782481459.78 total 80.08 last 96.05) - open/close 9053 (time 1782481469.79 total 90.09 last 92.74) total: 10000 open/close in 99.13 seconds: 100.88 ops/second - unlinked 0 (time 1782481482 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 878 (time 1782481561.46 total 10.02 last 87.66) - open/close 1818 (time 1782481571.46 total 20.02 last 93.96) - open/close 2813 (time 1782481581.46 total 30.02 last 99.50) - open/close 3819 (time 1782481591.46 total 40.02 last 100.58) - open/close 4746 (time 1782481601.47 total 50.03 last 92.63) - open/close 5656 (time 1782481611.48 total 60.04 last 90.87) - open/close 6686 (time 1782481621.49 total 70.05 last 102.96) - open/close 7616 (time 1782481631.50 total 80.06 last 92.86) - open/close 8636 (time 1782481641.52 total 90.08 last 101.88) - open/close 9585 (time 1782481651.52 total 100.08 last 94.84) total: 10000 open/close in 104.47 seconds: 95.72 ops/second - unlinked 0 (time 1782481660 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1068 (time 1782481732.55 total 10.01 last 106.72) - open/close 2131 (time 1782481742.55 total 20.01 last 106.26) - open/close 3099 (time 1782481752.56 total 30.02 last 96.68) - open/close 4111 (time 1782481762.57 total 40.03 last 101.15) - open/close 5127 (time 1782481772.57 total 50.03 last 101.54) - open/close 6106 (time 1782481782.58 total 60.04 last 97.87) - open/close 7079 (time 1782481792.58 total 70.04 last 97.29) - open/close 8106 (time 1782481802.59 total 80.04 last 102.64) - open/close 9199 (time 1782481812.59 total 90.05 last 109.29) total: 10000 open/close in 98.34 seconds: 101.69 ops/second - unlinked 0 (time 1782481824 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 996 (time 1782481894.32 total 10.01 last 99.51) - open/close 2060 (time 1782481904.32 total 20.01 last 106.40) - open/close 3203 (time 1782481914.33 total 30.02 last 114.22) - open/close 4329 (time 1782481924.34 total 40.03 last 112.49) - open/close 5441 (time 1782481934.34 total 50.03 last 111.15) - open/close 6612 (time 1782481944.35 total 60.04 last 117.04) - open/close 7772 (time 1782481954.35 total 70.04 last 115.99) - open/close 8866 (time 1782481964.35 total 80.04 last 109.37) - open/close 9940 (time 1782481974.36 total 90.04 last 107.34) total: 10000 open/close in 90.57 seconds: 110.41 ops/second - unlinked 0 (time 1782481978 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1048 (time 1782482049.23 total 10.01 last 104.74) - open/close 2229 (time 1782482059.24 total 20.01 last 117.99) - open/close 3448 (time 1782482069.24 total 30.02 last 121.86) - open/close 4685 (time 1782482079.25 total 40.02 last 123.65) - open/close 5761 (time 1782482089.25 total 50.03 last 107.57) - open/close 6722 (time 1782482099.26 total 60.03 last 96.02) - open/close 7810 (time 1782482109.26 total 70.04 last 108.78) - open/close 9032 (time 1782482119.26 total 80.04 last 122.18) total: 10000 open/close in 88.54 seconds: 112.95 ops/second - unlinked 0 (time 1782482130 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1234 (time 1782482193.62 total 10.00 last 123.36) - open/close 2377 (time 1782482203.63 total 20.01 last 114.26) - open/close 3401 (time 1782482213.64 total 30.02 last 102.30) - open/close 4570 (time 1782482223.65 total 40.03 last 116.71) - open/close 5616 (time 1782482233.65 total 50.03 last 104.59) - open/close 6705 (time 1782482243.66 total 60.04 last 108.86) - open/close 7843 (time 1782482253.66 total 70.04 last 113.78) - open/close 8931 (time 1782482263.67 total 80.05 last 108.66) - open/close 9989 (time 1782482273.68 total 90.06 last 105.75) total: 10000 open/close in 90.14 seconds: 110.94 ops/second - unlinked 0 (time 1782482276 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1038 (time 1782482338.90 total 10.01 last 103.69) - open/close 2074 (time 1782482348.91 total 20.02 last 103.56) - open/close 3229 (time 1782482358.91 total 30.02 last 115.47) - open/close 4354 (time 1782482368.92 total 40.02 last 112.45) - open/close 5391 (time 1782482378.92 total 50.02 last 103.67) - open/close 6466 (time 1782482388.92 total 60.03 last 107.49) - open/close 7487 (time 1782482398.93 total 70.03 last 102.02) - open/close 8651 (time 1782482408.93 total 80.03 last 116.38) - open/close 9837 (time 1782482418.93 total 90.04 last 118.57) total: 10000 open/close in 91.57 seconds: 109.20 ops/second - unlinked 0 (time 1782482423 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1242 (time 1782482486.52 total 10.00 last 124.18) - open/close 2379 (time 1782482496.53 total 20.01 last 113.66) - open/close 3585 (time 1782482506.53 total 30.01 last 120.56) - open/close 4741 (time 1782482516.54 total 40.02 last 115.51) - open/close 5915 (time 1782482526.54 total 50.02 last 117.38) - open/close 7255 (time 1782482536.55 total 60.02 last 133.94) - open/close 8445 (time 1782482546.55 total 70.03 last 118.96) - open/close 9749 (time 1782482556.55 total 80.03 last 130.37) total: 10000 open/close in 82.03 seconds: 121.90 ops/second - unlinked 0 (time 1782482561 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1254 (time 1782482620.86 total 10.01 last 125.23) - open/close 2420 (time 1782482630.87 total 20.02 last 116.51) - open/close 3594 (time 1782482640.88 total 30.03 last 117.28) - open/close 4884 (time 1782482650.88 total 40.03 last 128.96) - open/close 6104 (time 1782482660.89 total 50.04 last 121.97) - open/close 7273 (time 1782482670.89 total 60.04 last 116.88) - open/close 8412 (time 1782482680.89 total 70.04 last 113.90) - open/close 9554 (time 1782482690.89 total 80.04 last 114.18) total: 10000 open/close in 83.54 seconds: 119.71 ops/second - unlinked 0 (time 1782482696 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1234 (time 1782482755.61 total 10.00 last 123.35) - open/close 2593 (time 1782482765.61 total 20.01 last 135.84) - open/close 3854 (time 1782482775.61 total 30.01 last 126.08) - open/close 5099 (time 1782482785.62 total 40.01 last 124.47) - open/close 6387 (time 1782482795.63 total 50.02 last 128.66) - open/close 7629 (time 1782482805.63 total 60.03 last 124.15) - open/close 8917 (time 1782482815.64 total 70.03 last 128.72) total: 10000 open/close in 78.36 seconds: 127.61 ops/second - unlinked 0 (time 1782482826 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre On OST0, 20187 used inodes rc=0 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local oleg235-server: modules unloaded. pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2