== conf-sanity test 69: replace an OST with the same index ========================================================== 01:17:13 (1783142233) start mds service on oleg145-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: MODOPTS_LIBCFS= oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg145-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg145-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 957 (time 1783142318.99 total 10.01 last 95.64) - open/close 1922 (time 1783142329.00 total 20.01 last 96.47) - open/close 3073 (time 1783142339.00 total 30.02 last 115.03) - open/close 4054 (time 1783142349.00 total 40.02 last 98.07) - open/close 5016 (time 1783142359.01 total 50.02 last 96.15) - open/close 6030 (time 1783142369.01 total 60.03 last 101.38) - open/close 7059 (time 1783142379.02 total 70.04 last 102.80) - open/close 8074 (time 1783142389.03 total 80.04 last 101.42) - open/close 9103 (time 1783142399.03 total 90.04 last 102.88) total: 10000 open/close in 98.87 seconds: 101.14 ops/second - unlinked 0 (time 1783142411 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 997 (time 1783142481.04 total 10.01 last 99.55) - open/close 1968 (time 1783142491.05 total 20.02 last 97.05) - open/close 2928 (time 1783142501.05 total 30.02 last 95.97) - open/close 3916 (time 1783142511.05 total 40.02 last 98.79) - open/close 4920 (time 1783142521.06 total 50.03 last 100.30) - open/close 6013 (time 1783142531.07 total 60.04 last 109.26) - open/close 7162 (time 1783142541.07 total 70.04 last 114.83) - open/close 8254 (time 1783142551.07 total 80.04 last 109.20) - open/close 9489 (time 1783142561.08 total 90.05 last 123.40) total: 10000 open/close in 94.53 seconds: 105.79 ops/second - unlinked 0 (time 1783142567 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 936 (time 1783142629.72 total 10.00 last 93.60) - open/close 1819 (time 1783142639.72 total 20.00 last 88.27) - open/close 2852 (time 1783142649.73 total 30.01 last 103.24) - open/close 3789 (time 1783142659.73 total 40.01 last 93.68) - open/close 4726 (time 1783142669.74 total 50.02 last 93.64) - open/close 5743 (time 1783142679.74 total 60.02 last 101.67) - open/close 6751 (time 1783142689.74 total 70.02 last 100.78) - open/close 7754 (time 1783142699.75 total 80.03 last 100.25) - open/close 8808 (time 1783142709.75 total 90.03 last 105.35) - open/close 9838 (time 1783142719.76 total 100.04 last 102.95) total: 10000 open/close in 101.42 seconds: 98.60 ops/second - unlinked 0 (time 1783142723 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 944 (time 1783142787.72 total 10.00 last 94.38) - open/close 2031 (time 1783142797.72 total 20.00 last 108.67) - open/close 3077 (time 1783142807.73 total 30.02 last 104.47) - open/close 4207 (time 1783142817.74 total 40.03 last 112.90) - open/close 5367 (time 1783142827.74 total 50.03 last 115.97) - open/close 6513 (time 1783142837.74 total 60.03 last 114.60) - open/close 7707 (time 1783142847.74 total 70.03 last 119.39) - open/close 8892 (time 1783142857.75 total 80.04 last 118.39) total: 10000 open/close in 89.81 seconds: 111.35 ops/second - unlinked 0 (time 1783142870 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1074 (time 1783142936.02 total 10.00 last 107.35) - open/close 2087 (time 1783142946.02 total 20.01 last 101.27) - open/close 3257 (time 1783142956.03 total 30.01 last 116.99) - open/close 4313 (time 1783142966.04 total 40.02 last 105.44) - open/close 5462 (time 1783142976.05 total 50.04 last 114.76) - open/close 6634 (time 1783142986.07 total 60.05 last 117.04) - open/close 7953 (time 1783142996.08 total 70.07 last 131.65) - open/close 9048 (time 1783143006.09 total 80.07 last 109.45) total: 10000 open/close in 88.62 seconds: 112.84 ops/second - unlinked 0 (time 1783143016 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1110 (time 1783143080.66 total 10.01 last 110.94) - open/close 2203 (time 1783143090.66 total 20.01 last 109.25) - open/close 3301 (time 1783143100.67 total 30.02 last 109.73) - open/close 4565 (time 1783143110.67 total 40.02 last 126.38) - open/close 5706 (time 1783143120.69 total 50.04 last 113.90) - open/close 6859 (time 1783143130.69 total 60.04 last 115.24) - open/close 8116 (time 1783143140.70 total 70.05 last 125.63) - open/close 9219 (time 1783143150.70 total 80.05 last 110.27) total: 10000 open/close in 87.37 seconds: 114.45 ops/second - unlinked 0 (time 1783143160 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1111 (time 1783143218.98 total 10.00 last 111.09) - open/close 2333 (time 1783143229.00 total 20.02 last 121.94) - open/close 3393 (time 1783143239.00 total 30.02 last 105.98) - open/close 4468 (time 1783143249.00 total 40.02 last 107.50) - open/close 5621 (time 1783143259.01 total 50.03 last 115.19) - open/close 6768 (time 1783143269.02 total 60.04 last 114.66) - open/close 7831 (time 1783143279.02 total 70.05 last 106.21) - open/close 9028 (time 1783143289.03 total 80.05 last 119.59) total: 10000 open/close in 89.48 seconds: 111.75 ops/second - unlinked 0 (time 1783143301 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1285 (time 1783143356.81 total 10.01 last 128.39) - open/close 2587 (time 1783143366.81 total 20.01 last 130.17) - open/close 3848 (time 1783143376.82 total 30.02 last 126.03) - open/close 5232 (time 1783143386.83 total 40.03 last 138.28) - open/close 6847 (time 1783143396.83 total 50.03 last 161.41) - open/close 8178 (time 1783143406.84 total 60.03 last 133.04) - open/close 9826 (time 1783143416.84 total 70.04 last 164.75) total: 10000 open/close in 71.03 seconds: 140.79 ops/second - unlinked 0 (time 1783143419 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1849 (time 1783143463.39 total 10.01 last 184.80) - open/close 3678 (time 1783143473.39 total 20.01 last 182.87) - open/close 5414 (time 1783143483.39 total 30.01 last 173.53) - open/close 7420 (time 1783143493.40 total 40.02 last 200.50) - open/close 9141 (time 1783143503.40 total 50.02 last 172.05) total: 10000 open/close in 56.01 seconds: 178.54 ops/second - unlinked 0 (time 1783143511 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1480 (time 1783143555.63 total 10.00 last 147.95) - open/close 2932 (time 1783143565.64 total 20.01 last 145.13) - open/close 4416 (time 1783143575.64 total 30.01 last 148.37) - open/close 6929 (time 1783143585.64 total 40.01 last 251.22) - open/close 8367 (time 1783143595.65 total 50.02 last 143.75) - open/close 9614 (time 1783143605.65 total 60.02 last 124.65) total: 10000 open/close in 62.98 seconds: 158.77 ops/second - unlinked 0 (time 1783143610 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server unloading modules via unload_modules_local on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local oleg145-server: modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2