== conf-sanity test 69: replace an OST with the same index ========================================================== 19:06:51 (1785452811) start mds service on oleg627-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing load_modules_local oleg627-server: Loading modules from /home/green/git/lustre-release/lustre oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg627-server: detected 4 online CPUs by sysfs oleg627-server: MODOPTS_LIBCFS= oleg627-server: Force libcfs to create 2 CPU partitions oleg627-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg627-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg627-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg627-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg627-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 807 (time 1785452918.51 total 10.01 last 80.65) - open/close 1610 (time 1785452928.51 total 20.01 last 80.30) - open/close 2413 (time 1785452938.51 total 30.01 last 80.24) - open/close 3257 (time 1785452948.52 total 40.01 last 84.39) - open/close 4009 (time 1785452958.53 total 50.03 last 75.12) - open/close 4810 (time 1785452968.53 total 60.03 last 80.05) - open/close 5618 (time 1785452978.53 total 70.03 last 80.79) - open/close 6450 (time 1785452988.55 total 80.04 last 83.11) - open/close 7248 (time 1785452998.55 total 90.05 last 79.75) - open/close 8106 (time 1785453008.56 total 100.05 last 85.77) - open/close 8909 (time 1785453018.56 total 110.06 last 80.26) - open/close 9652 (time 1785453028.58 total 120.08 last 74.12) total: 10000 open/close in 124.68 seconds: 80.20 ops/second - unlinked 0 (time 1785453036 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 862 (time 1785453117.80 total 10.00 last 86.18) - open/close 1706 (time 1785453127.81 total 20.01 last 84.32) - open/close 2556 (time 1785453137.82 total 30.02 last 84.94) - open/close 3399 (time 1785453147.83 total 40.03 last 84.24) - open/close 4225 (time 1785453157.85 total 50.05 last 82.45) - open/close 5021 (time 1785453167.85 total 60.05 last 79.54) - open/close 5857 (time 1785453177.87 total 70.07 last 83.46) - open/close 6659 (time 1785453187.87 total 80.07 last 80.20) - open/close 7503 (time 1785453197.87 total 90.07 last 84.37) - open/close 8335 (time 1785453207.88 total 100.08 last 83.13) - open/close 9138 (time 1785453217.90 total 110.10 last 80.17) - open/close 9793 (time 1785453227.91 total 120.11 last 65.45) total: 10000 open/close in 122.59 seconds: 81.57 ops/second - unlinked 0 (time 1785453235 ; total 1 ; last 1) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 847 (time 1785453322.06 total 10.01 last 84.64) - open/close 1694 (time 1785453332.07 total 20.01 last 84.63) - open/close 2523 (time 1785453342.07 total 30.02 last 82.84) - open/close 3261 (time 1785453352.08 total 40.03 last 73.73) - open/close 4136 (time 1785453362.09 total 50.04 last 87.43) - open/close 5010 (time 1785453372.10 total 60.04 last 87.36) - open/close 5885 (time 1785453382.11 total 70.06 last 87.34) - open/close 6685 (time 1785453392.12 total 80.07 last 79.94) - open/close 7474 (time 1785453402.13 total 90.08 last 78.83) - open/close 8173 (time 1785453412.14 total 100.08 last 69.87) - open/close 9008 (time 1785453422.16 total 110.11 last 83.31) - open/close 9802 (time 1785453432.16 total 120.11 last 79.36) total: 10000 open/close in 122.69 seconds: 81.51 ops/second - unlinked 0 (time 1785453438 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 777 (time 1785453526.44 total 10.00 last 77.67) - open/close 1512 (time 1785453536.44 total 20.01 last 73.45) - open/close 2291 (time 1785453546.46 total 30.03 last 77.77) - open/close 3135 (time 1785453556.48 total 40.05 last 84.23) - open/close 3879 (time 1785453566.49 total 50.06 last 74.31) - open/close 4622 (time 1785453576.50 total 60.07 last 74.24) - open/close 5393 (time 1785453586.51 total 70.08 last 77.03) - open/close 6100 (time 1785453596.52 total 80.09 last 70.64) - open/close 6860 (time 1785453606.53 total 90.09 last 75.93) - open/close 7606 (time 1785453616.54 total 100.11 last 74.49) - open/close 8310 (time 1785453626.55 total 110.12 last 70.36) - open/close 9112 (time 1785453636.56 total 120.13 last 80.08) - open/close 9941 (time 1785453646.57 total 130.13 last 82.87) total: 10000 open/close in 130.77 seconds: 76.47 ops/second - unlinked 0 (time 1785453651 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 854 (time 1785453743.92 total 10.00 last 85.36) - open/close 1672 (time 1785453753.93 total 20.01 last 81.72) - open/close 2499 (time 1785453763.94 total 30.02 last 82.63) - open/close 3390 (time 1785453773.94 total 40.03 last 89.06) - open/close 4251 (time 1785453783.95 total 50.04 last 86.03) - open/close 5064 (time 1785453793.95 total 60.04 last 81.26) - open/close 5841 (time 1785453803.96 total 70.04 last 77.66) - open/close 6675 (time 1785453813.96 total 80.05 last 83.35) - open/close 7471 (time 1785453823.96 total 90.05 last 79.60) - open/close 8257 (time 1785453833.97 total 100.05 last 78.58) - open/close 9081 (time 1785453843.98 total 110.06 last 82.31) - open/close 9861 (time 1785453854.00 total 120.09 last 77.81) total: 10000 open/close in 122.33 seconds: 81.74 ops/second - unlinked 0 (time 1785453861 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 859 (time 1785453947.19 total 10.00 last 85.90) - open/close 1620 (time 1785453957.20 total 20.01 last 76.02) - open/close 2365 (time 1785453967.21 total 30.02 last 74.44) - open/close 3188 (time 1785453977.22 total 40.03 last 82.19) - open/close 3981 (time 1785453987.23 total 50.04 last 79.24) - open/close 4756 (time 1785453997.24 total 60.05 last 77.44) - open/close 5595 (time 1785454007.25 total 70.06 last 83.82) - open/close 6348 (time 1785454017.25 total 80.06 last 75.26) - open/close 7121 (time 1785454027.25 total 90.07 last 77.28) - open/close 7809 (time 1785454037.26 total 100.07 last 68.74) - open/close 8567 (time 1785454047.27 total 110.08 last 75.76) - open/close 9353 (time 1785454057.27 total 120.08 last 78.56) total: 10000 open/close in 129.37 seconds: 77.30 ops/second - unlinked 0 (time 1785454070 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 812 (time 1785454156.55 total 10.01 last 81.14) - open/close 1666 (time 1785454166.55 total 20.01 last 85.36) - open/close 2498 (time 1785454176.56 total 30.02 last 83.12) - open/close 3368 (time 1785454186.57 total 40.03 last 86.93) - open/close 4210 (time 1785454196.57 total 50.03 last 84.16) - open/close 5044 (time 1785454206.58 total 60.04 last 83.32) - open/close 5912 (time 1785454216.59 total 70.05 last 86.77) - open/close 6703 (time 1785454226.59 total 80.05 last 79.08) - open/close 7521 (time 1785454236.61 total 90.07 last 81.62) - open/close 8364 (time 1785454246.61 total 100.07 last 84.29) - open/close 9136 (time 1785454256.63 total 110.09 last 77.06) - open/close 9975 (time 1785454266.64 total 120.10 last 83.83) total: 10000 open/close in 120.32 seconds: 83.11 ops/second - unlinked 0 (time 1785454271 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 866 (time 1785454357.89 total 10.01 last 86.48) - open/close 1707 (time 1785454367.90 total 20.02 last 84.04) - open/close 2513 (time 1785454377.90 total 30.02 last 80.60) - open/close 3341 (time 1785454387.90 total 40.02 last 82.79) - open/close 4017 (time 1785454397.90 total 50.03 last 67.58) - open/close 4761 (time 1785454407.91 total 60.03 last 74.38) - open/close 5488 (time 1785454417.94 total 70.06 last 72.47) - open/close 6184 (time 1785454427.95 total 80.07 last 69.54) - open/close 6880 (time 1785454437.96 total 90.08 last 69.54) - open/close 7622 (time 1785454447.96 total 100.08 last 74.19) - open/close 8398 (time 1785454457.96 total 110.09 last 77.55) - open/close 9214 (time 1785454467.97 total 120.10 last 81.52) - open/close 9919 (time 1785454477.98 total 130.11 last 70.44) total: 10000 open/close in 131.17 seconds: 76.24 ops/second - unlinked 0 (time 1785454484 ; total 1 ; last 1) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 673 (time 1785454573.86 total 10.01 last 67.23) - open/close 1302 (time 1785454583.89 total 20.04 last 62.72) - open/close 2034 (time 1785454593.90 total 30.05 last 73.13) - open/close 2852 (time 1785454603.91 total 40.06 last 81.72) - open/close 3704 (time 1785454613.92 total 50.07 last 85.13) - open/close 4522 (time 1785454623.92 total 60.07 last 81.77) - open/close 5319 (time 1785454633.92 total 70.07 last 79.68) - open/close 6162 (time 1785454643.94 total 80.09 last 84.16) - open/close 6972 (time 1785454653.94 total 90.09 last 80.98) - open/close 7800 (time 1785454663.95 total 100.10 last 82.76) - open/close 8572 (time 1785454673.96 total 110.11 last 77.11) - open/close 9319 (time 1785454683.96 total 120.11 last 74.68) total: 10000 open/close in 128.66 seconds: 77.72 ops/second - unlinked 0 (time 1785454697 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 856 (time 1785454786.29 total 10.01 last 85.55) - open/close 1691 (time 1785454796.30 total 20.01 last 83.44) - open/close 2461 (time 1785454806.30 total 30.01 last 77.00) - open/close 3188 (time 1785454816.30 total 40.02 last 72.67) - open/close 3993 (time 1785454826.31 total 50.02 last 80.46) - open/close 4811 (time 1785454836.32 total 60.04 last 81.70) - open/close 5590 (time 1785454846.33 total 70.04 last 77.85) - open/close 6398 (time 1785454856.33 total 80.05 last 80.75) - open/close 7207 (time 1785454866.35 total 90.06 last 80.79) - open/close 8039 (time 1785454876.35 total 100.07 last 83.14) - open/close 8827 (time 1785454886.37 total 110.08 last 78.68) - open/close 9637 (time 1785454896.37 total 120.09 last 80.96) total: 10000 open/close in 124.07 seconds: 80.60 ops/second - unlinked 0 (time 1785454904 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.127@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg627-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg627-server: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg627-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg627-server: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg627-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.127@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.127@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.127@tcp sys.timeout=20 start ost1 service on oleg627-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg627-server: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg627-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg627-server: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg627-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg627-server unloading modules via unload_modules_local on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing unload_modules_local oleg627-server: modules unloaded. pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2