== conf-sanity test 69: replace an OST with the same index ========================================================== 04:54:11 (1787734451) start mds service on oleg454-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg454-server: detected 4 online CPUs by sysfs oleg454-server: MODOPTS_LIBCFS= oleg454-server: Force libcfs to create 2 CPU partitions oleg454-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg454-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg454-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg454-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 723 (time 1787734569.28 total 10.00 last 72.27) - open/close 1533 (time 1787734579.28 total 20.01 last 80.97) - open/close 2328 (time 1787734589.29 total 30.01 last 79.47) - open/close 2985 (time 1787734599.30 total 40.02 last 65.62) - open/close 3778 (time 1787734609.31 total 50.04 last 79.20) - open/close 4576 (time 1787734619.33 total 60.05 last 79.67) - open/close 5159 (time 1787734629.33 total 70.06 last 58.28) - open/close 5890 (time 1787734639.34 total 80.06 last 73.06) - open/close 6569 (time 1787734649.34 total 90.06 last 67.89) - open/close 7408 (time 1787734659.34 total 100.07 last 83.87) - open/close 8173 (time 1787734669.35 total 110.07 last 76.46) - open/close 8856 (time 1787734679.35 total 120.08 last 68.27) - open/close 9397 (time 1787734689.36 total 130.08 last 54.08) total: 10000 open/close in 138.22 seconds: 72.35 ops/second - unlinked 0 (time 1787734702 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 831 (time 1787734782.51 total 10.02 last 82.97) - open/close 1624 (time 1787734792.51 total 20.02 last 79.29) - open/close 2576 (time 1787734802.52 total 30.03 last 95.11) - open/close 3380 (time 1787734812.53 total 40.03 last 80.38) - open/close 4315 (time 1787734822.53 total 50.04 last 93.43) - open/close 5185 (time 1787734832.53 total 60.04 last 86.99) - open/close 6044 (time 1787734842.54 total 70.04 last 85.86) - open/close 6972 (time 1787734852.56 total 80.07 last 92.57) - open/close 7826 (time 1787734862.58 total 90.08 last 85.30) - open/close 8701 (time 1787734872.58 total 100.08 last 87.49) - open/close 9519 (time 1787734882.59 total 110.09 last 81.70) total: 10000 open/close in 115.78 seconds: 86.37 ops/second - unlinked 0 (time 1787734892 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 727 (time 1787734979.02 total 10.02 last 72.56) - open/close 1514 (time 1787734989.04 total 20.04 last 78.53) - open/close 2119 (time 1787734999.07 total 30.07 last 60.30) - open/close 2819 (time 1787735009.09 total 40.09 last 69.87) - open/close 3542 (time 1787735019.11 total 50.12 last 72.13) - open/close 4237 (time 1787735029.12 total 60.12 last 69.46) - open/close 4927 (time 1787735039.13 total 70.13 last 68.95) - open/close 5735 (time 1787735049.14 total 80.14 last 80.73) - open/close 6557 (time 1787735059.14 total 90.14 last 82.20) - open/close 7364 (time 1787735069.15 total 100.15 last 80.62) - open/close 8075 (time 1787735079.17 total 110.18 last 70.91) - open/close 8804 (time 1787735089.18 total 120.18 last 72.84) - open/close 9753 (time 1787735099.18 total 130.18 last 94.90) total: 10000 open/close in 133.03 seconds: 75.17 ops/second - unlinked 0 (time 1787735106 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 859 (time 1787735185.45 total 10.01 last 85.79) - open/close 1674 (time 1787735195.46 total 20.03 last 81.37) - open/close 2529 (time 1787735205.47 total 30.03 last 85.46) - open/close 3408 (time 1787735215.47 total 40.03 last 87.89) - open/close 4265 (time 1787735225.48 total 50.04 last 85.64) - open/close 5037 (time 1787735235.48 total 60.04 last 77.19) - open/close 5981 (time 1787735245.49 total 70.05 last 94.33) - open/close 6777 (time 1787735255.49 total 80.06 last 79.55) - open/close 7608 (time 1787735265.50 total 90.06 last 83.04) - open/close 8438 (time 1787735275.52 total 100.08 last 82.84) - open/close 8999 (time 1787735285.53 total 110.09 last 56.06) - open/close 9679 (time 1787735295.53 total 120.09 last 67.98) total: 10000 open/close in 124.17 seconds: 80.53 ops/second - unlinked 0 (time 1787735304 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 689 (time 1787735388.57 total 10.02 last 68.79) - open/close 1443 (time 1787735398.58 total 20.02 last 75.37) - open/close 2080 (time 1787735408.59 total 30.03 last 63.62) - open/close 3061 (time 1787735418.59 total 40.04 last 98.07) - open/close 3839 (time 1787735428.60 total 50.04 last 77.75) - open/close 4737 (time 1787735438.60 total 60.05 last 89.78) - open/close 5584 (time 1787735448.62 total 70.06 last 84.55) - open/close 6463 (time 1787735458.63 total 80.07 last 87.83) - open/close 7321 (time 1787735468.63 total 90.08 last 85.76) - open/close 8198 (time 1787735478.63 total 100.08 last 87.67) - open/close 9130 (time 1787735488.65 total 110.09 last 93.05) total: 10000 open/close in 119.53 seconds: 83.66 ops/second - unlinked 0 (time 1787735502 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 876 (time 1787735577.98 total 10.00 last 87.57) - open/close 1700 (time 1787735587.98 total 20.00 last 82.38) - open/close 2540 (time 1787735597.99 total 30.02 last 83.91) - open/close 3433 (time 1787735608.00 total 40.03 last 89.22) - open/close 4305 (time 1787735618.01 total 50.04 last 87.06) - open/close 5131 (time 1787735628.02 total 60.05 last 82.52) - open/close 5973 (time 1787735638.03 total 70.06 last 84.15) - open/close 6884 (time 1787735648.04 total 80.07 last 91.01) - open/close 7705 (time 1787735658.05 total 90.08 last 82.01) - open/close 8491 (time 1787735668.06 total 100.08 last 78.54) - open/close 8999 (time 1787735678.07 total 110.10 last 50.75) - open/close 9732 (time 1787735688.07 total 120.10 last 73.29) total: 10000 open/close in 122.95 seconds: 81.33 ops/second - unlinked 0 (time 1787735695 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 892 (time 1787735775.76 total 10.01 last 89.07) - open/close 1771 (time 1787735785.77 total 20.02 last 87.83) - open/close 2665 (time 1787735795.77 total 30.02 last 89.38) - open/close 3603 (time 1787735805.78 total 40.03 last 93.72) - open/close 4454 (time 1787735815.78 total 50.04 last 85.08) - open/close 5299 (time 1787735825.80 total 60.06 last 84.34) - open/close 6151 (time 1787735835.80 total 70.06 last 85.17) - open/close 7025 (time 1787735845.80 total 80.06 last 87.39) - open/close 7902 (time 1787735855.80 total 90.06 last 87.70) - open/close 8756 (time 1787735865.81 total 100.06 last 85.38) - open/close 9676 (time 1787735875.81 total 110.07 last 91.94) total: 10000 open/close in 113.90 seconds: 87.79 ops/second - unlinked 0 (time 1787735883 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 834 (time 1787735962.21 total 10.00 last 83.38) - open/close 1669 (time 1787735972.21 total 20.01 last 83.45) - open/close 2514 (time 1787735982.22 total 30.01 last 84.45) - open/close 3379 (time 1787735992.22 total 40.02 last 86.45) - open/close 4294 (time 1787736002.23 total 50.02 last 91.47) - open/close 5173 (time 1787736012.23 total 60.03 last 87.88) - open/close 6094 (time 1787736022.24 total 70.04 last 92.01) - open/close 6915 (time 1787736032.24 total 80.04 last 82.10) - open/close 7774 (time 1787736042.25 total 90.04 last 85.84) - open/close 8715 (time 1787736052.25 total 100.05 last 94.06) - open/close 9615 (time 1787736062.26 total 110.05 last 89.93) total: 10000 open/close in 114.25 seconds: 87.53 ops/second - unlinked 0 (time 1787736070 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 693 (time 1787736150.52 total 10.02 last 69.17) - open/close 1472 (time 1787736160.53 total 20.02 last 77.86) - open/close 2375 (time 1787736170.53 total 30.03 last 90.23) - open/close 3265 (time 1787736180.55 total 40.05 last 88.86) - open/close 4243 (time 1787736190.56 total 50.05 last 97.73) - open/close 5104 (time 1787736200.56 total 60.06 last 86.07) - open/close 5994 (time 1787736210.57 total 70.07 last 88.88) - open/close 6773 (time 1787736220.58 total 80.08 last 77.87) - open/close 7528 (time 1787736230.58 total 90.08 last 75.46) - open/close 8323 (time 1787736240.59 total 100.09 last 79.46) - open/close 8939 (time 1787736250.60 total 110.09 last 61.55) - open/close 9532 (time 1787736260.60 total 120.09 last 59.29) total: 10000 open/close in 126.41 seconds: 79.11 ops/second - unlinked 0 (time 1787736271 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 835 (time 1787736349.37 total 10.00 last 83.48) - open/close 1657 (time 1787736359.38 total 20.01 last 82.16) - open/close 2589 (time 1787736369.38 total 30.01 last 93.16) - open/close 3486 (time 1787736379.41 total 40.04 last 89.45) - open/close 4356 (time 1787736389.41 total 50.04 last 86.96) - open/close 5112 (time 1787736399.42 total 60.05 last 75.54) - open/close 6068 (time 1787736409.43 total 70.06 last 95.48) - open/close 6823 (time 1787736419.43 total 80.07 last 75.47) - open/close 7623 (time 1787736429.44 total 90.07 last 79.98) - open/close 8379 (time 1787736439.45 total 100.08 last 75.52) - open/close 9167 (time 1787736449.45 total 110.09 last 78.74) - open/close 9899 (time 1787736459.46 total 120.09 last 73.14) total: 10000 open/close in 121.19 seconds: 82.51 ops/second - unlinked 0 (time 1787736464 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.154@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg454-server: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg454-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg454-server: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg454-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.154@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.154@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.154@tcp sys.timeout=20 start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg454-server: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg454-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg454-server: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg454-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg454-server unloading modules via unload_modules_local on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local oleg454-server: modules unloaded. pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2