== conf-sanity test 69: replace an OST with the same index ========================================================== 11:04:02 (1783350242) start mds service on oleg221-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg221-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg221-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 874 (time 1783350357.26 total 10.00 last 87.38) - open/close 1703 (time 1783350367.27 total 20.01 last 82.84) - open/close 2473 (time 1783350377.27 total 30.01 last 76.98) - open/close 3305 (time 1783350387.28 total 40.02 last 83.10) - open/close 4010 (time 1783350397.28 total 50.03 last 70.48) - open/close 4701 (time 1783350407.30 total 60.04 last 68.98) - open/close 5467 (time 1783350417.31 total 70.05 last 76.55) - open/close 5999 (time 1783350427.31 total 80.05 last 53.19) - open/close 6834 (time 1783350437.31 total 90.06 last 83.46) - open/close 7705 (time 1783350447.31 total 100.06 last 87.10) - open/close 8460 (time 1783350457.32 total 110.07 last 75.44) - open/close 9140 (time 1783350467.33 total 120.07 last 67.95) - open/close 9846 (time 1783350477.33 total 130.08 last 70.56) total: 10000 open/close in 132.36 seconds: 75.55 ops/second - unlinked 0 (time 1783350484 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 624 (time 1783350573.23 total 10.01 last 62.31) - open/close 1301 (time 1783350583.23 total 20.02 last 67.68) - open/close 2011 (time 1783350593.23 total 30.02 last 70.99) - open/close 2799 (time 1783350603.24 total 40.02 last 78.76) - open/close 3523 (time 1783350613.24 total 50.03 last 72.35) - open/close 4385 (time 1783350623.25 total 60.04 last 86.12) - open/close 5148 (time 1783350633.26 total 70.04 last 76.26) - open/close 5838 (time 1783350643.26 total 80.05 last 68.96) - open/close 6629 (time 1783350653.26 total 90.05 last 79.10) - open/close 7345 (time 1783350663.27 total 100.05 last 71.57) - open/close 8135 (time 1783350673.28 total 110.06 last 78.93) - open/close 8971 (time 1783350683.29 total 120.08 last 83.51) - open/close 9730 (time 1783350693.29 total 130.08 last 75.86) total: 10000 open/close in 133.32 seconds: 75.01 ops/second - unlinked 0 (time 1783350700 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 870 (time 1783350788.18 total 10.01 last 86.90) - open/close 1685 (time 1783350798.19 total 20.02 last 81.42) - open/close 2553 (time 1783350808.19 total 30.03 last 86.75) - open/close 3390 (time 1783350818.21 total 40.04 last 83.59) - open/close 4143 (time 1783350828.21 total 50.05 last 75.24) - open/close 5061 (time 1783350838.22 total 60.05 last 91.75) - open/close 5871 (time 1783350848.23 total 70.06 last 80.92) - open/close 6776 (time 1783350858.23 total 80.06 last 90.47) - open/close 7629 (time 1783350868.23 total 90.07 last 85.28) - open/close 8491 (time 1783350878.24 total 100.07 last 86.16) - open/close 9457 (time 1783350888.25 total 110.08 last 96.51) total: 10000 open/close in 115.88 seconds: 86.29 ops/second - unlinked 0 (time 1783350897 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 926 (time 1783350974.77 total 10.01 last 92.54) - open/close 1905 (time 1783350984.78 total 20.02 last 97.80) - open/close 2868 (time 1783350994.79 total 30.02 last 96.24) - open/close 3890 (time 1783351004.79 total 40.03 last 102.17) - open/close 4813 (time 1783351014.80 total 50.03 last 92.21) - open/close 5721 (time 1783351024.81 total 60.04 last 90.72) - open/close 6659 (time 1783351034.81 total 70.05 last 93.76) - open/close 7689 (time 1783351044.82 total 80.05 last 102.95) - open/close 8691 (time 1783351054.82 total 90.06 last 100.17) - open/close 9623 (time 1783351064.83 total 100.07 last 93.10) total: 10000 open/close in 104.06 seconds: 96.10 ops/second - unlinked 0 (time 1783351072 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 930 (time 1783351142.11 total 10.00 last 92.99) - open/close 1826 (time 1783351152.12 total 20.01 last 89.52) - open/close 2730 (time 1783351162.12 total 30.01 last 90.37) - open/close 3695 (time 1783351172.13 total 40.02 last 96.48) - open/close 4669 (time 1783351182.13 total 50.02 last 97.39) - open/close 5675 (time 1783351192.13 total 60.02 last 100.60) - open/close 6750 (time 1783351202.13 total 70.02 last 107.45) - open/close 7712 (time 1783351212.13 total 80.02 last 96.19) - open/close 8674 (time 1783351222.14 total 90.03 last 96.11) - open/close 9600 (time 1783351232.14 total 100.03 last 92.59) total: 10000 open/close in 104.15 seconds: 96.01 ops/second - unlinked 0 (time 1783351239 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 906 (time 1783351312.78 total 10.01 last 90.54) - open/close 1878 (time 1783351322.79 total 20.02 last 97.11) - open/close 2860 (time 1783351332.79 total 30.02 last 98.16) - open/close 3923 (time 1783351342.79 total 40.02 last 106.30) - open/close 4908 (time 1783351352.80 total 50.03 last 98.45) - open/close 5952 (time 1783351362.82 total 60.05 last 104.17) - open/close 6844 (time 1783351372.83 total 70.06 last 89.12) - open/close 7910 (time 1783351382.83 total 80.06 last 106.59) - open/close 8892 (time 1783351392.83 total 90.06 last 98.17) - open/close 9986 (time 1783351402.84 total 100.07 last 109.31) total: 10000 open/close in 100.17 seconds: 99.83 ops/second - unlinked 0 (time 1783351405 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 943 (time 1783351479.08 total 10.01 last 94.25) - open/close 1854 (time 1783351489.09 total 20.02 last 90.99) - open/close 2790 (time 1783351499.10 total 30.02 last 93.54) - open/close 3844 (time 1783351509.10 total 40.02 last 105.39) - open/close 4841 (time 1783351519.11 total 50.03 last 99.64) - open/close 5696 (time 1783351529.11 total 60.03 last 85.46) - open/close 6619 (time 1783351539.11 total 70.04 last 92.28) - open/close 7697 (time 1783351549.12 total 80.04 last 107.76) - open/close 8619 (time 1783351559.12 total 90.04 last 92.20) - open/close 9592 (time 1783351569.12 total 100.04 last 97.29) total: 10000 open/close in 104.44 seconds: 95.75 ops/second - unlinked 0 (time 1783351576 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1026 (time 1783351642.50 total 10.00 last 102.59) - open/close 2048 (time 1783351652.50 total 20.00 last 102.16) - open/close 3128 (time 1783351662.50 total 30.01 last 107.96) - open/close 4107 (time 1783351672.51 total 40.01 last 97.87) - open/close 5158 (time 1783351682.52 total 50.02 last 104.97) - open/close 6166 (time 1783351692.53 total 60.03 last 100.71) - open/close 7212 (time 1783351702.53 total 70.04 last 104.58) - open/close 8421 (time 1783351712.54 total 80.04 last 120.84) - open/close 9546 (time 1783351722.54 total 90.05 last 112.44) total: 10000 open/close in 93.99 seconds: 106.40 ops/second - unlinked 0 (time 1783351728 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 989 (time 1783351794.06 total 10.02 last 98.75) - open/close 2030 (time 1783351804.07 total 20.02 last 104.00) - open/close 3189 (time 1783351814.07 total 30.03 last 115.83) - open/close 4217 (time 1783351824.08 total 40.04 last 102.67) - open/close 5201 (time 1783351834.09 total 50.04 last 98.39) - open/close 6188 (time 1783351844.09 total 60.05 last 98.67) - open/close 7348 (time 1783351854.09 total 70.05 last 115.98) - open/close 8497 (time 1783351864.09 total 80.05 last 114.86) - open/close 9627 (time 1783351874.10 total 90.06 last 112.93) total: 10000 open/close in 93.78 seconds: 106.63 ops/second - unlinked 0 (time 1783351880 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1304 (time 1783351941.63 total 10.01 last 130.32) - open/close 2454 (time 1783351951.63 total 20.01 last 114.98) - open/close 3723 (time 1783351961.64 total 30.01 last 126.83) - open/close 4899 (time 1783351971.65 total 40.03 last 117.46) - open/close 6046 (time 1783351981.65 total 50.03 last 114.69) - open/close 7221 (time 1783351991.65 total 60.03 last 117.48) - open/close 8323 (time 1783352001.66 total 70.03 last 110.16) - open/close 9542 (time 1783352011.66 total 80.04 last 121.82) total: 10000 open/close in 82.98 seconds: 120.51 ops/second - unlinked 0 (time 1783352017 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre On OST0, 17382 used inodes rc=0 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: modules unloaded. pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2