== conf-sanity test 69: replace an OST with the same index ========================================================== 08:40:57 (1787143257) start mds service on oleg260-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 732 (time 1787143388.14 total 10.00 last 73.19) - open/close 1431 (time 1787143398.16 total 20.03 last 69.70) - open/close 2092 (time 1787143408.17 total 30.03 last 66.08) - open/close 2826 (time 1787143418.18 total 40.04 last 73.33) - open/close 3525 (time 1787143428.18 total 50.04 last 69.88) - open/close 4406 (time 1787143438.20 total 60.06 last 87.94) - open/close 5278 (time 1787143448.20 total 70.06 last 87.20) - open/close 5870 (time 1787143458.20 total 80.07 last 59.18) - open/close 6558 (time 1787143468.22 total 90.09 last 68.66) - open/close 7395 (time 1787143478.24 total 100.11 last 83.53) - open/close 8232 (time 1787143488.24 total 110.11 last 83.68) - open/close 9036 (time 1787143498.25 total 120.12 last 80.33) - open/close 9807 (time 1787143508.26 total 130.12 last 77.08) total: 10000 open/close in 132.19 seconds: 75.65 ops/second - unlinked 0 (time 1787143514 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 793 (time 1787143600.46 total 10.00 last 79.29) - open/close 1711 (time 1787143610.46 total 20.00 last 91.80) - open/close 2714 (time 1787143620.46 total 30.01 last 100.26) - open/close 3239 (time 1787143630.47 total 40.02 last 52.44) - open/close 3802 (time 1787143640.48 total 50.03 last 56.24) - open/close 4776 (time 1787143650.49 total 60.03 last 97.36) - open/close 5595 (time 1787143660.49 total 70.03 last 81.89) - open/close 6695 (time 1787143670.50 total 80.04 last 109.95) - open/close 7591 (time 1787143680.50 total 90.04 last 89.59) - open/close 8618 (time 1787143690.50 total 100.04 last 102.68) - open/close 9592 (time 1787143700.51 total 110.05 last 97.29) total: 10000 open/close in 114.49 seconds: 87.34 ops/second - unlinked 0 (time 1787143709 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 839 (time 1787143787.21 total 10.01 last 83.84) - open/close 1810 (time 1787143797.22 total 20.02 last 96.99) - open/close 2610 (time 1787143807.25 total 30.05 last 79.77) - open/close 3505 (time 1787143817.25 total 40.05 last 89.45) - open/close 4391 (time 1787143827.26 total 50.06 last 88.55) - open/close 5104 (time 1787143837.28 total 60.08 last 71.13) - open/close 6202 (time 1787143847.29 total 70.09 last 109.70) - open/close 7030 (time 1787143857.30 total 80.10 last 82.76) - open/close 7994 (time 1787143867.30 total 90.10 last 96.37) - open/close 8804 (time 1787143877.31 total 100.11 last 80.92) - open/close 9368 (time 1787143887.32 total 110.12 last 56.32) total: 10000 open/close in 118.49 seconds: 84.39 ops/second - unlinked 0 (time 1787143900 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 929 (time 1787143977.55 total 10.01 last 92.77) - open/close 1769 (time 1787143987.57 total 20.03 last 83.84) - open/close 2520 (time 1787143997.60 total 30.06 last 74.91) - open/close 2928 (time 1787144007.60 total 40.06 last 40.79) - open/close 3618 (time 1787144017.61 total 50.07 last 68.94) - open/close 4600 (time 1787144027.61 total 60.07 last 98.17) - open/close 5666 (time 1787144037.62 total 70.08 last 106.52) - open/close 6648 (time 1787144047.62 total 80.08 last 98.17) - open/close 7609 (time 1787144057.63 total 90.09 last 96.04) - open/close 8764 (time 1787144067.64 total 100.10 last 115.44) - open/close 9890 (time 1787144077.64 total 110.10 last 112.56) total: 10000 open/close in 111.00 seconds: 90.09 ops/second - unlinked 0 (time 1787144082 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 951 (time 1787144160.23 total 10.01 last 95.05) - open/close 1793 (time 1787144170.24 total 20.01 last 84.14) - open/close 2593 (time 1787144180.24 total 30.02 last 79.96) - open/close 3572 (time 1787144190.24 total 40.02 last 97.86) - open/close 4685 (time 1787144200.25 total 50.03 last 111.26) - open/close 5682 (time 1787144210.25 total 60.03 last 99.65) - open/close 6533 (time 1787144220.26 total 70.04 last 85.05) - open/close 7245 (time 1787144230.27 total 80.05 last 71.12) - open/close 8025 (time 1787144240.28 total 90.05 last 77.97) - open/close 8627 (time 1787144250.28 total 100.06 last 60.17) - open/close 9216 (time 1787144260.29 total 110.07 last 58.84) - open/close 9917 (time 1787144270.32 total 120.10 last 69.87) total: 10000 open/close in 121.21 seconds: 82.50 ops/second - unlinked 0 (time 1787144275 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 536 (time 1787144371.21 total 10.02 last 53.51) - open/close 1367 (time 1787144381.22 total 20.02 last 83.03) - open/close 2468 (time 1787144391.23 total 30.03 last 110.00) - open/close 3344 (time 1787144401.23 total 40.04 last 87.59) - open/close 4310 (time 1787144411.23 total 50.04 last 96.54) - open/close 5237 (time 1787144421.24 total 60.04 last 92.68) - open/close 6108 (time 1787144431.24 total 70.05 last 87.07) - open/close 7042 (time 1787144441.25 total 80.05 last 93.33) - open/close 8035 (time 1787144451.26 total 90.07 last 99.13) - open/close 8681 (time 1787144461.27 total 100.08 last 64.56) - open/close 9584 (time 1787144471.28 total 110.08 last 90.26) total: 10000 open/close in 114.48 seconds: 87.35 ops/second - unlinked 0 (time 1787144479 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1090 (time 1787144547.83 total 10.01 last 108.91) - open/close 2102 (time 1787144557.84 total 20.01 last 101.18) - open/close 3124 (time 1787144567.84 total 30.01 last 102.17) - open/close 4072 (time 1787144577.84 total 40.02 last 94.75) - open/close 5043 (time 1787144587.85 total 50.02 last 97.05) - open/close 5965 (time 1787144597.85 total 60.02 last 92.19) - open/close 7091 (time 1787144607.86 total 70.03 last 112.53) - open/close 7892 (time 1787144617.87 total 80.04 last 79.99) - open/close 8888 (time 1787144627.87 total 90.04 last 99.59) - open/close 9981 (time 1787144637.87 total 100.05 last 109.27) total: 10000 open/close in 100.23 seconds: 99.77 ops/second - unlinked 0 (time 1787144641 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 933 (time 1787144713.28 total 10.00 last 93.27) - open/close 1879 (time 1787144723.28 total 20.01 last 94.56) - open/close 2844 (time 1787144733.28 total 30.01 last 96.48) - open/close 3744 (time 1787144743.30 total 40.03 last 89.81) - open/close 4676 (time 1787144753.32 total 50.04 last 93.10) - open/close 5761 (time 1787144763.33 total 60.05 last 108.37) - open/close 6890 (time 1787144773.33 total 70.06 last 112.87) - open/close 7936 (time 1787144783.34 total 80.06 last 104.53) - open/close 8912 (time 1787144793.34 total 90.07 last 97.53) - open/close 9919 (time 1787144803.35 total 100.07 last 100.66) total: 10000 open/close in 100.81 seconds: 99.20 ops/second - unlinked 0 (time 1787144808 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 822 (time 1787144880.12 total 10.01 last 82.12) - open/close 1697 (time 1787144890.14 total 20.02 last 87.40) - open/close 2738 (time 1787144900.14 total 30.03 last 104.03) - open/close 3500 (time 1787144910.15 total 40.03 last 76.16) - open/close 4340 (time 1787144920.16 total 50.04 last 83.93) - open/close 5220 (time 1787144930.16 total 60.05 last 87.94) - open/close 6065 (time 1787144940.18 total 70.07 last 84.36) - open/close 6813 (time 1787144950.18 total 80.07 last 74.76) - open/close 7802 (time 1787144960.19 total 90.07 last 98.86) - open/close 8703 (time 1787144970.19 total 100.08 last 90.09) - open/close 9494 (time 1787144980.22 total 110.11 last 78.82) total: 10000 open/close in 118.77 seconds: 84.20 ops/second - unlinked 0 (time 1787144992 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 997 (time 1787145060.60 total 10.01 last 99.62) - open/close 1960 (time 1787145070.61 total 20.01 last 96.28) - open/close 3084 (time 1787145080.62 total 30.02 last 112.29) - open/close 4139 (time 1787145090.62 total 40.02 last 105.48) - open/close 5019 (time 1787145100.63 total 50.03 last 87.93) - open/close 5856 (time 1787145110.63 total 60.03 last 83.68) - open/close 6727 (time 1787145120.63 total 70.03 last 87.09) - open/close 7540 (time 1787145130.63 total 80.04 last 81.28) - open/close 8607 (time 1787145140.64 total 90.04 last 106.66) - open/close 9575 (time 1787145150.65 total 100.05 last 96.68) total: 10000 open/close in 105.34 seconds: 94.93 ops/second - unlinked 0 (time 1787145160 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.160@tcp sys.timeout=20 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2