== conf-sanity test 69: replace an OST with the same index ========================================================== 19:49:56 (1787788196) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 789 (time 1787788312.81 total 10.01 last 78.79) - open/close 1553 (time 1787788322.82 total 20.02 last 76.32) - open/close 2286 (time 1787788332.84 total 30.04 last 73.21) - open/close 3052 (time 1787788342.84 total 40.05 last 76.53) - open/close 3821 (time 1787788352.85 total 50.05 last 76.89) - open/close 4679 (time 1787788362.85 total 60.06 last 85.72) - open/close 5507 (time 1787788372.86 total 70.06 last 82.73) - open/close 6360 (time 1787788382.87 total 80.07 last 85.27) - open/close 7287 (time 1787788392.88 total 90.08 last 92.61) - open/close 8131 (time 1787788402.90 total 100.10 last 84.17) - open/close 8757 (time 1787788412.91 total 110.11 last 62.55) - open/close 9486 (time 1787788422.91 total 120.11 last 72.89) total: 10000 open/close in 126.68 seconds: 78.94 ops/second - unlinked 0 (time 1787788434 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 761 (time 1787788524.13 total 10.01 last 76.05) - open/close 1633 (time 1787788534.13 total 20.02 last 87.12) - open/close 2466 (time 1787788544.13 total 30.02 last 83.29) - open/close 3300 (time 1787788554.14 total 40.02 last 83.35) - open/close 4010 (time 1787788564.15 total 50.03 last 70.95) - open/close 4876 (time 1787788574.15 total 60.04 last 86.54) - open/close 5732 (time 1787788584.15 total 70.04 last 85.60) - open/close 6467 (time 1787788594.18 total 80.06 last 73.34) - open/close 7253 (time 1787788604.18 total 90.06 last 78.59) - open/close 8160 (time 1787788614.19 total 100.07 last 90.61) - open/close 9079 (time 1787788624.19 total 110.07 last 91.85) - open/close 9818 (time 1787788634.20 total 120.08 last 73.88) total: 10000 open/close in 121.99 seconds: 81.98 ops/second - unlinked 0 (time 1787788640 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 831 (time 1787788726.52 total 10.01 last 83.04) - open/close 1668 (time 1787788736.53 total 20.02 last 83.62) - open/close 2558 (time 1787788746.53 total 30.02 last 88.97) - open/close 3397 (time 1787788756.54 total 40.03 last 83.84) - open/close 4266 (time 1787788766.55 total 50.03 last 86.85) - open/close 5149 (time 1787788776.55 total 60.04 last 88.26) - open/close 5953 (time 1787788786.55 total 70.04 last 80.38) - open/close 6738 (time 1787788796.55 total 80.04 last 78.49) - open/close 7541 (time 1787788806.56 total 90.05 last 80.24) - open/close 8434 (time 1787788816.56 total 100.05 last 89.29) - open/close 9203 (time 1787788826.57 total 110.06 last 76.83) total: 10000 open/close in 119.33 seconds: 83.80 ops/second - unlinked 0 (time 1787788840 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 746 (time 1787788928.69 total 10.02 last 74.46) - open/close 1529 (time 1787788938.69 total 20.02 last 78.29) - open/close 2389 (time 1787788948.70 total 30.03 last 85.93) - open/close 3249 (time 1787788958.70 total 40.03 last 85.99) - open/close 4065 (time 1787788968.71 total 50.04 last 81.55) - open/close 4879 (time 1787788978.72 total 60.05 last 81.32) - open/close 5571 (time 1787788988.72 total 70.05 last 69.18) - open/close 6428 (time 1787788998.74 total 80.07 last 85.51) - open/close 7196 (time 1787789008.74 total 90.07 last 76.79) - open/close 7903 (time 1787789018.75 total 100.08 last 70.63) - open/close 8632 (time 1787789028.75 total 110.08 last 72.90) - open/close 9395 (time 1787789038.75 total 120.08 last 76.28) total: 10000 open/close in 128.39 seconds: 77.89 ops/second - unlinked 0 (time 1787789052 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 724 (time 1787789138.19 total 10.01 last 72.31) - open/close 1454 (time 1787789148.19 total 20.01 last 72.99) - open/close 2146 (time 1787789158.19 total 30.02 last 69.18) - open/close 2922 (time 1787789168.21 total 40.03 last 77.51) - open/close 3733 (time 1787789178.21 total 50.04 last 81.04) - open/close 4343 (time 1787789188.22 total 60.04 last 60.95) - open/close 5148 (time 1787789198.22 total 70.05 last 80.48) - open/close 5943 (time 1787789208.23 total 80.06 last 79.43) - open/close 6795 (time 1787789218.24 total 90.06 last 85.17) - open/close 7613 (time 1787789228.24 total 100.06 last 81.77) - open/close 8392 (time 1787789238.24 total 110.06 last 77.90) - open/close 9179 (time 1787789248.24 total 120.07 last 78.66) total: 10000 open/close in 130.04 seconds: 76.90 ops/second - unlinked 0 (time 1787789263 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 782 (time 1787789361.23 total 10.00 last 78.19) - open/close 1555 (time 1787789371.23 total 20.01 last 77.26) - open/close 2295 (time 1787789381.25 total 30.02 last 73.88) - open/close 3105 (time 1787789391.26 total 40.03 last 80.92) - open/close 3893 (time 1787789401.27 total 50.05 last 78.67) - open/close 4602 (time 1787789411.28 total 60.06 last 70.84) - open/close 5366 (time 1787789421.29 total 70.07 last 76.35) - open/close 6130 (time 1787789431.30 total 80.08 last 76.32) - open/close 7187 (time 1787789441.31 total 90.09 last 105.60) - open/close 7977 (time 1787789451.33 total 100.10 last 78.85) - open/close 8697 (time 1787789461.34 total 110.11 last 71.93) - open/close 9435 (time 1787789471.34 total 120.12 last 73.79) total: 10000 open/close in 127.65 seconds: 78.34 ops/second - unlinked 0 (time 1787789483 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 804 (time 1787789562.26 total 10.01 last 80.35) - open/close 1548 (time 1787789572.27 total 20.01 last 74.35) - open/close 2221 (time 1787789582.28 total 30.03 last 67.21) - open/close 2920 (time 1787789592.28 total 40.03 last 69.86) - open/close 3782 (time 1787789602.29 total 50.04 last 86.17) - open/close 4633 (time 1787789612.29 total 60.04 last 85.07) - open/close 5520 (time 1787789622.30 total 70.05 last 88.63) - open/close 6424 (time 1787789632.30 total 80.05 last 90.37) - open/close 7306 (time 1787789642.31 total 90.06 last 88.13) - open/close 8217 (time 1787789652.31 total 100.06 last 91.07) - open/close 9221 (time 1787789662.32 total 110.07 last 100.35) total: 10000 open/close in 119.03 seconds: 84.01 ops/second - unlinked 0 (time 1787789675 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 905 (time 1787789756.17 total 10.00 last 90.46) - open/close 1829 (time 1787789766.18 total 20.02 last 92.29) - open/close 2836 (time 1787789776.19 total 30.02 last 100.65) - open/close 3704 (time 1787789786.20 total 40.03 last 86.69) - open/close 4597 (time 1787789796.21 total 50.04 last 89.25) - open/close 5566 (time 1787789806.21 total 60.04 last 96.87) - open/close 6383 (time 1787789816.22 total 70.05 last 81.61) - open/close 7281 (time 1787789826.23 total 80.06 last 89.74) - open/close 8191 (time 1787789836.23 total 90.06 last 90.99) - open/close 9022 (time 1787789846.25 total 100.08 last 82.97) - open/close 9819 (time 1787789856.25 total 110.08 last 79.69) total: 10000 open/close in 111.94 seconds: 89.33 ops/second - unlinked 0 (time 1787789862 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 834 (time 1787789940.25 total 10.00 last 83.38) - open/close 1714 (time 1787789950.25 total 20.00 last 87.99) - open/close 2537 (time 1787789960.27 total 30.02 last 82.16) - open/close 3423 (time 1787789970.28 total 40.03 last 88.49) - open/close 4186 (time 1787789980.30 total 50.05 last 76.15) - open/close 4984 (time 1787789990.30 total 60.05 last 79.79) - open/close 5846 (time 1787790000.31 total 70.06 last 86.12) - open/close 6830 (time 1787790010.32 total 80.07 last 98.35) - open/close 7832 (time 1787790020.32 total 90.07 last 100.14) - open/close 8768 (time 1787790030.32 total 100.07 last 93.60) - open/close 9549 (time 1787790040.33 total 110.08 last 78.08) total: 10000 open/close in 114.82 seconds: 87.09 ops/second - unlinked 0 (time 1787790049 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 862 (time 1787790127.67 total 10.01 last 86.14) - open/close 1826 (time 1787790137.67 total 20.01 last 96.40) - open/close 2639 (time 1787790147.67 total 30.01 last 81.30) - open/close 3498 (time 1787790157.68 total 40.01 last 85.84) - open/close 4243 (time 1787790167.69 total 50.03 last 74.42) - open/close 5019 (time 1787790177.69 total 60.03 last 77.59) - open/close 5864 (time 1787790187.70 total 70.03 last 84.44) - open/close 6658 (time 1787790197.70 total 80.03 last 79.38) - open/close 7472 (time 1787790207.71 total 90.05 last 81.30) - open/close 8337 (time 1787790217.71 total 100.05 last 86.50) - open/close 9261 (time 1787790227.72 total 110.06 last 92.31) total: 10000 open/close in 118.35 seconds: 84.50 ops/second - unlinked 0 (time 1787790239 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2