== conf-sanity test 69: replace an OST with the same index ========================================================== 21:39:13 (1781228353) start mds service on oleg435-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg435-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg435-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 692 (time 1781228449.52 total 10.01 last 69.14) - open/close 1426 (time 1781228459.53 total 20.02 last 73.33) - open/close 2057 (time 1781228469.54 total 30.02 last 63.07) - open/close 2699 (time 1781228479.54 total 40.03 last 64.18) - open/close 3424 (time 1781228489.56 total 50.04 last 72.41) - open/close 4162 (time 1781228499.56 total 60.05 last 73.75) - open/close 4910 (time 1781228509.57 total 70.05 last 74.75) - open/close 5676 (time 1781228519.58 total 80.06 last 76.54) - open/close 6401 (time 1781228529.58 total 90.07 last 72.44) - open/close 7080 (time 1781228539.59 total 100.07 last 67.89) - open/close 7805 (time 1781228549.59 total 110.07 last 72.47) - open/close 8488 (time 1781228559.60 total 120.09 last 68.22) - open/close 9155 (time 1781228569.62 total 130.11 last 66.57) - open/close 9755 (time 1781228579.64 total 140.12 last 59.91) total: 10000 open/close in 144.38 seconds: 69.26 ops/second - unlinked 0 (time 1781228588 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 681 (time 1781228687.98 total 10.01 last 68.05) - open/close 1401 (time 1781228697.99 total 20.01 last 71.96) - open/close 2158 (time 1781228708.00 total 30.02 last 75.66) - open/close 2870 (time 1781228718.00 total 40.03 last 71.14) - open/close 3524 (time 1781228728.02 total 50.04 last 65.28) - open/close 4173 (time 1781228738.02 total 60.05 last 64.89) - open/close 4873 (time 1781228748.03 total 70.06 last 69.93) - open/close 5560 (time 1781228758.05 total 80.07 last 68.60) - open/close 6208 (time 1781228768.06 total 90.08 last 64.72) - open/close 6904 (time 1781228778.08 total 100.10 last 69.50) - open/close 7618 (time 1781228788.08 total 110.11 last 71.35) - open/close 8287 (time 1781228798.08 total 120.11 last 66.89) - open/close 8960 (time 1781228808.09 total 130.12 last 67.23) - open/close 9658 (time 1781228818.10 total 140.12 last 69.77) total: 10000 open/close in 144.97 seconds: 68.98 ops/second - unlinked 0 (time 1781228828 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 695 (time 1781228930.96 total 10.00 last 69.48) - open/close 1438 (time 1781228940.96 total 20.01 last 74.25) - open/close 2126 (time 1781228950.96 total 30.01 last 68.80) - open/close 2795 (time 1781228960.97 total 40.01 last 66.88) - open/close 3506 (time 1781228970.97 total 50.02 last 71.05) - open/close 4165 (time 1781228980.98 total 60.02 last 65.89) - open/close 4797 (time 1781228990.99 total 70.03 last 63.12) - open/close 5445 (time 1781229000.99 total 80.04 last 64.75) - open/close 6158 (time 1781229011.00 total 90.04 last 71.28) - open/close 6880 (time 1781229021.00 total 100.05 last 72.16) - open/close 7623 (time 1781229031.00 total 110.05 last 74.28) - open/close 8251 (time 1781229041.02 total 120.06 last 62.73) - open/close 8886 (time 1781229051.02 total 130.07 last 63.48) - open/close 9555 (time 1781229061.03 total 140.07 last 66.84) total: 10000 open/close in 146.88 seconds: 68.08 ops/second - unlinked 0 (time 1781229072 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 623 (time 1781229170.00 total 10.01 last 62.26) - open/close 1255 (time 1781229180.01 total 20.01 last 63.17) - open/close 1837 (time 1781229190.01 total 30.02 last 58.16) - open/close 2422 (time 1781229200.02 total 40.02 last 58.46) - open/close 3004 (time 1781229210.03 total 50.03 last 58.14) - open/close 3738 (time 1781229220.04 total 60.05 last 73.31) - open/close 4414 (time 1781229230.05 total 70.06 last 67.53) - open/close 5026 (time 1781229240.06 total 80.06 last 61.16) - open/close 5663 (time 1781229250.06 total 90.07 last 63.69) - open/close 6156 (time 1781229260.08 total 100.08 last 49.23) - open/close 6672 (time 1781229270.09 total 110.09 last 51.55) - open/close 7273 (time 1781229280.09 total 120.10 last 60.05) - open/close 7923 (time 1781229290.10 total 130.11 last 64.96) - open/close 8565 (time 1781229300.11 total 140.11 last 64.14) - open/close 9230 (time 1781229310.12 total 150.13 last 66.42) - open/close 9925 (time 1781229320.13 total 160.14 last 69.44) total: 10000 open/close in 161.29 seconds: 62.00 ops/second - unlinked 0 (time 1781229325 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 683 (time 1781229430.18 total 10.01 last 68.26) - open/close 1298 (time 1781229440.21 total 20.03 last 61.33) - open/close 1988 (time 1781229450.21 total 30.04 last 68.98) - open/close 2622 (time 1781229460.22 total 40.04 last 63.38) - open/close 3320 (time 1781229470.22 total 50.04 last 69.79) - open/close 3982 (time 1781229480.22 total 60.05 last 66.17) - open/close 4640 (time 1781229490.23 total 70.06 last 65.74) - open/close 5188 (time 1781229500.25 total 80.07 last 54.72) - open/close 5839 (time 1781229510.25 total 90.07 last 65.08) - open/close 6345 (time 1781229520.26 total 100.08 last 50.57) - open/close 6768 (time 1781229530.26 total 110.08 last 42.30) - open/close 7158 (time 1781229540.29 total 120.11 last 38.88) - open/close 7661 (time 1781229550.30 total 130.13 last 50.22) - open/close 8129 (time 1781229560.33 total 140.15 last 46.68) - open/close 8607 (time 1781229570.35 total 150.17 last 47.71) - open/close 9186 (time 1781229580.36 total 160.18 last 57.85) - open/close 9813 (time 1781229590.37 total 170.19 last 62.62) total: 10000 open/close in 173.30 seconds: 57.70 ops/second - unlinked 0 (time 1781229598 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 655 (time 1781229707.01 total 10.00 last 65.50) - open/close 1298 (time 1781229717.02 total 20.01 last 64.24) - open/close 1954 (time 1781229727.03 total 30.02 last 65.54) - open/close 2622 (time 1781229737.04 total 40.03 last 66.75) - open/close 3269 (time 1781229747.04 total 50.03 last 64.68) - open/close 3885 (time 1781229757.04 total 60.03 last 61.60) - open/close 4559 (time 1781229767.05 total 70.04 last 67.33) - open/close 5284 (time 1781229777.06 total 80.05 last 72.42) - open/close 5923 (time 1781229787.07 total 90.06 last 63.84) - open/close 6612 (time 1781229797.09 total 100.08 last 68.77) - open/close 7260 (time 1781229807.10 total 110.09 last 64.71) - open/close 7813 (time 1781229817.12 total 120.11 last 55.19) - open/close 8324 (time 1781229827.13 total 130.12 last 51.07) - open/close 8841 (time 1781229837.14 total 140.13 last 51.63) - open/close 9372 (time 1781229847.14 total 150.13 last 53.10) - open/close 9955 (time 1781229857.16 total 160.15 last 58.20) total: 10000 open/close in 160.83 seconds: 62.18 ops/second - unlinked 0 (time 1781229863 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 587 (time 1781229967.78 total 10.00 last 58.69) - open/close 1239 (time 1781229977.78 total 20.00 last 65.20) - open/close 1825 (time 1781229987.80 total 30.02 last 58.51) - open/close 2339 (time 1781229997.81 total 40.02 last 51.37) - open/close 2856 (time 1781230007.81 total 50.03 last 51.66) - open/close 3433 (time 1781230017.84 total 60.05 last 57.56) - open/close 4016 (time 1781230027.84 total 70.06 last 58.27) - open/close 4651 (time 1781230037.85 total 80.07 last 63.45) - open/close 5291 (time 1781230047.86 total 90.07 last 63.97) - open/close 6008 (time 1781230057.86 total 100.08 last 71.65) - open/close 6679 (time 1781230067.88 total 110.09 last 67.00) - open/close 7318 (time 1781230077.88 total 120.10 last 63.86) - open/close 7983 (time 1781230087.90 total 130.11 last 66.43) - open/close 8618 (time 1781230097.91 total 140.12 last 63.43) - open/close 9364 (time 1781230107.91 total 150.13 last 74.57) total: 10000 open/close in 159.55 seconds: 62.68 ops/second - unlinked 0 (time 1781230122 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 594 (time 1781230230.01 total 10.01 last 59.37) - open/close 1264 (time 1781230240.01 total 20.01 last 66.98) - open/close 1910 (time 1781230250.02 total 30.02 last 64.53) - open/close 2548 (time 1781230260.04 total 40.04 last 63.68) - open/close 3174 (time 1781230270.05 total 50.05 last 62.54) - open/close 3808 (time 1781230280.06 total 60.06 last 63.35) - open/close 4411 (time 1781230290.08 total 70.08 last 60.15) - open/close 5071 (time 1781230300.09 total 80.09 last 65.93) - open/close 5707 (time 1781230310.11 total 90.11 last 63.51) - open/close 6380 (time 1781230320.11 total 100.11 last 67.28) - open/close 7048 (time 1781230330.11 total 110.11 last 66.79) - open/close 7695 (time 1781230340.13 total 120.12 last 64.62) - open/close 8325 (time 1781230350.13 total 130.12 last 63.00) - open/close 8997 (time 1781230360.13 total 140.13 last 67.15) - open/close 9652 (time 1781230370.15 total 150.15 last 65.38) total: 10000 open/close in 154.89 seconds: 64.56 ops/second - unlinked 0 (time 1781230380 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 667 (time 1781230488.27 total 10.01 last 66.66) - open/close 1323 (time 1781230498.30 total 20.04 last 65.37) - open/close 1970 (time 1781230508.31 total 30.04 last 64.68) - open/close 2600 (time 1781230518.31 total 40.05 last 62.97) - open/close 3277 (time 1781230528.32 total 50.06 last 67.65) - open/close 3857 (time 1781230538.32 total 60.06 last 57.99) - open/close 4535 (time 1781230548.34 total 70.07 last 67.69) - open/close 5136 (time 1781230558.34 total 80.08 last 60.05) - open/close 5694 (time 1781230568.35 total 90.09 last 55.75) - open/close 6337 (time 1781230578.38 total 100.11 last 64.15) - open/close 7031 (time 1781230588.41 total 110.15 last 69.18) - open/close 7626 (time 1781230598.42 total 120.16 last 59.42) - open/close 8226 (time 1781230608.44 total 130.18 last 59.92) - open/close 8883 (time 1781230618.46 total 140.19 last 65.57) - open/close 9577 (time 1781230628.46 total 150.20 last 69.37) total: 10000 open/close in 156.67 seconds: 63.83 ops/second - unlinked 0 (time 1781230640 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 612 (time 1781230742.34 total 10.01 last 61.16) - open/close 1242 (time 1781230752.35 total 20.01 last 62.97) - open/close 1895 (time 1781230762.36 total 30.02 last 65.22) - open/close 2578 (time 1781230772.37 total 40.03 last 68.25) - open/close 3265 (time 1781230782.39 total 50.05 last 68.53) - open/close 3926 (time 1781230792.40 total 60.06 last 66.05) - open/close 4536 (time 1781230802.41 total 70.08 last 60.90) - open/close 5239 (time 1781230812.42 total 80.08 last 70.27) - open/close 5933 (time 1781230822.43 total 90.09 last 69.34) - open/close 6643 (time 1781230832.44 total 100.10 last 70.91) - open/close 7280 (time 1781230842.45 total 110.11 last 63.65) - open/close 7966 (time 1781230852.46 total 120.12 last 68.52) - open/close 8666 (time 1781230862.48 total 130.14 last 69.87) - open/close 9377 (time 1781230872.50 total 140.16 last 70.97) total: 10000 open/close in 149.27 seconds: 66.99 ops/second - unlinked 0 (time 1781230886 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded. pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2