== conf-sanity test 69: replace an OST with the same index ========================================================== 20:20:15 (1775866815) start mds service on oleg101-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg101-server: detected 4 online CPUs by sysfs oleg101-server: MODOPTS_LIBCFS= oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg101-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg101-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 937 (time 1775866919.92 total 10.00 last 93.66) - open/close 1922 (time 1775866929.92 total 20.01 last 98.47) - open/close 3015 (time 1775866939.93 total 30.01 last 109.24) - open/close 4108 (time 1775866949.93 total 40.02 last 109.26) - open/close 5176 (time 1775866959.94 total 50.02 last 106.74) - open/close 6158 (time 1775866969.94 total 60.02 last 98.19) - open/close 7177 (time 1775866979.94 total 70.03 last 101.88) - open/close 8198 (time 1775866989.94 total 80.03 last 102.06) - open/close 9201 (time 1775866999.94 total 90.03 last 100.29) total: 10000 open/close in 97.48 seconds: 102.59 ops/second - unlinked 0 (time 1775867011 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 943 (time 1775867084.50 total 10.00 last 94.28) - open/close 1915 (time 1775867094.51 total 20.00 last 97.18) - open/close 2943 (time 1775867104.52 total 30.01 last 102.71) - open/close 3936 (time 1775867114.53 total 40.03 last 99.15) - open/close 4930 (time 1775867124.54 total 50.04 last 99.30) - open/close 5908 (time 1775867134.55 total 60.05 last 97.71) - open/close 6971 (time 1775867144.56 total 70.05 last 106.22) - open/close 8034 (time 1775867154.56 total 80.06 last 106.26) - open/close 9069 (time 1775867164.56 total 90.06 last 103.45) total: 10000 open/close in 98.64 seconds: 101.38 ops/second - unlinked 0 (time 1775867176 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1173 (time 1775867244.31 total 10.00 last 117.25) - open/close 2285 (time 1775867254.32 total 20.01 last 111.09) - open/close 3405 (time 1775867264.33 total 30.02 last 111.97) - open/close 4525 (time 1775867274.33 total 40.02 last 111.97) - open/close 5605 (time 1775867284.34 total 50.03 last 107.92) - open/close 6690 (time 1775867294.34 total 60.03 last 108.50) - open/close 7751 (time 1775867304.35 total 70.04 last 105.95) - open/close 8274 (time 1775867314.36 total 80.05 last 52.27) - open/close 8846 (time 1775867324.36 total 90.05 last 57.18) - open/close 9455 (time 1775867334.36 total 100.06 last 60.87) total: 10000 open/close in 109.66 seconds: 91.19 ops/second - unlinked 0 (time 1775867348 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 906 (time 1775867426.83 total 10.00 last 90.59) - open/close 1872 (time 1775867436.84 total 20.01 last 96.52) - open/close 2923 (time 1775867446.84 total 30.01 last 105.09) - open/close 3928 (time 1775867456.85 total 40.03 last 100.34) - open/close 4999 (time 1775867466.85 total 50.03 last 107.08) - open/close 6119 (time 1775867476.86 total 60.03 last 111.96) - open/close 7140 (time 1775867486.86 total 70.03 last 102.07) - open/close 7997 (time 1775867496.87 total 80.04 last 85.64) - open/close 9086 (time 1775867506.88 total 90.05 last 108.81) total: 10000 open/close in 98.23 seconds: 101.80 ops/second - unlinked 0 (time 1775867518 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 902 (time 1775867593.69 total 10.01 last 90.13) - open/close 1868 (time 1775867603.69 total 20.01 last 96.58) - open/close 2956 (time 1775867613.70 total 30.01 last 108.75) - open/close 3678 (time 1775867623.70 total 40.02 last 72.19) - open/close 4781 (time 1775867633.70 total 50.02 last 110.27) - open/close 5898 (time 1775867643.70 total 60.02 last 111.69) - open/close 6967 (time 1775867653.70 total 70.02 last 106.88) - open/close 7960 (time 1775867663.71 total 80.02 last 99.27) - open/close 8811 (time 1775867673.71 total 90.03 last 85.05) - open/close 9565 (time 1775867683.72 total 100.04 last 75.36) total: 10000 open/close in 108.05 seconds: 92.55 ops/second - unlinked 0 (time 1775867697 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1019 (time 1775867769.57 total 10.01 last 101.84) - open/close 2063 (time 1775867779.58 total 20.01 last 104.33) - open/close 2996 (time 1775867789.59 total 30.02 last 93.22) - open/close 4023 (time 1775867799.59 total 40.03 last 102.64) - open/close 4790 (time 1775867809.60 total 50.03 last 76.65) - open/close 5483 (time 1775867819.61 total 60.04 last 69.25) - open/close 6173 (time 1775867829.61 total 70.04 last 68.97) - open/close 6972 (time 1775867839.61 total 80.05 last 79.89) - open/close 7851 (time 1775867849.62 total 90.05 last 87.85) - open/close 8785 (time 1775867859.63 total 100.06 last 93.29) - open/close 9923 (time 1775867869.64 total 110.07 last 113.73) total: 10000 open/close in 110.73 seconds: 90.31 ops/second - unlinked 0 (time 1775867874 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 600 (time 1775867948.73 total 10.00 last 59.98) - open/close 1320 (time 1775867958.74 total 20.02 last 71.91) - open/close 2191 (time 1775867968.75 total 30.02 last 87.03) - open/close 3305 (time 1775867978.76 total 40.03 last 111.33) - open/close 4227 (time 1775867988.76 total 50.03 last 92.17) - open/close 5133 (time 1775867998.76 total 60.04 last 90.56) - open/close 5998 (time 1775868008.77 total 70.04 last 86.46) - open/close 6874 (time 1775868018.77 total 80.05 last 87.57) - open/close 7587 (time 1775868028.80 total 90.08 last 71.09) - open/close 8348 (time 1775868038.81 total 100.08 last 76.05) - open/close 9199 (time 1775868048.83 total 110.10 last 84.91) - open/close 9727 (time 1775868058.84 total 120.11 last 52.77) total: 10000 open/close in 125.91 seconds: 79.42 ops/second - unlinked 0 (time 1775868071 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1079 (time 1775868152.08 total 10.00 last 107.89) - open/close 2130 (time 1775868162.08 total 20.01 last 105.05) - open/close 3096 (time 1775868172.09 total 30.01 last 96.56) - open/close 4139 (time 1775868182.09 total 40.02 last 104.24) - open/close 5139 (time 1775868192.10 total 50.02 last 99.96) - open/close 6152 (time 1775868202.10 total 60.03 last 101.22) - open/close 7107 (time 1775868212.11 total 70.03 last 95.45) - open/close 7992 (time 1775868222.11 total 80.03 last 88.49) - open/close 8871 (time 1775868232.11 total 90.03 last 87.89) - open/close 9814 (time 1775868242.12 total 100.04 last 94.24) total: 10000 open/close in 102.15 seconds: 97.89 ops/second - unlinked 0 (time 1775868248 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 763 (time 1775868332.92 total 10.00 last 76.26) - open/close 1552 (time 1775868342.94 total 20.02 last 78.78) - open/close 2149 (time 1775868352.94 total 30.02 last 59.70) - open/close 2797 (time 1775868362.95 total 40.04 last 64.70) - open/close 3574 (time 1775868372.96 total 50.04 last 77.68) - open/close 4468 (time 1775868382.97 total 60.05 last 89.31) - open/close 5214 (time 1775868392.98 total 70.06 last 74.52) - open/close 6009 (time 1775868402.99 total 80.07 last 79.41) - open/close 6824 (time 1775868413.00 total 90.08 last 81.42) - open/close 7608 (time 1775868423.00 total 100.08 last 78.38) - open/close 8316 (time 1775868433.01 total 110.10 last 70.70) - open/close 9011 (time 1775868443.02 total 120.11 last 69.45) - open/close 9716 (time 1775868453.03 total 130.11 last 70.47) total: 10000 open/close in 133.28 seconds: 75.03 ops/second - unlinked 0 (time 1775868461 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 547 (time 1775868540.66 total 10.01 last 54.67) - open/close 1055 (time 1775868550.67 total 20.02 last 50.75) - open/close 1630 (time 1775868560.67 total 30.02 last 57.46) - open/close 2337 (time 1775868570.69 total 40.04 last 70.61) - open/close 3110 (time 1775868580.69 total 50.04 last 77.29) - open/close 3867 (time 1775868590.69 total 60.04 last 75.66) - open/close 4844 (time 1775868600.72 total 70.06 last 97.48) - open/close 5675 (time 1775868610.72 total 80.07 last 83.09) - open/close 6363 (time 1775868620.72 total 90.07 last 68.76) - open/close 7042 (time 1775868630.74 total 100.08 last 67.82) - open/close 7901 (time 1775868640.74 total 110.09 last 85.83) - open/close 8551 (time 1775868650.75 total 120.09 last 64.98) - open/close 9305 (time 1775868660.75 total 130.10 last 75.34) total: 10000 open/close in 139.32 seconds: 71.78 ops/second - unlinked 0 (time 1775868676 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg101-server: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg101-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg101-server: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg101-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.101@tcp sys.timeout=20 start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg101-server: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg101-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg101-server: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg101-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg101-server unloading modules via unload_modules_local on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local oleg101-server: modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2