== conf-sanity test 69: replace an OST with the same index ========================================================== 23:15:41 (1769055341) start mds service on oleg424-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg424-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 716 (time 1769055465.45 total 10.01 last 71.51) - open/close 1342 (time 1769055475.45 total 20.01 last 62.60) - open/close 2028 (time 1769055485.45 total 30.02 last 68.58) - open/close 2658 (time 1769055495.45 total 40.02 last 63.00) - open/close 3430 (time 1769055505.45 total 50.02 last 77.15) - open/close 4189 (time 1769055515.46 total 60.03 last 75.86) - open/close 4739 (time 1769055525.47 total 70.04 last 54.93) - open/close 5375 (time 1769055535.48 total 80.04 last 63.57) - open/close 6021 (time 1769055545.48 total 90.05 last 64.57) - open/close 6694 (time 1769055555.48 total 100.05 last 67.28) - open/close 7455 (time 1769055565.49 total 110.06 last 76.05) - open/close 8253 (time 1769055575.50 total 120.06 last 79.75) - open/close 8953 (time 1769055585.50 total 130.07 last 69.95) - open/close 9554 (time 1769055595.51 total 140.07 last 60.09) total: 10000 open/close in 146.03 seconds: 68.48 ops/second - unlinked 0 (time 1769055605 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 678 (time 1769055698.74 total 10.00 last 67.79) - open/close 1419 (time 1769055708.74 total 20.00 last 74.08) - open/close 2118 (time 1769055718.75 total 30.01 last 69.82) - open/close 2784 (time 1769055728.76 total 40.03 last 66.50) - open/close 3430 (time 1769055738.80 total 50.06 last 64.39) - open/close 4088 (time 1769055748.80 total 60.07 last 65.76) - open/close 4921 (time 1769055758.82 total 70.09 last 83.15) - open/close 5685 (time 1769055768.85 total 80.12 last 76.17) - open/close 6315 (time 1769055778.87 total 90.13 last 62.91) - open/close 7059 (time 1769055788.88 total 100.15 last 74.27) - open/close 7860 (time 1769055798.89 total 110.16 last 80.04) - open/close 8696 (time 1769055808.90 total 120.17 last 83.51) - open/close 9419 (time 1769055818.91 total 130.17 last 72.23) total: 10000 open/close in 137.46 seconds: 72.75 ops/second - unlinked 0 (time 1769055831 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 639 (time 1769055927.72 total 10.00 last 63.90) - open/close 1354 (time 1769055937.73 total 20.00 last 71.49) - open/close 2073 (time 1769055947.73 total 30.01 last 71.85) - open/close 2742 (time 1769055957.74 total 40.02 last 66.82) - open/close 3403 (time 1769055967.75 total 50.03 last 66.04) - open/close 4158 (time 1769055977.78 total 60.05 last 75.33) - open/close 4944 (time 1769055987.78 total 70.05 last 78.58) - open/close 5723 (time 1769055997.80 total 80.07 last 77.76) - open/close 6467 (time 1769056007.80 total 90.08 last 74.35) - open/close 7167 (time 1769056017.81 total 100.09 last 69.94) - open/close 7817 (time 1769056027.82 total 110.09 last 64.95) - open/close 8420 (time 1769056037.82 total 120.10 last 60.29) - open/close 9105 (time 1769056047.83 total 130.11 last 68.41) - open/close 9745 (time 1769056057.86 total 140.14 last 63.82) total: 10000 open/close in 143.49 seconds: 69.69 ops/second - unlinked 0 (time 1769056066 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 509 (time 1769056167.73 total 10.01 last 50.86) - open/close 1154 (time 1769056177.76 total 20.04 last 64.32) - open/close 1921 (time 1769056187.78 total 30.06 last 76.55) - open/close 2694 (time 1769056197.78 total 40.06 last 77.30) - open/close 3351 (time 1769056207.78 total 50.06 last 65.70) - open/close 3960 (time 1769056217.79 total 60.06 last 60.88) - open/close 4717 (time 1769056227.79 total 70.07 last 75.66) - open/close 5428 (time 1769056237.80 total 80.07 last 71.04) - open/close 6112 (time 1769056247.82 total 90.09 last 68.27) - open/close 6859 (time 1769056257.83 total 100.10 last 74.62) - open/close 7509 (time 1769056267.84 total 110.11 last 64.97) - open/close 8099 (time 1769056277.86 total 120.14 last 58.84) - open/close 8756 (time 1769056287.88 total 130.15 last 65.61) - open/close 9370 (time 1769056297.88 total 140.15 last 61.39) total: 10000 open/close in 149.67 seconds: 66.81 ops/second - unlinked 0 (time 1769056312 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 670 (time 1769056416.79 total 10.01 last 66.96) - open/close 1220 (time 1769056426.80 total 20.01 last 54.99) - open/close 1776 (time 1769056436.82 total 30.03 last 55.49) - open/close 2482 (time 1769056446.83 total 40.04 last 70.53) - open/close 3110 (time 1769056456.83 total 50.04 last 62.76) - open/close 3915 (time 1769056466.84 total 60.06 last 80.42) - open/close 4703 (time 1769056476.85 total 70.06 last 78.79) - open/close 5416 (time 1769056486.85 total 80.06 last 71.25) - open/close 6073 (time 1769056496.87 total 90.08 last 65.57) - open/close 6676 (time 1769056506.88 total 100.09 last 60.28) - open/close 7321 (time 1769056516.88 total 110.09 last 64.48) - open/close 7883 (time 1769056526.89 total 120.10 last 56.15) - open/close 8503 (time 1769056536.90 total 130.11 last 61.96) - open/close 9303 (time 1769056546.91 total 140.12 last 79.91) - open/close 9964 (time 1769056556.92 total 150.13 last 66.01) total: 10000 open/close in 151.04 seconds: 66.21 ops/second - unlinked 0 (time 1769056563 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 707 (time 1769056656.95 total 10.01 last 70.60) - open/close 1407 (time 1769056666.96 total 20.02 last 69.98) - open/close 2128 (time 1769056676.96 total 30.02 last 72.08) - open/close 2936 (time 1769056686.96 total 40.03 last 80.75) - open/close 3618 (time 1769056696.97 total 50.03 last 68.18) - open/close 4299 (time 1769056707.00 total 60.06 last 67.87) - open/close 4891 (time 1769056717.00 total 70.06 last 59.19) - open/close 5496 (time 1769056727.02 total 80.08 last 60.42) - open/close 5972 (time 1769056737.02 total 90.08 last 47.58) - open/close 6517 (time 1769056747.03 total 100.09 last 54.48) - open/close 7077 (time 1769056757.03 total 110.09 last 55.98) - open/close 7732 (time 1769056767.03 total 120.09 last 65.46) - open/close 8431 (time 1769056777.05 total 130.11 last 69.82) - open/close 9108 (time 1769056787.05 total 140.11 last 67.67) - open/close 9732 (time 1769056797.06 total 150.12 last 62.36) total: 10000 open/close in 154.22 seconds: 64.84 ops/second - unlinked 0 (time 1769056806 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 623 (time 1769056910.24 total 10.01 last 62.26) - open/close 1183 (time 1769056920.25 total 20.02 last 55.92) - open/close 1891 (time 1769056930.26 total 30.03 last 70.74) - open/close 2609 (time 1769056940.26 total 40.03 last 71.79) - open/close 3310 (time 1769056950.27 total 50.04 last 70.06) - open/close 4013 (time 1769056960.28 total 60.05 last 70.21) - open/close 4646 (time 1769056970.29 total 70.06 last 63.26) - open/close 5345 (time 1769056980.29 total 80.06 last 69.89) - open/close 6023 (time 1769056990.31 total 90.08 last 67.66) - open/close 6773 (time 1769057000.33 total 100.10 last 74.88) - open/close 7316 (time 1769057010.34 total 110.11 last 54.21) - open/close 7872 (time 1769057020.35 total 120.11 last 55.59) - open/close 8591 (time 1769057030.36 total 130.13 last 71.81) - open/close 9196 (time 1769057040.36 total 140.13 last 60.50) - open/close 9912 (time 1769057050.36 total 150.13 last 71.59) total: 10000 open/close in 151.45 seconds: 66.03 ops/second - unlinked 0 (time 1769057056 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 618 (time 1769057150.69 total 10.03 last 61.64) - open/close 1338 (time 1769057160.69 total 20.03 last 71.94) - open/close 2032 (time 1769057170.70 total 30.03 last 69.39) - open/close 2881 (time 1769057180.70 total 40.04 last 84.89) - open/close 3767 (time 1769057190.72 total 50.06 last 88.41) - open/close 4571 (time 1769057200.72 total 60.06 last 80.35) - open/close 5454 (time 1769057210.73 total 70.07 last 88.27) - open/close 6315 (time 1769057220.73 total 80.07 last 86.09) - open/close 7034 (time 1769057230.73 total 90.07 last 71.89) - open/close 7809 (time 1769057240.73 total 100.07 last 77.47) - open/close 8620 (time 1769057250.73 total 110.07 last 81.09) - open/close 9438 (time 1769057260.74 total 120.08 last 81.76) total: 10000 open/close in 128.28 seconds: 77.95 ops/second - unlinked 0 (time 1769057273 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 822 (time 1769057362.67 total 10.00 last 82.19) - open/close 1649 (time 1769057372.67 total 20.00 last 82.69) - open/close 2458 (time 1769057382.69 total 30.01 last 80.81) - open/close 3199 (time 1769057392.69 total 40.01 last 74.09) - open/close 3970 (time 1769057402.69 total 50.02 last 77.05) - open/close 4646 (time 1769057412.70 total 60.03 last 67.55) - open/close 5389 (time 1769057422.71 total 70.04 last 74.21) - open/close 6203 (time 1769057432.73 total 80.05 last 81.28) - open/close 6973 (time 1769057442.73 total 90.06 last 76.97) - open/close 7809 (time 1769057452.74 total 100.07 last 83.54) - open/close 8735 (time 1769057462.74 total 110.07 last 92.55) - open/close 9632 (time 1769057472.74 total 120.07 last 89.70) total: 10000 open/close in 124.21 seconds: 80.51 ops/second - unlinked 0 (time 1769057480 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 718 (time 1769057591.52 total 10.01 last 71.75) - open/close 1266 (time 1769057601.52 total 20.01 last 54.76) - open/close 1932 (time 1769057611.53 total 30.02 last 66.57) - open/close 2583 (time 1769057621.54 total 40.03 last 65.05) - open/close 3209 (time 1769057631.55 total 50.04 last 62.51) - open/close 3876 (time 1769057641.58 total 60.07 last 66.48) - open/close 4543 (time 1769057651.58 total 70.07 last 66.69) - open/close 5075 (time 1769057661.59 total 80.09 last 53.14) - open/close 5572 (time 1769057671.60 total 90.09 last 49.66) - open/close 6243 (time 1769057681.60 total 100.10 last 67.09) - open/close 6878 (time 1769057691.62 total 110.11 last 63.40) - open/close 7406 (time 1769057701.63 total 120.12 last 52.73) - open/close 7978 (time 1769057711.64 total 130.13 last 57.18) - open/close 8530 (time 1769057721.66 total 140.15 last 55.10) - open/close 9313 (time 1769057731.67 total 150.16 last 78.20) - open/close 9943 (time 1769057741.67 total 160.17 last 62.96) total: 10000 open/close in 160.83 seconds: 62.18 ops/second - unlinked 0 (time 1769057747 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2