== conf-sanity test 69: replace an OST with the same index ========================================================== 10:52:46 (1785077566) start mds service on oleg444-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg444-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg444-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 916 (time 1785077678.24 total 10.01 last 91.48) - open/close 1737 (time 1785077688.24 total 20.02 last 82.05) - open/close 2623 (time 1785077698.25 total 30.03 last 88.54) - open/close 3512 (time 1785077708.28 total 40.05 last 88.68) - open/close 4123 (time 1785077718.30 total 50.07 last 60.98) - open/close 4657 (time 1785077728.31 total 60.08 last 53.34) - open/close 5364 (time 1785077738.31 total 70.09 last 70.66) - open/close 6115 (time 1785077748.34 total 80.11 last 74.93) - open/close 6847 (time 1785077758.36 total 90.14 last 73.01) - open/close 7661 (time 1785077768.37 total 100.14 last 81.35) - open/close 8565 (time 1785077778.38 total 110.16 last 90.27) - open/close 9425 (time 1785077788.39 total 120.17 last 85.91) total: 10000 open/close in 127.05 seconds: 78.71 ops/second - unlinked 0 (time 1785077799 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 693 (time 1785077891.10 total 10.01 last 69.24) - open/close 1408 (time 1785077901.11 total 20.02 last 71.46) - open/close 2117 (time 1785077911.12 total 30.02 last 70.86) - open/close 2927 (time 1785077921.12 total 40.02 last 81.00) - open/close 3577 (time 1785077931.12 total 50.03 last 64.95) - open/close 4285 (time 1785077941.13 total 60.03 last 70.77) - open/close 5073 (time 1785077951.14 total 70.05 last 78.70) - open/close 5837 (time 1785077961.15 total 80.05 last 76.35) - open/close 6716 (time 1785077971.16 total 90.06 last 87.81) - open/close 7472 (time 1785077981.17 total 100.08 last 75.49) - open/close 8217 (time 1785077991.18 total 110.09 last 74.43) - open/close 9025 (time 1785078001.19 total 120.09 last 80.75) - open/close 9786 (time 1785078011.20 total 130.10 last 76.05) total: 10000 open/close in 133.49 seconds: 74.91 ops/second - unlinked 0 (time 1785078021 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 861 (time 1785078116.91 total 10.00 last 86.10) - open/close 1670 (time 1785078126.91 total 20.00 last 80.89) - open/close 2416 (time 1785078136.93 total 30.02 last 74.49) - open/close 3178 (time 1785078146.93 total 40.02 last 76.17) - open/close 3919 (time 1785078156.94 total 50.03 last 74.05) - open/close 4736 (time 1785078166.97 total 60.06 last 81.43) - open/close 5473 (time 1785078176.98 total 70.07 last 73.64) - open/close 6304 (time 1785078186.98 total 80.07 last 83.09) - open/close 6977 (time 1785078196.98 total 90.07 last 67.28) - open/close 7684 (time 1785078206.99 total 100.08 last 70.64) - open/close 8576 (time 1785078217.00 total 110.09 last 89.15) - open/close 9378 (time 1785078227.00 total 120.09 last 80.18) total: 10000 open/close in 128.68 seconds: 77.71 ops/second - unlinked 0 (time 1785078241 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 743 (time 1785078332.50 total 10.03 last 74.08) - open/close 1450 (time 1785078342.51 total 20.04 last 70.61) - open/close 2135 (time 1785078352.52 total 30.05 last 68.44) - open/close 2777 (time 1785078362.52 total 40.06 last 64.18) - open/close 3576 (time 1785078372.53 total 50.07 last 79.82) - open/close 4346 (time 1785078382.53 total 60.07 last 76.98) - open/close 5117 (time 1785078392.54 total 70.08 last 77.04) - open/close 5837 (time 1785078402.55 total 80.09 last 71.93) - open/close 6588 (time 1785078412.56 total 90.09 last 75.08) - open/close 7442 (time 1785078422.56 total 100.10 last 85.35) - open/close 8149 (time 1785078432.57 total 110.10 last 70.67) - open/close 8884 (time 1785078442.57 total 120.10 last 73.50) - open/close 9678 (time 1785078452.57 total 130.11 last 79.34) total: 10000 open/close in 134.12 seconds: 74.56 ops/second - unlinked 0 (time 1785078460 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 707 (time 1785078550.52 total 10.00 last 70.69) - open/close 1435 (time 1785078560.53 total 20.01 last 72.74) - open/close 2155 (time 1785078570.54 total 30.02 last 71.93) - open/close 2787 (time 1785078580.54 total 40.02 last 63.18) - open/close 3419 (time 1785078590.55 total 50.02 last 63.19) - open/close 4004 (time 1785078600.55 total 60.03 last 58.45) - open/close 4610 (time 1785078610.56 total 70.04 last 60.54) - open/close 5326 (time 1785078620.57 total 80.04 last 71.59) - open/close 6244 (time 1785078630.57 total 90.05 last 91.78) - open/close 7125 (time 1785078640.57 total 100.05 last 88.04) - open/close 7959 (time 1785078650.59 total 110.07 last 83.27) - open/close 8744 (time 1785078660.59 total 120.07 last 78.46) - open/close 9623 (time 1785078670.60 total 130.08 last 87.88) total: 10000 open/close in 134.43 seconds: 74.39 ops/second - unlinked 0 (time 1785078679 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 889 (time 1785078766.69 total 10.00 last 88.89) - open/close 1618 (time 1785078776.72 total 20.03 last 72.72) - open/close 2183 (time 1785078786.74 total 30.05 last 56.38) - open/close 2833 (time 1785078796.74 total 40.05 last 65.00) - open/close 3713 (time 1785078806.74 total 50.05 last 87.99) - open/close 4611 (time 1785078816.75 total 60.06 last 89.67) - open/close 5491 (time 1785078826.76 total 70.07 last 87.92) - open/close 6354 (time 1785078836.77 total 80.08 last 86.21) - open/close 7337 (time 1785078846.78 total 90.09 last 98.25) - open/close 8096 (time 1785078856.79 total 100.10 last 75.83) - open/close 8954 (time 1785078866.80 total 110.11 last 85.68) - open/close 9723 (time 1785078876.81 total 120.12 last 76.81) total: 10000 open/close in 124.61 seconds: 80.25 ops/second - unlinked 0 (time 1785078887 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 1014 (time 1785078973.00 total 10.00 last 101.36) - open/close 1861 (time 1785078983.01 total 20.01 last 84.67) - open/close 2719 (time 1785078993.02 total 30.02 last 85.72) - open/close 3474 (time 1785079003.02 total 40.02 last 75.45) - open/close 4059 (time 1785079013.03 total 50.03 last 58.45) - open/close 4740 (time 1785079023.06 total 60.06 last 67.93) - open/close 5346 (time 1785079033.06 total 70.06 last 60.58) - open/close 6062 (time 1785079043.07 total 80.07 last 71.52) - open/close 6816 (time 1785079053.07 total 90.07 last 75.39) - open/close 7656 (time 1785079063.09 total 100.09 last 83.85) - open/close 8434 (time 1785079073.10 total 110.10 last 77.72) - open/close 9060 (time 1785079083.11 total 120.11 last 62.54) - open/close 9795 (time 1785079093.12 total 130.13 last 73.38) total: 10000 open/close in 133.03 seconds: 75.17 ops/second - unlinked 0 (time 1785079100 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 603 (time 1785079184.84 total 10.01 last 60.23) - open/close 1188 (time 1785079194.84 total 20.01 last 58.49) - open/close 1755 (time 1785079204.87 total 30.04 last 56.57) - open/close 2400 (time 1785079214.90 total 40.07 last 64.30) - open/close 3318 (time 1785079224.90 total 50.07 last 91.74) - open/close 4149 (time 1785079234.90 total 60.08 last 83.08) - open/close 5006 (time 1785079244.91 total 70.08 last 85.69) - open/close 5966 (time 1785079254.91 total 80.08 last 95.95) - open/close 6909 (time 1785079264.92 total 90.09 last 94.24) - open/close 7728 (time 1785079274.92 total 100.09 last 81.88) - open/close 8491 (time 1785079284.93 total 110.10 last 76.23) - open/close 9283 (time 1785079294.93 total 120.10 last 79.19) total: 10000 open/close in 128.15 seconds: 78.04 ops/second - unlinked 0 (time 1785079307 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 564 (time 1785079391.97 total 10.00 last 56.38) - open/close 1354 (time 1785079401.97 total 20.01 last 78.96) - open/close 2128 (time 1785079411.98 total 30.02 last 77.34) - open/close 2931 (time 1785079421.98 total 40.02 last 80.27) - open/close 3920 (time 1785079431.99 total 50.02 last 98.88) - open/close 4785 (time 1785079441.99 total 60.02 last 86.49) - open/close 5715 (time 1785079451.99 total 70.03 last 92.93) - open/close 6638 (time 1785079461.99 total 80.03 last 92.30) - open/close 7597 (time 1785079472.00 total 90.04 last 95.83) - open/close 8452 (time 1785079482.00 total 100.04 last 85.49) - open/close 9361 (time 1785079492.00 total 110.04 last 90.89) total: 10000 open/close in 116.57 seconds: 85.79 ops/second - unlinked 0 (time 1785079502 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 831 (time 1785079582.52 total 10.01 last 83.00) - open/close 1593 (time 1785079592.53 total 20.02 last 76.16) - open/close 2522 (time 1785079602.54 total 30.02 last 92.85) - open/close 3393 (time 1785079612.54 total 40.02 last 87.09) - open/close 4291 (time 1785079622.54 total 50.03 last 89.74) - open/close 5248 (time 1785079632.55 total 60.03 last 95.67) - open/close 5953 (time 1785079642.55 total 70.04 last 70.46) - open/close 6674 (time 1785079652.55 total 80.04 last 72.09) - open/close 7544 (time 1785079662.56 total 90.05 last 86.97) - open/close 8461 (time 1785079672.57 total 100.05 last 91.63) - open/close 9408 (time 1785079682.57 total 110.06 last 94.64) total: 10000 open/close in 116.62 seconds: 85.75 ops/second - unlinked 0 (time 1785079693 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2