== conf-sanity test 69: replace an OST with the same index ========================================================== 13:06:19 (1786813579) start mds service on oleg116-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg116-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg116-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 585 (time 1786813692.71 total 10.02 last 58.36) - open/close 1475 (time 1786813702.72 total 20.04 last 88.89) - open/close 2334 (time 1786813712.73 total 30.04 last 85.83) - open/close 3283 (time 1786813722.73 total 40.05 last 94.86) - open/close 4370 (time 1786813732.75 total 50.06 last 108.55) - open/close 5280 (time 1786813742.75 total 60.06 last 90.99) - open/close 6081 (time 1786813752.75 total 70.07 last 80.06) - open/close 7070 (time 1786813762.76 total 80.07 last 98.87) - open/close 8006 (time 1786813772.76 total 90.08 last 93.55) - open/close 8930 (time 1786813782.77 total 100.08 last 92.37) - open/close 9855 (time 1786813792.77 total 110.09 last 92.45) total: 10000 open/close in 111.78 seconds: 89.46 ops/second - unlinked 0 (time 1786813799 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 959 (time 1786813873.48 total 10.01 last 95.79) - open/close 1899 (time 1786813883.48 total 20.02 last 93.95) - open/close 2901 (time 1786813893.49 total 30.02 last 100.15) - open/close 3823 (time 1786813903.50 total 40.03 last 92.12) - open/close 4806 (time 1786813913.50 total 50.03 last 98.26) - open/close 5843 (time 1786813923.51 total 60.04 last 103.61) - open/close 6910 (time 1786813933.51 total 70.05 last 106.65) - open/close 7923 (time 1786813943.52 total 80.05 last 101.25) - open/close 9057 (time 1786813953.52 total 90.06 last 113.36) - open/close 9977 (time 1786813963.53 total 100.07 last 91.90) total: 10000 open/close in 100.31 seconds: 99.69 ops/second - unlinked 0 (time 1786813967 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1067 (time 1786814038.08 total 10.02 last 106.53) - open/close 2063 (time 1786814048.08 total 20.02 last 99.58) - open/close 3152 (time 1786814058.08 total 30.02 last 108.88) - open/close 4228 (time 1786814068.09 total 40.02 last 107.56) - open/close 5228 (time 1786814078.09 total 50.03 last 99.94) - open/close 6290 (time 1786814088.10 total 60.03 last 106.15) - open/close 7533 (time 1786814098.10 total 70.04 last 124.27) - open/close 8557 (time 1786814108.12 total 80.05 last 102.24) - open/close 9656 (time 1786814118.13 total 90.06 last 109.79) total: 10000 open/close in 93.59 seconds: 106.85 ops/second - unlinked 0 (time 1786814125 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 986 (time 1786814199.18 total 10.01 last 98.53) - open/close 1903 (time 1786814209.18 total 20.01 last 91.70) - open/close 2683 (time 1786814219.20 total 30.03 last 77.83) - open/close 3317 (time 1786814229.20 total 40.03 last 63.38) - open/close 4158 (time 1786814239.21 total 50.04 last 84.02) - open/close 5177 (time 1786814249.22 total 60.05 last 101.82) - open/close 6147 (time 1786814259.23 total 70.06 last 96.88) - open/close 7020 (time 1786814269.24 total 80.07 last 87.25) - open/close 7722 (time 1786814279.26 total 90.09 last 70.03) - open/close 8556 (time 1786814289.26 total 100.09 last 83.39) - open/close 9526 (time 1786814299.27 total 110.10 last 96.88) total: 10000 open/close in 114.98 seconds: 86.97 ops/second - unlinked 0 (time 1786814308 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1144 (time 1786814389.27 total 10.03 last 114.10) - open/close 2218 (time 1786814399.29 total 20.04 last 107.27) - open/close 2960 (time 1786814409.29 total 30.04 last 74.18) - open/close 3739 (time 1786814419.29 total 40.05 last 77.87) - open/close 4690 (time 1786814429.31 total 50.06 last 94.94) - open/close 5595 (time 1786814439.31 total 60.06 last 90.49) - open/close 6249 (time 1786814449.32 total 70.07 last 65.34) - open/close 7039 (time 1786814459.34 total 80.09 last 78.87) - open/close 7927 (time 1786814469.35 total 90.10 last 88.69) - open/close 8806 (time 1786814479.35 total 100.11 last 87.86) - open/close 9607 (time 1786814489.36 total 110.11 last 80.07) total: 10000 open/close in 114.42 seconds: 87.40 ops/second - unlinked 0 (time 1786814499 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 878 (time 1786814584.51 total 10.01 last 87.70) - open/close 1558 (time 1786814594.52 total 20.01 last 67.99) - open/close 2444 (time 1786814604.52 total 30.02 last 88.56) - open/close 3422 (time 1786814614.52 total 40.02 last 97.76) - open/close 4262 (time 1786814624.53 total 50.03 last 83.95) - open/close 4994 (time 1786814634.53 total 60.03 last 73.18) - open/close 5660 (time 1786814644.54 total 70.04 last 66.54) - open/close 6384 (time 1786814654.55 total 80.05 last 72.36) - open/close 7193 (time 1786814664.56 total 90.06 last 80.82) - open/close 8070 (time 1786814674.56 total 100.06 last 87.65) - open/close 9070 (time 1786814684.57 total 110.06 last 99.99) total: 10000 open/close in 118.01 seconds: 84.74 ops/second - unlinked 0 (time 1786814697 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 815 (time 1786814775.14 total 10.01 last 81.45) - open/close 1743 (time 1786814785.14 total 20.01 last 92.79) - open/close 2601 (time 1786814795.16 total 30.02 last 85.67) - open/close 3429 (time 1786814805.17 total 40.03 last 82.72) - open/close 3993 (time 1786814815.18 total 50.04 last 56.36) - open/close 4816 (time 1786814825.18 total 60.05 last 82.24) - open/close 5785 (time 1786814835.18 total 70.05 last 96.89) - open/close 6617 (time 1786814845.20 total 80.06 last 83.08) - open/close 7473 (time 1786814855.22 total 90.08 last 85.43) - open/close 8330 (time 1786814865.22 total 100.09 last 85.67) - open/close 9158 (time 1786814875.24 total 110.10 last 82.66) total: 10000 open/close in 119.18 seconds: 83.91 ops/second - unlinked 0 (time 1786814888 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 781 (time 1786814974.73 total 10.01 last 78.02) - open/close 1544 (time 1786814984.75 total 20.02 last 76.19) - open/close 2379 (time 1786814994.76 total 30.04 last 83.38) - open/close 3125 (time 1786815004.77 total 40.05 last 74.51) - open/close 3907 (time 1786815014.78 total 50.06 last 78.15) - open/close 4732 (time 1786815024.79 total 60.07 last 82.42) - open/close 5523 (time 1786815034.79 total 70.07 last 79.09) - open/close 6413 (time 1786815044.79 total 80.07 last 88.97) - open/close 7295 (time 1786815054.79 total 90.07 last 88.18) - open/close 8376 (time 1786815064.80 total 100.08 last 108.02) - open/close 9521 (time 1786815074.80 total 110.08 last 114.48) total: 10000 open/close in 116.07 seconds: 86.15 ops/second - unlinked 0 (time 1786815084 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 661 (time 1786815171.30 total 10.01 last 66.01) - open/close 1568 (time 1786815181.31 total 20.03 last 90.59) - open/close 2524 (time 1786815191.32 total 30.03 last 95.58) - open/close 3470 (time 1786815201.32 total 40.03 last 94.57) - open/close 4337 (time 1786815211.32 total 50.03 last 86.69) - open/close 5129 (time 1786815221.33 total 60.04 last 79.16) - open/close 5853 (time 1786815231.33 total 70.04 last 72.37) - open/close 6724 (time 1786815241.35 total 80.06 last 86.95) - open/close 7595 (time 1786815251.36 total 90.07 last 87.02) - open/close 8503 (time 1786815261.36 total 100.07 last 90.77) - open/close 9330 (time 1786815271.36 total 110.07 last 82.68) total: 10000 open/close in 116.86 seconds: 85.57 ops/second - unlinked 0 (time 1786815282 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 855 (time 1786815364.40 total 10.01 last 85.42) - open/close 1506 (time 1786815374.41 total 20.02 last 65.01) - open/close 2408 (time 1786815384.42 total 30.03 last 90.14) - open/close 3408 (time 1786815394.42 total 40.04 last 99.93) - open/close 4345 (time 1786815404.43 total 50.05 last 93.60) - open/close 5145 (time 1786815414.44 total 60.05 last 79.98) - open/close 5993 (time 1786815424.44 total 70.06 last 84.74) - open/close 6799 (time 1786815434.45 total 80.07 last 80.53) - open/close 7681 (time 1786815444.46 total 90.07 last 88.14) - open/close 8537 (time 1786815454.47 total 100.08 last 85.51) - open/close 9559 (time 1786815464.47 total 110.09 last 102.18) total: 10000 open/close in 115.58 seconds: 86.52 ops/second - unlinked 0 (time 1786815474 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.116@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.116@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.116@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.116@tcp sys.timeout=20 start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local oleg116-server: modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2