== conf-sanity test 69: replace an OST with the same index ========================================================== 15:24:28 (1786649068) start mds service on oleg112-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' llite/lustre options: 'enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg112-server: detected 4 online CPUs by sysfs oleg112-server: MODOPTS_LIBCFS= oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg112-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg112-server: llite/lustre options: 'enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg112-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg112-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 710 (time 1786649191.73 total 10.00 last 70.97) - open/close 1493 (time 1786649201.75 total 20.02 last 78.19) - open/close 2270 (time 1786649211.76 total 30.03 last 77.64) - open/close 3154 (time 1786649221.76 total 40.03 last 88.36) - open/close 3827 (time 1786649231.76 total 50.03 last 67.28) - open/close 4635 (time 1786649241.77 total 60.04 last 80.77) - open/close 5432 (time 1786649251.77 total 70.04 last 79.68) - open/close 6037 (time 1786649261.78 total 80.05 last 60.46) - open/close 6868 (time 1786649271.78 total 90.05 last 83.09) - open/close 7642 (time 1786649281.78 total 100.05 last 77.40) - open/close 8453 (time 1786649291.78 total 110.05 last 81.07) - open/close 9313 (time 1786649301.79 total 120.06 last 85.89) total: 10000 open/close in 129.20 seconds: 77.40 ops/second - unlinked 0 (time 1786649315 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 841 (time 1786649391.19 total 10.00 last 84.09) - open/close 1729 (time 1786649401.20 total 20.01 last 88.74) - open/close 2526 (time 1786649411.21 total 30.02 last 79.62) - open/close 3363 (time 1786649421.21 total 40.02 last 83.69) - open/close 4253 (time 1786649431.22 total 50.03 last 88.91) - open/close 5014 (time 1786649441.22 total 60.03 last 76.07) - open/close 5635 (time 1786649451.23 total 70.04 last 62.06) - open/close 6425 (time 1786649461.23 total 80.04 last 78.97) - open/close 7228 (time 1786649471.24 total 90.05 last 80.22) - open/close 7936 (time 1786649481.25 total 100.06 last 70.74) - open/close 8735 (time 1786649491.25 total 110.06 last 79.87) - open/close 9607 (time 1786649501.26 total 120.07 last 87.15) total: 10000 open/close in 125.05 seconds: 79.97 ops/second - unlinked 0 (time 1786649511 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 887 (time 1786649590.72 total 10.00 last 88.67) - open/close 1911 (time 1786649600.73 total 20.02 last 102.26) - open/close 2802 (time 1786649610.74 total 30.03 last 89.00) - open/close 3662 (time 1786649620.75 total 40.03 last 85.95) - open/close 4448 (time 1786649630.75 total 50.04 last 78.56) - open/close 5347 (time 1786649640.76 total 60.05 last 89.82) - open/close 6119 (time 1786649650.76 total 70.05 last 77.19) - open/close 6972 (time 1786649660.77 total 80.05 last 85.28) - open/close 7953 (time 1786649670.77 total 90.06 last 98.02) - open/close 8894 (time 1786649680.78 total 100.07 last 94.03) - open/close 9730 (time 1786649690.78 total 110.07 last 83.58) total: 10000 open/close in 113.23 seconds: 88.32 ops/second - unlinked 0 (time 1786649698 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 892 (time 1786649784.09 total 10.01 last 89.11) - open/close 1656 (time 1786649794.10 total 20.02 last 76.34) - open/close 2537 (time 1786649804.10 total 30.02 last 88.10) - open/close 3413 (time 1786649814.11 total 40.03 last 87.54) - open/close 4367 (time 1786649824.12 total 50.03 last 95.33) - open/close 5303 (time 1786649834.15 total 60.06 last 93.32) - open/close 6237 (time 1786649844.15 total 70.06 last 93.38) - open/close 7282 (time 1786649854.15 total 80.07 last 104.49) - open/close 8299 (time 1786649864.15 total 90.07 last 101.65) - open/close 9219 (time 1786649874.17 total 100.09 last 91.83) total: 10000 open/close in 107.67 seconds: 92.88 ops/second - unlinked 0 (time 1786649886 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 889 (time 1786649967.04 total 10.01 last 88.85) - open/close 1807 (time 1786649977.05 total 20.02 last 91.70) - open/close 2672 (time 1786649987.05 total 30.02 last 86.47) - open/close 3573 (time 1786649997.06 total 40.02 last 90.07) - open/close 4412 (time 1786650007.06 total 50.03 last 83.86) - open/close 5398 (time 1786650017.07 total 60.03 last 98.53) - open/close 6314 (time 1786650027.07 total 70.04 last 91.55) - open/close 7197 (time 1786650037.07 total 80.04 last 88.29) - open/close 8062 (time 1786650047.10 total 90.07 last 86.29) - open/close 9093 (time 1786650057.10 total 100.07 last 103.07) total: 10000 open/close in 108.66 seconds: 92.03 ops/second - unlinked 0 (time 1786650069 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 963 (time 1786650143.49 total 10.01 last 96.24) - open/close 1892 (time 1786650153.49 total 20.01 last 92.89) - open/close 2801 (time 1786650163.50 total 30.01 last 90.86) - open/close 3810 (time 1786650173.51 total 40.02 last 100.79) - open/close 4868 (time 1786650183.51 total 50.03 last 105.76) - open/close 5710 (time 1786650193.52 total 60.03 last 84.15) - open/close 6456 (time 1786650203.52 total 70.04 last 74.55) - open/close 7366 (time 1786650213.53 total 80.04 last 90.98) - open/close 8288 (time 1786650223.53 total 90.05 last 92.17) - open/close 9257 (time 1786650233.54 total 100.06 last 96.80) total: 10000 open/close in 108.78 seconds: 91.93 ops/second - unlinked 0 (time 1786650247 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1055 (time 1786650325.87 total 10.00 last 105.49) - open/close 1885 (time 1786650335.87 total 20.00 last 82.98) - open/close 2966 (time 1786650345.88 total 30.01 last 108.02) - open/close 4014 (time 1786650355.88 total 40.01 last 104.78) - open/close 4794 (time 1786650365.90 total 50.02 last 77.90) - open/close 5704 (time 1786650375.90 total 60.03 last 90.94) - open/close 6450 (time 1786650385.92 total 70.04 last 74.50) - open/close 7427 (time 1786650395.92 total 80.04 last 97.70) - open/close 8228 (time 1786650405.92 total 90.05 last 80.04) - open/close 9130 (time 1786650415.93 total 100.06 last 90.16) total: 10000 open/close in 108.13 seconds: 92.48 ops/second - unlinked 0 (time 1786650427 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 801 (time 1786650514.87 total 10.01 last 80.03) - open/close 1623 (time 1786650524.88 total 20.02 last 82.08) - open/close 2590 (time 1786650534.88 total 30.03 last 96.68) - open/close 3470 (time 1786650544.90 total 40.04 last 87.87) - open/close 4334 (time 1786650554.91 total 50.05 last 86.35) - open/close 5318 (time 1786650564.92 total 60.06 last 98.25) - open/close 6277 (time 1786650574.93 total 70.07 last 95.81) - open/close 7180 (time 1786650584.94 total 80.08 last 90.22) - open/close 7864 (time 1786650594.95 total 90.09 last 68.35) - open/close 8725 (time 1786650604.96 total 100.10 last 86.02) - open/close 9430 (time 1786650614.98 total 110.12 last 70.36) total: 10000 open/close in 117.73 seconds: 84.94 ops/second - unlinked 0 (time 1786650627 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 971 (time 1786650704.55 total 10.01 last 97.00) - open/close 1934 (time 1786650714.56 total 20.01 last 96.27) - open/close 2856 (time 1786650724.56 total 30.02 last 92.13) - open/close 3908 (time 1786650734.57 total 40.02 last 105.16) - open/close 4867 (time 1786650744.58 total 50.03 last 95.82) - open/close 5835 (time 1786650754.58 total 60.04 last 96.76) - open/close 6719 (time 1786650764.59 total 70.04 last 88.35) - open/close 7578 (time 1786650774.59 total 80.05 last 85.84) - open/close 8302 (time 1786650784.61 total 90.06 last 72.31) - open/close 9164 (time 1786650794.61 total 100.07 last 86.14) total: 10000 open/close in 109.10 seconds: 91.66 ops/second - unlinked 0 (time 1786650807 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 769 (time 1786650892.76 total 10.00 last 76.87) - open/close 1660 (time 1786650902.77 total 20.01 last 89.01) - open/close 2147 (time 1786650912.77 total 30.02 last 48.69) - open/close 2739 (time 1786650922.77 total 40.02 last 59.19) - open/close 3590 (time 1786650932.78 total 50.02 last 85.04) - open/close 4386 (time 1786650942.79 total 60.03 last 79.54) - open/close 5260 (time 1786650952.80 total 70.05 last 87.25) - open/close 6187 (time 1786650962.80 total 80.05 last 92.68) - open/close 6984 (time 1786650972.81 total 90.06 last 79.66) - open/close 8032 (time 1786650982.82 total 100.06 last 104.74) - open/close 8963 (time 1786650992.82 total 110.07 last 93.05) - open/close 9966 (time 1786651002.82 total 120.07 last 100.26) total: 10000 open/close in 120.35 seconds: 83.09 ops/second - unlinked 0 (time 1786651007 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg112-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 start ost1 service on oleg112-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg112-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2