== conf-sanity test 69: replace an OST with the same index ========================================================== 19:29:50 (1786750190) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg433-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg433-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 551 (time 1786750302.52 total 10.01 last 55.04) - open/close 1015 (time 1786750312.53 total 20.02 last 46.37) - open/close 1670 (time 1786750322.54 total 30.02 last 65.44) - open/close 2395 (time 1786750332.56 total 40.05 last 72.35) - open/close 3050 (time 1786750342.58 total 50.06 last 65.38) - open/close 3695 (time 1786750352.58 total 60.07 last 64.46) - open/close 4328 (time 1786750362.59 total 70.08 last 63.27) - open/close 4966 (time 1786750372.61 total 80.09 last 63.68) - open/close 5739 (time 1786750382.61 total 90.10 last 77.28) - open/close 6450 (time 1786750392.62 total 100.10 last 71.06) - open/close 7109 (time 1786750402.62 total 110.10 last 65.90) - open/close 7745 (time 1786750412.62 total 120.11 last 63.58) - open/close 8420 (time 1786750422.63 total 130.12 last 67.43) - open/close 9086 (time 1786750432.64 total 140.13 last 66.53) - open/close 9779 (time 1786750442.64 total 150.13 last 69.27) total: 10000 open/close in 153.08 seconds: 65.33 ops/second - unlinked 0 (time 1786750450 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 735 (time 1786750541.83 total 10.01 last 73.46) - open/close 1412 (time 1786750551.85 total 20.02 last 67.59) - open/close 2080 (time 1786750561.85 total 30.02 last 66.79) - open/close 2864 (time 1786750571.86 total 40.03 last 78.33) - open/close 3655 (time 1786750581.87 total 50.04 last 79.05) - open/close 4351 (time 1786750591.87 total 60.04 last 69.58) - open/close 5046 (time 1786750601.89 total 70.06 last 69.38) - open/close 5675 (time 1786750611.90 total 80.07 last 62.82) - open/close 6282 (time 1786750621.91 total 90.08 last 60.65) - open/close 6934 (time 1786750631.91 total 100.09 last 65.16) - open/close 7574 (time 1786750641.92 total 110.09 last 63.95) - open/close 8127 (time 1786750651.93 total 120.10 last 55.25) - open/close 8802 (time 1786750661.94 total 130.11 last 67.43) - open/close 9382 (time 1786750671.97 total 140.14 last 57.83) total: 10000 open/close in 148.93 seconds: 67.15 ops/second - unlinked 0 (time 1786750685 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 756 (time 1786750791.23 total 10.01 last 75.56) - open/close 1441 (time 1786750801.23 total 20.01 last 68.47) - open/close 2197 (time 1786750811.23 total 30.01 last 75.60) - open/close 2965 (time 1786750821.25 total 40.02 last 76.71) - open/close 3765 (time 1786750831.25 total 50.03 last 79.96) - open/close 4571 (time 1786750841.26 total 60.03 last 80.55) - open/close 5332 (time 1786750851.27 total 70.04 last 76.03) - open/close 6053 (time 1786750861.27 total 80.05 last 72.05) - open/close 6826 (time 1786750871.28 total 90.05 last 77.28) - open/close 7572 (time 1786750881.29 total 100.06 last 74.52) - open/close 8337 (time 1786750891.29 total 110.06 last 76.50) - open/close 9052 (time 1786750901.29 total 120.07 last 71.47) - open/close 9867 (time 1786750911.30 total 130.07 last 81.44) total: 10000 open/close in 132.35 seconds: 75.56 ops/second - unlinked 0 (time 1786750919 ; total 1 ; last 1) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 731 (time 1786751020.22 total 10.01 last 73.05) - open/close 1420 (time 1786751030.24 total 20.02 last 68.78) - open/close 2063 (time 1786751040.24 total 30.03 last 64.26) - open/close 2833 (time 1786751050.25 total 40.04 last 76.94) - open/close 3543 (time 1786751060.26 total 50.05 last 70.94) - open/close 4267 (time 1786751070.28 total 60.07 last 72.26) - open/close 4882 (time 1786751080.31 total 70.09 last 61.33) - open/close 5560 (time 1786751090.31 total 80.10 last 67.77) - open/close 6238 (time 1786751100.31 total 90.10 last 67.79) - open/close 6992 (time 1786751110.32 total 100.11 last 75.34) - open/close 7676 (time 1786751120.34 total 110.13 last 68.23) - open/close 8412 (time 1786751130.35 total 120.14 last 73.57) - open/close 9162 (time 1786751140.36 total 130.15 last 74.91) - open/close 9770 (time 1786751150.37 total 140.16 last 60.75) total: 10000 open/close in 143.84 seconds: 69.52 ops/second - unlinked 0 (time 1786751160 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 438 (time 1786751268.82 total 10.01 last 43.77) - open/close 1020 (time 1786751278.82 total 20.01 last 58.20) - open/close 1793 (time 1786751288.83 total 30.01 last 77.25) - open/close 2597 (time 1786751298.83 total 40.02 last 80.37) - open/close 3345 (time 1786751308.84 total 50.02 last 74.76) - open/close 4052 (time 1786751318.84 total 60.03 last 70.67) - open/close 4731 (time 1786751328.85 total 70.04 last 67.83) - open/close 5546 (time 1786751338.86 total 80.05 last 81.43) - open/close 6258 (time 1786751348.86 total 90.05 last 71.18) - open/close 6963 (time 1786751358.88 total 100.07 last 70.37) - open/close 7735 (time 1786751368.89 total 110.08 last 77.10) - open/close 8443 (time 1786751378.90 total 120.09 last 70.73) - open/close 9118 (time 1786751388.91 total 130.09 last 67.49) - open/close 9819 (time 1786751398.91 total 140.09 last 70.10) total: 10000 open/close in 142.18 seconds: 70.34 ops/second - unlinked 0 (time 1786751406 ; total 1 ; last 1) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 776 (time 1786751498.24 total 10.00 last 77.56) - open/close 1581 (time 1786751508.25 total 20.02 last 80.41) - open/close 2256 (time 1786751518.25 total 30.02 last 67.46) - open/close 3087 (time 1786751528.26 total 40.03 last 83.07) - open/close 3853 (time 1786751538.26 total 50.03 last 76.58) - open/close 4677 (time 1786751548.27 total 60.04 last 82.34) - open/close 5478 (time 1786751558.28 total 70.05 last 80.02) - open/close 6280 (time 1786751568.28 total 80.05 last 80.16) - open/close 7112 (time 1786751578.29 total 90.06 last 83.15) - open/close 7821 (time 1786751588.31 total 100.07 last 70.77) - open/close 8481 (time 1786751598.34 total 110.11 last 65.74) - open/close 9206 (time 1786751608.35 total 120.12 last 72.48) - open/close 9927 (time 1786751618.35 total 130.12 last 72.07) total: 10000 open/close in 131.14 seconds: 76.25 ops/second - unlinked 0 (time 1786751624 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 752 (time 1786751716.28 total 10.00 last 75.17) - open/close 1479 (time 1786751726.29 total 20.01 last 72.63) - open/close 2254 (time 1786751736.29 total 30.01 last 77.49) - open/close 3016 (time 1786751746.30 total 40.02 last 76.19) - open/close 3807 (time 1786751756.31 total 50.03 last 79.02) - open/close 4562 (time 1786751766.31 total 60.03 last 75.44) - open/close 5269 (time 1786751776.32 total 70.04 last 70.64) - open/close 6019 (time 1786751786.32 total 80.04 last 74.98) - open/close 6939 (time 1786751796.34 total 90.06 last 91.82) - open/close 7626 (time 1786751806.35 total 100.07 last 68.65) - open/close 8443 (time 1786751816.35 total 110.07 last 81.69) - open/close 9315 (time 1786751826.36 total 120.08 last 87.10) total: 10000 open/close in 128.26 seconds: 77.97 ops/second - unlinked 0 (time 1786751839 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 584 (time 1786751922.62 total 10.02 last 58.30) - open/close 1237 (time 1786751932.63 total 20.03 last 65.24) - open/close 1916 (time 1786751942.66 total 30.06 last 67.69) - open/close 2786 (time 1786751952.67 total 40.06 last 86.93) - open/close 3663 (time 1786751962.67 total 50.07 last 87.67) - open/close 4517 (time 1786751972.69 total 60.08 last 85.26) - open/close 5323 (time 1786751982.70 total 70.09 last 80.55) - open/close 6058 (time 1786751992.72 total 80.12 last 73.29) - open/close 6800 (time 1786752002.73 total 90.12 last 74.19) - open/close 7598 (time 1786752012.74 total 100.13 last 79.69) - open/close 8416 (time 1786752022.75 total 110.14 last 81.71) - open/close 9118 (time 1786752032.76 total 120.16 last 70.11) - open/close 9700 (time 1786752042.78 total 130.17 last 58.12) total: 10000 open/close in 134.66 seconds: 74.26 ops/second - unlinked 0 (time 1786752053 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 805 (time 1786752140.68 total 10.00 last 80.46) - open/close 1644 (time 1786752150.69 total 20.01 last 83.82) - open/close 2399 (time 1786752160.69 total 30.02 last 75.46) - open/close 3096 (time 1786752170.70 total 40.03 last 69.61) - open/close 3674 (time 1786752180.72 total 50.05 last 57.69) - open/close 4302 (time 1786752190.72 total 60.05 last 62.79) - open/close 5025 (time 1786752200.73 total 70.06 last 72.25) - open/close 5893 (time 1786752210.74 total 80.07 last 86.71) - open/close 6675 (time 1786752220.75 total 90.08 last 78.15) - open/close 7354 (time 1786752230.75 total 100.08 last 67.88) - open/close 7931 (time 1786752240.77 total 110.10 last 57.61) - open/close 8643 (time 1786752250.77 total 120.10 last 71.18) - open/close 9333 (time 1786752260.78 total 130.11 last 68.91) total: 10000 open/close in 139.13 seconds: 71.88 ops/second - unlinked 0 (time 1786752275 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 614 (time 1786752371.30 total 10.00 last 61.39) - open/close 1315 (time 1786752381.31 total 20.01 last 70.05) - open/close 1993 (time 1786752391.32 total 30.02 last 67.75) - open/close 2591 (time 1786752401.34 total 40.04 last 59.69) - open/close 3319 (time 1786752411.34 total 50.04 last 72.77) - open/close 3942 (time 1786752421.35 total 60.05 last 62.26) - open/close 4651 (time 1786752431.36 total 70.06 last 70.83) - open/close 5319 (time 1786752441.37 total 80.07 last 66.73) - open/close 6072 (time 1786752451.38 total 90.07 last 75.25) - open/close 6858 (time 1786752461.38 total 100.08 last 78.58) - open/close 7470 (time 1786752471.39 total 110.09 last 61.14) - open/close 8307 (time 1786752481.39 total 120.09 last 83.66) - open/close 9072 (time 1786752491.40 total 130.10 last 76.45) - open/close 9704 (time 1786752501.40 total 140.10 last 63.18) total: 10000 open/close in 144.56 seconds: 69.18 ops/second - unlinked 0 (time 1786752511 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2