== conf-sanity test 69: replace an OST with the same index ========================================================== 09:39:31 (1779975571) start mds service on oleg214-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg214-server: detected 4 online CPUs by sysfs oleg214-server: MODOPTS_LIBCFS= oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg214-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 821 (time 1779975693.32 total 10.01 last 82.05) - open/close 1730 (time 1779975703.32 total 20.01 last 90.86) - open/close 2532 (time 1779975713.33 total 30.02 last 80.16) - open/close 3280 (time 1779975723.33 total 40.02 last 74.75) - open/close 4003 (time 1779975733.35 total 50.04 last 72.16) - open/close 4537 (time 1779975743.37 total 60.06 last 53.32) - open/close 5131 (time 1779975753.38 total 70.07 last 59.33) - open/close 5577 (time 1779975763.39 total 80.08 last 44.55) - open/close 6315 (time 1779975773.40 total 90.09 last 73.78) - open/close 6990 (time 1779975783.42 total 100.10 last 67.37) - open/close 7570 (time 1779975793.42 total 110.10 last 57.99) - open/close 8344 (time 1779975803.43 total 120.12 last 77.32) - open/close 9062 (time 1779975813.45 total 130.14 last 71.63) - open/close 9637 (time 1779975823.46 total 140.14 last 57.48) total: 10000 open/close in 147.07 seconds: 68.00 ops/second - unlinked 0 (time 1779975837 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 836 (time 1779975946.69 total 10.00 last 83.59) - open/close 1501 (time 1779975956.70 total 20.01 last 66.47) - open/close 2341 (time 1779975966.71 total 30.02 last 83.86) - open/close 2984 (time 1779975976.71 total 40.02 last 64.29) - open/close 3649 (time 1779975986.72 total 50.03 last 66.50) - open/close 4447 (time 1779975996.72 total 60.03 last 79.73) - open/close 5184 (time 1779976006.74 total 70.05 last 73.61) - open/close 5999 (time 1779976016.74 total 80.05 last 81.46) - open/close 6730 (time 1779976026.76 total 90.07 last 72.94) - open/close 7357 (time 1779976036.77 total 100.08 last 62.67) - open/close 8101 (time 1779976046.78 total 110.09 last 74.32) - open/close 8876 (time 1779976056.79 total 120.10 last 77.42) - open/close 9573 (time 1779976066.79 total 130.10 last 69.69) total: 10000 open/close in 136.75 seconds: 73.13 ops/second - unlinked 0 (time 1779976078 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 704 (time 1779976182.44 total 10.03 last 70.18) - open/close 1456 (time 1779976192.45 total 20.04 last 75.12) - open/close 2081 (time 1779976202.45 total 30.04 last 62.48) - open/close 2838 (time 1779976212.46 total 40.05 last 75.64) - open/close 3568 (time 1779976222.48 total 50.08 last 72.83) - open/close 4360 (time 1779976232.49 total 60.08 last 79.15) - open/close 5046 (time 1779976242.49 total 70.08 last 68.60) - open/close 5709 (time 1779976252.50 total 80.09 last 66.22) - open/close 6320 (time 1779976262.50 total 90.10 last 61.09) - open/close 7058 (time 1779976272.54 total 100.13 last 73.57) - open/close 7877 (time 1779976282.54 total 110.13 last 81.85) - open/close 8722 (time 1779976292.55 total 120.14 last 84.45) - open/close 9673 (time 1779976302.55 total 130.14 last 95.10) total: 10000 open/close in 133.70 seconds: 74.79 ops/second - unlinked 0 (time 1779976309 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 700 (time 1779976405.88 total 10.00 last 69.99) - open/close 1432 (time 1779976415.89 total 20.01 last 73.13) - open/close 2027 (time 1779976425.90 total 30.02 last 59.47) - open/close 2599 (time 1779976435.91 total 40.03 last 57.11) - open/close 3254 (time 1779976445.92 total 50.04 last 65.46) - open/close 3916 (time 1779976455.93 total 60.04 last 66.16) - open/close 4679 (time 1779976465.93 total 70.05 last 76.26) - open/close 5422 (time 1779976475.94 total 80.06 last 74.25) - open/close 6160 (time 1779976485.94 total 90.06 last 73.78) - open/close 6796 (time 1779976495.95 total 100.07 last 63.56) - open/close 7319 (time 1779976505.95 total 110.07 last 52.27) - open/close 8102 (time 1779976515.96 total 120.08 last 78.28) - open/close 8833 (time 1779976525.97 total 130.09 last 73.00) - open/close 9724 (time 1779976535.98 total 140.10 last 89.00) total: 10000 open/close in 143.81 seconds: 69.54 ops/second - unlinked 0 (time 1779976544 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 754 (time 1779976640.66 total 10.01 last 75.33) - open/close 1564 (time 1779976650.67 total 20.02 last 80.89) - open/close 2335 (time 1779976660.67 total 30.02 last 77.09) - open/close 3078 (time 1779976670.68 total 40.03 last 74.25) - open/close 3895 (time 1779976680.69 total 50.04 last 81.62) - open/close 4557 (time 1779976690.70 total 60.05 last 66.12) - open/close 5310 (time 1779976700.71 total 70.06 last 75.25) - open/close 6040 (time 1779976710.71 total 80.07 last 72.95) - open/close 6799 (time 1779976720.72 total 90.07 last 75.85) - open/close 7569 (time 1779976730.72 total 100.07 last 77.00) - open/close 8274 (time 1779976740.73 total 110.09 last 70.41) - open/close 8981 (time 1779976750.74 total 120.09 last 70.64) - open/close 9761 (time 1779976760.75 total 130.10 last 77.94) total: 10000 open/close in 133.90 seconds: 74.68 ops/second - unlinked 0 (time 1779976768 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 519 (time 1779976861.37 total 10.01 last 51.87) - open/close 1289 (time 1779976871.37 total 20.01 last 76.99) - open/close 2019 (time 1779976881.38 total 30.02 last 72.94) - open/close 2783 (time 1779976891.38 total 40.02 last 76.37) - open/close 3605 (time 1779976901.38 total 50.02 last 82.19) - open/close 4428 (time 1779976911.38 total 60.02 last 82.29) - open/close 5080 (time 1779976921.39 total 70.03 last 65.14) - open/close 5829 (time 1779976931.40 total 80.04 last 74.85) - open/close 6640 (time 1779976941.41 total 90.04 last 81.06) - open/close 7476 (time 1779976951.41 total 100.04 last 83.59) - open/close 8412 (time 1779976961.41 total 110.04 last 93.59) - open/close 9306 (time 1779976971.41 total 120.05 last 89.37) total: 10000 open/close in 129.07 seconds: 77.47 ops/second - unlinked 0 (time 1779976985 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 918 (time 1779977090.48 total 10.01 last 91.73) - open/close 1645 (time 1779977100.49 total 20.01 last 72.66) - open/close 2534 (time 1779977110.50 total 30.02 last 88.83) - open/close 3455 (time 1779977120.50 total 40.03 last 92.07) - open/close 4219 (time 1779977130.51 total 50.04 last 76.31) - open/close 5014 (time 1779977140.52 total 60.05 last 79.44) - open/close 5844 (time 1779977150.53 total 70.05 last 82.96) - open/close 6580 (time 1779977160.53 total 80.06 last 73.53) - open/close 7269 (time 1779977170.54 total 90.06 last 68.90) - open/close 8000 (time 1779977180.54 total 100.06 last 73.07) - open/close 8817 (time 1779977190.54 total 110.07 last 81.67) - open/close 9713 (time 1779977200.54 total 120.07 last 89.59) total: 10000 open/close in 124.16 seconds: 80.54 ops/second - unlinked 0 (time 1779977210 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 703 (time 1779977305.96 total 10.01 last 70.23) - open/close 1521 (time 1779977315.97 total 20.02 last 81.73) - open/close 2388 (time 1779977325.97 total 30.03 last 86.64) - open/close 3116 (time 1779977335.98 total 40.03 last 72.76) - open/close 3953 (time 1779977345.98 total 50.03 last 83.69) - open/close 4716 (time 1779977355.99 total 60.04 last 76.24) - open/close 5641 (time 1779977365.99 total 70.04 last 92.48) - open/close 6485 (time 1779977375.99 total 80.05 last 84.38) - open/close 7339 (time 1779977386.00 total 90.05 last 85.34) - open/close 8207 (time 1779977396.01 total 100.07 last 86.69) - open/close 8997 (time 1779977406.02 total 110.07 last 78.93) - open/close 9724 (time 1779977416.03 total 120.08 last 72.66) total: 10000 open/close in 123.48 seconds: 80.98 ops/second - unlinked 0 (time 1779977423 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 771 (time 1779977520.91 total 10.01 last 77.06) - open/close 1635 (time 1779977530.92 total 20.02 last 86.31) - open/close 2350 (time 1779977540.95 total 30.05 last 71.28) - open/close 2895 (time 1779977550.97 total 40.06 last 54.41) - open/close 3633 (time 1779977560.98 total 50.07 last 73.71) - open/close 4498 (time 1779977570.98 total 60.07 last 86.49) - open/close 5197 (time 1779977580.99 total 70.08 last 69.83) - open/close 5927 (time 1779977590.99 total 80.08 last 73.00) - open/close 6497 (time 1779977601.01 total 90.10 last 56.91) - open/close 7260 (time 1779977611.02 total 100.11 last 76.24) - open/close 8044 (time 1779977621.02 total 110.11 last 78.39) - open/close 8859 (time 1779977631.03 total 120.12 last 81.42) - open/close 9616 (time 1779977641.06 total 130.15 last 75.51) total: 10000 open/close in 135.07 seconds: 74.04 ops/second - unlinked 0 (time 1779977650 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 849 (time 1779977735.73 total 10.01 last 84.84) - open/close 1583 (time 1779977745.74 total 20.01 last 73.37) - open/close 2475 (time 1779977755.74 total 30.02 last 89.13) - open/close 3491 (time 1779977765.75 total 40.03 last 101.52) - open/close 4358 (time 1779977775.76 total 50.03 last 86.66) - open/close 5349 (time 1779977785.76 total 60.03 last 99.07) - open/close 6194 (time 1779977795.76 total 70.04 last 84.45) - open/close 7151 (time 1779977805.77 total 80.04 last 95.67) - open/close 7963 (time 1779977815.77 total 90.05 last 81.14) - open/close 8869 (time 1779977825.78 total 100.05 last 90.58) - open/close 9599 (time 1779977835.78 total 110.06 last 72.94) total: 10000 open/close in 114.38 seconds: 87.43 ops/second - unlinked 0 (time 1779977844 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: modules unloaded. pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2