== conf-sanity test 69: replace an OST with the same index ========================================================== 11:00:27 (1787842827) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg318-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg318-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 865 (time 1787842927.78 total 10.00 last 86.48) - open/close 1832 (time 1787842937.78 total 20.01 last 96.66) - open/close 2803 (time 1787842947.80 total 30.03 last 96.90) - open/close 3760 (time 1787842957.81 total 40.03 last 95.65) - open/close 4702 (time 1787842967.81 total 50.04 last 94.17) - open/close 5519 (time 1787842977.82 total 60.04 last 81.66) - open/close 6486 (time 1787842987.83 total 70.05 last 96.58) - open/close 7426 (time 1787842997.84 total 80.06 last 93.96) - open/close 8328 (time 1787843007.84 total 90.06 last 90.18) - open/close 9293 (time 1787843017.85 total 100.07 last 96.40) total: 10000 open/close in 107.45 seconds: 93.07 ops/second - unlinked 0 (time 1787843028 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 935 (time 1787843101.31 total 10.00 last 93.46) - open/close 1776 (time 1787843111.31 total 20.01 last 84.05) - open/close 2681 (time 1787843121.32 total 30.02 last 90.45) - open/close 3517 (time 1787843131.33 total 40.03 last 83.50) - open/close 4453 (time 1787843141.34 total 50.03 last 93.55) - open/close 5367 (time 1787843151.34 total 60.03 last 91.39) - open/close 6183 (time 1787843161.34 total 70.04 last 81.57) - open/close 7090 (time 1787843171.35 total 80.05 last 90.62) - open/close 8028 (time 1787843181.36 total 90.05 last 93.74) - open/close 9047 (time 1787843191.36 total 100.06 last 101.86) total: 10000 open/close in 109.66 seconds: 91.19 ops/second - unlinked 0 (time 1787843204 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 931 (time 1787843282.99 total 10.00 last 93.08) - open/close 1804 (time 1787843293.00 total 20.01 last 87.26) - open/close 2651 (time 1787843303.00 total 30.01 last 84.65) - open/close 3466 (time 1787843313.00 total 40.01 last 81.50) - open/close 4395 (time 1787843323.01 total 50.02 last 92.84) - open/close 5352 (time 1787843333.01 total 60.02 last 95.66) - open/close 6278 (time 1787843343.02 total 70.03 last 92.57) - open/close 7086 (time 1787843353.02 total 80.03 last 80.77) - open/close 7985 (time 1787843363.03 total 90.04 last 89.84) - open/close 8822 (time 1787843373.04 total 100.05 last 83.57) - open/close 9648 (time 1787843383.04 total 110.05 last 82.59) total: 10000 open/close in 114.38 seconds: 87.43 ops/second - unlinked 0 (time 1787843391 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 771 (time 1787843468.35 total 10.00 last 77.07) - open/close 1563 (time 1787843478.36 total 20.01 last 79.13) - open/close 2511 (time 1787843488.36 total 30.01 last 94.80) - open/close 3444 (time 1787843498.36 total 40.02 last 93.26) - open/close 4342 (time 1787843508.37 total 50.02 last 89.76) - open/close 5319 (time 1787843518.38 total 60.03 last 97.61) - open/close 6306 (time 1787843528.39 total 70.04 last 98.62) - open/close 7235 (time 1787843538.39 total 80.04 last 92.85) - open/close 8201 (time 1787843548.40 total 90.05 last 96.56) - open/close 9079 (time 1787843558.40 total 100.05 last 87.76) - open/close 9988 (time 1787843568.42 total 110.07 last 90.72) total: 10000 open/close in 110.17 seconds: 90.77 ops/second - unlinked 0 (time 1787843572 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 817 (time 1787843649.56 total 10.00 last 81.68) - open/close 1666 (time 1787843659.57 total 20.02 last 84.79) - open/close 2536 (time 1787843669.58 total 30.02 last 86.95) - open/close 3462 (time 1787843679.60 total 40.04 last 92.39) - open/close 4262 (time 1787843689.62 total 50.07 last 79.83) - open/close 5180 (time 1787843699.63 total 60.08 last 91.70) - open/close 6045 (time 1787843709.64 total 70.08 last 86.49) - open/close 6909 (time 1787843719.64 total 80.08 last 86.38) - open/close 7831 (time 1787843729.65 total 90.09 last 92.12) - open/close 8756 (time 1787843739.65 total 100.09 last 92.50) - open/close 9647 (time 1787843749.66 total 110.10 last 89.01) total: 10000 open/close in 114.01 seconds: 87.71 ops/second - unlinked 0 (time 1787843757 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 903 (time 1787843833.72 total 10.00 last 90.26) - open/close 1760 (time 1787843843.72 total 20.01 last 85.64) - open/close 2647 (time 1787843853.72 total 30.01 last 88.69) - open/close 3470 (time 1787843863.73 total 40.02 last 82.25) - open/close 4295 (time 1787843873.74 total 50.03 last 82.38) - open/close 5174 (time 1787843883.75 total 60.03 last 87.88) - open/close 6071 (time 1787843893.75 total 70.04 last 89.65) - open/close 6900 (time 1787843903.76 total 80.05 last 82.87) - open/close 7769 (time 1787843913.77 total 90.05 last 86.83) - open/close 8618 (time 1787843923.78 total 100.07 last 84.79) - open/close 9481 (time 1787843933.79 total 110.07 last 86.24) total: 10000 open/close in 115.96 seconds: 86.24 ops/second - unlinked 0 (time 1787843943 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 939 (time 1787844016.31 total 10.00 last 93.87) - open/close 1866 (time 1787844026.31 total 20.00 last 92.69) - open/close 2736 (time 1787844036.31 total 30.01 last 86.99) - open/close 3670 (time 1787844046.32 total 40.01 last 93.35) - open/close 4620 (time 1787844056.33 total 50.02 last 94.92) - open/close 5604 (time 1787844066.33 total 60.02 last 98.35) - open/close 6511 (time 1787844076.34 total 70.03 last 90.63) - open/close 7487 (time 1787844086.34 total 80.04 last 97.56) - open/close 8438 (time 1787844096.34 total 90.04 last 95.09) - open/close 9357 (time 1787844106.35 total 100.04 last 91.83) total: 10000 open/close in 106.96 seconds: 93.49 ops/second - unlinked 0 (time 1787844117 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 950 (time 1787844194.21 total 10.01 last 94.95) - open/close 1803 (time 1787844204.22 total 20.01 last 85.23) - open/close 2688 (time 1787844214.24 total 30.04 last 88.26) - open/close 3520 (time 1787844224.25 total 40.05 last 83.16) - open/close 4449 (time 1787844234.26 total 50.05 last 92.85) - open/close 5368 (time 1787844244.27 total 60.06 last 91.79) - open/close 6347 (time 1787844254.28 total 70.07 last 97.80) - open/close 7307 (time 1787844264.29 total 80.08 last 95.89) - open/close 8231 (time 1787844274.29 total 90.09 last 92.34) - open/close 9120 (time 1787844284.30 total 100.10 last 88.83) total: 10000 open/close in 109.34 seconds: 91.46 ops/second - unlinked 0 (time 1787844297 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 976 (time 1787844371.58 total 10.01 last 97.54) - open/close 1964 (time 1787844381.58 total 20.01 last 98.78) - open/close 2989 (time 1787844391.59 total 30.01 last 102.44) - open/close 3913 (time 1787844401.60 total 40.02 last 92.32) - open/close 4866 (time 1787844411.61 total 50.04 last 95.13) - open/close 5703 (time 1787844421.63 total 60.06 last 83.56) - open/close 6581 (time 1787844431.63 total 70.06 last 87.76) - open/close 7486 (time 1787844441.65 total 80.08 last 90.35) - open/close 8362 (time 1787844451.66 total 90.09 last 87.52) - open/close 9289 (time 1787844461.66 total 100.09 last 92.67) total: 10000 open/close in 107.98 seconds: 92.61 ops/second - unlinked 0 (time 1787844472 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 707 (time 1787844551.97 total 10.00 last 70.67) - open/close 1565 (time 1787844561.97 total 20.01 last 85.79) - open/close 2500 (time 1787844571.98 total 30.01 last 93.43) - open/close 3447 (time 1787844581.98 total 40.02 last 94.63) - open/close 4303 (time 1787844591.99 total 50.02 last 85.57) - open/close 5157 (time 1787844601.99 total 60.03 last 85.39) - open/close 5950 (time 1787844611.99 total 70.03 last 79.26) - open/close 6856 (time 1787844621.99 total 80.03 last 90.58) - open/close 7765 (time 1787844632.01 total 90.04 last 90.81) - open/close 8733 (time 1787844642.01 total 100.05 last 96.76) - open/close 9589 (time 1787844652.01 total 110.05 last 85.58) total: 10000 open/close in 114.12 seconds: 87.62 ops/second - unlinked 0 (time 1787844660 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg318-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg318-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg318-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg318-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg318-server unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2