== conf-sanity test 69: replace an OST with the same index ========================================================== 06:42:48 (1756809768) start mds service on oleg420-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1461 (time 1756809832.07 total 10.01 last 146.02) - open/close 2653 (time 1756809842.08 total 20.02 last 119.06) - open/close 3894 (time 1756809852.08 total 30.02 last 124.07) - open/close 5439 (time 1756809862.09 total 40.02 last 154.46) - open/close 6930 (time 1756809872.09 total 50.03 last 149.00) - open/close 8338 (time 1756809882.10 total 60.03 last 140.72) - open/close 9893 (time 1756809892.10 total 70.04 last 155.48) total: 10000 open/close in 70.53 seconds: 141.78 ops/second - unlinked 0 (time 1756809894 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1941 (time 1756809942.36 total 10.00 last 194.02) - open/close 3211 (time 1756809952.37 total 20.01 last 126.94) - open/close 4665 (time 1756809962.37 total 30.02 last 145.31) - open/close 6798 (time 1756809972.38 total 40.02 last 213.28) - open/close 8434 (time 1756809982.38 total 50.02 last 163.47) total: 10000 open/close in 58.49 seconds: 170.97 ops/second - unlinked 0 (time 1756809992 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2395 (time 1756810028.45 total 10.00 last 239.44) - open/close 4540 (time 1756810038.45 total 20.01 last 214.43) - open/close 5776 (time 1756810048.46 total 30.01 last 123.49) - open/close 6730 (time 1756810058.46 total 40.02 last 95.39) - open/close 7698 (time 1756810068.46 total 50.02 last 96.79) - open/close 9075 (time 1756810078.47 total 60.02 last 137.58) total: 10000 open/close in 66.53 seconds: 150.30 ops/second - unlinked 0 (time 1756810086 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1124 (time 1756810138.93 total 10.01 last 112.28) - open/close 2564 (time 1756810148.93 total 20.01 last 143.98) - open/close 3920 (time 1756810158.94 total 30.01 last 135.58) - open/close 5305 (time 1756810168.94 total 40.01 last 138.50) - open/close 6856 (time 1756810178.94 total 50.02 last 155.02) - open/close 8209 (time 1756810188.94 total 60.02 last 135.28) - open/close 9325 (time 1756810198.95 total 70.03 last 111.49) total: 10000 open/close in 75.30 seconds: 132.81 ops/second - unlinked 0 (time 1756810206 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1124 (time 1756810267.03 total 10.01 last 112.33) - open/close 2268 (time 1756810277.03 total 20.01 last 114.39) - open/close 3495 (time 1756810287.03 total 30.01 last 122.70) - open/close 5205 (time 1756810297.03 total 40.01 last 170.97) - open/close 7128 (time 1756810307.03 total 50.01 last 192.29) - open/close 9506 (time 1756810317.03 total 60.01 last 237.79) total: 10000 open/close in 61.80 seconds: 161.82 ops/second - unlinked 0 (time 1756810320 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2214 (time 1756810354.22 total 10.00 last 221.40) - open/close 3891 (time 1756810364.23 total 20.00 last 167.62) - open/close 5618 (time 1756810374.23 total 30.01 last 172.64) - open/close 7294 (time 1756810384.23 total 40.01 last 167.53) - open/close 9293 (time 1756810394.24 total 50.01 last 199.87) total: 10000 open/close in 52.66 seconds: 189.90 ops/second - unlinked 0 (time 1756810398 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1697 (time 1756810437.60 total 10.00 last 169.62) - open/close 3681 (time 1756810447.60 total 20.01 last 198.36) - open/close 5460 (time 1756810457.61 total 30.01 last 177.84) - open/close 7192 (time 1756810467.61 total 40.01 last 173.14) - open/close 9065 (time 1756810477.61 total 50.02 last 187.26) total: 10000 open/close in 53.55 seconds: 186.75 ops/second - unlinked 0 (time 1756810482 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1704 (time 1756810523.58 total 10.00 last 170.39) - open/close 3495 (time 1756810533.58 total 20.00 last 179.05) - open/close 5250 (time 1756810543.58 total 30.00 last 175.48) - open/close 6991 (time 1756810553.59 total 40.01 last 174.05) - open/close 8864 (time 1756810563.59 total 50.01 last 187.27) total: 10000 open/close in 56.12 seconds: 178.20 ops/second - unlinked 0 (time 1756810571 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1706 (time 1756810612.90 total 10.00 last 170.59) - open/close 3087 (time 1756810622.91 total 20.01 last 138.02) - open/close 4482 (time 1756810632.92 total 30.01 last 139.41) - open/close 5887 (time 1756810642.92 total 40.02 last 140.46) - open/close 7166 (time 1756810652.93 total 50.03 last 127.78) - open/close 8471 (time 1756810662.93 total 60.03 last 130.49) - open/close 9847 (time 1756810672.93 total 70.03 last 137.58) total: 10000 open/close in 71.11 seconds: 140.62 ops/second - unlinked 0 (time 1756810675 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1301 (time 1756810726.24 total 10.00 last 130.09) - open/close 3140 (time 1756810736.24 total 20.01 last 183.77) - open/close 4653 (time 1756810746.25 total 30.01 last 151.27) - open/close 6884 (time 1756810756.25 total 40.01 last 223.00) - open/close 8676 (time 1756810766.26 total 50.02 last 179.10) total: 10000 open/close in 57.24 seconds: 174.71 ops/second - unlinked 0 (time 1756810775 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre On OST0, 13322 used inodes rc=0 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2