== conf-sanity test 69: replace an OST with the same index ========================================================== 13:13:56 (1758734036) start mds service on oleg405-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg405-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions oleg405-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg405-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg405-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1403 (time 1758734095.64 total 10.01 last 140.19) - open/close 2872 (time 1758734105.65 total 20.01 last 146.79) - open/close 4096 (time 1758734115.65 total 30.02 last 122.32) - open/close 5488 (time 1758734125.66 total 40.03 last 139.13) - open/close 7155 (time 1758734135.66 total 50.03 last 166.69) - open/close 8865 (time 1758734145.67 total 60.03 last 170.91) total: 10000 open/close in 67.89 seconds: 147.30 ops/second - unlinked 0 (time 1758734155 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1150 (time 1758734216.78 total 10.02 last 114.82) - open/close 2404 (time 1758734226.78 total 20.02 last 125.39) - open/close 3722 (time 1758734236.78 total 30.02 last 131.78) - open/close 5343 (time 1758734246.78 total 40.02 last 162.02) - open/close 6718 (time 1758734256.79 total 50.03 last 137.43) - open/close 7709 (time 1758734266.80 total 60.04 last 99.02) - open/close 8716 (time 1758734276.80 total 70.04 last 100.66) - open/close 9799 (time 1758734286.80 total 80.04 last 108.28) total: 10000 open/close in 81.87 seconds: 122.14 ops/second - unlinked 0 (time 1758734290 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1505 (time 1758734336.55 total 10.01 last 150.40) - open/close 3125 (time 1758734346.55 total 20.01 last 162.00) - open/close 4596 (time 1758734356.56 total 30.01 last 147.08) - open/close 6142 (time 1758734366.56 total 40.01 last 154.57) - open/close 7700 (time 1758734376.56 total 50.01 last 155.76) - open/close 9367 (time 1758734386.56 total 60.02 last 166.63) total: 10000 open/close in 63.65 seconds: 157.10 ops/second - unlinked 0 (time 1758734392 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1754 (time 1758734439.40 total 10.01 last 175.28) - open/close 3247 (time 1758734449.40 total 20.01 last 149.27) - open/close 4771 (time 1758734459.40 total 30.01 last 152.35) - open/close 6409 (time 1758734469.41 total 40.02 last 163.70) - open/close 8185 (time 1758734479.41 total 50.02 last 177.55) - open/close 9902 (time 1758734489.42 total 60.03 last 171.61) total: 10000 open/close in 60.68 seconds: 164.80 ops/second - unlinked 0 (time 1758734491 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1572 (time 1758734536.35 total 10.00 last 157.16) - open/close 3164 (time 1758734546.35 total 20.00 last 159.18) - open/close 4709 (time 1758734556.36 total 30.01 last 154.43) - open/close 6096 (time 1758734566.36 total 40.01 last 138.67) - open/close 7466 (time 1758734576.36 total 50.01 last 136.95) - open/close 8963 (time 1758734586.37 total 60.02 last 149.61) total: 10000 open/close in 67.89 seconds: 147.29 ops/second - unlinked 0 (time 1758734596 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1705 (time 1758734644.19 total 10.00 last 170.48) - open/close 3404 (time 1758734654.19 total 20.01 last 169.79) - open/close 4824 (time 1758734664.20 total 30.01 last 141.94) - open/close 6187 (time 1758734674.20 total 40.01 last 136.29) - open/close 7838 (time 1758734684.20 total 50.01 last 165.09) - open/close 9429 (time 1758734694.20 total 60.01 last 159.10) total: 10000 open/close in 63.85 seconds: 156.63 ops/second - unlinked 0 (time 1758734699 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1536 (time 1758734741.58 total 10.01 last 153.51) - open/close 3172 (time 1758734751.59 total 20.01 last 163.58) - open/close 4858 (time 1758734761.59 total 30.01 last 168.54) - open/close 6789 (time 1758734771.59 total 40.01 last 193.08) - open/close 8748 (time 1758734781.59 total 50.01 last 195.90) total: 10000 open/close in 57.82 seconds: 172.94 ops/second - unlinked 0 (time 1758734791 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1877 (time 1758734835.16 total 10.00 last 187.61) - open/close 3608 (time 1758734845.16 total 20.01 last 173.00) - open/close 5531 (time 1758734855.17 total 30.01 last 192.24) - open/close 7172 (time 1758734865.17 total 40.02 last 164.01) - open/close 9135 (time 1758734875.17 total 50.02 last 196.29) total: 10000 open/close in 54.53 seconds: 183.38 ops/second - unlinked 0 (time 1758734881 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2368 (time 1758734918.07 total 10.00 last 236.74) - open/close 4841 (time 1758734928.08 total 20.01 last 247.20) - open/close 7236 (time 1758734938.08 total 30.01 last 239.44) - open/close 9571 (time 1758734948.08 total 40.01 last 233.40) total: 10000 open/close in 41.82 seconds: 239.11 ops/second - unlinked 0 (time 1758734951 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2305 (time 1758734988.35 total 10.00 last 230.50) - open/close 4759 (time 1758734998.36 total 20.00 last 245.34) - open/close 7225 (time 1758735008.36 total 30.00 last 246.57) - open/close 9432 (time 1758735018.36 total 40.00 last 220.70) total: 10000 open/close in 43.40 seconds: 230.44 ops/second - unlinked 0 (time 1758735023 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg405-server: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg405-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg405-server: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg405-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre On OST0, 14722 used inodes rc=0 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.105@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.105@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg405-server: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg405-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg405-server: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg405-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg405-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg405-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2