== conf-sanity test 69: replace an OST with the same index ========================================================== 22:34:33 (1786415673) start mds service on oleg109-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions oleg109-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg109-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg109-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg109-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1095 (time 1786415789.27 total 10.01 last 109.42) - open/close 2066 (time 1786415799.28 total 20.02 last 97.01) - open/close 2991 (time 1786415809.28 total 30.02 last 92.47) - open/close 3898 (time 1786415819.29 total 40.03 last 90.64) - open/close 4891 (time 1786415829.29 total 50.03 last 99.30) - open/close 5776 (time 1786415839.29 total 60.03 last 88.48) - open/close 6577 (time 1786415849.29 total 70.03 last 80.08) - open/close 7491 (time 1786415859.29 total 80.03 last 91.39) - open/close 8508 (time 1786415869.30 total 90.04 last 101.63) - open/close 9340 (time 1786415879.33 total 100.07 last 82.93) total: 10000 open/close in 108.33 seconds: 92.31 ops/second - unlinked 0 (time 1786415892 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 874 (time 1786415972.01 total 10.01 last 87.32) - open/close 1771 (time 1786415982.01 total 20.01 last 89.70) - open/close 2643 (time 1786415992.03 total 30.03 last 87.04) - open/close 3457 (time 1786416002.05 total 40.04 last 81.27) - open/close 4296 (time 1786416012.06 total 50.06 last 83.77) - open/close 5099 (time 1786416022.07 total 60.07 last 80.22) - open/close 5940 (time 1786416032.07 total 70.07 last 84.08) - open/close 6562 (time 1786416042.08 total 80.08 last 62.16) - open/close 7409 (time 1786416052.08 total 90.08 last 84.69) - open/close 8329 (time 1786416062.08 total 100.08 last 91.99) - open/close 9105 (time 1786416072.09 total 110.09 last 77.53) - open/close 9935 (time 1786416082.09 total 120.09 last 82.97) total: 10000 open/close in 120.89 seconds: 82.72 ops/second - unlinked 0 (time 1786416087 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 845 (time 1786416166.85 total 10.01 last 84.43) - open/close 1604 (time 1786416176.86 total 20.01 last 75.86) - open/close 2429 (time 1786416186.86 total 30.01 last 82.48) - open/close 3139 (time 1786416196.86 total 40.02 last 70.99) - open/close 3918 (time 1786416206.87 total 50.02 last 77.85) - open/close 4664 (time 1786416216.88 total 60.03 last 74.53) - open/close 5530 (time 1786416226.88 total 70.03 last 86.58) - open/close 6337 (time 1786416236.88 total 80.04 last 80.68) - open/close 7264 (time 1786416246.88 total 90.04 last 92.68) - open/close 8042 (time 1786416256.89 total 100.04 last 77.79) - open/close 8999 (time 1786416266.89 total 110.04 last 95.69) - open/close 9886 (time 1786416276.89 total 120.04 last 88.68) total: 10000 open/close in 121.31 seconds: 82.43 ops/second - unlinked 0 (time 1786416282 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 811 (time 1786416367.66 total 10.02 last 80.96) - open/close 1600 (time 1786416377.67 total 20.02 last 78.84) - open/close 2342 (time 1786416387.67 total 30.03 last 74.20) - open/close 3232 (time 1786416397.67 total 40.03 last 88.98) - open/close 4233 (time 1786416407.68 total 50.03 last 100.04) - open/close 5307 (time 1786416417.68 total 60.04 last 107.37) - open/close 6246 (time 1786416427.68 total 70.04 last 93.85) - open/close 7243 (time 1786416437.69 total 80.05 last 99.61) - open/close 8144 (time 1786416447.70 total 90.06 last 90.04) - open/close 9094 (time 1786416457.71 total 100.07 last 94.89) - open/close 9969 (time 1786416467.71 total 110.07 last 87.46) total: 10000 open/close in 110.32 seconds: 90.65 ops/second - unlinked 0 (time 1786416472 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 857 (time 1786416556.39 total 10.01 last 85.64) - open/close 1757 (time 1786416566.40 total 20.01 last 89.92) - open/close 2636 (time 1786416576.41 total 30.03 last 87.79) - open/close 3653 (time 1786416586.43 total 40.04 last 101.55) - open/close 4335 (time 1786416596.43 total 50.04 last 68.20) - open/close 5171 (time 1786416606.44 total 60.06 last 83.48) - open/close 5975 (time 1786416616.45 total 70.07 last 80.34) - open/close 6760 (time 1786416626.46 total 80.08 last 78.42) - open/close 7561 (time 1786416636.48 total 90.10 last 79.92) - open/close 8271 (time 1786416646.49 total 100.10 last 70.97) - open/close 9147 (time 1786416656.50 total 110.12 last 87.47) - open/close 9942 (time 1786416666.51 total 120.13 last 79.41) total: 10000 open/close in 120.75 seconds: 82.82 ops/second - unlinked 0 (time 1786416672 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 800 (time 1786416756.18 total 10.02 last 79.85) - open/close 1654 (time 1786416766.19 total 20.03 last 85.33) - open/close 2514 (time 1786416776.19 total 30.03 last 85.97) - open/close 3374 (time 1786416786.20 total 40.03 last 85.98) - open/close 4363 (time 1786416796.20 total 50.04 last 98.84) - open/close 5276 (time 1786416806.22 total 60.06 last 91.13) - open/close 6224 (time 1786416816.23 total 70.07 last 94.73) - open/close 7195 (time 1786416826.24 total 80.08 last 96.99) - open/close 8085 (time 1786416836.24 total 90.08 last 88.98) - open/close 9076 (time 1786416846.24 total 100.08 last 99.09) - open/close 9821 (time 1786416856.25 total 110.09 last 74.45) total: 10000 open/close in 112.26 seconds: 89.08 ops/second - unlinked 0 (time 1786416863 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 683 (time 1786416944.54 total 10.01 last 68.21) - open/close 1586 (time 1786416954.55 total 20.02 last 90.26) - open/close 2510 (time 1786416964.55 total 30.02 last 92.38) - open/close 3436 (time 1786416974.56 total 40.03 last 92.53) - open/close 4435 (time 1786416984.56 total 50.03 last 99.84) - open/close 5281 (time 1786416994.57 total 60.04 last 84.58) - open/close 6097 (time 1786417004.57 total 70.04 last 81.55) - open/close 6971 (time 1786417014.58 total 80.05 last 87.30) - open/close 7829 (time 1786417024.59 total 90.06 last 85.77) - open/close 8803 (time 1786417034.59 total 100.06 last 97.34) - open/close 9798 (time 1786417044.60 total 110.07 last 99.44) total: 10000 open/close in 111.90 seconds: 89.37 ops/second - unlinked 0 (time 1786417050 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 852 (time 1786417127.60 total 10.02 last 85.05) - open/close 1802 (time 1786417137.60 total 20.02 last 94.98) - open/close 2774 (time 1786417147.61 total 30.02 last 97.17) - open/close 3711 (time 1786417157.63 total 40.04 last 93.53) - open/close 4547 (time 1786417167.63 total 50.05 last 83.56) - open/close 5439 (time 1786417177.64 total 60.06 last 89.10) - open/close 6401 (time 1786417187.65 total 70.06 last 96.14) - open/close 7408 (time 1786417197.66 total 80.07 last 100.63) - open/close 8313 (time 1786417207.67 total 90.09 last 90.37) - open/close 9218 (time 1786417217.67 total 100.09 last 90.47) total: 10000 open/close in 108.48 seconds: 92.19 ops/second - unlinked 0 (time 1786417230 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 677 (time 1786417309.65 total 10.00 last 67.67) - open/close 1636 (time 1786417319.65 total 20.01 last 95.86) - open/close 2474 (time 1786417329.65 total 30.01 last 83.80) - open/close 3348 (time 1786417339.66 total 40.02 last 87.33) - open/close 4305 (time 1786417349.67 total 50.02 last 95.63) - open/close 5265 (time 1786417359.68 total 60.04 last 95.87) - open/close 6335 (time 1786417369.69 total 70.05 last 106.92) - open/close 7384 (time 1786417379.69 total 80.05 last 104.85) - open/close 8319 (time 1786417389.70 total 90.06 last 93.44) - open/close 9192 (time 1786417399.70 total 100.06 last 87.27) total: 10000 open/close in 107.88 seconds: 92.70 ops/second - unlinked 0 (time 1786417411 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 795 (time 1786417490.92 total 10.01 last 79.39) - open/close 1729 (time 1786417500.92 total 20.02 last 93.38) - open/close 2718 (time 1786417510.93 total 30.02 last 98.82) - open/close 3617 (time 1786417520.93 total 40.03 last 89.87) - open/close 4492 (time 1786417530.94 total 50.03 last 87.46) - open/close 5364 (time 1786417540.95 total 60.05 last 87.04) - open/close 6254 (time 1786417550.96 total 70.05 last 88.97) - open/close 7356 (time 1786417560.96 total 80.06 last 110.16) - open/close 8332 (time 1786417570.97 total 90.06 last 97.55) - open/close 9342 (time 1786417580.97 total 100.06 last 100.99) total: 10000 open/close in 106.86 seconds: 93.58 ops/second - unlinked 0 (time 1786417591 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.109@tcp sys.timeout=20 start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg109-server: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg109-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg109-server unloading modules via unload_modules_local on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local oleg109-server: modules unloaded. pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2