== conf-sanity test 69: replace an OST with the same index ========================================================== 14:51:46 (1761245506) start mds service on oleg102-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing load_modules_local oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg102-server: Loading modules from /home/green/git/lustre-release/lustre oleg102-server: detected 4 online CPUs by sysfs oleg102-server: MODOPTS_LIBCFS= oleg102-server: Force libcfs to create 2 CPU partitions oleg102-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg102-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg102-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 736 (time 1761245597.01 total 10.00 last 73.58) - open/close 1500 (time 1761245607.02 total 20.01 last 76.34) - open/close 2233 (time 1761245617.03 total 30.02 last 73.20) - open/close 2973 (time 1761245627.04 total 40.03 last 73.96) - open/close 3710 (time 1761245637.05 total 50.04 last 73.60) - open/close 4435 (time 1761245647.06 total 60.06 last 72.38) - open/close 5163 (time 1761245657.07 total 70.07 last 72.74) - open/close 5845 (time 1761245667.08 total 80.07 last 68.17) - open/close 6505 (time 1761245677.08 total 90.07 last 66.00) - open/close 7243 (time 1761245687.09 total 100.08 last 73.72) - open/close 7998 (time 1761245697.09 total 110.09 last 75.46) - open/close 8773 (time 1761245707.10 total 120.09 last 77.47) - open/close 9542 (time 1761245717.10 total 130.09 last 76.87) total: 10000 open/close in 136.27 seconds: 73.39 ops/second - unlinked 0 (time 1761245726 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 668 (time 1761245815.86 total 10.00 last 66.78) - open/close 1429 (time 1761245825.86 total 20.01 last 76.08) - open/close 2217 (time 1761245835.88 total 30.02 last 78.70) - open/close 2998 (time 1761245845.88 total 40.02 last 78.07) - open/close 3767 (time 1761245855.88 total 50.03 last 76.87) - open/close 4533 (time 1761245865.90 total 60.04 last 76.52) - open/close 5228 (time 1761245875.91 total 70.05 last 69.43) - open/close 5956 (time 1761245885.91 total 80.05 last 72.79) - open/close 6712 (time 1761245895.91 total 90.05 last 75.59) - open/close 7461 (time 1761245905.91 total 100.05 last 74.87) - open/close 8136 (time 1761245915.92 total 110.06 last 67.44) - open/close 8746 (time 1761245925.94 total 120.08 last 60.91) - open/close 9424 (time 1761245935.94 total 130.08 last 67.76) total: 10000 open/close in 137.95 seconds: 72.49 ops/second - unlinked 0 (time 1761245947 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 734 (time 1761246036.20 total 10.00 last 73.40) - open/close 1439 (time 1761246046.22 total 20.02 last 70.38) - open/close 2159 (time 1761246056.24 total 30.04 last 71.88) - open/close 2853 (time 1761246066.25 total 40.05 last 69.32) - open/close 3504 (time 1761246076.26 total 50.06 last 64.99) - open/close 4165 (time 1761246086.27 total 60.07 last 66.09) - open/close 4881 (time 1761246096.27 total 70.07 last 71.60) - open/close 5608 (time 1761246106.27 total 80.07 last 72.68) - open/close 6333 (time 1761246116.27 total 90.07 last 72.48) - open/close 7023 (time 1761246126.28 total 100.08 last 68.90) - open/close 7716 (time 1761246136.29 total 110.09 last 69.25) - open/close 8386 (time 1761246146.30 total 120.10 last 66.94) - open/close 9095 (time 1761246156.30 total 130.10 last 70.90) - open/close 9846 (time 1761246166.31 total 140.11 last 75.05) total: 10000 open/close in 142.08 seconds: 70.38 ops/second - unlinked 0 (time 1761246172 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 804 (time 1761246268.98 total 10.01 last 80.32) - open/close 1466 (time 1761246278.98 total 20.01 last 66.17) - open/close 2195 (time 1761246288.98 total 30.02 last 72.88) - open/close 2918 (time 1761246298.99 total 40.02 last 72.28) - open/close 3671 (time 1761246309.00 total 50.04 last 75.17) - open/close 4384 (time 1761246319.01 total 60.05 last 71.24) - open/close 5078 (time 1761246329.03 total 70.06 last 69.28) - open/close 5715 (time 1761246339.04 total 80.08 last 63.60) - open/close 6458 (time 1761246349.06 total 90.09 last 74.18) - open/close 7185 (time 1761246359.07 total 100.10 last 72.66) - open/close 7974 (time 1761246369.07 total 110.10 last 78.88) - open/close 8718 (time 1761246379.08 total 120.11 last 74.33) - open/close 9513 (time 1761246389.08 total 130.11 last 79.49) total: 10000 open/close in 136.57 seconds: 73.23 ops/second - unlinked 0 (time 1761246399 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 702 (time 1761246494.25 total 10.01 last 70.14) - open/close 1428 (time 1761246504.26 total 20.01 last 72.59) - open/close 2157 (time 1761246514.26 total 30.02 last 72.85) - open/close 2869 (time 1761246524.26 total 40.02 last 71.20) - open/close 3593 (time 1761246534.27 total 50.03 last 72.33) - open/close 4352 (time 1761246544.29 total 60.04 last 75.81) - open/close 5102 (time 1761246554.30 total 70.05 last 74.93) - open/close 5795 (time 1761246564.31 total 80.06 last 69.21) - open/close 6523 (time 1761246574.31 total 90.06 last 72.80) - open/close 7255 (time 1761246584.31 total 100.06 last 73.20) - open/close 8004 (time 1761246594.32 total 110.07 last 74.81) - open/close 8643 (time 1761246604.32 total 120.08 last 63.89) - open/close 9333 (time 1761246614.33 total 130.09 last 68.94) total: 10000 open/close in 139.36 seconds: 71.75 ops/second - unlinked 0 (time 1761246627 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 747 (time 1761246717.63 total 10.00 last 74.68) - open/close 1538 (time 1761246727.64 total 20.01 last 79.03) - open/close 2366 (time 1761246737.66 total 30.03 last 82.62) - open/close 3058 (time 1761246747.67 total 40.04 last 69.16) - open/close 3771 (time 1761246757.67 total 50.05 last 71.26) - open/close 4463 (time 1761246767.68 total 60.06 last 69.12) - open/close 5193 (time 1761246777.71 total 70.08 last 72.82) - open/close 5892 (time 1761246787.72 total 80.09 last 69.83) - open/close 6631 (time 1761246797.73 total 90.10 last 73.84) - open/close 7386 (time 1761246807.73 total 100.10 last 75.48) - open/close 8144 (time 1761246817.74 total 110.11 last 75.74) - open/close 8839 (time 1761246827.75 total 120.12 last 69.41) - open/close 9585 (time 1761246837.77 total 130.14 last 74.48) total: 10000 open/close in 135.60 seconds: 73.75 ops/second - unlinked 0 (time 1761246847 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 831 (time 1761246934.02 total 10.00 last 83.09) - open/close 1636 (time 1761246944.02 total 20.00 last 80.48) - open/close 2462 (time 1761246954.03 total 30.01 last 82.58) - open/close 3224 (time 1761246964.03 total 40.01 last 76.17) - open/close 4007 (time 1761246974.04 total 50.02 last 78.24) - open/close 4786 (time 1761246984.04 total 60.02 last 77.86) - open/close 5585 (time 1761246994.05 total 70.03 last 79.82) - open/close 6343 (time 1761247004.06 total 80.04 last 75.76) - open/close 7117 (time 1761247014.07 total 90.05 last 77.32) - open/close 7918 (time 1761247024.08 total 100.06 last 79.98) - open/close 8724 (time 1761247034.09 total 110.07 last 80.56) - open/close 9527 (time 1761247044.10 total 120.08 last 80.22) total: 10000 open/close in 125.82 seconds: 79.48 ops/second - unlinked 0 (time 1761247052 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 699 (time 1761247138.99 total 10.01 last 69.85) - open/close 1363 (time 1761247148.99 total 20.01 last 66.37) - open/close 2099 (time 1761247159.00 total 30.02 last 73.51) - open/close 2839 (time 1761247169.01 total 40.03 last 73.96) - open/close 3593 (time 1761247179.01 total 50.03 last 75.38) - open/close 4293 (time 1761247189.02 total 60.04 last 69.96) - open/close 4880 (time 1761247199.02 total 70.04 last 58.69) - open/close 5527 (time 1761247209.02 total 80.05 last 64.66) - open/close 6288 (time 1761247219.03 total 90.05 last 76.04) - open/close 7048 (time 1761247229.03 total 100.05 last 76.00) - open/close 7835 (time 1761247239.04 total 110.07 last 78.61) - open/close 8635 (time 1761247249.05 total 120.07 last 79.94) - open/close 9419 (time 1761247259.07 total 130.09 last 78.29) total: 10000 open/close in 137.73 seconds: 72.61 ops/second - unlinked 0 (time 1761247270 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 735 (time 1761247358.84 total 10.02 last 73.33) - open/close 1460 (time 1761247368.85 total 20.03 last 72.44) - open/close 2215 (time 1761247378.86 total 30.04 last 75.45) - open/close 2994 (time 1761247388.87 total 40.05 last 77.83) - open/close 3774 (time 1761247398.88 total 50.05 last 77.94) - open/close 4532 (time 1761247408.89 total 60.07 last 75.67) - open/close 5317 (time 1761247418.90 total 70.07 last 78.49) - open/close 6070 (time 1761247428.90 total 80.08 last 75.29) - open/close 6800 (time 1761247438.92 total 90.10 last 72.83) - open/close 7581 (time 1761247448.92 total 100.10 last 78.10) - open/close 8374 (time 1761247458.93 total 110.11 last 79.22) - open/close 9113 (time 1761247468.95 total 120.12 last 73.78) - open/close 9850 (time 1761247478.96 total 130.14 last 73.59) total: 10000 open/close in 132.29 seconds: 75.59 ops/second - unlinked 0 (time 1761247485 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 729 (time 1761247575.03 total 10.01 last 72.83) - open/close 1436 (time 1761247585.05 total 20.03 last 70.56) - open/close 2221 (time 1761247595.06 total 30.04 last 78.41) - open/close 2999 (time 1761247605.06 total 40.05 last 77.77) - open/close 3674 (time 1761247615.07 total 50.05 last 67.48) - open/close 4319 (time 1761247625.07 total 60.05 last 64.47) - open/close 5098 (time 1761247635.08 total 70.07 last 77.79) - open/close 5870 (time 1761247645.10 total 80.08 last 77.11) - open/close 6623 (time 1761247655.11 total 90.09 last 75.19) - open/close 7345 (time 1761247665.11 total 100.09 last 72.20) - open/close 8089 (time 1761247675.12 total 110.10 last 74.34) - open/close 8801 (time 1761247685.13 total 120.11 last 71.15) - open/close 9557 (time 1761247695.14 total 130.12 last 75.51) total: 10000 open/close in 136.45 seconds: 73.29 ops/second - unlinked 0 (time 1761247704 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg102-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg102-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg102-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg102-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg102-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing unload_modules_local oleg102-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg102-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2