== conf-sanity test 69: replace an OST with the same index ========================================================== 14:50:36 (1764791436) start mds service on oleg238-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg238-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1807 (time 1764791514.17 total 10.00 last 180.70) - open/close 4882 (time 1764791524.17 total 20.00 last 307.48) - open/close 8769 (time 1764791534.17 total 30.00 last 388.61) total: 10000 open/close in 33.14 seconds: 301.73 ops/second - unlinked 0 (time 1764791538 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 4176 (time 1764791570.25 total 10.00 last 417.58) - open/close 5600 (time 1764791580.26 total 20.01 last 142.26) - open/close 6678 (time 1764791590.26 total 30.01 last 107.77) - open/close 7724 (time 1764791600.26 total 40.02 last 104.55) - open/close 8947 (time 1764791610.27 total 50.02 last 122.20) total: 10000 open/close in 54.90 seconds: 182.13 ops/second - unlinked 0 (time 1764791616 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2044 (time 1764791647.11 total 10.00 last 204.37) - open/close 4662 (time 1764791657.11 total 20.01 last 261.67) - open/close 7871 (time 1764791667.12 total 30.01 last 320.80) total: 10000 open/close in 35.73 seconds: 279.87 ops/second - unlinked 0 (time 1764791674 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 783 (time 1764791739.01 total 10.03 last 78.10) - open/close 1581 (time 1764791749.02 total 20.03 last 79.76) - open/close 2373 (time 1764791759.03 total 30.04 last 79.15) - open/close 2909 (time 1764791769.04 total 40.05 last 53.51) - open/close 3518 (time 1764791779.04 total 50.05 last 60.89) - open/close 4232 (time 1764791789.06 total 60.07 last 71.30) - open/close 5175 (time 1764791799.06 total 70.07 last 94.24) - open/close 6125 (time 1764791809.07 total 80.08 last 94.93) - open/close 7113 (time 1764791819.07 total 90.08 last 98.77) - open/close 7967 (time 1764791829.08 total 100.09 last 85.37) - open/close 8865 (time 1764791839.08 total 110.09 last 89.79) - open/close 9787 (time 1764791849.08 total 120.09 last 92.17) total: 10000 open/close in 122.42 seconds: 81.69 ops/second - unlinked 0 (time 1764791854 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 871 (time 1764791934.57 total 10.01 last 87.01) - open/close 1601 (time 1764791944.57 total 20.02 last 72.97) - open/close 2233 (time 1764791954.58 total 30.02 last 63.15) - open/close 2941 (time 1764791964.59 total 40.03 last 70.74) - open/close 3711 (time 1764791974.59 total 50.03 last 76.99) - open/close 4561 (time 1764791984.59 total 60.03 last 85.00) - open/close 5429 (time 1764791994.59 total 70.03 last 86.78) - open/close 6072 (time 1764792004.60 total 80.04 last 64.26) - open/close 6696 (time 1764792014.62 total 90.06 last 62.29) - open/close 7491 (time 1764792024.63 total 100.07 last 79.44) - open/close 8327 (time 1764792034.64 total 110.08 last 83.50) - open/close 9156 (time 1764792044.65 total 120.09 last 82.77) total: 10000 open/close in 129.89 seconds: 76.99 ops/second - unlinked 0 (time 1764792058 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 764 (time 1764792140.44 total 10.00 last 76.36) - open/close 1370 (time 1764792150.46 total 20.03 last 60.48) - open/close 1936 (time 1764792160.49 total 30.06 last 56.41) - open/close 2651 (time 1764792170.50 total 40.07 last 71.44) - open/close 3326 (time 1764792180.50 total 50.07 last 67.46) - open/close 4085 (time 1764792190.51 total 60.08 last 75.83) - open/close 4809 (time 1764792200.52 total 70.09 last 72.35) - open/close 5605 (time 1764792210.52 total 80.09 last 79.58) - open/close 6463 (time 1764792220.53 total 90.09 last 85.78) - open/close 7273 (time 1764792230.53 total 100.10 last 80.94) - open/close 8159 (time 1764792240.54 total 110.11 last 88.54) - open/close 9107 (time 1764792250.54 total 120.11 last 94.75) - open/close 9933 (time 1764792260.55 total 130.12 last 82.58) total: 10000 open/close in 130.74 seconds: 76.49 ops/second - unlinked 0 (time 1764792264 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 468 (time 1764792351.43 total 10.02 last 46.72) - open/close 1248 (time 1764792361.44 total 20.02 last 77.96) - open/close 1998 (time 1764792371.44 total 30.03 last 74.95) - open/close 2699 (time 1764792381.45 total 40.04 last 70.05) - open/close 3504 (time 1764792391.45 total 50.04 last 80.49) - open/close 4309 (time 1764792401.45 total 60.04 last 80.48) - open/close 5082 (time 1764792411.46 total 70.05 last 77.23) - open/close 5936 (time 1764792421.47 total 80.05 last 85.37) - open/close 6794 (time 1764792431.47 total 90.06 last 85.75) - open/close 7615 (time 1764792441.47 total 100.06 last 82.10) - open/close 8361 (time 1764792451.47 total 110.06 last 74.58) - open/close 9243 (time 1764792461.49 total 120.08 last 88.08) total: 10000 open/close in 128.60 seconds: 77.76 ops/second - unlinked 0 (time 1764792473 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 655 (time 1764792557.68 total 10.00 last 65.48) - open/close 1340 (time 1764792567.69 total 20.01 last 68.43) - open/close 2245 (time 1764792577.69 total 30.02 last 90.49) - open/close 3088 (time 1764792587.70 total 40.03 last 84.21) - open/close 4001 (time 1764792597.71 total 50.03 last 91.23) - open/close 4879 (time 1764792607.71 total 60.03 last 87.80) - open/close 5782 (time 1764792617.71 total 70.04 last 90.24) - open/close 6650 (time 1764792627.72 total 80.04 last 86.76) - open/close 7433 (time 1764792637.73 total 90.06 last 78.20) - open/close 8300 (time 1764792647.74 total 100.07 last 86.60) - open/close 9141 (time 1764792657.75 total 110.07 last 84.07) - open/close 9904 (time 1764792667.76 total 120.09 last 76.19) total: 10000 open/close in 121.46 seconds: 82.33 ops/second - unlinked 0 (time 1764792673 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 772 (time 1764792763.69 total 10.01 last 77.15) - open/close 1739 (time 1764792773.69 total 20.01 last 96.65) - open/close 2618 (time 1764792783.69 total 30.01 last 87.90) - open/close 3487 (time 1764792793.70 total 40.01 last 86.87) - open/close 4369 (time 1764792803.71 total 50.03 last 88.10) - open/close 5319 (time 1764792813.71 total 60.03 last 95.00) - open/close 6275 (time 1764792823.72 total 70.04 last 95.50) - open/close 6979 (time 1764792833.73 total 80.05 last 70.31) - open/close 7720 (time 1764792843.74 total 90.06 last 74.06) - open/close 8471 (time 1764792853.74 total 100.06 last 75.07) - open/close 9148 (time 1764792863.75 total 110.07 last 67.62) - open/close 9763 (time 1764792873.76 total 120.08 last 61.47) total: 10000 open/close in 123.61 seconds: 80.90 ops/second - unlinked 0 (time 1764792881 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 925 (time 1764792963.98 total 10.01 last 92.40) - open/close 1834 (time 1764792973.99 total 20.02 last 90.86) - open/close 2735 (time 1764792983.99 total 30.02 last 90.06) - open/close 3620 (time 1764792994.00 total 40.03 last 88.44) - open/close 4480 (time 1764793004.01 total 50.04 last 85.92) - open/close 5391 (time 1764793014.02 total 60.05 last 91.01) - open/close 6324 (time 1764793024.03 total 70.06 last 93.21) - open/close 7032 (time 1764793034.05 total 80.08 last 70.63) - open/close 7792 (time 1764793044.06 total 90.08 last 75.98) - open/close 8492 (time 1764793054.06 total 100.09 last 69.96) - open/close 9231 (time 1764793064.07 total 110.10 last 73.82) total: 10000 open/close in 118.50 seconds: 84.39 ops/second - unlinked 0 (time 1764793076 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2