== conf-sanity test 69: replace an OST with the same index ========================================================== 22:12:52 (1767323572) start mds service on oleg322-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg322-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1196 (time 1767323678.94 total 10.00 last 119.56) - open/close 2489 (time 1767323688.94 total 20.01 last 129.25) - open/close 3891 (time 1767323698.95 total 30.01 last 140.13) - open/close 5140 (time 1767323708.95 total 40.02 last 124.86) - open/close 6430 (time 1767323718.96 total 50.02 last 128.93) - open/close 7861 (time 1767323728.96 total 60.02 last 143.08) - open/close 9358 (time 1767323738.96 total 70.03 last 149.65) total: 10000 open/close in 73.62 seconds: 135.83 ops/second - unlinked 0 (time 1767323744 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1300 (time 1767323793.26 total 10.02 last 129.72) - open/close 2936 (time 1767323803.26 total 20.02 last 163.58) - open/close 4654 (time 1767323813.27 total 30.02 last 171.79) - open/close 6467 (time 1767323823.27 total 40.02 last 181.29) - open/close 8112 (time 1767323833.27 total 50.03 last 164.45) - open/close 9684 (time 1767323843.27 total 60.03 last 157.13) total: 10000 open/close in 62.38 seconds: 160.30 ops/second - unlinked 0 (time 1767323847 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1248 (time 1767323894.24 total 10.00 last 124.76) - open/close 2723 (time 1767323904.25 total 20.01 last 147.39) - open/close 4611 (time 1767323914.25 total 30.01 last 188.79) - open/close 6110 (time 1767323924.25 total 40.01 last 149.90) - open/close 7528 (time 1767323934.25 total 50.01 last 141.77) - open/close 9113 (time 1767323944.25 total 60.01 last 158.50) total: 10000 open/close in 65.91 seconds: 151.73 ops/second - unlinked 0 (time 1767323951 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1456 (time 1767323994.97 total 10.00 last 145.58) - open/close 2794 (time 1767324004.98 total 20.01 last 133.73) - open/close 4488 (time 1767324014.98 total 30.01 last 169.36) - open/close 5956 (time 1767324024.98 total 40.01 last 146.79) - open/close 7363 (time 1767324034.98 total 50.01 last 140.67) - open/close 8917 (time 1767324044.98 total 60.01 last 155.39) total: 10000 open/close in 65.42 seconds: 152.85 ops/second - unlinked 0 (time 1767324052 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1493 (time 1767324090.41 total 10.01 last 149.22) - open/close 3195 (time 1767324100.41 total 20.01 last 170.17) - open/close 5153 (time 1767324110.41 total 30.01 last 195.77) - open/close 7180 (time 1767324120.41 total 40.01 last 202.65) - open/close 8609 (time 1767324130.42 total 50.02 last 142.83) total: 10000 open/close in 58.36 seconds: 171.34 ops/second - unlinked 0 (time 1767324140 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1341 (time 1767324181.55 total 10.00 last 134.07) - open/close 2808 (time 1767324191.56 total 20.01 last 146.54) - open/close 4412 (time 1767324201.56 total 30.02 last 160.35) - open/close 5902 (time 1767324211.57 total 40.02 last 148.88) - open/close 7640 (time 1767324221.57 total 50.03 last 173.75) - open/close 9493 (time 1767324231.58 total 60.03 last 185.28) total: 10000 open/close in 62.45 seconds: 160.12 ops/second - unlinked 0 (time 1767324235 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1506 (time 1767324273.43 total 10.00 last 150.57) - open/close 3487 (time 1767324283.44 total 20.01 last 198.02) - open/close 5058 (time 1767324293.44 total 30.01 last 157.08) - open/close 7039 (time 1767324303.44 total 40.01 last 198.03) - open/close 8794 (time 1767324313.45 total 50.02 last 175.42) total: 10000 open/close in 55.94 seconds: 178.75 ops/second - unlinked 0 (time 1767324321 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1525 (time 1767324362.12 total 10.01 last 152.41) - open/close 3301 (time 1767324372.13 total 20.01 last 177.52) - open/close 4773 (time 1767324382.13 total 30.01 last 147.17) - open/close 6337 (time 1767324392.13 total 40.02 last 156.34) - open/close 7660 (time 1767324402.13 total 50.02 last 132.29) - open/close 9199 (time 1767324412.14 total 60.02 last 153.85) total: 10000 open/close in 65.09 seconds: 153.62 ops/second - unlinked 0 (time 1767324418 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2086 (time 1767324457.55 total 10.00 last 208.59) - open/close 4093 (time 1767324467.56 total 20.00 last 200.65) - open/close 6190 (time 1767324477.56 total 30.00 last 209.67) - open/close 8532 (time 1767324487.56 total 40.01 last 234.12) total: 10000 open/close in 47.71 seconds: 209.61 ops/second - unlinked 0 (time 1767324497 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1858 (time 1767324535.46 total 10.00 last 185.77) - open/close 3467 (time 1767324545.47 total 20.00 last 160.86) - open/close 5292 (time 1767324555.47 total 30.01 last 182.42) - open/close 7516 (time 1767324565.47 total 40.01 last 222.31) - open/close 9528 (time 1767324575.48 total 50.02 last 201.15) total: 10000 open/close in 52.69 seconds: 189.78 ops/second - unlinked 0 (time 1767324579 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.122@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg322-server: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg322-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg322-server: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg322-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre On OST0, 14613 used inodes rc=0 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.122@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.122@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.122@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg322-server: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg322-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg322-server: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg322-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local oleg322-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg322-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2