== conf-sanity test 69: replace an OST with the same index ========================================================== 10:05:27 (1784124327) start mds service on oleg418-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: MODOPTS_LIBCFS= oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg418-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg418-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 2492 (time 1784124382.10 total 10.00 last 249.19) - open/close 4514 (time 1784124392.10 total 20.00 last 202.17) - open/close 6716 (time 1784124402.11 total 30.01 last 220.12) - open/close 9079 (time 1784124412.11 total 40.01 last 236.28) total: 10000 open/close in 43.91 seconds: 227.74 ops/second - unlinked 0 (time 1784124417 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1876 (time 1784124458.87 total 10.00 last 187.53) - open/close 4146 (time 1784124468.87 total 20.00 last 227.00) - open/close 6008 (time 1784124478.87 total 30.01 last 186.13) - open/close 6847 (time 1784124488.87 total 40.01 last 83.89) - open/close 8159 (time 1784124498.88 total 50.01 last 131.13) - open/close 9035 (time 1784124508.88 total 60.01 last 87.60) - open/close 9861 (time 1784124518.88 total 70.02 last 82.58) total: 10000 open/close in 71.20 seconds: 140.45 ops/second - unlinked 0 (time 1784124523 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1198 (time 1784124584.45 total 10.01 last 119.68) - open/close 2527 (time 1784124594.45 total 20.01 last 132.90) - open/close 3725 (time 1784124604.45 total 30.01 last 119.79) - open/close 4868 (time 1784124614.45 total 40.01 last 114.26) - open/close 6084 (time 1784124624.45 total 50.02 last 121.58) - open/close 7446 (time 1784124634.46 total 60.02 last 136.14) - open/close 8659 (time 1784124644.46 total 70.02 last 121.28) - open/close 9899 (time 1784124654.46 total 80.02 last 124.00) total: 10000 open/close in 80.94 seconds: 123.55 ops/second - unlinked 0 (time 1784124657 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1656 (time 1784124709.74 total 10.00 last 165.57) - open/close 3314 (time 1784124719.74 total 20.01 last 165.73) - open/close 4817 (time 1784124729.74 total 30.01 last 150.26) - open/close 6375 (time 1784124739.75 total 40.02 last 155.69) - open/close 7141 (time 1784124749.76 total 50.02 last 76.56) - open/close 7790 (time 1784124759.76 total 60.03 last 64.86) - open/close 8294 (time 1784124769.76 total 70.03 last 50.39) - open/close 8877 (time 1784124779.77 total 80.03 last 58.27) - open/close 9391 (time 1784124789.77 total 90.04 last 51.39) - open/close 9988 (time 1784124799.77 total 100.04 last 59.70) total: 10000 open/close in 100.19 seconds: 99.81 ops/second - unlinked 0 (time 1784124803 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1124 (time 1784124864.65 total 10.01 last 112.29) - open/close 2328 (time 1784124874.66 total 20.01 last 120.36) - open/close 3581 (time 1784124884.66 total 30.02 last 125.23) - open/close 4700 (time 1784124894.67 total 40.02 last 111.87) - open/close 5942 (time 1784124904.68 total 50.03 last 124.09) - open/close 7339 (time 1784124914.68 total 60.04 last 139.64) - open/close 8454 (time 1784124924.68 total 70.04 last 111.45) - open/close 9696 (time 1784124934.69 total 80.04 last 124.18) total: 10000 open/close in 82.87 seconds: 120.67 ops/second - unlinked 0 (time 1784124940 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1148 (time 1784125005.25 total 10.00 last 114.79) - open/close 2166 (time 1784125015.25 total 20.01 last 101.75) - open/close 3203 (time 1784125025.26 total 30.01 last 103.64) - open/close 4305 (time 1784125035.27 total 40.03 last 110.05) - open/close 5368 (time 1784125045.28 total 50.03 last 106.26) - open/close 6403 (time 1784125055.29 total 60.04 last 103.40) - open/close 7454 (time 1784125065.29 total 70.05 last 105.03) - open/close 8655 (time 1784125075.29 total 80.05 last 120.10) - open/close 9920 (time 1784125085.30 total 90.05 last 126.46) total: 10000 open/close in 90.64 seconds: 110.33 ops/second - unlinked 0 (time 1784125087 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1497 (time 1784125140.42 total 10.00 last 149.69) - open/close 3057 (time 1784125150.42 total 20.00 last 155.95) - open/close 4828 (time 1784125160.42 total 30.01 last 177.08) - open/close 6348 (time 1784125170.42 total 40.01 last 151.96) - open/close 7902 (time 1784125180.43 total 50.01 last 155.38) - open/close 9308 (time 1784125190.43 total 60.01 last 140.60) total: 10000 open/close in 64.09 seconds: 156.03 ops/second - unlinked 0 (time 1784125196 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 2136 (time 1784125240.37 total 10.00 last 213.53) - open/close 3907 (time 1784125250.37 total 20.01 last 177.05) - open/close 5928 (time 1784125260.38 total 30.01 last 202.00) - open/close 7709 (time 1784125270.39 total 40.02 last 177.96) - open/close 9765 (time 1784125280.39 total 50.02 last 205.51) total: 10000 open/close in 51.48 seconds: 194.24 ops/second - unlinked 0 (time 1784125283 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2846 (time 1784125322.35 total 10.00 last 284.57) - open/close 5384 (time 1784125332.35 total 20.00 last 253.78) - open/close 7556 (time 1784125342.35 total 30.00 last 217.15) - open/close 9373 (time 1784125352.36 total 40.01 last 181.66) total: 10000 open/close in 42.77 seconds: 233.80 ops/second - unlinked 0 (time 1784125356 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2369 (time 1784125389.72 total 10.00 last 236.82) - open/close 4894 (time 1784125399.73 total 20.00 last 252.47) - open/close 6969 (time 1784125409.73 total 30.01 last 207.42) - open/close 9262 (time 1784125419.73 total 40.01 last 229.24) total: 10000 open/close in 43.90 seconds: 227.81 ops/second - unlinked 0 (time 1784125425 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg418-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 12 sec oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre On OST0, 12036 used inodes rc=0 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.118@tcp sys.timeout=20 start ost1 service on oleg418-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec oleg418-server: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg418-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules via unload_modules_local on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local oleg418-server: modules unloaded. pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2