== conf-sanity test 69: replace an OST with the same index ========================================================== 18:17:26 (1786400246) start mds service on oleg207-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg207-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg207-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 802 (time 1786400368.36 total 10.01 last 80.14) - open/close 1471 (time 1786400378.38 total 20.03 last 66.75) - open/close 2123 (time 1786400388.40 total 30.05 last 65.09) - open/close 2649 (time 1786400398.41 total 40.06 last 52.53) - open/close 3330 (time 1786400408.41 total 50.06 last 68.08) - open/close 4085 (time 1786400418.43 total 60.07 last 75.41) - open/close 4803 (time 1786400428.43 total 70.08 last 71.77) - open/close 5495 (time 1786400438.45 total 80.09 last 69.09) - open/close 6128 (time 1786400448.45 total 90.09 last 63.30) - open/close 6818 (time 1786400458.45 total 100.10 last 68.97) - open/close 7477 (time 1786400468.46 total 110.10 last 65.86) - open/close 8248 (time 1786400478.46 total 120.11 last 77.05) - open/close 8975 (time 1786400488.47 total 130.12 last 72.66) - open/close 9687 (time 1786400498.48 total 140.12 last 71.14) total: 10000 open/close in 144.15 seconds: 69.37 ops/second - unlinked 0 (time 1786400507 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 849 (time 1786400599.11 total 10.01 last 84.85) - open/close 1532 (time 1786400609.11 total 20.01 last 68.29) - open/close 2227 (time 1786400619.11 total 30.01 last 69.49) - open/close 3009 (time 1786400629.12 total 40.02 last 78.13) - open/close 3702 (time 1786400639.13 total 50.03 last 69.25) - open/close 4318 (time 1786400649.13 total 60.03 last 61.59) - open/close 5086 (time 1786400659.13 total 70.03 last 76.78) - open/close 5862 (time 1786400669.14 total 80.03 last 77.58) - open/close 6487 (time 1786400679.14 total 90.04 last 62.47) - open/close 7116 (time 1786400689.15 total 100.05 last 62.85) - open/close 7858 (time 1786400699.16 total 110.05 last 74.14) - open/close 8596 (time 1786400709.16 total 120.06 last 73.75) - open/close 9532 (time 1786400719.16 total 130.06 last 93.59) total: 10000 open/close in 135.85 seconds: 73.61 ops/second - unlinked 0 (time 1786400729 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 697 (time 1786400823.50 total 10.01 last 69.65) - open/close 1394 (time 1786400833.50 total 20.01 last 69.69) - open/close 2269 (time 1786400843.51 total 30.02 last 87.45) - open/close 2933 (time 1786400853.52 total 40.03 last 66.31) - open/close 3643 (time 1786400863.53 total 50.03 last 70.97) - open/close 4436 (time 1786400873.54 total 60.04 last 79.20) - open/close 5241 (time 1786400883.54 total 70.05 last 80.47) - open/close 6018 (time 1786400893.55 total 80.05 last 77.68) - open/close 6671 (time 1786400903.56 total 90.07 last 65.21) - open/close 7418 (time 1786400913.56 total 100.07 last 74.69) - open/close 8122 (time 1786400923.57 total 110.08 last 70.33) - open/close 8786 (time 1786400933.57 total 120.08 last 66.38) - open/close 9580 (time 1786400943.58 total 130.09 last 79.32) total: 10000 open/close in 135.60 seconds: 73.74 ops/second - unlinked 0 (time 1786400954 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 860 (time 1786401047.01 total 10.02 last 85.86) - open/close 1445 (time 1786401057.01 total 20.02 last 58.48) - open/close 2203 (time 1786401067.01 total 30.02 last 75.79) - open/close 2938 (time 1786401077.02 total 40.02 last 73.48) - open/close 3630 (time 1786401087.02 total 50.03 last 69.17) - open/close 4353 (time 1786401097.02 total 60.03 last 72.28) - open/close 5040 (time 1786401107.03 total 70.04 last 68.66) - open/close 5658 (time 1786401117.04 total 80.05 last 61.74) - open/close 6293 (time 1786401127.05 total 90.06 last 63.44) - open/close 7065 (time 1786401137.05 total 100.06 last 77.18) - open/close 7801 (time 1786401147.06 total 110.06 last 73.57) - open/close 8629 (time 1786401157.06 total 120.07 last 82.77) - open/close 9426 (time 1786401167.06 total 130.07 last 79.66) total: 10000 open/close in 137.25 seconds: 72.86 ops/second - unlinked 0 (time 1786401179 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 819 (time 1786401272.16 total 10.01 last 81.82) - open/close 1524 (time 1786401282.16 total 20.01 last 70.46) - open/close 2309 (time 1786401292.17 total 30.02 last 78.48) - open/close 2974 (time 1786401302.17 total 40.02 last 66.49) - open/close 3807 (time 1786401312.17 total 50.02 last 83.25) - open/close 4691 (time 1786401322.18 total 60.03 last 88.31) - open/close 5418 (time 1786401332.21 total 70.06 last 72.50) - open/close 6234 (time 1786401342.23 total 80.08 last 81.47) - open/close 7078 (time 1786401352.24 total 90.09 last 84.28) - open/close 7712 (time 1786401362.24 total 100.09 last 63.40) - open/close 8553 (time 1786401372.25 total 110.10 last 84.05) - open/close 9273 (time 1786401382.26 total 120.11 last 71.92) total: 10000 open/close in 129.75 seconds: 77.07 ops/second - unlinked 0 (time 1786401396 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 734 (time 1786401486.21 total 10.01 last 73.36) - open/close 1472 (time 1786401496.22 total 20.01 last 73.77) - open/close 2109 (time 1786401506.22 total 30.01 last 63.67) - open/close 2714 (time 1786401516.23 total 40.02 last 60.47) - open/close 3337 (time 1786401526.23 total 50.02 last 62.28) - open/close 4058 (time 1786401536.24 total 60.03 last 72.04) - open/close 4769 (time 1786401546.25 total 70.04 last 71.04) - open/close 5489 (time 1786401556.25 total 80.04 last 72.00) - open/close 6258 (time 1786401566.25 total 90.05 last 76.84) - open/close 7058 (time 1786401576.27 total 100.06 last 79.90) - open/close 7821 (time 1786401586.27 total 110.06 last 76.30) - open/close 8677 (time 1786401596.27 total 120.06 last 85.56) - open/close 9455 (time 1786401606.28 total 130.07 last 77.76) total: 10000 open/close in 139.42 seconds: 71.72 ops/second - unlinked 0 (time 1786401620 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 907 (time 1786401712.21 total 10.01 last 90.65) - open/close 1749 (time 1786401722.22 total 20.02 last 84.11) - open/close 2486 (time 1786401732.22 total 30.02 last 73.69) - open/close 3159 (time 1786401742.23 total 40.02 last 67.28) - open/close 3719 (time 1786401752.23 total 50.03 last 55.96) - open/close 4220 (time 1786401762.24 total 60.03 last 50.08) - open/close 5012 (time 1786401772.24 total 70.04 last 79.16) - open/close 5849 (time 1786401782.25 total 80.05 last 83.64) - open/close 6647 (time 1786401792.25 total 90.05 last 79.79) - open/close 7410 (time 1786401802.25 total 100.05 last 76.27) - open/close 8232 (time 1786401812.27 total 110.06 last 82.10) - open/close 9006 (time 1786401822.28 total 120.07 last 77.33) - open/close 9731 (time 1786401832.28 total 130.08 last 72.47) total: 10000 open/close in 133.52 seconds: 74.90 ops/second - unlinked 0 (time 1786401840 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 713 (time 1786401926.91 total 10.00 last 71.28) - open/close 1492 (time 1786401936.91 total 20.01 last 77.87) - open/close 2271 (time 1786401946.92 total 30.02 last 77.83) - open/close 3155 (time 1786401956.93 total 40.03 last 88.31) - open/close 3889 (time 1786401966.94 total 50.03 last 73.37) - open/close 4801 (time 1786401976.95 total 60.05 last 91.06) - open/close 5618 (time 1786401986.96 total 70.05 last 81.63) - open/close 6487 (time 1786401996.96 total 80.05 last 86.90) - open/close 7356 (time 1786402006.97 total 90.07 last 86.78) - open/close 8146 (time 1786402016.98 total 100.07 last 78.97) - open/close 9002 (time 1786402026.98 total 110.07 last 85.59) - open/close 9894 (time 1786402036.98 total 120.07 last 89.20) total: 10000 open/close in 121.51 seconds: 82.29 ops/second - unlinked 0 (time 1786402042 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 816 (time 1786402122.00 total 10.03 last 81.37) - open/close 1668 (time 1786402132.01 total 20.03 last 85.16) - open/close 2490 (time 1786402142.02 total 30.05 last 82.10) - open/close 3320 (time 1786402152.04 total 40.06 last 82.89) - open/close 4054 (time 1786402162.04 total 50.06 last 73.38) - open/close 5025 (time 1786402172.04 total 60.06 last 97.09) - open/close 5851 (time 1786402182.05 total 70.07 last 82.53) - open/close 6725 (time 1786402192.05 total 80.08 last 87.36) - open/close 7526 (time 1786402202.07 total 90.09 last 79.98) - open/close 8216 (time 1786402212.07 total 100.10 last 68.95) - open/close 8939 (time 1786402222.08 total 110.11 last 72.23) - open/close 9762 (time 1786402232.08 total 120.11 last 82.30) total: 10000 open/close in 123.79 seconds: 80.78 ops/second - unlinked 0 (time 1786402240 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 931 (time 1786402318.99 total 10.00 last 93.06) - open/close 1797 (time 1786402329.00 total 20.01 last 86.57) - open/close 2605 (time 1786402339.01 total 30.02 last 80.67) - open/close 3370 (time 1786402349.02 total 40.03 last 76.44) - open/close 4199 (time 1786402359.03 total 50.04 last 82.81) - open/close 4855 (time 1786402369.04 total 60.05 last 65.56) - open/close 5821 (time 1786402379.04 total 70.05 last 96.56) - open/close 6585 (time 1786402389.04 total 80.05 last 76.39) - open/close 7410 (time 1786402399.05 total 90.06 last 82.43) - open/close 8262 (time 1786402409.05 total 100.06 last 85.19) - open/close 9141 (time 1786402419.06 total 110.07 last 87.82) total: 10000 open/close in 118.73 seconds: 84.22 ops/second - unlinked 0 (time 1786402432 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2