== conf-sanity test 69: replace an OST with the same index ========================================================== 06:46:35 (1786185995) start mds service on oleg153-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg153-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg153-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg153-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 998 (time 1786186122.67 total 10.00 last 99.77) - open/close 2093 (time 1786186132.67 total 20.01 last 109.44) - open/close 3001 (time 1786186142.68 total 30.02 last 90.73) - open/close 3808 (time 1786186152.68 total 40.02 last 80.70) - open/close 4607 (time 1786186162.70 total 50.04 last 79.73) - open/close 5481 (time 1786186172.70 total 60.04 last 87.38) - open/close 6354 (time 1786186182.72 total 70.05 last 87.19) - open/close 7165 (time 1786186192.72 total 80.06 last 81.04) - open/close 7827 (time 1786186202.73 total 90.07 last 66.17) - open/close 8698 (time 1786186212.73 total 100.07 last 87.06) - open/close 9407 (time 1786186222.73 total 110.07 last 70.89) total: 10000 open/close in 117.18 seconds: 85.34 ops/second - unlinked 0 (time 1786186234 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 876 (time 1786186314.73 total 10.01 last 87.53) - open/close 1859 (time 1786186324.73 total 20.01 last 98.29) - open/close 2780 (time 1786186334.74 total 30.02 last 91.98) - open/close 3694 (time 1786186344.75 total 40.03 last 91.32) - open/close 4637 (time 1786186354.76 total 50.04 last 94.26) - open/close 5587 (time 1786186364.76 total 60.04 last 95.00) - open/close 6500 (time 1786186374.76 total 70.04 last 91.28) - open/close 7425 (time 1786186384.77 total 80.05 last 92.44) - open/close 8179 (time 1786186394.77 total 90.05 last 75.40) - open/close 9071 (time 1786186404.77 total 100.05 last 89.17) - open/close 9940 (time 1786186414.79 total 110.07 last 86.71) total: 10000 open/close in 110.79 seconds: 90.26 ops/second - unlinked 0 (time 1786186420 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 789 (time 1786186497.31 total 10.02 last 78.78) - open/close 1843 (time 1786186507.33 total 20.03 last 105.24) - open/close 2731 (time 1786186517.34 total 30.04 last 88.73) - open/close 3563 (time 1786186527.34 total 40.04 last 83.16) - open/close 4425 (time 1786186537.36 total 50.06 last 86.03) - open/close 5239 (time 1786186547.37 total 60.07 last 81.32) - open/close 6202 (time 1786186557.38 total 70.08 last 96.26) - open/close 7203 (time 1786186567.38 total 80.09 last 100.02) - open/close 8188 (time 1786186577.39 total 90.09 last 98.43) - open/close 9142 (time 1786186587.39 total 100.09 last 95.39) - open/close 9887 (time 1786186597.40 total 110.10 last 74.43) total: 10000 open/close in 111.71 seconds: 89.52 ops/second - unlinked 0 (time 1786186604 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 938 (time 1786186683.63 total 10.01 last 93.74) - open/close 1833 (time 1786186693.64 total 20.01 last 89.43) - open/close 2728 (time 1786186703.64 total 30.02 last 89.48) - open/close 3691 (time 1786186713.64 total 40.02 last 96.24) - open/close 4693 (time 1786186723.64 total 50.02 last 100.20) - open/close 5745 (time 1786186733.65 total 60.03 last 105.14) - open/close 6718 (time 1786186743.67 total 70.05 last 97.08) - open/close 7531 (time 1786186753.68 total 80.06 last 81.27) - open/close 8224 (time 1786186763.71 total 90.09 last 69.06) - open/close 9191 (time 1786186773.71 total 100.09 last 96.69) - open/close 9932 (time 1786186783.72 total 110.09 last 74.08) total: 10000 open/close in 110.50 seconds: 90.50 ops/second - unlinked 0 (time 1786186788 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 789 (time 1786186867.30 total 10.01 last 78.86) - open/close 1653 (time 1786186877.31 total 20.01 last 86.38) - open/close 2539 (time 1786186887.31 total 30.01 last 88.55) - open/close 3489 (time 1786186897.31 total 40.01 last 95.00) - open/close 4543 (time 1786186907.31 total 50.01 last 105.39) - open/close 5462 (time 1786186917.32 total 60.02 last 91.84) - open/close 6451 (time 1786186927.33 total 70.04 last 98.74) - open/close 7478 (time 1786186937.34 total 80.04 last 102.63) - open/close 8455 (time 1786186947.35 total 90.05 last 97.65) - open/close 9541 (time 1786186957.35 total 100.05 last 108.58) total: 10000 open/close in 104.89 seconds: 95.34 ops/second - unlinked 0 (time 1786186966 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 910 (time 1786187047.16 total 10.01 last 90.89) - open/close 1917 (time 1786187057.17 total 20.02 last 100.60) - open/close 2734 (time 1786187067.17 total 30.03 last 81.67) - open/close 3591 (time 1786187077.18 total 40.03 last 85.69) - open/close 4472 (time 1786187087.18 total 50.03 last 88.07) - open/close 5371 (time 1786187097.18 total 60.03 last 89.90) - open/close 6212 (time 1786187107.18 total 70.03 last 84.07) - open/close 7019 (time 1786187117.19 total 80.04 last 80.64) - open/close 8010 (time 1786187127.19 total 90.05 last 99.06) - open/close 8932 (time 1786187137.20 total 100.05 last 92.15) - open/close 9795 (time 1786187147.21 total 110.06 last 86.21) total: 10000 open/close in 112.80 seconds: 88.65 ops/second - unlinked 0 (time 1786187155 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 602 (time 1786187236.50 total 10.01 last 60.16) - open/close 1461 (time 1786187246.50 total 20.02 last 85.83) - open/close 2312 (time 1786187256.51 total 30.02 last 85.05) - open/close 3135 (time 1786187266.52 total 40.03 last 82.25) - open/close 3917 (time 1786187276.52 total 50.03 last 78.17) - open/close 4531 (time 1786187286.52 total 60.03 last 61.38) - open/close 5454 (time 1786187296.53 total 70.04 last 92.27) - open/close 6504 (time 1786187306.53 total 80.04 last 104.97) - open/close 7440 (time 1786187316.54 total 90.05 last 93.52) - open/close 8319 (time 1786187326.55 total 100.06 last 87.82) - open/close 9204 (time 1786187336.55 total 110.06 last 88.47) - open/close 9998 (time 1786187346.56 total 120.07 last 79.36) total: 10000 open/close in 120.07 seconds: 83.28 ops/second - unlinked 0 (time 1786187351 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 725 (time 1786187446.60 total 10.00 last 72.47) - open/close 1318 (time 1786187456.61 total 20.02 last 59.23) - open/close 2021 (time 1786187466.63 total 30.03 last 70.16) - open/close 2817 (time 1786187476.63 total 40.04 last 79.54) - open/close 3528 (time 1786187486.64 total 50.05 last 71.05) - open/close 4327 (time 1786187496.65 total 60.05 last 79.86) - open/close 5149 (time 1786187506.65 total 70.06 last 82.15) - open/close 6062 (time 1786187516.65 total 80.06 last 91.29) - open/close 6796 (time 1786187526.66 total 90.07 last 73.36) - open/close 7552 (time 1786187536.67 total 100.08 last 75.50) - open/close 8386 (time 1786187546.69 total 110.10 last 83.24) - open/close 9198 (time 1786187556.71 total 120.12 last 81.00) total: 10000 open/close in 129.35 seconds: 77.31 ops/second - unlinked 0 (time 1786187570 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 863 (time 1786187658.21 total 10.02 last 86.14) - open/close 1432 (time 1786187668.22 total 20.03 last 56.83) - open/close 2113 (time 1786187678.23 total 30.04 last 68.04) - open/close 2969 (time 1786187688.24 total 40.06 last 85.45) - open/close 3889 (time 1786187698.25 total 50.06 last 91.98) - open/close 4774 (time 1786187708.25 total 60.06 last 88.49) - open/close 5621 (time 1786187718.25 total 70.06 last 84.67) - open/close 6370 (time 1786187728.25 total 80.07 last 74.88) - open/close 7292 (time 1786187738.26 total 90.07 last 92.15) - open/close 8045 (time 1786187748.26 total 100.07 last 75.27) - open/close 8785 (time 1786187758.27 total 110.09 last 73.92) - open/close 9551 (time 1786187768.28 total 120.09 last 76.54) total: 10000 open/close in 126.05 seconds: 79.33 ops/second - unlinked 0 (time 1786187780 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 782 (time 1786187866.61 total 10.01 last 78.11) - open/close 1674 (time 1786187876.62 total 20.01 last 89.18) - open/close 2560 (time 1786187886.62 total 30.02 last 88.55) - open/close 3351 (time 1786187896.62 total 40.02 last 79.10) - open/close 4369 (time 1786187906.63 total 50.03 last 101.73) - open/close 5065 (time 1786187916.63 total 60.03 last 69.60) - open/close 6007 (time 1786187926.64 total 70.04 last 94.10) - open/close 6843 (time 1786187936.64 total 80.04 last 83.56) - open/close 7596 (time 1786187946.66 total 90.06 last 75.19) - open/close 8386 (time 1786187956.67 total 100.06 last 78.95) - open/close 9172 (time 1786187966.67 total 110.07 last 78.57) - open/close 9833 (time 1786187976.68 total 120.08 last 66.04) total: 10000 open/close in 122.87 seconds: 81.39 ops/second - unlinked 0 (time 1786187984 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg153-server unloading modules via unload_modules_local on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local oleg153-server: modules unloaded. pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2