== conf-sanity test 69: replace an OST with the same index ========================================================== 22:15:46 (1746065746) start mds service on oleg135-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 882 (time 1746065815.21 total 10.01 last 88.14) - open/close 1730 (time 1746065825.23 total 20.02 last 84.68) - open/close 2559 (time 1746065835.23 total 30.02 last 82.89) - open/close 3407 (time 1746065845.24 total 40.03 last 84.72) - open/close 4454 (time 1746065855.24 total 50.03 last 104.66) - open/close 5436 (time 1746065865.24 total 60.04 last 98.15) - open/close 6529 (time 1746065875.25 total 70.04 last 109.26) - open/close 7415 (time 1746065885.25 total 80.04 last 88.59) - open/close 8273 (time 1746065895.26 total 90.05 last 85.75) - open/close 9092 (time 1746065905.26 total 100.05 last 81.86) - open/close 9918 (time 1746065915.27 total 110.06 last 82.52) total: 10000 open/close in 110.85 seconds: 90.21 ops/second - unlinked 0 (time 1746065918 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1017 (time 1746065995.18 total 10.00 last 101.66) - open/close 1942 (time 1746066005.19 total 20.01 last 92.42) - open/close 2840 (time 1746066015.19 total 30.02 last 89.75) - open/close 3758 (time 1746066025.20 total 40.02 last 91.77) - open/close 4508 (time 1746066035.20 total 50.02 last 74.99) - open/close 5242 (time 1746066045.20 total 60.03 last 73.36) - open/close 5913 (time 1746066055.21 total 70.03 last 67.06) - open/close 6647 (time 1746066065.21 total 80.04 last 73.36) - open/close 7309 (time 1746066075.22 total 90.04 last 66.19) - open/close 8063 (time 1746066085.22 total 100.04 last 75.37) - open/close 8828 (time 1746066095.23 total 110.05 last 76.45) - open/close 9557 (time 1746066105.24 total 120.06 last 72.81) total: 10000 open/close in 126.11 seconds: 79.29 ops/second - unlinked 0 (time 1746066114 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 651 (time 1746066207.23 total 10.01 last 65.03) - open/close 1338 (time 1746066217.25 total 20.03 last 68.57) - open/close 2088 (time 1746066227.27 total 30.06 last 74.79) - open/close 2826 (time 1746066237.28 total 40.06 last 73.73) - open/close 3533 (time 1746066247.29 total 50.07 last 70.65) - open/close 4307 (time 1746066257.30 total 60.09 last 77.30) - open/close 5248 (time 1746066267.32 total 70.10 last 93.98) - open/close 6147 (time 1746066277.32 total 80.10 last 89.85) - open/close 6931 (time 1746066287.32 total 90.11 last 78.38) - open/close 7654 (time 1746066297.34 total 100.12 last 72.19) - open/close 8427 (time 1746066307.36 total 110.14 last 77.16) - open/close 9154 (time 1746066317.37 total 120.15 last 72.59) - open/close 9954 (time 1746066327.39 total 130.17 last 79.90) total: 10000 open/close in 130.71 seconds: 76.50 ops/second - unlinked 0 (time 1746066330 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 815 (time 1746066418.12 total 10.01 last 81.43) - open/close 1610 (time 1746066428.12 total 20.01 last 79.48) - open/close 2434 (time 1746066438.12 total 30.01 last 82.39) - open/close 3239 (time 1746066448.12 total 40.02 last 80.48) - open/close 4083 (time 1746066458.13 total 50.02 last 84.37) - open/close 4995 (time 1746066468.14 total 60.03 last 91.10) - open/close 5730 (time 1746066478.16 total 70.05 last 73.38) - open/close 6583 (time 1746066488.16 total 80.05 last 85.25) - open/close 7387 (time 1746066498.16 total 90.06 last 80.37) - open/close 8135 (time 1746066508.17 total 100.06 last 74.76) - open/close 9005 (time 1746066518.19 total 110.08 last 86.87) - open/close 9775 (time 1746066528.19 total 120.08 last 76.95) total: 10000 open/close in 122.60 seconds: 81.56 ops/second - unlinked 0 (time 1746066533 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 937 (time 1746066621.92 total 10.01 last 93.57) - open/close 1809 (time 1746066631.94 total 20.03 last 87.05) - open/close 2626 (time 1746066641.95 total 30.04 last 81.65) - open/close 3498 (time 1746066651.95 total 40.04 last 87.18) - open/close 4429 (time 1746066661.95 total 50.04 last 93.08) - open/close 5300 (time 1746066671.96 total 60.05 last 87.05) - open/close 6096 (time 1746066681.96 total 70.05 last 79.56) - open/close 6983 (time 1746066691.96 total 80.05 last 88.70) - open/close 7762 (time 1746066701.96 total 90.05 last 77.87) - open/close 8501 (time 1746066711.98 total 100.07 last 73.78) - open/close 9263 (time 1746066721.98 total 110.07 last 76.19) - open/close 9968 (time 1746066732.01 total 120.10 last 70.31) total: 10000 open/close in 120.47 seconds: 83.01 ops/second - unlinked 0 (time 1746066735 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 893 (time 1746066822.97 total 10.00 last 89.26) - open/close 1712 (time 1746066832.98 total 20.01 last 81.82) - open/close 2704 (time 1746066842.99 total 30.03 last 99.07) - open/close 3577 (time 1746066852.99 total 40.03 last 87.29) - open/close 4531 (time 1746066863.00 total 50.03 last 95.39) - open/close 5441 (time 1746066873.01 total 60.04 last 90.88) - open/close 6313 (time 1746066883.01 total 70.05 last 87.16) - open/close 7352 (time 1746066893.02 total 80.05 last 103.84) - open/close 8395 (time 1746066903.02 total 90.06 last 104.27) - open/close 9275 (time 1746066913.03 total 100.06 last 87.94) total: 10000 open/close in 108.61 seconds: 92.07 ops/second - unlinked 0 (time 1746066924 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 771 (time 1746067013.61 total 10.00 last 77.07) - open/close 1504 (time 1746067023.62 total 20.02 last 73.22) - open/close 2308 (time 1746067033.63 total 30.03 last 80.29) - open/close 3151 (time 1746067043.64 total 40.03 last 84.25) - open/close 3917 (time 1746067053.64 total 50.03 last 76.60) - open/close 4678 (time 1746067063.67 total 60.07 last 75.84) - open/close 5393 (time 1746067073.68 total 70.08 last 71.45) - open/close 6227 (time 1746067083.68 total 80.08 last 83.37) - open/close 7144 (time 1746067093.69 total 90.09 last 91.61) - open/close 7979 (time 1746067103.69 total 100.09 last 83.49) - open/close 8720 (time 1746067113.70 total 110.10 last 74.04) - open/close 9427 (time 1746067123.71 total 120.11 last 70.64) total: 10000 open/close in 127.22 seconds: 78.61 ops/second - unlinked 0 (time 1746067133 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 1048 (time 1746067215.41 total 10.01 last 104.74) - open/close 2108 (time 1746067225.42 total 20.01 last 105.92) - open/close 3152 (time 1746067235.42 total 30.02 last 104.38) - open/close 4079 (time 1746067245.43 total 40.02 last 92.64) - open/close 4973 (time 1746067255.43 total 50.03 last 89.34) - open/close 5863 (time 1746067265.44 total 60.03 last 88.97) - open/close 6742 (time 1746067275.44 total 70.03 last 87.89) - open/close 7619 (time 1746067285.44 total 80.03 last 87.70) - open/close 8574 (time 1746067295.44 total 90.04 last 95.45) - open/close 9386 (time 1746067305.45 total 100.04 last 81.17) total: 10000 open/close in 106.62 seconds: 93.79 ops/second - unlinked 0 (time 1746067314 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 879 (time 1746067389.96 total 10.00 last 87.87) - open/close 1731 (time 1746067399.96 total 20.01 last 85.16) - open/close 2633 (time 1746067409.96 total 30.01 last 90.20) - open/close 3590 (time 1746067419.97 total 40.01 last 95.65) - open/close 4489 (time 1746067429.98 total 50.02 last 89.81) - open/close 5457 (time 1746067439.98 total 60.02 last 96.79) - open/close 6462 (time 1746067449.98 total 70.03 last 100.45) - open/close 7369 (time 1746067459.98 total 80.03 last 90.70) - open/close 8394 (time 1746067469.99 total 90.04 last 102.42) - open/close 9432 (time 1746067479.99 total 100.04 last 103.78) total: 10000 open/close in 105.52 seconds: 94.77 ops/second - unlinked 0 (time 1746067487 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1205 (time 1746067552.34 total 10.00 last 120.44) - open/close 2322 (time 1746067562.34 total 20.01 last 111.63) - open/close 3166 (time 1746067572.34 total 30.01 last 84.39) - open/close 4058 (time 1746067582.35 total 40.02 last 89.15) - open/close 4959 (time 1746067592.37 total 50.04 last 89.93) - open/close 5881 (time 1746067602.37 total 60.04 last 92.18) - open/close 6828 (time 1746067612.38 total 70.05 last 94.57) - open/close 7688 (time 1746067622.38 total 80.05 last 85.99) - open/close 8544 (time 1746067632.41 total 90.08 last 85.38) - open/close 9385 (time 1746067642.41 total 100.08 last 84.09) total: 10000 open/close in 108.19 seconds: 92.43 ops/second - unlinked 0 (time 1746067653 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre On OST0, 10709 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2