== conf-sanity test 69: replace an OST with the same index ========================================================== 17:50:17 (1746136217) start mds service on oleg225-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg225-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1282 (time 1746136262.86 total 10.01 last 128.08) - open/close 2385 (time 1746136272.87 total 20.02 last 110.23) - open/close 3957 (time 1746136282.87 total 30.02 last 157.14) - open/close 5382 (time 1746136292.88 total 40.02 last 142.47) - open/close 6773 (time 1746136302.89 total 50.03 last 138.94) - open/close 7885 (time 1746136312.89 total 60.04 last 111.14) - open/close 9253 (time 1746136322.90 total 70.04 last 136.75) total: 10000 open/close in 76.24 seconds: 131.17 ops/second - unlinked 0 (time 1746136330 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1191 (time 1746136380.94 total 10.00 last 119.04) - open/close 2516 (time 1746136390.94 total 20.01 last 132.47) - open/close 3932 (time 1746136400.95 total 30.02 last 141.46) - open/close 5163 (time 1746136410.96 total 40.02 last 123.08) - open/close 6522 (time 1746136420.96 total 50.02 last 135.82) - open/close 7745 (time 1746136430.98 total 60.04 last 122.11) - open/close 8763 (time 1746136440.98 total 70.05 last 101.73) - open/close 9815 (time 1746136451.00 total 80.06 last 105.06) total: 10000 open/close in 81.99 seconds: 121.97 ops/second - unlinked 0 (time 1746136454 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1077 (time 1746136524.55 total 10.00 last 107.67) - open/close 2164 (time 1746136534.56 total 20.01 last 108.65) - open/close 3259 (time 1746136544.57 total 30.02 last 109.40) - open/close 4205 (time 1746136554.57 total 40.02 last 94.58) - open/close 5499 (time 1746136564.57 total 50.02 last 129.39) - open/close 6544 (time 1746136574.58 total 60.03 last 104.43) - open/close 7646 (time 1746136584.58 total 70.03 last 110.18) - open/close 8872 (time 1746136594.59 total 80.03 last 122.52) total: 10000 open/close in 87.62 seconds: 114.13 ops/second - unlinked 0 (time 1746136603 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1289 (time 1746136672.06 total 10.00 last 128.89) - open/close 2661 (time 1746136682.07 total 20.01 last 137.00) - open/close 4085 (time 1746136692.07 total 30.02 last 142.35) - open/close 5926 (time 1746136702.08 total 40.02 last 184.00) - open/close 7164 (time 1746136712.09 total 50.04 last 123.62) - open/close 8655 (time 1746136722.10 total 60.04 last 149.04) total: 10000 open/close in 68.47 seconds: 146.05 ops/second - unlinked 0 (time 1746136732 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1666 (time 1746136773.63 total 10.00 last 166.52) - open/close 3341 (time 1746136783.64 total 20.01 last 167.34) - open/close 5123 (time 1746136793.65 total 30.02 last 178.10) - open/close 6895 (time 1746136803.65 total 40.02 last 177.14) - open/close 8962 (time 1746136813.65 total 50.02 last 206.68) total: 10000 open/close in 57.90 seconds: 172.71 ops/second - unlinked 0 (time 1746136823 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1346 (time 1746136877.52 total 10.01 last 134.44) - open/close 3101 (time 1746136887.53 total 20.02 last 175.31) - open/close 4552 (time 1746136897.53 total 30.02 last 145.09) - open/close 6118 (time 1746136907.53 total 40.02 last 156.60) - open/close 7482 (time 1746136917.53 total 50.03 last 136.31) total: 10000 open/close in 59.03 seconds: 169.39 ops/second - unlinked 0 (time 1746136927 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2167 (time 1746136963.07 total 10.00 last 216.64) - open/close 4017 (time 1746136973.08 total 20.01 last 184.90) - open/close 5352 (time 1746136983.08 total 30.01 last 133.44) - open/close 6799 (time 1746136993.09 total 40.02 last 144.59) - open/close 8378 (time 1746137003.09 total 50.02 last 157.84) total: 10000 open/close in 58.89 seconds: 169.80 ops/second - unlinked 0 (time 1746137013 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1617 (time 1746137066.75 total 10.00 last 161.69) - open/close 3163 (time 1746137076.75 total 20.00 last 154.58) - open/close 4447 (time 1746137086.76 total 30.01 last 128.28) - open/close 5856 (time 1746137096.76 total 40.01 last 140.85) - open/close 7228 (time 1746137106.77 total 50.02 last 137.14) - open/close 8577 (time 1746137116.77 total 60.02 last 134.83) - open/close 9730 (time 1746137126.78 total 70.03 last 115.27) total: 10000 open/close in 72.72 seconds: 137.51 ops/second - unlinked 0 (time 1746137131 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2006 (time 1746137172.16 total 10.00 last 200.58) - open/close 4105 (time 1746137182.16 total 20.01 last 209.81) - open/close 5870 (time 1746137192.16 total 30.01 last 176.45) - open/close 7962 (time 1746137202.17 total 40.01 last 209.09) - open/close 9799 (time 1746137212.17 total 50.02 last 183.62) total: 10000 open/close in 51.13 seconds: 195.56 ops/second - unlinked 0 (time 1746137214 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1526 (time 1746137263.71 total 10.01 last 152.49) - open/close 3100 (time 1746137273.72 total 20.02 last 157.24) - open/close 4445 (time 1746137283.72 total 30.02 last 134.49) - open/close 6120 (time 1746137293.72 total 40.03 last 167.38) - open/close 7891 (time 1746137303.73 total 50.03 last 177.08) - open/close 9633 (time 1746137313.73 total 60.03 last 174.11) total: 10000 open/close in 62.47 seconds: 160.08 ops/second - unlinked 0 (time 1746137317 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.125@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg225-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre On OST0, 17661 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2