== conf-sanity test 69: replace an OST with the same index ========================================================== 09:48:51 (1746625731) start mds service on oleg356-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: MODOPTS_LIBCFS= oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2688 (time 1746625771.51 total 10.01 last 268.63) - open/close 6049 (time 1746625781.51 total 20.01 last 335.99) - open/close 8069 (time 1746625791.52 total 30.01 last 201.93) total: 10000 open/close in 39.89 seconds: 250.69 ops/second - unlinked 0 (time 1746625802 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2396 (time 1746625839.25 total 10.00 last 239.59) - open/close 5023 (time 1746625849.25 total 20.00 last 262.61) - open/close 7249 (time 1746625859.25 total 30.01 last 222.54) - open/close 9517 (time 1746625869.25 total 40.01 last 226.80) total: 10000 open/close in 41.85 seconds: 238.93 ops/second - unlinked 0 (time 1746625872 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2623 (time 1746625905.36 total 10.00 last 262.20) - open/close 5392 (time 1746625915.37 total 20.01 last 276.73) - open/close 8267 (time 1746625925.37 total 30.01 last 287.49) total: 10000 open/close in 36.47 seconds: 274.22 ops/second - unlinked 0 (time 1746625932 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2352 (time 1746625962.70 total 10.01 last 234.87) - open/close 4618 (time 1746625972.70 total 20.02 last 226.54) - open/close 7606 (time 1746625982.70 total 30.02 last 298.78) total: 10000 open/close in 36.85 seconds: 271.37 ops/second - unlinked 0 (time 1746625990 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 3383 (time 1746626020.22 total 10.00 last 338.25) - open/close 7498 (time 1746626030.22 total 20.00 last 411.42) total: 10000 open/close in 27.42 seconds: 364.72 ops/second - unlinked 0 (time 1746626038 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3638 (time 1746626066.27 total 10.00 last 363.74) - open/close 6628 (time 1746626076.27 total 20.00 last 298.93) - open/close 9390 (time 1746626086.28 total 30.01 last 276.05) total: 10000 open/close in 32.64 seconds: 306.34 ops/second - unlinked 0 (time 1746626089 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2692 (time 1746626118.47 total 10.00 last 269.18) - open/close 5694 (time 1746626128.47 total 20.00 last 300.17) - open/close 9056 (time 1746626138.47 total 30.00 last 336.18) total: 10000 open/close in 32.63 seconds: 306.43 ops/second - unlinked 0 (time 1746626141 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2887 (time 1746626173.61 total 10.00 last 288.67) - open/close 6389 (time 1746626183.62 total 20.00 last 350.10) - open/close 9994 (time 1746626193.62 total 30.00 last 360.48) total: 10000 open/close in 30.01 seconds: 333.17 ops/second - unlinked 0 (time 1746626194 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 2746 (time 1746626221.68 total 10.00 last 274.57) - open/close 5258 (time 1746626231.68 total 20.00 last 251.15) - open/close 7354 (time 1746626241.68 total 30.01 last 209.53) total: 10000 open/close in 39.45 seconds: 253.50 ops/second - unlinked 0 (time 1746626252 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2587 (time 1746626284.70 total 10.00 last 258.68) - open/close 5218 (time 1746626294.70 total 20.00 last 263.05) - open/close 8187 (time 1746626304.71 total 30.01 last 296.78) total: 10000 open/close in 36.03 seconds: 277.52 ops/second - unlinked 0 (time 1746626311 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.156@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg356-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre On OST0, 17741 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server unloading modules via unload_modules_local on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2