== conf-sanity test 69: replace an OST with the same index ========================================================== 18:19:45 (1736983185) start mds service on oleg344-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg344-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg344-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 2320 (time 1736983246.28 total 10.00 last 231.98) - open/close 4145 (time 1736983256.28 total 20.00 last 182.43) - open/close 5794 (time 1736983266.28 total 30.01 last 164.89) - open/close 7339 (time 1736983276.29 total 40.01 last 154.47) - open/close 8925 (time 1736983286.29 total 50.01 last 158.59) total: 10000 open/close in 55.88 seconds: 178.96 ops/second - unlinked 0 (time 1736983293 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2470 (time 1736983329.75 total 10.00 last 246.97) - open/close 5056 (time 1736983339.75 total 20.00 last 258.55) - open/close 7404 (time 1736983349.75 total 30.00 last 234.77) total: 10000 open/close in 39.30 seconds: 254.44 ops/second - unlinked 0 (time 1736983360 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2453 (time 1736983392.66 total 10.00 last 245.28) - open/close 5275 (time 1736983402.66 total 20.00 last 282.11) - open/close 8031 (time 1736983412.66 total 30.01 last 275.52) total: 10000 open/close in 37.14 seconds: 269.25 ops/second - unlinked 0 (time 1736983420 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2674 (time 1736983454.36 total 10.00 last 267.35) - open/close 5121 (time 1736983464.36 total 20.01 last 244.62) - open/close 7626 (time 1736983474.37 total 30.01 last 250.44) total: 10000 open/close in 39.28 seconds: 254.59 ops/second - unlinked 0 (time 1736983484 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2891 (time 1736983514.63 total 10.00 last 289.06) - open/close 5863 (time 1736983524.64 total 20.00 last 297.15) - open/close 8534 (time 1736983534.64 total 30.00 last 267.07) total: 10000 open/close in 36.01 seconds: 277.70 ops/second - unlinked 0 (time 1736983541 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3109 (time 1736983572.26 total 10.00 last 310.83) - open/close 5759 (time 1736983582.26 total 20.00 last 265.00) - open/close 8331 (time 1736983592.26 total 30.01 last 257.11) total: 10000 open/close in 36.29 seconds: 275.55 ops/second - unlinked 0 (time 1736983599 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2979 (time 1736983628.34 total 10.00 last 297.78) - open/close 5429 (time 1736983638.34 total 20.01 last 244.93) - open/close 8041 (time 1736983648.34 total 30.01 last 261.18) total: 10000 open/close in 37.12 seconds: 269.42 ops/second - unlinked 0 (time 1736983656 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2654 (time 1736983686.74 total 10.00 last 265.39) - open/close 5480 (time 1736983696.74 total 20.00 last 282.50) - open/close 8220 (time 1736983706.74 total 30.01 last 273.93) total: 10000 open/close in 35.99 seconds: 277.83 ops/second - unlinked 0 (time 1736983713 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3214 (time 1736983741.94 total 10.00 last 321.33) - open/close 6341 (time 1736983751.94 total 20.00 last 312.66) - open/close 9170 (time 1736983761.95 total 30.01 last 282.80) total: 10000 open/close in 33.23 seconds: 300.98 ops/second - unlinked 0 (time 1736983766 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2464 (time 1736983799.60 total 10.00 last 246.28) - open/close 5454 (time 1736983809.60 total 20.01 last 298.95) - open/close 8598 (time 1736983819.60 total 30.01 last 314.37) total: 10000 open/close in 35.39 seconds: 282.60 ops/second - unlinked 0 (time 1736983826 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.144@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg344-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg344-server: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg344-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 12 sec oleg344-server: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg344-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre On OST0, 10397 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server LNET ready to unload unloading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local oleg344-server: LNET ready to unload modules unloaded. pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2