== conf-sanity test 69: replace an OST with the same index ========================================================== 07:25:54 (1788866754) start mds service on oleg209-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: kmemleak disabled oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg209-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg209-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 747 (time 1788866872.77 total 10.00 last 74.69) - open/close 1496 (time 1788866882.78 total 20.00 last 74.88) - open/close 2251 (time 1788866892.78 total 30.01 last 75.46) - open/close 3073 (time 1788866902.80 total 40.02 last 82.09) - open/close 3837 (time 1788866912.80 total 50.03 last 76.35) - open/close 4682 (time 1788866922.81 total 60.04 last 84.46) - open/close 5464 (time 1788866932.82 total 70.05 last 78.11) - open/close 6248 (time 1788866942.82 total 80.05 last 78.37) - open/close 7087 (time 1788866952.83 total 90.06 last 83.81) - open/close 7880 (time 1788866962.85 total 100.08 last 79.17) - open/close 8595 (time 1788866972.86 total 110.09 last 71.44) - open/close 9418 (time 1788866982.87 total 120.10 last 82.20) total: 10000 open/close in 125.99 seconds: 79.37 ops/second - unlinked 0 (time 1788866993 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 763 (time 1788867085.09 total 10.00 last 76.28) - open/close 1474 (time 1788867095.10 total 20.01 last 71.06) - open/close 2211 (time 1788867105.10 total 30.02 last 73.64) - open/close 2957 (time 1788867115.11 total 40.02 last 74.54) - open/close 3648 (time 1788867125.13 total 50.05 last 68.95) - open/close 4442 (time 1788867135.14 total 60.05 last 79.36) - open/close 5211 (time 1788867145.14 total 70.05 last 76.89) - open/close 5948 (time 1788867155.14 total 80.05 last 73.67) - open/close 6679 (time 1788867165.16 total 90.07 last 73.01) - open/close 7392 (time 1788867175.16 total 100.07 last 71.28) - open/close 8105 (time 1788867185.17 total 110.08 last 71.23) - open/close 8795 (time 1788867195.17 total 120.08 last 68.99) - open/close 9510 (time 1788867205.17 total 130.08 last 71.48) total: 10000 open/close in 136.42 seconds: 73.30 ops/second - unlinked 0 (time 1788867216 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 646 (time 1788867312.96 total 10.00 last 64.59) - open/close 1283 (time 1788867322.98 total 20.02 last 63.57) - open/close 1919 (time 1788867332.99 total 30.03 last 63.56) - open/close 2592 (time 1788867343.00 total 40.04 last 67.21) - open/close 3163 (time 1788867353.01 total 50.05 last 57.06) - open/close 3979 (time 1788867363.01 total 60.05 last 81.55) - open/close 4610 (time 1788867373.02 total 70.06 last 63.09) - open/close 5275 (time 1788867383.02 total 80.06 last 66.49) - open/close 6022 (time 1788867393.02 total 90.06 last 74.68) - open/close 6780 (time 1788867403.04 total 100.08 last 75.66) - open/close 7580 (time 1788867413.05 total 110.09 last 79.85) - open/close 8390 (time 1788867423.06 total 120.10 last 80.94) - open/close 9102 (time 1788867433.08 total 130.12 last 71.06) - open/close 9787 (time 1788867443.10 total 140.14 last 68.40) total: 10000 open/close in 143.17 seconds: 69.85 ops/second - unlinked 0 (time 1788867452 ; total 1 ; last 1) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 631 (time 1788867553.00 total 10.00 last 63.08) - open/close 1248 (time 1788867563.02 total 20.03 last 61.56) - open/close 1957 (time 1788867573.03 total 30.04 last 70.82) - open/close 2791 (time 1788867583.03 total 40.04 last 83.39) - open/close 3570 (time 1788867593.05 total 50.05 last 77.79) - open/close 4332 (time 1788867603.05 total 60.05 last 76.19) - open/close 4954 (time 1788867613.05 total 70.06 last 62.14) - open/close 5599 (time 1788867623.06 total 80.07 last 64.47) - open/close 6301 (time 1788867633.08 total 90.08 last 70.07) - open/close 7054 (time 1788867643.08 total 100.09 last 75.27) - open/close 7696 (time 1788867653.08 total 110.09 last 64.19) - open/close 8180 (time 1788867663.10 total 120.11 last 48.32) - open/close 8759 (time 1788867673.13 total 130.13 last 57.74) - open/close 9331 (time 1788867683.15 total 140.15 last 57.09) total: 10000 open/close in 149.89 seconds: 66.71 ops/second - unlinked 0 (time 1788867698 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 742 (time 1788867787.13 total 10.01 last 74.15) - open/close 1489 (time 1788867797.13 total 20.01 last 74.67) - open/close 2337 (time 1788867807.14 total 30.02 last 84.75) - open/close 3101 (time 1788867817.14 total 40.02 last 76.39) - open/close 3944 (time 1788867827.14 total 50.02 last 84.28) - open/close 4655 (time 1788867837.14 total 60.02 last 71.08) - open/close 5361 (time 1788867847.15 total 70.03 last 70.57) - open/close 6145 (time 1788867857.16 total 80.04 last 78.31) - open/close 6875 (time 1788867867.17 total 90.05 last 72.94) - open/close 7719 (time 1788867877.17 total 100.05 last 84.34) - open/close 8387 (time 1788867887.18 total 110.06 last 66.79) - open/close 9116 (time 1788867897.19 total 120.07 last 72.83) - open/close 9833 (time 1788867907.19 total 130.07 last 71.69) total: 10000 open/close in 132.19 seconds: 75.65 ops/second - unlinked 0 (time 1788867914 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 833 (time 1788868018.99 total 10.01 last 83.23) - open/close 1532 (time 1788868029.00 total 20.02 last 69.80) - open/close 2231 (time 1788868039.02 total 30.04 last 69.77) - open/close 2885 (time 1788868049.03 total 40.05 last 65.36) - open/close 3555 (time 1788868059.04 total 50.06 last 66.89) - open/close 4292 (time 1788868069.07 total 60.09 last 73.51) - open/close 4901 (time 1788868079.07 total 70.09 last 60.89) - open/close 5566 (time 1788868089.08 total 80.10 last 66.47) - open/close 6388 (time 1788868099.08 total 90.10 last 82.18) - open/close 7163 (time 1788868109.09 total 100.11 last 77.43) - open/close 7841 (time 1788868119.12 total 110.15 last 67.55) - open/close 8486 (time 1788868129.14 total 120.16 last 64.43) - open/close 9180 (time 1788868139.15 total 130.17 last 69.29) - open/close 9765 (time 1788868149.17 total 140.19 last 58.41) total: 10000 open/close in 143.29 seconds: 69.79 ops/second - unlinked 0 (time 1788868157 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 811 (time 1788868247.29 total 10.00 last 81.10) - open/close 1551 (time 1788868257.30 total 20.00 last 73.97) - open/close 2247 (time 1788868267.30 total 30.01 last 69.56) - open/close 2955 (time 1788868277.31 total 40.01 last 70.77) - open/close 3811 (time 1788868287.31 total 50.02 last 85.55) - open/close 4575 (time 1788868297.33 total 60.04 last 76.25) - open/close 5381 (time 1788868307.34 total 70.04 last 80.56) - open/close 6171 (time 1788868317.34 total 80.05 last 78.94) - open/close 7000 (time 1788868327.35 total 90.05 last 82.89) - open/close 7825 (time 1788868337.35 total 100.05 last 82.49) - open/close 8602 (time 1788868347.35 total 110.06 last 77.67) - open/close 9289 (time 1788868357.35 total 120.06 last 68.69) total: 10000 open/close in 130.01 seconds: 76.92 ops/second - unlinked 0 (time 1788868372 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 896 (time 1788868456.55 total 10.01 last 89.55) - open/close 1735 (time 1788868466.58 total 20.03 last 83.66) - open/close 2651 (time 1788868476.61 total 30.06 last 91.33) - open/close 3405 (time 1788868486.62 total 40.08 last 75.27) - open/close 4091 (time 1788868496.63 total 50.09 last 68.56) - open/close 4991 (time 1788868506.63 total 60.09 last 89.96) - open/close 5686 (time 1788868516.66 total 70.11 last 69.35) - open/close 6545 (time 1788868526.66 total 80.12 last 85.86) - open/close 7386 (time 1788868536.66 total 90.12 last 84.09) - open/close 8235 (time 1788868546.67 total 100.12 last 84.85) - open/close 9018 (time 1788868556.68 total 110.14 last 78.17) - open/close 9711 (time 1788868566.70 total 120.15 last 69.22) total: 10000 open/close in 123.98 seconds: 80.66 ops/second - unlinked 0 (time 1788868575 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second