== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:48:29 (1789710509) start mds service on oleg258-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 18 00:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing load_modules_local oleg258-server: kmemleak disabled oleg258-server: Loading modules from /home/green/git/lustre-release/lustre oleg258-server: detected 4 online CPUs by sysfs oleg258-server: MODOPTS_LIBCFS= oleg258-server: Force libcfs to create 2 CPU partitions oleg258-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg258-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg258-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg258-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1789710550.98 total 10.02 last 13.47) - open/close 244 (time 1789710560.98 total 20.02 last 10.90) - open/close 417 (time 1789710571.39 total 30.44 last 16.61) - open/close 679 (time 1789710581.40 total 40.44 last 26.20) - open/close 961 (time 1789710591.40 total 50.45 last 28.18) - open/close 1246 (time 1789710601.87 total 60.91 last 27.23) - open/close 1506 (time 1789710612.87 total 71.92 last 23.63) - open/close 1744 (time 1789710623.36 total 82.41 last 22.68) - open/close 2223 (time 1789710633.36 total 92.41 last 47.90) - open/close 2754 (time 1789710643.39 total 102.44 last 52.95) - open/close 2988 (time 1789710653.50 total 112.55 last 23.14) - open/close 3148 (time 1789710663.62 total 122.66 last 15.82) - open/close 3535 (time 1789710674.63 total 133.67 last 35.15) - open/close 3932 (time 1789710684.64 total 143.68 last 39.66) - open/close 4330 (time 1789710694.64 total 153.68 last 39.79) - open/close 4846 (time 1789710705.22 total 164.26 last 48.79) - open/close 5210 (time 1789710715.22 total 174.26 last 36.40) - open/close 5886 (time 1789710726.14 total 185.19 last 61.85) - open/close 6512 (time 1789710736.51 total 195.56 last 60.38) - open/close 7051 (time 1789710746.52 total 205.56 last 53.89) - open/close 7212 (time 1789710756.68 total 215.72 last 15.85) - open/close 7572 (time 1789710767.55 total 226.60 last 33.10) - open/close 8159 (time 1789710777.99 total 237.03 last 56.25) - open/close 8664 (time 1789710788.16 total 247.20 last 49.64) - open/close 9269 (time 1789710798.21 total 257.26 last 60.18) - open/close 9815 (time 1789710808.21 total 267.26 last 54.60) - open/close 10000 (time 1789710812.89 total 271.93 last 39.56) - open/close 10479 (time 1789710823.68 total 282.73 last 44.38) - open/close 11038 (time 1789710833.73 total 292.77 last 55.64) - open/close 11644 (time 1789710844.61 total 303.66 last 55.68) - open/close 12252 (time 1789710854.61 total 313.66 last 60.80) - open/close 12835 (time 1789710864.77 total 323.81 last 57.40) - open/close 13384 (time 1789710875.65 total 334.69 last 50.46) - open/close 13986 (time 1789710885.66 total 344.70 last 60.15) - open/close 14981 (time 1789710895.66 total 354.70 last 99.49) - open/close 15960 (time 1789710905.66 total 364.70 last 97.89) - open/close 16734 (time 1789710915.66 total 374.71 last 77.36) - open/close 17707 (time 1789710925.67 total 384.72 last 97.22) - open/close 18952 (time 1789710935.68 total 394.72 last 124.41) - open/close 20000 (time 1789710942.82 total 401.87 last 146.67) - open/close 21339 (time 1789710952.82 total 411.87 last 133.89) - open/close 22787 (time 1789710962.83 total 421.87 last 144.77) - open/close 24205 (time 1789710972.83 total 431.88 last 141.72) - open/close 25522 (time 1789710982.85 total 441.90 last 131.43) - open/close 26876 (time 1789710992.86 total 451.91 last 135.26) - open/close 28239 (time 1789711002.86 total 461.91 last 136.30) - open/close 29869 (time 1789711012.87 total 471.91 last 162.92) - open/close 31403 (time 1789711022.88 total 481.92 last 153.25) - open/close 32612 (time 1789711032.88 total 491.92 last 120.90) - open/close 33795 (time 1789711042.89 total 501.94 last 118.17) - open/close 34902 (time 1789711052.89 total 511.94 last 110.66) - open/close 36030 (time 1789711062.91 total 521.95 last 112.65) - open/close 37249 (time 1789711072.91 total 531.95 last 121.88) - open/close 38618 (time 1789711082.91 total 541.96 last 136.86) - open/close 40000 (time 1789711090.70 total 549.75 last 177.44) - open/close 41930 (time 1789711100.71 total 559.75 last 192.90) - open/close 43814 (time 1789711110.71 total 569.75 last 188.36) - open/close 45565 (time 1789711120.71 total 579.75 last 175.08) - open/close 47618 (time 1789711130.71 total 589.76 last 205.28) total: 50000 open/close in 598.69 seconds: 83.52 ops/second - unlinked 0 (time 1789711140 ; total 0 ; last 0) - unlinked 10000 (time 1789711157 ; total 17 ; last 17) - unlinked 20000 (time 1789711175 ; total 35 ; last 18) - unlinked 30000 (time 1789711192 ; total 52 ; last 17) - unlinked 40000 (time 1789711215 ; total 75 ; last 23) total: 50000 unlinks in 100 seconds: 500.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server stop mds service on oleg258-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server unloading modules via unload_modules_local on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing unload_modules_local oleg258-server: kmemleak disabled modules unloaded. pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2