== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:38:15 (1790393895) start mds service on oleg211-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1932936 Sep 25 22:08 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: kmemleak disabled oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg211-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg211-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 457 (time 1790393936.14 total 10.12 last 45.14) - open/close 1114 (time 1790393946.25 total 20.23 last 64.99) - open/close 1582 (time 1790393956.30 total 30.28 last 46.57) - open/close 1869 (time 1790393968.85 total 42.83 last 22.88) - open/close 2280 (time 1790393979.60 total 53.58 last 38.22) - open/close 2843 (time 1790393989.60 total 63.59 last 56.29) - open/close 3453 (time 1790393999.69 total 73.67 last 60.46) - open/close 4063 (time 1790394009.70 total 83.68 last 60.98) - open/close 4687 (time 1790394020.11 total 94.09 last 59.90) - open/close 5326 (time 1790394030.42 total 104.40 last 62.01) - open/close 5908 (time 1790394040.66 total 114.64 last 56.83) - open/close 6379 (time 1790394051.67 total 125.65 last 42.79) - open/close 6766 (time 1790394062.16 total 136.14 last 36.90) - open/close 7246 (time 1790394072.85 total 146.83 last 44.89) - open/close 7840 (time 1790394083.34 total 157.32 last 56.61) - open/close 8374 (time 1790394093.59 total 167.57 last 52.13) - open/close 8964 (time 1790394103.76 total 177.74 last 58.00) - open/close 9552 (time 1790394116.05 total 190.03 last 47.84) - open/close 10000 (time 1790394123.90 total 197.88 last 57.08) - open/close 10569 (time 1790394134.86 total 208.84 last 51.90) - open/close 11053 (time 1790394145.42 total 219.40 last 45.82) - open/close 11526 (time 1790394156.18 total 230.16 last 43.98) - open/close 11938 (time 1790394167.12 total 241.10 last 37.65) - open/close 12668 (time 1790394177.93 total 251.91 last 67.51) - open/close 13256 (time 1790394187.98 total 261.96 last 58.51) - open/close 13919 (time 1790394198.16 total 272.14 last 65.15) - open/close 14355 (time 1790394208.21 total 282.20 last 43.36) - open/close 15004 (time 1790394218.77 total 292.75 last 61.50) - open/close 15561 (time 1790394229.00 total 302.99 last 54.41) - open/close 16207 (time 1790394239.50 total 313.48 last 61.54) - open/close 16532 (time 1790394250.44 total 324.43 last 29.69) - open/close 19122 (time 1790394260.45 total 334.43 last 258.98) - open/close 20000 (time 1790394264.44 total 338.43 last 219.57) - open/close 22402 (time 1790394274.45 total 348.43 last 240.14) - open/close 24370 (time 1790394284.45 total 358.43 last 196.75) - open/close 26357 (time 1790394294.45 total 368.44 last 198.60) - open/close 28374 (time 1790394304.46 total 378.44 last 201.64) - open/close 30000 (time 1790394312.68 total 386.66 last 197.81) - open/close 32263 (time 1790394322.68 total 396.66 last 226.27) - open/close 34983 (time 1790394332.68 total 406.66 last 271.96) - open/close 38223 (time 1790394342.68 total 416.66 last 323.96) - open/close 40000 (time 1790394347.69 total 421.67 last 354.60) - open/close 43005 (time 1790394357.70 total 431.68 last 300.41) - open/close 46118 (time 1790394367.79 total 441.77 last 308.33) - open/close 48732 (time 1790394377.80 total 451.78 last 261.27) total: 50000 open/close in 456.84 seconds: 109.45 ops/second - unlinked 0 (time 1790394384 ; total 0 ; last 0) - unlinked 10000 (time 1790394405 ; total 21 ; last 21) - unlinked 20000 (time 1790394425 ; total 41 ; last 20) - unlinked 30000 (time 1790394447 ; total 63 ; last 22) - unlinked 40000 (time 1790394472 ; total 88 ; last 25) total: 50000 unlinks in 111 seconds: 450.450439 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local oleg211-server: kmemleak disabled modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2