== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:20:15 (1789449615) start mds service on oleg316-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1933448 Sep 14 13:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: kmemleak disabled oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: MODOPTS_LIBCFS= oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg316-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg316-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg316-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg316-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg316-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 217 (time 1789449721.93 total 10.02 last 21.65) - open/close 343 (time 1789449732.65 total 20.74 last 11.76) - open/close 544 (time 1789449742.67 total 30.76 last 20.06) - open/close 820 (time 1789449753.29 total 41.38 last 25.98) - open/close 1073 (time 1789449763.99 total 52.09 last 23.64) - open/close 1245 (time 1789449774.61 total 62.70 last 16.20) - open/close 1370 (time 1789449784.62 total 72.71 last 12.49) - open/close 1563 (time 1789449795.16 total 83.25 last 18.30) - open/close 1724 (time 1789449805.52 total 93.61 last 15.54) - open/close 1949 (time 1789449815.53 total 103.62 last 22.48) - open/close 2099 (time 1789449825.55 total 113.64 last 14.97) - open/close 2274 (time 1789449835.57 total 123.67 last 17.46) - open/close 2417 (time 1789449846.24 total 134.34 last 13.40) - open/close 2634 (time 1789449856.25 total 144.34 last 21.69) - open/close 2896 (time 1789449866.58 total 154.67 last 25.37) - open/close 3208 (time 1789449876.58 total 164.67 last 31.19) - open/close 3415 (time 1789449887.00 total 175.10 last 19.86) - open/close 3773 (time 1789449897.56 total 185.66 last 33.90) - open/close 4032 (time 1789449908.57 total 196.66 last 23.53) - open/close 4265 (time 1789449918.58 total 206.67 last 23.28) - open/close 4500 (time 1789449928.78 total 216.88 last 23.03) - open/close 4752 (time 1789449938.83 total 226.92 last 25.08) - open/close 4920 (time 1789449949.78 total 237.87 last 15.35) - open/close 5169 (time 1789449959.78 total 247.88 last 24.88) - open/close 5402 (time 1789449970.39 total 258.48 last 21.98) - open/close 5667 (time 1789449980.49 total 268.58 last 26.23) - open/close 5932 (time 1789449990.49 total 278.59 last 26.49) - open/close 6092 (time 1789450000.51 total 288.61 last 15.97) - open/close 6258 (time 1789450010.54 total 298.64 last 16.56) - open/close 6425 (time 1789450021.02 total 309.11 last 15.94) - open/close 6552 (time 1789450031.70 total 319.79 last 11.90) - open/close 6768 (time 1789450042.77 total 330.86 last 19.50) - open/close 7018 (time 1789450053.14 total 341.24 last 24.10) - open/close 7326 (time 1789450063.15 total 351.24 last 30.79) - open/close 7577 (time 1789450073.16 total 361.26 last 25.06) - open/close 7831 (time 1789450083.23 total 371.33 last 25.23) - open/close 8090 (time 1789450093.71 total 381.80 last 24.72) - open/close 8367 (time 1789450104.15 total 392.24 last 26.54) - open/close 8637 (time 1789450114.15 total 402.25 last 26.99) - open/close 8853 (time 1789450125.05 total 413.14 last 19.83) - open/close 9105 (time 1789450135.18 total 423.27 last 24.87) - open/close 9317 (time 1789450145.74 total 433.84 last 20.07) - open/close 9590 (time 1789450156.12 total 444.21 last 26.32) - open/close 9893 (time 1789450166.94 total 455.03 last 28.00) - open/close 10000 (time 1789450170.74 total 458.83 last 28.17) - open/close 10173 (time 1789450180.82 total 468.91 last 17.17) - open/close 10415 (time 1789450190.87 total 478.96 last 24.08) - open/close 10679 (time 1789450200.88 total 488.97 last 26.38) - open/close 10919 (time 1789450211.61 total 499.70 last 22.36) - open/close 11185 (time 1789450221.77 total 509.87 last 26.17) - open/close 11371 (time 1789450232.28 total 520.37 last 17.71) - open/close 11642 (time 1789450243.02 total 531.11 last 25.23) - open/close 11926 (time 1789450253.20 total 541.30 last 27.89) - open/close 12225 (time 1789450263.22 total 551.31 last 29.85) - open/close 12468 (time 1789450273.23 total 561.32 last 24.28) - open/close 12644 (time 1789450283.48 total 571.58 last 17.17) - open/close 12880 (time 1789450293.58 total 581.67 last 23.38) - open/close 13142 (time 1789450303.64 total 591.74 last 26.02) - open/close 13337 (time 1789450313.88 total 601.97 last 19.05) - open/close 13563 (time 1789450323.91 total 612.00 last 22.53) - open/close 13738 (time 1789450333.93 total 622.02 last 17.47) - open/close 13979 (time 1789450343.94 total 632.04 last 24.07) - open/close 14225 (time 1789450354.64 total 642.74 last 22.99) - open/close 14513 (time 1789450365.71 total 653.81 last 26.02) - open/close 14748 (time 1789450376.66 total 664.75 last 21.47) - open/close 14997 (time 1789450386.78 total 674.87 last 24.61) - open/close 15236 (time 1789450396.78 total 684.87 last 23.90) - open/close 15473 (time 1789450407.18 total 695.27 last 22.79) - open/close 15755 (time 1789450417.18 total 705.28 last 28.18) - open/close 15972 (time 1789450427.54 total 715.63 last 20.95) - open/close 16240 (time 1789450438.49 total 726.58 last 24.48) - open/close 16509 (time 1789450449.44 total 737.53 last 24.56) - open/close 16782 (time 1789450459.45 total 747.55 last 27.26) - open/close 16983 (time 1789450469.66 total 757.76 last 19.68) - open/close 17173 (time 1789450480.62 total 768.72 last 17.34) - open/close 17422 (time 1789450490.62 total 778.72 last 24.89) - open/close 17669 (time 1789450501.59 total 789.68 last 22.53) - open/close 17865 (time 1789450512.39 total 800.49 last 18.14) - open/close 18077 (time 1789450523.06 total 811.15 last 19.88) - open/close 18184 (time 1789450533.41 total 821.51 last 10.33) - open/close 18335 (time 1789450543.42 total 831.51 last 15.09) - open/close 18624 (time 1789450553.60 total 841.69 last 28.39) - open/close 18951 (time 1789450563.66 total 851.76 last 32.49) - open/close 19240 (time 1789450574.62 total 862.71 last 26.37) - open/close 19505 (time 1789450585.43 total 873.53 last 24.51) - open/close 19753 (time 1789450596.33 total 884.42 last 22.77) - open/close 20000 (time 1789450599.09 total 887.19 last 89.29) - open/close 20957 (time 1789450609.10 total 897.20 last 95.60) - open/close 21978 (time 1789450619.11 total 907.21 last 101.99) - open/close 22919 (time 1789450629.12 total 917.21 last 94.05) - open/close 23695 (time 1789450639.13 total 927.23 last 77.48) - open/close 24589 (time 1789450649.14 total 937.24 last 89.32) - open/close 25429 (time 1789450659.14 total 947.24 last 83.99) - open/close 26178 (time 1789450669.16 total 957.25 last 74.81) - open/close 26807 (time 1789450679.16 total 967.25 last 62.90) - open/close 27532 (time 1789450689.16 total 977.25 last 72.48) - open/close 28377 (time 1789450699.17 total 987.27 last 84.40) - open/close 29352 (time 1789450709.17 total 997.27 last 97.47) - open/close 30000 (time 1789450715.97 total 1004.06 last 95.42) - open/close 30985 (time 1789450725.97 total 1014.06 last 98.49) - open/close 31917 (time 1789450735.98 total 1024.07 last 93.12) - open/close 32765 (time 1789450745.98 total 1034.08 last 84.73) - open/close 33560 (time 1789450756.00 total 1044.09 last 79.37) - open/close 34406 (time 1789450766.01 total 1054.10 last 84.53) - open/close 35294 (time 1789450776.02 total 1064.11 last 88.74) - open/close 35941 (time 1789450786.02 total 1074.11 last 64.69) - open/close 36839 (time 1789450796.04 total 1084.13 last 89.63) - open/close 37632 (time 1789450806.05 total 1094.14 last 79.21) - open/close 38529 (time 1789450816.05 total 1104.14 last 89.70) - open/close 39454 (time 1789450826.06 total 1114.16 last 92.37) - open/close 40000 (time 1789450831.94 total 1120.04 last 92.85) - open/close 40914 (time 1789450841.95 total 1130.05 last 91.32) - open/close 41867 (time 1789450851.96 total 1140.05 last 95.21) - open/close 42850 (time 1789450861.96 total 1150.06 last 98.28) - open/close 43786 (time 1789450871.97 total 1160.06 last 93.56) - open/close 44736 (time 1789450881.98 total 1170.07 last 94.93) - open/close 45700 (time 1789450891.98 total 1180.08 last 96.32) - open/close 46584 (time 1789450902.01 total 1190.10 last 88.19) - open/close 47538 (time 1789450912.02 total 1200.11 last 95.29) - open/close 48490 (time 1789450922.02 total 1210.11 last 95.19) - open/close 49366 (time 1789450932.02 total 1220.12 last 87.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1222.94 seconds: 40.58 ops/second - unlinked 0 (time 1789450939 ; total 0 ; last 0) - unlinked 10000 (time 1789451058 ; total 119 ; last 119) - unlinked 20000 (time 1789451173 ; total 234 ; last 115) - unlinked 30000 (time 1789451243 ; total 304 ; last 70) - unlinked 40000 (time 1789451319 ; total 380 ; last 76) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 441 seconds: 112.541946 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local oleg316-server: kmemleak disabled modules unloaded. pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2