== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:11:36 (1789780296) start mds service on oleg316-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 18 15:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: kmemleak disabled oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: MODOPTS_LIBCFS= oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg316-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg316-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg316-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg316-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg316-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1789780420.83 total 10.81 last 22.20) - open/close 476 (time 1789780431.03 total 21.01 last 23.14) - open/close 705 (time 1789780441.31 total 31.29 last 22.27) - open/close 917 (time 1789780451.44 total 41.42 last 20.93) - open/close 1180 (time 1789780461.46 total 51.43 last 26.26) - open/close 1480 (time 1789780471.46 total 61.44 last 29.98) - open/close 1712 (time 1789780482.03 total 72.00 last 21.97) - open/close 1982 (time 1789780492.04 total 82.02 last 26.95) - open/close 2194 (time 1789780502.05 total 92.02 last 21.19) - open/close 2421 (time 1789780512.06 total 102.04 last 22.67) - open/close 2744 (time 1789780523.10 total 113.08 last 29.26) - open/close 3068 (time 1789780534.05 total 124.03 last 29.59) - open/close 3317 (time 1789780544.62 total 134.60 last 23.55) - open/close 3622 (time 1789780555.57 total 145.55 last 27.86) - open/close 3797 (time 1789780566.07 total 156.04 last 16.68) - open/close 3895 (time 1789780576.50 total 166.47 last 9.40) - open/close 4001 (time 1789780586.50 total 176.48 last 10.59) - open/close 4234 (time 1789780596.51 total 186.49 last 23.28) - open/close 4561 (time 1789780606.53 total 196.50 last 32.66) - open/close 4836 (time 1789780617.20 total 207.17 last 25.77) - open/close 5048 (time 1789780628.27 total 218.25 last 19.14) - open/close 5285 (time 1789780638.38 total 228.35 last 23.46) - open/close 5486 (time 1789780648.39 total 238.36 last 20.09) - open/close 5686 (time 1789780658.39 total 248.36 last 19.99) - open/close 5833 (time 1789780668.39 total 258.37 last 14.70) - open/close 6022 (time 1789780678.40 total 268.37 last 18.89) - open/close 6184 (time 1789780690.11 total 280.09 last 13.83) - open/close 6328 (time 1789780700.99 total 290.96 last 13.24) - open/close 6583 (time 1789780710.99 total 300.96 last 25.50) - open/close 6674 (time 1789780720.99 total 310.96 last 9.10) - open/close 6908 (time 1789780731.69 total 321.67 last 21.87) - open/close 7073 (time 1789780742.06 total 332.03 last 15.92) - open/close 7286 (time 1789780752.22 total 342.20 last 20.96) - open/close 7592 (time 1789780762.22 total 352.20 last 30.59) - open/close 7826 (time 1789780772.24 total 362.22 last 23.36) - open/close 7936 (time 1789780782.49 total 372.47 last 10.73) - open/close 8145 (time 1789780792.82 total 382.80 last 20.24) - open/close 8224 (time 1789780802.86 total 392.84 last 7.87) - open/close 8340 (time 1789780813.66 total 403.63 last 10.74) - open/close 8572 (time 1789780823.66 total 413.64 last 23.19) - open/close 8723 (time 1789780834.53 total 424.51 last 13.89) - open/close 8993 (time 1789780844.54 total 434.51 last 26.98) - open/close 9160 (time 1789780855.21 total 445.19 last 15.65) - open/close 9434 (time 1789780865.22 total 455.19 last 27.38) - open/close 9706 (time 1789780875.22 total 465.19 last 27.19) - open/close 9908 (time 1789780886.06 total 476.04 last 18.63) - open/close 10000 (time 1789780889.70 total 479.68 last 25.27) - open/close 10234 (time 1789780899.71 total 489.69 last 23.38) - open/close 10501 (time 1789780910.20 total 500.18 last 25.45) - open/close 10739 (time 1789780920.21 total 510.18 last 23.79) - open/close 10905 (time 1789780930.24 total 520.22 last 16.55) - open/close 11126 (time 1789780940.96 total 530.93 last 20.63) - open/close 11404 (time 1789780951.39 total 541.37 last 26.64) - open/close 11505 (time 1789780968.38 total 558.35 last 5.95) - open/close 11723 (time 1789780978.62 total 568.60 last 21.28) - open/close 11871 (time 1789780989.69 total 579.67 last 13.37) - open/close 12134 (time 1789780999.86 total 589.84 last 25.86) - open/close 12367 (time 1789781010.14 total 600.12 last 22.67) - open/close 12641 (time 1789781020.15 total 610.13 last 27.37) - open/close 12868 (time 1789781030.94 total 620.92 last 21.04) - open/close 13066 (time 1789781041.07 total 631.05 last 19.55) - open/close 13293 (time 1789781051.71 total 641.68 last 21.34) - open/close 13444 (time 1789781061.86 total 651.84 last 14.87) - open/close 13709 (time 1789781072.44 total 662.41 last 25.06) - open/close 13966 (time 1789781082.44 total 672.41 last 25.69) - open/close 14197 (time 1789781092.77 total 682.74 last 22.37) - open/close 14463 (time 1789781103.53 total 693.51 last 24.71) - open/close 14634 (time 1789781113.91 total 703.88 last 16.48) - open/close 14768 (time 1789781125.36 total 715.33 last 11.70) - open/close 15055 (time 1789781136.36 total 726.34 last 26.08) - open/close 15280 (time 1789781146.47 total 736.44 last 22.27) - open/close 15490 (time 1789781156.47 total 746.44 last 21.00) - open/close 15630 (time 1789781166.47 total 756.44 last 14.00) - open/close 15793 (time 1789781176.61 total 766.58 last 16.08) - open/close 16030 (time 1789781186.63 total 776.60 last 23.65) - open/close 16257 (time 1789781196.64 total 786.61 last 22.67) - open/close 16512 (time 1789781207.03 total 797.01 last 24.53) - open/close 16761 (time 1789781218.08 total 808.06 last 22.53) - open/close 16981 (time 1789781228.09 total 818.06 last 21.99) - open/close 17268 (time 1789781238.10 total 828.07 last 28.68) - open/close 17423 (time 1789781248.37 total 838.35 last 15.09) - open/close 17624 (time 1789781258.40 total 848.37 last 20.05) - open/close 17898 (time 1789781268.40 total 858.37 last 27.39) - open/close 18158 (time 1789781278.40 total 868.38 last 25.99) - open/close 18361 (time 1789781288.41 total 878.38 last 20.28) - open/close 18529 (time 1789781299.31 total 889.29 last 15.41) - open/close 18650 (time 1789781309.48 total 899.46 last 11.90) - open/close 18653 (time 1789781319.73 total 909.71 last 0.29) - open/close 18796 (time 1789781330.74 total 920.71 last 12.99) - open/close 19058 (time 1789781340.84 total 930.81 last 25.94) - open/close 19317 (time 1789781351.85 total 941.83 last 23.52) - open/close 19513 (time 1789781361.86 total 951.83 last 19.59) - open/close 20000 (time 1789781370.61 total 960.58 last 55.68) - open/close 20673 (time 1789781380.62 total 970.59 last 67.21) - open/close 21414 (time 1789781390.62 total 980.60 last 74.05) - open/close 22136 (time 1789781400.63 total 990.60 last 72.19) - open/close 22916 (time 1789781410.63 total 1000.60 last 77.98) - open/close 23651 (time 1789781420.64 total 1010.62 last 73.40) - open/close 24395 (time 1789781430.67 total 1020.64 last 74.20) - open/close 25117 (time 1789781440.68 total 1030.65 last 72.15) - open/close 25883 (time 1789781450.68 total 1040.65 last 76.60) - open/close 26624 (time 1789781460.68 total 1050.66 last 74.04) - open/close 27445 (time 1789781470.69 total 1060.66 last 82.06) - open/close 28212 (time 1789781480.69 total 1070.67 last 76.67) - open/close 28928 (time 1789781490.70 total 1080.67 last 71.59) - open/close 29633 (time 1789781500.70 total 1090.68 last 70.44) - open/close 30000 (time 1789781506.42 total 1096.40 last 64.18) - open/close 30623 (time 1789781516.43 total 1106.40 last 62.27) - open/close 31117 (time 1789781526.43 total 1116.41 last 49.38) - open/close 31766 (time 1789781536.43 total 1126.41 last 64.90) - open/close 32347 (time 1789781546.44 total 1136.42 last 58.02) - open/close 32954 (time 1789781556.46 total 1146.43 last 60.62) - open/close 33586 (time 1789781566.46 total 1156.44 last 63.17) - open/close 34112 (time 1789781576.48 total 1166.45 last 52.52) - open/close 34799 (time 1789781586.48 total 1176.46 last 68.66) - open/close 35432 (time 1789781596.48 total 1186.46 last 63.29) - open/close 36150 (time 1789781606.49 total 1196.46 last 71.77) - open/close 36903 (time 1789781616.50 total 1206.48 last 75.20) - open/close 37667 (time 1789781626.51 total 1216.49 last 76.30) - open/close 38317 (time 1789781636.53 total 1226.50 last 64.93) - open/close 39053 (time 1789781646.53 total 1236.51 last 73.54) - open/close 39765 (time 1789781656.54 total 1246.51 last 71.17) - open/close 40000 (time 1789781659.33 total 1249.30 last 84.22) - open/close 40726 (time 1789781669.33 total 1259.31 last 72.57) - open/close 41425 (time 1789781679.34 total 1269.32 last 69.84) - open/close 42163 (time 1789781689.35 total 1279.32 last 73.74) - open/close 42912 (time 1789781699.35 total 1289.33 last 74.89) - open/close 43675 (time 1789781709.37 total 1299.35 last 76.15) - open/close 44484 (time 1789781719.38 total 1309.36 last 80.82) - open/close 45198 (time 1789781729.38 total 1319.36 last 71.40) - open/close 45928 (time 1789781739.38 total 1329.36 last 72.99) - open/close 46724 (time 1789781749.39 total 1339.36 last 79.53) - open/close 47473 (time 1789781759.39 total 1349.37 last 74.88) - open/close 48252 (time 1789781769.40 total 1359.38 last 77.84) - open/close 49020 (time 1789781779.40 total 1369.38 last 76.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1377.61 seconds: 36.03 ops/second - unlinked 0 (time 1789781791 ; total 0 ; last 0) - unlinked 10000 (time 1789781927 ; total 136 ; last 136) - unlinked 20000 (time 1789782064 ; total 273 ; last 137) - unlinked 30000 (time 1789782211 ; total 420 ; last 147) - unlinked 40000 (time 1789782355 ; total 564 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 708 seconds: 70.100281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local oleg316-server: kmemleak disabled modules unloaded. pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2