== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:57:43 (1789268263) start mds service on oleg439-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:11 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: kmemleak disabled oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg439-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg439-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg439-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 140 (time 1789268383.51 total 10.00 last 14.00) - open/close 392 (time 1789268393.96 total 20.45 last 24.13) - open/close 639 (time 1789268403.97 total 30.46 last 24.67) - open/close 864 (time 1789268413.98 total 40.47 last 22.48) - open/close 1020 (time 1789268424.42 total 50.91 last 14.93) - open/close 1161 (time 1789268434.44 total 60.93 last 14.08) - open/close 1310 (time 1789268445.12 total 71.61 last 13.95) - open/close 1577 (time 1789268455.13 total 81.62 last 26.68) - open/close 1831 (time 1789268465.13 total 91.62 last 25.39) - open/close 1915 (time 1789268475.16 total 101.65 last 8.37) - open/close 2131 (time 1789268486.05 total 112.54 last 19.84) - open/close 2385 (time 1789268496.73 total 123.22 last 23.77) - open/close 2667 (time 1789268506.74 total 133.23 last 28.16) - open/close 2931 (time 1789268517.35 total 143.85 last 24.88) - open/close 3178 (time 1789268527.36 total 153.85 last 24.70) - open/close 3374 (time 1789268538.34 total 164.83 last 17.85) - open/close 3470 (time 1789268548.35 total 174.85 last 9.58) - open/close 3686 (time 1789268558.37 total 184.86 last 21.57) - open/close 3881 (time 1789268568.38 total 194.87 last 19.47) - open/close 4075 (time 1789268578.79 total 205.28 last 18.64) - open/close 4364 (time 1789268588.84 total 215.33 last 28.77) - open/close 4637 (time 1789268598.85 total 225.34 last 27.27) - open/close 4881 (time 1789268609.01 total 235.50 last 24.02) - open/close 5127 (time 1789268619.35 total 245.84 last 23.78) - open/close 5372 (time 1789268630.18 total 256.67 last 22.62) - open/close 5475 (time 1789268640.48 total 266.97 last 10.00) - open/close 5650 (time 1789268650.99 total 277.48 last 16.66) - open/close 5922 (time 1789268661.85 total 288.34 last 25.05) - open/close 6139 (time 1789268671.86 total 298.35 last 21.68) - open/close 6207 (time 1789268682.79 total 309.28 last 6.22) - open/close 6412 (time 1789268692.98 total 319.47 last 20.12) - open/close 6604 (time 1789268703.96 total 330.45 last 17.49) - open/close 6860 (time 1789268715.52 total 342.01 last 22.13) - open/close 6953 (time 1789268725.78 total 352.27 last 9.07) - open/close 7142 (time 1789268735.78 total 362.27 last 18.90) - open/close 7402 (time 1789268745.79 total 372.28 last 25.99) - open/close 7557 (time 1789268756.58 total 383.07 last 14.35) - open/close 7630 (time 1789268767.48 total 393.97 last 6.70) - open/close 7875 (time 1789268777.49 total 403.98 last 24.49) - open/close 8005 (time 1789268787.54 total 414.03 last 12.93) - open/close 8282 (time 1789268798.59 total 425.08 last 25.08) - open/close 8550 (time 1789268809.24 total 435.73 last 25.14) - open/close 8785 (time 1789268819.88 total 446.37 last 22.10) - open/close 9020 (time 1789268830.95 total 457.44 last 21.22) - open/close 9324 (time 1789268840.96 total 467.45 last 30.35) - open/close 9581 (time 1789268850.97 total 477.46 last 25.68) - open/close 9851 (time 1789268861.73 total 488.22 last 25.10) - open/close 10000 (time 1789268868.71 total 495.20 last 21.32) - open/close 10058 (time 1789268878.96 total 505.45 last 5.66) - open/close 10149 (time 1789268889.27 total 515.77 last 8.82) - open/close 10355 (time 1789268899.53 total 526.03 last 20.08) - open/close 10514 (time 1789268909.81 total 536.30 last 15.47) - open/close 10683 (time 1789268920.26 total 546.75 last 16.18) - open/close 10963 (time 1789268930.26 total 556.75 last 27.99) - open/close 11254 (time 1789268941.01 total 567.50 last 27.06) - open/close 11436 (time 1789268951.64 total 578.13 last 17.12) - open/close 11667 (time 1789268962.13 total 588.62 last 22.02) - open/close 11844 (time 1789268972.64 total 599.13 last 16.85) - open/close 11992 (time 1789268982.70 total 609.19 last 14.71) - open/close 12261 (time 1789268992.70 total 619.19 last 26.89) - open/close 12462 (time 1789269003.17 total 629.66 last 19.20) - open/close 12727 (time 1789269013.18 total 639.67 last 26.48) - open/close 12934 (time 1789269023.21 total 649.70 last 20.64) - open/close 13177 (time 1789269033.23 total 659.72 last 24.26) - open/close 13467 (time 1789269044.25 total 670.74 last 26.30) - open/close 13666 (time 1789269054.26 total 680.75 last 19.87) - open/close 13755 (time 1789269065.13 total 691.62 last 8.19) - open/close 13999 (time 1789269075.15 total 701.64 last 24.37) - open/close 14285 (time 1789269085.61 total 712.10 last 27.34) - open/close 14498 (time 1789269096.57 total 723.06 last 19.43) - open/close 14651 (time 1789269106.60 total 733.09 last 15.25) - open/close 14771 (time 1789269117.02 total 743.51 last 11.52) - open/close 14867 (time 1789269127.64 total 754.13 last 9.04) - open/close 15021 (time 1789269139.95 total 766.44 last 12.52) - open/close 15238 (time 1789269149.95 total 776.44 last 21.69) - open/close 15332 (time 1789269160.17 total 786.66 last 9.20) - open/close 15407 (time 1789269170.43 total 796.92 last 7.31) - open/close 15485 (time 1789269180.77 total 807.26 last 7.54) - open/close 15709 (time 1789269191.09 total 817.58 last 21.69) - open/close 15932 (time 1789269201.76 total 828.25 last 20.91) - open/close 16053 (time 1789269211.77 total 838.26 last 12.08) - open/close 16220 (time 1789269222.04 total 848.53 last 16.26) - open/close 16483 (time 1789269232.06 total 858.55 last 26.25) - open/close 16664 (time 1789269242.84 total 869.33 last 16.79) - open/close 16795 (time 1789269253.15 total 879.64 last 12.70) - open/close 16956 (time 1789269264.23 total 890.72 last 14.53) - open/close 17077 (time 1789269274.24 total 900.73 last 12.09) - open/close 17227 (time 1789269284.51 total 911.00 last 14.60) - open/close 17477 (time 1789269294.52 total 921.01 last 24.98) - open/close 17594 (time 1789269304.63 total 931.12 last 11.58) - open/close 17600 (time 1789269314.92 total 941.41 last 0.58) - open/close 17870 (time 1789269325.87 total 952.36 last 24.64) - open/close 18076 (time 1789269336.50 total 962.99 last 19.38) - open/close 18352 (time 1789269347.49 total 973.99 last 25.11) - open/close 18554 (time 1789269357.50 total 983.99 last 20.18) - open/close 18778 (time 1789269367.71 total 994.21 last 21.94) - open/close 19012 (time 1789269377.74 total 1004.23 last 23.33) - open/close 19204 (time 1789269387.75 total 1014.24 last 19.19) - open/close 19436 (time 1789269398.26 total 1024.75 last 22.08) - open/close 19713 (time 1789269408.26 total 1034.75 last 27.70) - open/close 20000 (time 1789269410.89 total 1037.38 last 109.11) - open/close 20735 (time 1789269420.89 total 1047.38 last 73.48) - open/close 21577 (time 1789269430.91 total 1057.40 last 84.01) - open/close 22432 (time 1789269440.92 total 1067.41 last 85.46) - open/close 23136 (time 1789269450.94 total 1077.43 last 70.27) - open/close 23689 (time 1789269460.94 total 1087.43 last 55.28) - open/close 24364 (time 1789269470.94 total 1097.43 last 67.49) - open/close 25275 (time 1789269480.95 total 1107.44 last 91.05) - open/close 26136 (time 1789269490.96 total 1117.45 last 86.01) - open/close 26919 (time 1789269500.96 total 1127.45 last 78.28) - open/close 27752 (time 1789269510.97 total 1137.46 last 83.25) - open/close 28526 (time 1789269520.97 total 1147.46 last 77.36) - open/close 29117 (time 1789269530.97 total 1157.46 last 59.09) - open/close 29910 (time 1789269540.98 total 1167.48 last 79.20) - open/close 30759 (time 1789269550.99 total 1177.48 last 84.86) - open/close 31606 (time 1789269561.00 total 1187.49 last 84.65) - open/close 32429 (time 1789269571.00 total 1197.49 last 82.25) - open/close 33308 (time 1789269581.00 total 1207.49 last 87.87) - open/close 34219 (time 1789269591.01 total 1217.50 last 91.05) - open/close 35054 (time 1789269601.03 total 1227.52 last 83.35) - open/close 35995 (time 1789269611.03 total 1237.52 last 94.09) - open/close 36941 (time 1789269621.05 total 1247.54 last 94.44) - open/close 37665 (time 1789269631.05 total 1257.54 last 72.35) - open/close 38552 (time 1789269641.06 total 1267.55 last 88.66) - open/close 39311 (time 1789269651.08 total 1277.57 last 75.75) - open/close 40000 (time 1789269658.92 total 1285.41 last 87.84) - open/close 40798 (time 1789269668.93 total 1295.42 last 79.73) - open/close 41642 (time 1789269678.94 total 1305.43 last 84.27) - open/close 42469 (time 1789269688.95 total 1315.44 last 82.68) - open/close 43314 (time 1789269698.95 total 1325.44 last 84.43) - open/close 44167 (time 1789269708.97 total 1335.46 last 85.19) - open/close 44924 (time 1789269718.97 total 1345.46 last 75.66) - open/close 45637 (time 1789269728.98 total 1355.47 last 71.24) - open/close 46348 (time 1789269738.99 total 1365.48 last 71.04) - open/close 47258 (time 1789269748.99 total 1375.48 last 90.98) - open/close 48041 (time 1789269758.99 total 1385.48 last 78.28) - open/close 48891 (time 1789269769.00 total 1395.49 last 84.96) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1404.89 seconds: 35.33 ops/second - unlinked 0 (time 1789269782 ; total 0 ; last 0) - unlinked 10000 (time 1789269908 ; total 126 ; last 126) - unlinked 20000 (time 1789270039 ; total 257 ; last 131) - unlinked 30000 (time 1789270164 ; total 382 ; last 125) - unlinked 40000 (time 1789270282 ; total 500 ; last 118) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 611 seconds: 81.229134 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server unloading modules via unload_modules_local on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local oleg439-server: kmemleak disabled modules unloaded. pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2