== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:21:51 (1788474111) start mds service on oleg217-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921072 Sep 3 15:45 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: kmemleak disabled oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg217-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg217-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg217-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg217-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 92 (time 1788474252.11 total 10.56 last 8.71) - open/close 317 (time 1788474263.03 total 21.48 last 20.59) - open/close 373 (time 1788474273.04 total 31.49 last 5.60) - open/close 617 (time 1788474283.77 total 42.22 last 22.75) - open/close 784 (time 1788474294.28 total 52.73 last 15.88) - open/close 926 (time 1788474304.73 total 63.18 last 13.60) - open/close 1156 (time 1788474314.73 total 73.18 last 22.99) - open/close 1392 (time 1788474327.83 total 86.28 last 18.01) - open/close 1563 (time 1788474337.94 total 96.39 last 16.92) - open/close 1727 (time 1788474347.96 total 106.41 last 16.37) - open/close 1864 (time 1788474358.61 total 117.06 last 12.86) - open/close 2062 (time 1788474368.63 total 127.08 last 19.76) - open/close 2221 (time 1788474378.94 total 137.39 last 15.43) - open/close 2344 (time 1788474389.35 total 147.80 last 11.82) - open/close 2475 (time 1788474399.37 total 157.82 last 13.07) - open/close 2617 (time 1788474411.36 total 169.81 last 11.84) - open/close 2765 (time 1788474421.63 total 180.08 last 14.41) - open/close 2866 (time 1788474431.63 total 190.08 last 10.10) - open/close 3021 (time 1788474442.24 total 200.69 last 14.62) - open/close 3131 (time 1788474452.24 total 210.69 last 11.00) - open/close 3264 (time 1788474462.34 total 220.79 last 13.17) - open/close 3407 (time 1788474472.36 total 230.81 last 14.27) - open/close 3570 (time 1788474482.36 total 240.81 last 16.29) - open/close 3669 (time 1788474493.13 total 251.58 last 9.19) - open/close 3933 (time 1788474503.14 total 261.59 last 26.38) - open/close 4115 (time 1788474513.48 total 271.93 last 17.60) - open/close 4354 (time 1788474523.50 total 281.95 last 23.86) - open/close 4509 (time 1788474533.50 total 291.96 last 15.49) - open/close 4745 (time 1788474543.51 total 301.96 last 23.58) - open/close 5014 (time 1788474554.36 total 312.82 last 24.79) - open/close 5298 (time 1788474564.38 total 322.83 last 28.37) - open/close 5489 (time 1788474574.39 total 332.84 last 19.08) - open/close 5687 (time 1788474584.90 total 343.35 last 18.84) - open/close 5935 (time 1788474595.58 total 354.03 last 23.22) - open/close 6169 (time 1788474606.48 total 364.93 last 21.47) - open/close 6348 (time 1788474616.50 total 374.95 last 17.85) - open/close 6528 (time 1788474626.82 total 385.27 last 17.44) - open/close 6644 (time 1788474637.68 total 396.13 last 10.68) - open/close 6814 (time 1788474647.69 total 406.14 last 16.99) - open/close 7047 (time 1788474657.70 total 416.15 last 23.28) - open/close 7291 (time 1788474667.91 total 426.36 last 23.90) - open/close 7522 (time 1788474677.94 total 436.39 last 23.02) - open/close 7734 (time 1788474687.99 total 446.44 last 21.09) - open/close 7890 (time 1788474698.00 total 456.45 last 15.60) - open/close 8014 (time 1788474708.00 total 466.46 last 12.39) - open/close 8080 (time 1788474718.78 total 477.23 last 6.12) - open/close 8317 (time 1788474728.85 total 487.31 last 23.53) - open/close 8621 (time 1788474738.86 total 497.31 last 30.38) - open/close 8841 (time 1788474749.18 total 507.63 last 21.31) - open/close 9018 (time 1788474760.20 total 518.65 last 16.07) - open/close 9201 (time 1788474771.97 total 530.42 last 15.55) - open/close 9290 (time 1788474781.98 total 540.43 last 8.88) - open/close 9479 (time 1788474792.64 total 551.09 last 17.74) - open/close 9540 (time 1788474806.24 total 564.69 last 4.49) - open/close 9670 (time 1788474817.17 total 575.62 last 11.89) - open/close 9881 (time 1788474827.20 total 585.65 last 21.03) - open/close 10000 (time 1788474831.27 total 589.72 last 29.23) - open/close 10229 (time 1788474841.28 total 599.73 last 22.88) - open/close 10427 (time 1788474853.52 total 611.97 last 16.18) - open/close 10749 (time 1788474863.53 total 621.98 last 32.15) - open/close 10959 (time 1788474873.63 total 632.08 last 20.81) - open/close 11101 (time 1788474883.78 total 642.23 last 13.98) - open/close 11272 (time 1788474894.63 total 653.08 last 15.77) - open/close 11462 (time 1788474905.40 total 663.85 last 17.63) - open/close 11596 (time 1788474915.97 total 674.42 last 12.68) - open/close 11681 (time 1788474928.25 total 686.70 last 6.92) - open/close 11816 (time 1788474938.58 total 697.03 last 13.07) - open/close 11881 (time 1788474949.59 total 708.04 last 5.90) - open/close 12020 (time 1788474960.41 total 718.86 last 12.85) - open/close 12256 (time 1788474970.41 total 728.86 last 23.60) - open/close 12493 (time 1788474981.32 total 739.77 last 21.72) - open/close 12663 (time 1788474991.93 total 750.38 last 16.02) - open/close 12904 (time 1788475004.51 total 762.96 last 19.16) - open/close 13102 (time 1788475014.52 total 772.97 last 19.79) - open/close 13266 (time 1788475024.53 total 782.98 last 16.38) - open/close 13463 (time 1788475034.54 total 792.99 last 19.68) - open/close 13678 (time 1788475044.54 total 802.99 last 21.49) - open/close 13954 (time 1788475055.48 total 813.93 last 25.22) - open/close 14174 (time 1788475065.51 total 823.96 last 21.95) - open/close 14401 (time 1788475075.51 total 833.96 last 22.69) - open/close 14548 (time 1788475086.59 total 845.04 last 13.26) - open/close 14805 (time 1788475097.48 total 855.93 last 23.60) - open/close 15039 (time 1788475108.04 total 866.49 last 22.17) - open/close 15282 (time 1788475118.92 total 877.37 last 22.34) - open/close 15544 (time 1788475128.92 total 887.37 last 26.19) - open/close 15657 (time 1788475139.12 total 897.57 last 11.08) - open/close 15906 (time 1788475149.12 total 907.57 last 24.89) - open/close 16019 (time 1788475159.61 total 918.06 last 10.77) - open/close 16257 (time 1788475169.80 total 928.25 last 23.37) - open/close 16352 (time 1788475179.82 total 938.27 last 9.48) - open/close 16455 (time 1788475190.59 total 949.04 last 9.56) - open/close 16575 (time 1788475201.53 total 959.98 last 10.97) - open/close 16647 (time 1788475212.03 total 970.48 last 6.86) - open/close 16883 (time 1788475222.54 total 980.99 last 22.45) - open/close 17136 (time 1788475233.02 total 991.48 last 24.13) - open/close 17333 (time 1788475243.79 total 1002.24 last 18.30) - open/close 17433 (time 1788475253.82 total 1012.27 last 9.97) - open/close 17620 (time 1788475264.41 total 1022.86 last 17.66) - open/close 17787 (time 1788475275.01 total 1033.46 last 15.75) - open/close 18164 (time 1788475285.01 total 1043.46 last 37.69) - open/close 18836 (time 1788475295.03 total 1053.48 last 67.06) - open/close 19543 (time 1788475305.06 total 1063.51 last 70.52) - open/close 20000 (time 1788475312.10 total 1070.55 last 64.94) - open/close 20602 (time 1788475322.11 total 1080.56 last 60.12) - open/close 21263 (time 1788475332.11 total 1090.56 last 66.06) - open/close 21897 (time 1788475342.12 total 1100.57 last 63.38) - open/close 22537 (time 1788475352.13 total 1110.58 last 63.91) - open/close 23263 (time 1788475362.14 total 1120.59 last 72.51) - open/close 23811 (time 1788475372.15 total 1130.60 last 54.74) - open/close 24355 (time 1788475382.17 total 1140.62 last 54.32) - open/close 24919 (time 1788475392.17 total 1150.62 last 56.39) - open/close 25727 (time 1788475402.19 total 1160.64 last 80.65) - open/close 26537 (time 1788475412.19 total 1170.64 last 80.99) - open/close 27274 (time 1788475422.19 total 1180.64 last 73.68) - open/close 28124 (time 1788475432.21 total 1190.66 last 84.84) - open/close 28912 (time 1788475442.23 total 1200.68 last 78.65) - open/close 29751 (time 1788475452.25 total 1210.70 last 83.77) - open/close 30000 (time 1788475456.41 total 1214.86 last 59.81) - open/close 30726 (time 1788475466.42 total 1224.87 last 72.56) - open/close 31454 (time 1788475476.42 total 1234.87 last 72.74) - open/close 32175 (time 1788475486.44 total 1244.89 last 72.01) - open/close 32871 (time 1788475496.44 total 1254.89 last 69.57) - open/close 33677 (time 1788475506.45 total 1264.90 last 80.55) - open/close 34397 (time 1788475516.45 total 1274.90 last 71.97) - open/close 34973 (time 1788475526.45 total 1284.90 last 57.59) - open/close 35667 (time 1788475536.47 total 1294.92 last 69.31) - open/close 36451 (time 1788475546.47 total 1304.92 last 78.37) - open/close 37228 (time 1788475556.48 total 1314.93 last 77.62) - open/close 38012 (time 1788475566.50 total 1324.95 last 78.28) - open/close 38716 (time 1788475576.50 total 1334.95 last 70.36) - open/close 39369 (time 1788475586.50 total 1344.95 last 65.30) - open/close 40000 (time 1788475595.65 total 1354.10 last 68.97) - open/close 40692 (time 1788475605.68 total 1364.13 last 68.98) - open/close 41394 (time 1788475615.70 total 1374.15 last 70.11) - open/close 42141 (time 1788475625.70 total 1384.15 last 74.69) - open/close 42854 (time 1788475635.70 total 1394.15 last 71.27) - open/close 43574 (time 1788475645.73 total 1404.18 last 71.83) - open/close 44372 (time 1788475655.74 total 1414.19 last 79.72) - open/close 45131 (time 1788475665.75 total 1424.20 last 75.81) - open/close 45959 (time 1788475675.75 total 1434.20 last 82.77) - open/close 46658 (time 1788475685.75 total 1444.20 last 69.89) - open/close 47353 (time 1788475695.77 total 1454.22 last 69.41) - open/close 48149 (time 1788475705.78 total 1464.23 last 79.48) - open/close 48777 (time 1788475715.78 total 1474.23 last 62.79) - open/close 49414 (time 1788475725.79 total 1484.24 last 63.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1487.37 seconds: 33.37 ops/second - unlinked 0 (time 1788475734 ; total 0 ; last 0) - unlinked 10000 (time 1788475880 ; total 146 ; last 146) - unlinked 20000 (time 1788476025 ; total 291 ; last 145) - unlinked 30000 (time 1788476161 ; total 427 ; last 136) - unlinked 40000 (time 1788476298 ; total 564 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 682 seconds: 72.772728 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg217-server unloading modules via unload_modules_local on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local oleg217-server: kmemleak disabled modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2