== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:14:13 (1788617653) start mds service on oleg310-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 17:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: kmemleak disabled oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg310-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg310-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg310-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 277 (time 1788617769.94 total 11.06 last 25.04) - open/close 475 (time 1788617780.84 total 21.96 last 18.18) - open/close 751 (time 1788617791.52 total 32.64 last 25.84) - open/close 1056 (time 1788617802.01 total 43.13 last 29.05) - open/close 1316 (time 1788617812.02 total 53.14 last 25.99) - open/close 1555 (time 1788617822.89 total 64.01 last 21.98) - open/close 1742 (time 1788617833.64 total 74.76 last 17.39) - open/close 1885 (time 1788617844.18 total 85.30 last 13.57) - open/close 2092 (time 1788617854.19 total 95.31 last 20.68) - open/close 2280 (time 1788617865.74 total 106.86 last 16.27) - open/close 2453 (time 1788617875.75 total 116.87 last 17.29) - open/close 2681 (time 1788617885.75 total 126.87 last 22.80) - open/close 2844 (time 1788617895.89 total 137.01 last 16.08) - open/close 3099 (time 1788617905.90 total 147.02 last 25.48) - open/close 3351 (time 1788617916.63 total 157.75 last 23.48) - open/close 3622 (time 1788617926.64 total 167.76 last 27.07) - open/close 3863 (time 1788617936.93 total 178.05 last 23.42) - open/close 4105 (time 1788617947.11 total 188.23 last 23.76) - open/close 4369 (time 1788617957.12 total 198.24 last 26.39) - open/close 4574 (time 1788617967.13 total 208.25 last 20.47) - open/close 4768 (time 1788617977.16 total 218.28 last 19.35) - open/close 5027 (time 1788617987.17 total 228.29 last 25.88) - open/close 5238 (time 1788617997.18 total 238.30 last 21.08) - open/close 5522 (time 1788618007.25 total 248.37 last 28.21) - open/close 5765 (time 1788618017.87 total 258.99 last 22.87) - open/close 5992 (time 1788618028.23 total 269.35 last 21.91) - open/close 6196 (time 1788618038.24 total 279.36 last 20.38) - open/close 6448 (time 1788618049.06 total 290.18 last 23.30) - open/close 6676 (time 1788618059.28 total 300.40 last 22.29) - open/close 6912 (time 1788618069.73 total 310.85 last 22.60) - open/close 7136 (time 1788618080.02 total 321.14 last 21.77) - open/close 7361 (time 1788618090.26 total 331.38 last 21.97) - open/close 7573 (time 1788618100.28 total 341.40 last 21.16) - open/close 7783 (time 1788618110.37 total 351.49 last 20.81) - open/close 7986 (time 1788618120.59 total 361.71 last 19.87) - open/close 8166 (time 1788618130.59 total 371.71 last 17.99) - open/close 8328 (time 1788618140.61 total 381.73 last 16.17) - open/close 8449 (time 1788618151.52 total 392.64 last 11.09) - open/close 8652 (time 1788618162.53 total 403.65 last 18.43) - open/close 8851 (time 1788618172.71 total 413.83 last 19.56) - open/close 9111 (time 1788618183.44 total 424.56 last 24.24) - open/close 9317 (time 1788618194.38 total 435.50 last 18.82) - open/close 9514 (time 1788618204.40 total 445.52 last 19.66) - open/close 9774 (time 1788618215.12 total 456.24 last 24.26) - open/close 10000 (time 1788618222.68 total 463.80 last 29.90) - open/close 10244 (time 1788618232.69 total 473.81 last 24.38) - open/close 10508 (time 1788618242.70 total 483.82 last 26.38) - open/close 10717 (time 1788618252.77 total 493.89 last 20.74) - open/close 10918 (time 1788618263.82 total 504.94 last 18.20) - open/close 11156 (time 1788618274.26 total 515.38 last 22.80) - open/close 11294 (time 1788618284.27 total 525.39 last 13.78) - open/close 11480 (time 1788618294.27 total 535.39 last 18.60) - open/close 11720 (time 1788618304.93 total 546.05 last 22.53) - open/close 11966 (time 1788618315.92 total 557.04 last 22.37) - open/close 12243 (time 1788618325.93 total 567.05 last 27.68) - open/close 12491 (time 1788618335.94 total 577.06 last 24.78) - open/close 12678 (time 1788618345.97 total 587.09 last 18.65) - open/close 12951 (time 1788618355.98 total 597.10 last 27.27) - open/close 13209 (time 1788618365.98 total 607.10 last 25.78) - open/close 13417 (time 1788618375.99 total 617.11 last 20.78) - open/close 13622 (time 1788618386.01 total 627.13 last 20.46) - open/close 13870 (time 1788618396.02 total 637.14 last 24.79) - open/close 14106 (time 1788618406.10 total 647.22 last 23.40) - open/close 14355 (time 1788618416.34 total 657.46 last 24.32) - open/close 14617 (time 1788618426.71 total 667.83 last 25.27) - open/close 14868 (time 1788618436.73 total 677.85 last 25.03) - open/close 15122 (time 1788618446.88 total 688.00 last 25.03) - open/close 15352 (time 1788618457.17 total 698.29 last 22.36) - open/close 15631 (time 1788618467.17 total 708.29 last 27.90) - open/close 15872 (time 1788618477.47 total 718.59 last 23.39) - open/close 16046 (time 1788618488.34 total 729.46 last 16.01) - open/close 16190 (time 1788618498.72 total 739.84 last 13.88) - open/close 16354 (time 1788618509.27 total 750.39 last 15.55) - open/close 16560 (time 1788618519.29 total 760.41 last 20.56) - open/close 16759 (time 1788618531.99 total 773.11 last 15.67) - open/close 16926 (time 1788618542.50 total 783.62 last 15.89) - open/close 17039 (time 1788618553.68 total 794.80 last 10.10) - open/close 17207 (time 1788618565.40 total 806.52 last 14.34) - open/close 17409 (time 1788618575.41 total 816.53 last 20.18) - open/close 17584 (time 1788618586.21 total 827.33 last 16.21) - open/close 17775 (time 1788618597.13 total 838.25 last 17.49) - open/close 18083 (time 1788618607.13 total 848.25 last 30.77) - open/close 18281 (time 1788618617.14 total 858.26 last 19.80) - open/close 18420 (time 1788618627.31 total 868.43 last 13.67) - open/close 18666 (time 1788618637.32 total 878.44 last 24.56) - open/close 18818 (time 1788618647.57 total 888.69 last 14.82) - open/close 19075 (time 1788618658.25 total 899.37 last 24.06) - open/close 19285 (time 1788618669.24 total 910.36 last 19.12) - open/close 19586 (time 1788618680.20 total 921.32 last 27.45) - open/close 19797 (time 1788618691.09 total 932.21 last 19.37) - open/close 20000 (time 1788618699.22 total 940.34 last 24.98) - open/close 20178 (time 1788618709.78 total 950.90 last 16.85) - open/close 20483 (time 1788618719.78 total 960.90 last 30.50) - open/close 21243 (time 1788618729.79 total 970.91 last 75.93) - open/close 21997 (time 1788618739.80 total 980.92 last 75.33) - open/close 22712 (time 1788618749.80 total 990.92 last 71.49) - open/close 23468 (time 1788618759.81 total 1000.93 last 75.55) - open/close 24222 (time 1788618769.82 total 1010.94 last 75.32) - open/close 24974 (time 1788618779.82 total 1020.94 last 75.16) - open/close 25622 (time 1788618789.84 total 1030.96 last 64.72) - open/close 26380 (time 1788618799.85 total 1040.97 last 75.72) - open/close 27041 (time 1788618809.87 total 1050.99 last 65.94) - open/close 27668 (time 1788618819.87 total 1060.99 last 62.68) - open/close 28276 (time 1788618829.89 total 1071.01 last 60.70) - open/close 28915 (time 1788618839.90 total 1081.02 last 63.85) - open/close 29671 (time 1788618849.90 total 1091.02 last 75.56) - open/close 30000 (time 1788618854.26 total 1095.38 last 75.57) - open/close 30770 (time 1788618864.26 total 1105.38 last 76.96) - open/close 31595 (time 1788618874.27 total 1115.39 last 82.45) - open/close 32354 (time 1788618884.27 total 1125.39 last 75.86) - open/close 33064 (time 1788618894.30 total 1135.42 last 70.82) - open/close 33735 (time 1788618904.37 total 1145.49 last 66.65) - open/close 34416 (time 1788618914.38 total 1155.50 last 68.01) - open/close 34963 (time 1788618924.43 total 1165.55 last 54.46) - open/close 35557 (time 1788618934.43 total 1175.55 last 59.38) - open/close 36234 (time 1788618944.43 total 1185.55 last 67.68) - open/close 36894 (time 1788618954.45 total 1195.57 last 65.88) - open/close 37545 (time 1788618964.46 total 1205.58 last 65.06) - open/close 38278 (time 1788618974.46 total 1215.58 last 73.30) - open/close 39042 (time 1788618984.46 total 1225.58 last 76.37) - open/close 39733 (time 1788618994.47 total 1235.59 last 69.01) - open/close 40000 (time 1788618998.22 total 1239.34 last 71.32) - open/close 40710 (time 1788619008.22 total 1249.34 last 70.99) - open/close 41428 (time 1788619018.22 total 1259.34 last 71.75) - open/close 42135 (time 1788619028.24 total 1269.36 last 70.59) - open/close 42882 (time 1788619038.24 total 1279.36 last 74.69) - open/close 43635 (time 1788619048.24 total 1289.36 last 75.28) - open/close 44408 (time 1788619058.24 total 1299.36 last 77.30) - open/close 45142 (time 1788619068.25 total 1309.37 last 73.38) - open/close 45850 (time 1788619078.25 total 1319.37 last 70.75) - open/close 46613 (time 1788619088.26 total 1329.38 last 76.27) - open/close 47318 (time 1788619098.27 total 1339.39 last 70.41) - open/close 48076 (time 1788619108.28 total 1349.40 last 75.76) - open/close 48941 (time 1788619118.28 total 1359.40 last 86.48) - open/close 49606 (time 1788619128.28 total 1369.40 last 66.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1369.86 seconds: 36.23 ops/second - unlinked 0 (time 1788619133 ; total 0 ; last 0) - unlinked 10000 (time 1788619292 ; total 159 ; last 159) - unlinked 20000 (time 1788619438 ; total 305 ; last 146) - unlinked 30000 (time 1788619581 ; total 448 ; last 143) - unlinked 40000 (time 1788619727 ; total 594 ; last 146) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 727 seconds: 68.268227 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: kmemleak disabled modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2