== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:15:06 (1790784906) start mds service on oleg141-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Sep 30 01:47 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: kmemleak disabled oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg141-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg141-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg141-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 72 (time 1790785025.39 total 10.42 last 6.91) - open/close 324 (time 1790785037.04 total 22.07 last 21.63) - open/close 564 (time 1790785047.28 total 32.31 last 23.44) - open/close 653 (time 1790785057.40 total 42.43 last 8.79) - open/close 904 (time 1790785068.07 total 53.10 last 23.53) - open/close 1183 (time 1790785078.15 total 63.17 last 27.69) - open/close 1437 (time 1790785088.15 total 73.18 last 25.39) - open/close 1643 (time 1790785100.20 total 85.23 last 17.09) - open/close 1937 (time 1790785110.22 total 95.25 last 29.35) - open/close 2113 (time 1790785120.69 total 105.72 last 16.81) - open/close 2399 (time 1790785130.70 total 115.72 last 28.59) - open/close 2584 (time 1790785141.30 total 126.32 last 17.46) - open/close 2740 (time 1790785151.71 total 136.74 last 14.98) - open/close 3062 (time 1790785162.42 total 147.45 last 30.05) - open/close 3342 (time 1790785173.48 total 158.50 last 25.34) - open/close 3640 (time 1790785183.49 total 168.52 last 29.75) - open/close 3907 (time 1790785195.23 total 180.26 last 22.74) - open/close 4169 (time 1790785205.61 total 190.64 last 25.24) - open/close 4403 (time 1790785216.04 total 201.06 last 22.45) - open/close 4617 (time 1790785226.17 total 211.20 last 21.11) - open/close 4782 (time 1790785236.46 total 221.48 last 16.05) - open/close 4894 (time 1790785246.47 total 231.50 last 11.18) - open/close 5040 (time 1790785256.48 total 241.51 last 14.59) - open/close 5251 (time 1790785268.32 total 253.35 last 17.82) - open/close 5362 (time 1790785280.69 total 265.72 last 8.97) - open/close 5551 (time 1790785291.34 total 276.36 last 17.76) - open/close 5728 (time 1790785301.35 total 286.38 last 17.67) - open/close 5919 (time 1790785312.24 total 297.27 last 17.54) - open/close 6119 (time 1790785322.31 total 307.34 last 19.85) - open/close 6318 (time 1790785334.52 total 319.55 last 16.30) - open/close 6521 (time 1790785344.53 total 329.56 last 20.27) - open/close 6792 (time 1790785354.54 total 339.56 last 27.09) - open/close 6913 (time 1790785365.10 total 350.13 last 11.45) - open/close 7170 (time 1790785376.11 total 361.14 last 23.34) - open/close 7493 (time 1790785386.73 total 371.75 last 30.43) - open/close 7715 (time 1790785396.75 total 381.78 last 22.14) - open/close 7950 (time 1790785406.77 total 391.79 last 23.47) - open/close 8228 (time 1790785417.36 total 402.39 last 26.23) - open/close 8448 (time 1790785427.76 total 412.78 last 21.17) - open/close 8689 (time 1790785438.08 total 423.11 last 23.34) - open/close 8796 (time 1790785448.35 total 433.38 last 10.42) - open/close 8999 (time 1790785458.37 total 443.40 last 20.26) - open/close 9153 (time 1790785469.11 total 454.14 last 14.34) - open/close 9404 (time 1790785479.12 total 464.14 last 25.08) - open/close 9545 (time 1790785489.38 total 474.41 last 13.74) - open/close 9659 (time 1790785501.35 total 486.38 last 9.52) - open/close 9853 (time 1790785512.37 total 497.40 last 17.61) - open/close 10000 (time 1790785515.89 total 500.92 last 41.70) - open/close 10057 (time 1790785526.31 total 511.34 last 5.47) - open/close 10291 (time 1790785536.32 total 521.34 last 23.39) - open/close 10489 (time 1790785546.33 total 531.35 last 19.78) - open/close 10713 (time 1790785556.36 total 541.38 last 22.34) - open/close 10969 (time 1790785566.36 total 551.39 last 25.58) - open/close 11073 (time 1790785576.37 total 561.39 last 10.40) - open/close 11233 (time 1790785586.37 total 571.39 last 16.00) - open/close 11489 (time 1790785596.38 total 581.40 last 25.57) - open/close 11601 (time 1790785606.70 total 591.73 last 10.85) - open/close 11693 (time 1790785617.66 total 602.68 last 8.40) - open/close 11827 (time 1790785627.91 total 612.94 last 13.07) - open/close 12066 (time 1790785637.95 total 622.98 last 23.80) - open/close 12346 (time 1790785647.96 total 632.98 last 27.98) - open/close 12533 (time 1790785657.97 total 643.00 last 18.66) - open/close 12668 (time 1790785668.45 total 653.48 last 12.89) - open/close 12825 (time 1790785679.10 total 664.13 last 14.74) - open/close 12993 (time 1790785691.56 total 676.59 last 13.48) - open/close 13244 (time 1790785702.50 total 687.52 last 22.95) - open/close 13470 (time 1790785712.76 total 697.79 last 22.02) - open/close 13587 (time 1790785724.31 total 709.33 last 10.13) - open/close 13739 (time 1790785734.31 total 719.34 last 15.20) - open/close 13897 (time 1790785745.40 total 730.43 last 14.24) - open/close 14121 (time 1790785756.28 total 741.30 last 20.60) - open/close 14392 (time 1790785766.99 total 752.01 last 25.31) - open/close 14516 (time 1790785777.91 total 762.93 last 11.36) - open/close 14707 (time 1790785787.95 total 772.98 last 19.01) - open/close 14889 (time 1790785798.95 total 783.98 last 16.54) - open/close 15077 (time 1790785809.93 total 794.96 last 17.12) - open/close 15320 (time 1790785822.76 total 807.78 last 18.95) - open/close 15568 (time 1790785833.80 total 818.83 last 22.45) - open/close 15808 (time 1790785846.73 total 831.75 last 18.57) - open/close 15945 (time 1790785856.73 total 841.76 last 13.70) - open/close 16055 (time 1790785866.74 total 851.77 last 10.99) - open/close 16301 (time 1790785876.75 total 861.78 last 24.57) - open/close 16427 (time 1790785890.85 total 875.87 last 8.94) - open/close 16610 (time 1790785900.85 total 885.88 last 18.29) - open/close 16882 (time 1790785911.33 total 896.36 last 25.95) - open/close 17094 (time 1790785921.58 total 906.60 last 20.69) - open/close 17364 (time 1790785932.45 total 917.47 last 24.84) - open/close 17566 (time 1790785943.22 total 928.24 last 18.76) - open/close 17757 (time 1790785953.25 total 938.27 last 19.04) - open/close 18024 (time 1790785963.27 total 948.30 last 26.64) - open/close 18275 (time 1790785973.35 total 958.37 last 24.91) - open/close 18446 (time 1790785983.71 total 968.74 last 16.50) - open/close 18575 (time 1790785993.99 total 979.02 last 12.55) - open/close 18717 (time 1790786006.45 total 991.48 last 11.40) - open/close 18809 (time 1790786016.70 total 1001.73 last 8.97) - open/close 18993 (time 1790786026.71 total 1011.74 last 18.38) - open/close 19706 (time 1790786036.72 total 1021.75 last 71.28) - open/close 20000 (time 1790786040.21 total 1025.23 last 84.28) - open/close 20687 (time 1790786050.21 total 1035.24 last 68.67) - open/close 21438 (time 1790786060.22 total 1045.25 last 75.03) - open/close 22150 (time 1790786070.23 total 1055.25 last 71.15) - open/close 22700 (time 1790786080.23 total 1065.25 last 55.00) - open/close 23257 (time 1790786090.23 total 1075.26 last 55.67) - open/close 23929 (time 1790786100.23 total 1085.26 last 67.20) - open/close 24507 (time 1790786110.26 total 1095.29 last 57.65) - open/close 25157 (time 1790786120.29 total 1105.31 last 64.83) - open/close 25924 (time 1790786130.29 total 1115.31 last 76.70) - open/close 26545 (time 1790786140.29 total 1125.32 last 62.07) - open/close 27281 (time 1790786150.29 total 1135.32 last 73.59) - open/close 28161 (time 1790786160.30 total 1145.33 last 87.96) - open/close 28916 (time 1790786170.31 total 1155.33 last 75.44) - open/close 29635 (time 1790786180.31 total 1165.33 last 71.89) - open/close 30000 (time 1790786185.30 total 1170.33 last 73.04) - open/close 30858 (time 1790786195.30 total 1180.33 last 85.80) - open/close 31593 (time 1790786205.31 total 1190.34 last 73.43) - open/close 32259 (time 1790786215.32 total 1200.35 last 66.57) - open/close 33053 (time 1790786225.32 total 1210.35 last 79.35) - open/close 33854 (time 1790786235.33 total 1220.36 last 80.06) - open/close 34579 (time 1790786245.33 total 1230.36 last 72.49) - open/close 35392 (time 1790786255.33 total 1240.36 last 81.28) - open/close 36136 (time 1790786265.35 total 1250.37 last 74.31) - open/close 36696 (time 1790786275.35 total 1260.37 last 55.99) - open/close 37484 (time 1790786285.35 total 1270.38 last 78.77) - open/close 38272 (time 1790786295.35 total 1280.38 last 78.77) - open/close 39094 (time 1790786305.36 total 1290.39 last 82.15) - open/close 39928 (time 1790786315.37 total 1300.40 last 83.33) - open/close 40662 (time 1790786325.37 total 1310.40 last 73.40) - open/close 41412 (time 1790786335.37 total 1320.40 last 74.98) - open/close 42099 (time 1790786345.38 total 1330.41 last 68.65) - open/close 42980 (time 1790786355.38 total 1340.41 last 88.08) - open/close 43831 (time 1790786365.40 total 1350.42 last 84.99) - open/close 44637 (time 1790786375.42 total 1360.45 last 80.41) - open/close 45528 (time 1790786385.43 total 1370.45 last 89.04) - open/close 46325 (time 1790786395.43 total 1380.46 last 79.66) - open/close 47202 (time 1790786405.44 total 1390.46 last 87.65) - open/close 48048 (time 1790786415.44 total 1400.47 last 84.57) - open/close 48843 (time 1790786425.45 total 1410.47 last 79.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1420.38 seconds: 34.94 ops/second - unlinked 0 (time 1790786439 ; total 0 ; last 0) - unlinked 10000 (time 1790786563 ; total 124 ; last 124) - unlinked 20000 (time 1790786685 ; total 246 ; last 122) - unlinked 30000 (time 1790786807 ; total 368 ; last 122) - unlinked 40000 (time 1790786920 ; total 481 ; last 113) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 556 seconds: 89.264389 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local oleg141-server: kmemleak disabled modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2