== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:45:27 (1788817527) start mds service on oleg455-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931936 Sep 7 16:07 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: kmemleak disabled oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg455-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg455-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg455-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg455-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 217 (time 1788817645.73 total 10.68 last 20.31) - open/close 470 (time 1788817655.73 total 20.69 last 25.29) - open/close 688 (time 1788817665.74 total 30.69 last 21.78) - open/close 850 (time 1788817675.75 total 40.70 last 16.20) - open/close 1087 (time 1788817688.23 total 53.18 last 18.99) - open/close 1297 (time 1788817698.28 total 63.23 last 20.89) - open/close 1494 (time 1788817709.15 total 74.11 last 18.12) - open/close 1690 (time 1788817719.17 total 84.12 last 19.57) - open/close 1930 (time 1788817730.02 total 94.97 last 22.12) - open/close 2163 (time 1788817740.36 total 105.31 last 22.53) - open/close 2314 (time 1788817750.37 total 115.32 last 15.08) - open/close 2435 (time 1788817761.10 total 126.05 last 11.28) - open/close 2688 (time 1788817771.11 total 136.06 last 25.27) - open/close 2718 (time 1788817781.23 total 146.18 last 2.97) - open/close 2774 (time 1788817791.80 total 156.75 last 5.30) - open/close 2888 (time 1788817804.21 total 169.16 last 9.18) - open/close 3151 (time 1788817815.08 total 180.03 last 24.19) - open/close 3378 (time 1788817825.91 total 190.86 last 20.96) - open/close 3598 (time 1788817835.95 total 200.91 last 21.90) - open/close 3715 (time 1788817846.20 total 211.15 last 11.43) - open/close 3922 (time 1788817856.23 total 221.18 last 20.62) - open/close 4025 (time 1788817866.25 total 231.20 last 10.28) - open/close 4181 (time 1788817876.26 total 241.21 last 15.59) - open/close 4317 (time 1788817886.30 total 251.25 last 13.54) - open/close 4418 (time 1788817896.33 total 261.28 last 10.07) - open/close 4578 (time 1788817907.34 total 272.29 last 14.53) - open/close 4691 (time 1788817918.40 total 283.35 last 10.22) - open/close 4782 (time 1788817929.47 total 294.42 last 8.22) - open/close 4952 (time 1788817939.57 total 304.52 last 16.83) - open/close 5128 (time 1788817950.06 total 315.01 last 16.77) - open/close 5293 (time 1788817960.10 total 325.05 last 16.43) - open/close 5460 (time 1788817970.77 total 335.72 last 15.65) - open/close 5599 (time 1788817981.37 total 346.32 last 13.12) - open/close 5726 (time 1788817991.37 total 356.32 last 12.69) - open/close 5840 (time 1788818001.42 total 366.37 last 11.35) - open/close 5946 (time 1788818011.76 total 376.71 last 10.25) - open/close 6115 (time 1788818022.71 total 387.66 last 15.43) - open/close 6191 (time 1788818032.82 total 397.77 last 7.51) - open/close 6403 (time 1788818043.37 total 408.32 last 20.10) - open/close 6639 (time 1788818054.19 total 419.14 last 21.82) - open/close 6843 (time 1788818066.04 total 430.99 last 17.21) - open/close 7098 (time 1788818076.05 total 441.00 last 25.48) - open/close 7296 (time 1788818086.06 total 451.02 last 19.76) - open/close 7538 (time 1788818096.74 total 461.69 last 22.67) - open/close 7769 (time 1788818109.42 total 474.37 last 18.22) - open/close 7900 (time 1788818119.66 total 484.61 last 12.79) - open/close 8058 (time 1788818130.49 total 495.44 last 14.59) - open/close 8187 (time 1788818140.51 total 505.46 last 12.87) - open/close 8352 (time 1788818150.53 total 515.48 last 16.46) - open/close 8450 (time 1788818161.00 total 525.96 last 9.35) - open/close 8601 (time 1788818171.88 total 536.83 last 13.88) - open/close 8660 (time 1788818182.50 total 547.45 last 5.56) - open/close 8877 (time 1788818193.24 total 558.19 last 20.20) - open/close 9035 (time 1788818203.78 total 568.73 last 15.00) - open/close 9274 (time 1788818213.80 total 578.75 last 23.84) - open/close 9471 (time 1788818223.80 total 588.75 last 19.70) - open/close 9579 (time 1788818233.82 total 598.78 last 10.77) - open/close 9676 (time 1788818244.79 total 609.74 last 8.85) - open/close 9737 (time 1788818255.74 total 620.69 last 5.57) - open/close 9865 (time 1788818266.54 total 631.49 last 11.86) - open/close 10000 (time 1788818272.17 total 637.12 last 23.95) - open/close 10106 (time 1788818282.77 total 647.72 last 10.00) - open/close 10279 (time 1788818292.77 total 657.72 last 17.29) - open/close 10343 (time 1788818304.62 total 669.58 last 5.40) - open/close 10456 (time 1788818315.31 total 680.26 last 10.57) - open/close 10615 (time 1788818326.64 total 691.59 last 14.03) - open/close 10853 (time 1788818336.67 total 701.62 last 23.74) - open/close 10919 (time 1788818346.67 total 711.62 last 6.60) - open/close 10995 (time 1788818356.67 total 721.62 last 7.60) - open/close 11061 (time 1788818366.67 total 731.62 last 6.60) - open/close 11203 (time 1788818376.68 total 741.63 last 14.19) - open/close 11347 (time 1788818386.68 total 751.63 last 14.39) - open/close 11516 (time 1788818397.22 total 762.17 last 16.04) - open/close 11643 (time 1788818407.22 total 772.18 last 12.69) - open/close 11855 (time 1788818417.24 total 782.19 last 21.18) - open/close 12045 (time 1788818427.24 total 792.19 last 18.99) - open/close 12150 (time 1788818437.43 total 802.38 last 10.30) - open/close 12293 (time 1788818447.86 total 812.81 last 13.71) - open/close 12506 (time 1788818458.15 total 823.11 last 20.70) - open/close 12694 (time 1788818468.16 total 833.11 last 18.79) - open/close 12783 (time 1788818478.16 total 843.11 last 8.90) - open/close 12995 (time 1788818488.17 total 853.12 last 21.18) - open/close 13216 (time 1788818498.74 total 863.69 last 20.91) - open/close 13425 (time 1788818508.77 total 873.72 last 20.83) - open/close 13631 (time 1788818518.78 total 883.73 last 20.59) - open/close 13908 (time 1788818529.12 total 894.07 last 26.77) - open/close 14133 (time 1788818539.94 total 904.89 last 20.79) - open/close 14374 (time 1788818549.94 total 914.89 last 24.10) - open/close 14598 (time 1788818560.33 total 925.28 last 21.58) - open/close 14813 (time 1788818570.33 total 935.28 last 21.48) - open/close 15048 (time 1788818580.34 total 945.29 last 23.48) - open/close 15257 (time 1788818590.35 total 955.30 last 20.90) - open/close 15482 (time 1788818600.36 total 965.31 last 22.47) - open/close 15719 (time 1788818610.79 total 975.75 last 22.71) - open/close 15972 (time 1788818620.80 total 985.75 last 25.29) - open/close 16110 (time 1788818632.17 total 997.12 last 12.13) - open/close 16366 (time 1788818642.18 total 1007.13 last 25.58) - open/close 16528 (time 1788818652.92 total 1017.87 last 15.08) - open/close 16679 (time 1788818662.93 total 1027.88 last 15.09) - open/close 16829 (time 1788818674.36 total 1039.31 last 13.13) - open/close 17029 (time 1788818684.85 total 1049.80 last 19.06) - open/close 17330 (time 1788818695.53 total 1060.48 last 28.20) - open/close 17582 (time 1788818706.03 total 1070.98 last 24.00) - open/close 18110 (time 1788818716.04 total 1080.99 last 52.71) - open/close 18825 (time 1788818726.05 total 1091.00 last 71.48) - open/close 19565 (time 1788818736.07 total 1101.02 last 73.85) - open/close 20000 (time 1788818742.10 total 1107.05 last 72.06) - open/close 20684 (time 1788818752.11 total 1117.06 last 68.35) - open/close 21343 (time 1788818762.12 total 1127.07 last 65.85) - open/close 21962 (time 1788818772.12 total 1137.08 last 61.85) - open/close 22554 (time 1788818782.13 total 1147.08 last 59.17) - open/close 23101 (time 1788818792.13 total 1157.08 last 54.69) - open/close 23704 (time 1788818802.13 total 1167.08 last 60.30) - open/close 24342 (time 1788818812.15 total 1177.10 last 63.71) - open/close 25020 (time 1788818822.15 total 1187.10 last 67.78) - open/close 25700 (time 1788818832.15 total 1197.10 last 67.98) - open/close 26394 (time 1788818842.16 total 1207.11 last 69.37) - open/close 27131 (time 1788818852.17 total 1217.12 last 73.62) - open/close 27817 (time 1788818862.19 total 1227.14 last 68.47) - open/close 28499 (time 1788818872.19 total 1237.14 last 68.17) - open/close 29237 (time 1788818882.19 total 1247.14 last 73.78) - open/close 29968 (time 1788818892.20 total 1257.15 last 73.05) - open/close 30616 (time 1788818902.21 total 1267.16 last 64.77) - open/close 31224 (time 1788818912.21 total 1277.17 last 60.74) - open/close 31848 (time 1788818922.22 total 1287.17 last 62.34) - open/close 32497 (time 1788818932.23 total 1297.18 last 64.89) - open/close 33213 (time 1788818942.23 total 1307.18 last 71.59) - open/close 33928 (time 1788818952.28 total 1317.23 last 71.13) - open/close 34637 (time 1788818962.29 total 1327.24 last 70.81) - open/close 35348 (time 1788818972.29 total 1337.24 last 71.09) - open/close 36019 (time 1788818982.30 total 1347.26 last 67.02) - open/close 36850 (time 1788818992.32 total 1357.27 last 82.94) - open/close 37654 (time 1788819002.34 total 1367.29 last 80.29) - open/close 38317 (time 1788819012.34 total 1377.30 last 66.25) - open/close 39026 (time 1788819022.35 total 1387.30 last 70.89) - open/close 39782 (time 1788819032.35 total 1397.30 last 75.55) - open/close 40000 (time 1788819035.21 total 1400.16 last 76.26) - open/close 40768 (time 1788819045.21 total 1410.16 last 76.79) - open/close 41456 (time 1788819055.22 total 1420.17 last 68.74) - open/close 42147 (time 1788819065.23 total 1430.18 last 69.04) - open/close 42921 (time 1788819075.23 total 1440.18 last 77.39) - open/close 43625 (time 1788819085.24 total 1450.19 last 70.36) - open/close 44306 (time 1788819095.25 total 1460.20 last 68.04) - open/close 45017 (time 1788819105.27 total 1470.22 last 70.95) - open/close 45652 (time 1788819115.27 total 1480.22 last 63.47) - open/close 46398 (time 1788819125.28 total 1490.23 last 74.54) - open/close 47117 (time 1788819135.30 total 1500.25 last 71.73) - open/close 47854 (time 1788819145.32 total 1510.27 last 73.61) - open/close 48558 (time 1788819155.32 total 1520.27 last 70.36) - open/close 49266 (time 1788819165.32 total 1530.27 last 70.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1535.35 seconds: 32.33 ops/second - unlinked 0 (time 1788819174 ; total 0 ; last 0) - unlinked 10000 (time 1788819321 ; total 147 ; last 147) - unlinked 20000 (time 1788819481 ; total 307 ; last 160) - unlinked 30000 (time 1788819636 ; total 462 ; last 155) - unlinked 40000 (time 1788819772 ; total 598 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 723 seconds: 68.645920 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: kmemleak disabled modules unloaded. pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2