== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:25:15 (1788855915) start mds service on oleg350-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 7 20:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: kmemleak disabled oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg350-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1788856027.88 total 10.01 last 18.19) - open/close 385 (time 1788856037.88 total 20.01 last 20.29) - open/close 541 (time 1788856047.89 total 30.02 last 15.58) - open/close 680 (time 1788856058.25 total 40.38 last 13.42) - open/close 831 (time 1788856069.31 total 51.44 last 13.65) - open/close 929 (time 1788856079.36 total 61.49 last 9.75) - open/close 1024 (time 1788856089.65 total 71.78 last 9.23) - open/close 1140 (time 1788856101.57 total 83.70 last 9.74) - open/close 1367 (time 1788856111.82 total 93.95 last 22.13) - open/close 1592 (time 1788856122.11 total 104.23 last 21.88) - open/close 1851 (time 1788856132.12 total 114.25 last 25.85) - open/close 2051 (time 1788856142.13 total 124.26 last 19.99) - open/close 2235 (time 1788856152.15 total 134.28 last 18.36) - open/close 2438 (time 1788856162.17 total 144.30 last 20.25) - open/close 2705 (time 1788856173.19 total 155.32 last 24.23) - open/close 2909 (time 1788856185.08 total 167.21 last 17.16) - open/close 3098 (time 1788856195.98 total 178.11 last 17.35) - open/close 3349 (time 1788856206.46 total 188.59 last 23.94) - open/close 3541 (time 1788856217.01 total 199.14 last 18.20) - open/close 3790 (time 1788856227.25 total 209.38 last 24.32) - open/close 3992 (time 1788856237.59 total 219.71 last 19.55) - open/close 4086 (time 1788856247.59 total 229.72 last 9.39) - open/close 4299 (time 1788856259.09 total 241.22 last 18.52) - open/close 4522 (time 1788856269.85 total 251.97 last 20.74) - open/close 4776 (time 1788856280.83 total 262.96 last 23.12) - open/close 4989 (time 1788856291.34 total 273.47 last 20.28) - open/close 5267 (time 1788856302.15 total 284.28 last 25.71) - open/close 5467 (time 1788856314.94 total 297.07 last 15.64) - open/close 5706 (time 1788856325.25 total 307.38 last 23.18) - open/close 5926 (time 1788856335.26 total 317.39 last 21.98) - open/close 6184 (time 1788856345.27 total 327.40 last 25.77) - open/close 6402 (time 1788856357.37 total 339.50 last 18.01) - open/close 6601 (time 1788856367.61 total 349.74 last 19.44) - open/close 6866 (time 1788856378.67 total 360.80 last 23.94) - open/close 7033 (time 1788856388.68 total 370.81 last 16.68) - open/close 7224 (time 1788856398.70 total 380.82 last 19.08) - open/close 7346 (time 1788856408.97 total 391.09 last 11.88) - open/close 7488 (time 1788856419.11 total 401.24 last 14.00) - open/close 7650 (time 1788856429.38 total 411.51 last 15.77) - open/close 7812 (time 1788856440.39 total 422.51 last 14.72) - open/close 7955 (time 1788856450.45 total 432.58 last 14.21) - open/close 8127 (time 1788856461.52 total 443.65 last 15.53) - open/close 8207 (time 1788856471.93 total 454.06 last 7.69) - open/close 8335 (time 1788856482.69 total 464.82 last 11.89) - open/close 8486 (time 1788856493.14 total 475.27 last 14.45) - open/close 8632 (time 1788856503.60 total 485.73 last 13.95) - open/close 8840 (time 1788856513.61 total 495.74 last 20.79) - open/close 8907 (time 1788856524.28 total 506.41 last 6.27) - open/close 9152 (time 1788856534.30 total 516.43 last 24.46) - open/close 9279 (time 1788856545.36 total 527.49 last 11.48) - open/close 9419 (time 1788856557.63 total 539.76 last 11.41) - open/close 9547 (time 1788856567.69 total 549.82 last 12.73) - open/close 9663 (time 1788856578.36 total 560.49 last 10.87) - open/close 9896 (time 1788856588.75 total 570.88 last 22.43) - open/close 10000 (time 1788856592.95 total 575.08 last 24.75) - open/close 10229 (time 1788856603.46 total 585.59 last 21.80) - open/close 10391 (time 1788856613.82 total 595.95 last 15.63) - open/close 10612 (time 1788856623.83 total 605.96 last 22.07) - open/close 10846 (time 1788856634.31 total 616.43 last 22.34) - open/close 11051 (time 1788856644.34 total 626.47 last 20.43) - open/close 11275 (time 1788856654.79 total 636.91 last 21.45) - open/close 11480 (time 1788856664.80 total 646.93 last 20.46) - open/close 11740 (time 1788856675.45 total 657.57 last 24.43) - open/close 11997 (time 1788856685.91 total 668.04 last 24.55) - open/close 12165 (time 1788856695.95 total 678.07 last 16.74) - open/close 12387 (time 1788856706.64 total 688.77 last 20.75) - open/close 12495 (time 1788856716.67 total 698.80 last 10.77) - open/close 12722 (time 1788856727.42 total 709.54 last 21.12) - open/close 12852 (time 1788856737.42 total 719.55 last 12.99) - open/close 13052 (time 1788856748.41 total 730.54 last 18.20) - open/close 13297 (time 1788856759.03 total 741.16 last 23.07) - open/close 13411 (time 1788856769.52 total 751.65 last 10.87) - open/close 13638 (time 1788856779.54 total 761.67 last 22.65) - open/close 13815 (time 1788856791.23 total 773.36 last 15.14) - open/close 13886 (time 1788856801.24 total 783.36 last 7.10) - open/close 14057 (time 1788856811.64 total 793.77 last 16.43) - open/close 14235 (time 1788856821.66 total 803.78 last 17.78) - open/close 14474 (time 1788856832.44 total 814.57 last 22.16) - open/close 14629 (time 1788856842.45 total 824.58 last 15.48) - open/close 14753 (time 1788856852.67 total 834.80 last 12.13) - open/close 14884 (time 1788856863.31 total 845.44 last 12.32) - open/close 15051 (time 1788856874.12 total 856.25 last 15.44) - open/close 15221 (time 1788856885.00 total 867.12 last 15.63) - open/close 15334 (time 1788856896.01 total 878.14 last 10.26) - open/close 15442 (time 1788856906.69 total 888.82 last 10.11) - open/close 15567 (time 1788856917.16 total 899.29 last 11.94) - open/close 15725 (time 1788856928.22 total 910.35 last 14.29) - open/close 15902 (time 1788856940.38 total 922.51 last 14.55) - open/close 16016 (time 1788856951.18 total 933.31 last 10.56) - open/close 16199 (time 1788856961.86 total 943.98 last 17.14) - open/close 16363 (time 1788856972.60 total 954.73 last 15.26) - open/close 16501 (time 1788856982.62 total 964.75 last 13.77) - open/close 16682 (time 1788856992.63 total 974.76 last 18.09) - open/close 16934 (time 1788857003.27 total 985.40 last 23.67) - open/close 17110 (time 1788857014.17 total 996.29 last 16.16) - open/close 17292 (time 1788857024.18 total 1006.31 last 18.18) - open/close 17448 (time 1788857034.19 total 1016.32 last 15.59) - open/close 17535 (time 1788857044.74 total 1026.87 last 8.24) - open/close 17568 (time 1788857055.03 total 1037.16 last 3.21) - open/close 17729 (time 1788857065.42 total 1047.54 last 15.50) - open/close 17797 (time 1788857076.23 total 1058.35 last 6.29) - open/close 17976 (time 1788857087.05 total 1069.18 last 16.54) - open/close 18236 (time 1788857097.06 total 1079.19 last 25.97) - open/close 18957 (time 1788857107.06 total 1089.19 last 72.10) - open/close 19645 (time 1788857117.07 total 1099.20 last 68.73) - open/close 20000 (time 1788857123.48 total 1105.61 last 55.38) - open/close 20631 (time 1788857133.49 total 1115.61 last 63.07) - open/close 21267 (time 1788857143.50 total 1125.62 last 63.54) - open/close 21792 (time 1788857153.50 total 1135.63 last 52.47) - open/close 22274 (time 1788857163.52 total 1145.65 last 48.12) - open/close 22848 (time 1788857173.52 total 1155.65 last 57.37) - open/close 23388 (time 1788857183.53 total 1165.66 last 53.97) - open/close 23988 (time 1788857193.54 total 1175.67 last 59.93) - open/close 24523 (time 1788857203.55 total 1185.67 last 53.47) - open/close 25031 (time 1788857213.55 total 1195.68 last 50.78) - open/close 25423 (time 1788857223.56 total 1205.69 last 39.17) - open/close 25970 (time 1788857233.59 total 1215.72 last 54.52) - open/close 26737 (time 1788857243.61 total 1225.73 last 76.59) - open/close 27449 (time 1788857253.62 total 1235.75 last 71.08) - open/close 28222 (time 1788857263.63 total 1245.76 last 77.21) - open/close 28918 (time 1788857273.65 total 1255.78 last 69.47) - open/close 29621 (time 1788857283.67 total 1265.80 last 70.16) - open/close 30000 (time 1788857288.87 total 1271.00 last 72.89) - open/close 30750 (time 1788857298.88 total 1281.00 last 74.98) - open/close 31490 (time 1788857308.90 total 1291.03 last 73.80) - open/close 32154 (time 1788857318.92 total 1301.05 last 66.31) - open/close 32796 (time 1788857328.94 total 1311.07 last 64.07) - open/close 33541 (time 1788857338.96 total 1321.09 last 74.33) - open/close 34223 (time 1788857348.97 total 1331.10 last 68.15) - open/close 34963 (time 1788857358.97 total 1341.10 last 73.98) - open/close 35699 (time 1788857368.97 total 1351.10 last 73.59) - open/close 36336 (time 1788857378.98 total 1361.11 last 63.66) - open/close 37109 (time 1788857388.98 total 1371.11 last 77.29) - open/close 37764 (time 1788857398.99 total 1381.12 last 65.45) - open/close 38317 (time 1788857408.99 total 1391.12 last 55.28) - open/close 39066 (time 1788857419.00 total 1401.13 last 74.86) - open/close 39770 (time 1788857429.00 total 1411.13 last 70.40) - open/close 40000 (time 1788857432.60 total 1414.73 last 63.82) - open/close 40691 (time 1788857442.60 total 1424.73 last 69.09) - open/close 41361 (time 1788857452.61 total 1434.74 last 66.95) - open/close 42036 (time 1788857462.62 total 1444.75 last 67.46) - open/close 42728 (time 1788857472.62 total 1454.75 last 69.19) - open/close 43477 (time 1788857482.62 total 1464.75 last 74.88) - open/close 44221 (time 1788857492.62 total 1474.75 last 74.38) - open/close 44948 (time 1788857502.63 total 1484.76 last 72.66) - open/close 45660 (time 1788857512.63 total 1494.76 last 71.18) - open/close 46345 (time 1788857522.63 total 1504.76 last 68.49) - open/close 47037 (time 1788857532.64 total 1514.77 last 69.19) - open/close 47821 (time 1788857542.64 total 1524.77 last 78.38) - open/close 48577 (time 1788857552.65 total 1534.78 last 75.54) - open/close 49100 (time 1788857562.66 total 1544.79 last 52.25) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1554.28 seconds: 31.93 ops/second - unlinked 0 (time 1788857577 ; total 0 ; last 0) - unlinked 10000 (time 1788857718 ; total 141 ; last 141) - unlinked 20000 (time 1788857857 ; total 280 ; last 139) - unlinked 30000 (time 1788857996 ; total 419 ; last 139) - unlinked 40000 (time 1788858131 ; total 554 ; last 135) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 682 seconds: 72.772728 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: kmemleak disabled modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2