== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:20:37 (1790950837) start mds service on oleg139-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1946968 Oct 2 07:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: kmemleak disabled oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: MODOPTS_LIBCFS= oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg139-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg139-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg139-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 265 (time 1790950940.21 total 10.36 last 25.58) - open/close 511 (time 1790950950.21 total 20.36 last 24.59) - open/close 716 (time 1790950960.22 total 30.37 last 20.48) - open/close 961 (time 1790950970.30 total 40.45 last 24.32) - open/close 1082 (time 1790950980.33 total 50.48 last 12.05) - open/close 1306 (time 1790950990.34 total 60.49 last 22.38) - open/close 1579 (time 1790951000.35 total 70.50 last 27.29) - open/close 1791 (time 1790951010.50 total 80.65 last 20.89) - open/close 1999 (time 1790951020.86 total 91.01 last 20.07) - open/close 2201 (time 1790951030.87 total 101.02 last 20.19) - open/close 2410 (time 1790951040.87 total 111.02 last 20.88) - open/close 2606 (time 1790951050.90 total 121.05 last 19.54) - open/close 2831 (time 1790951060.91 total 131.06 last 22.49) - open/close 3058 (time 1790951070.91 total 141.06 last 22.69) - open/close 3150 (time 1790951080.93 total 151.08 last 9.18) - open/close 3378 (time 1790951091.45 total 161.60 last 21.67) - open/close 3502 (time 1790951102.01 total 172.16 last 11.74) - open/close 3861 (time 1790951114.93 total 185.08 last 27.79) - open/close 3919 (time 1790951126.58 total 196.73 last 4.98) - open/close 4180 (time 1790951136.70 total 206.85 last 25.80) - open/close 4322 (time 1790951147.07 total 217.22 last 13.68) - open/close 4491 (time 1790951157.08 total 227.23 last 16.89) - open/close 4717 (time 1790951167.37 total 237.52 last 21.96) - open/close 4983 (time 1790951177.38 total 247.53 last 26.56) - open/close 5192 (time 1790951187.70 total 257.85 last 20.25) - open/close 5384 (time 1790951198.53 total 268.68 last 17.74) - open/close 5604 (time 1790951209.52 total 279.67 last 20.01) - open/close 5841 (time 1790951219.53 total 289.68 last 23.67) - open/close 6014 (time 1790951230.83 total 300.98 last 15.31) - open/close 6274 (time 1790951241.39 total 311.54 last 24.62) - open/close 6452 (time 1790951251.96 total 322.11 last 16.85) - open/close 6594 (time 1790951262.21 total 332.36 last 13.85) - open/close 6754 (time 1790951272.95 total 343.10 last 14.89) - open/close 6855 (time 1790951283.13 total 353.28 last 9.92) - open/close 7037 (time 1790951293.14 total 363.29 last 18.18) - open/close 7206 (time 1790951304.07 total 374.22 last 15.46) - open/close 7376 (time 1790951316.85 total 387.00 last 13.30) - open/close 7609 (time 1790951326.87 total 397.02 last 23.24) - open/close 7843 (time 1790951337.21 total 407.36 last 22.63) - open/close 8047 (time 1790951347.22 total 417.37 last 20.40) - open/close 8290 (time 1790951357.22 total 427.37 last 24.29) - open/close 8486 (time 1790951367.22 total 437.37 last 19.60) - open/close 8666 (time 1790951377.92 total 448.07 last 16.83) - open/close 8907 (time 1790951388.36 total 458.51 last 23.09) - open/close 9127 (time 1790951398.37 total 468.52 last 21.97) - open/close 9367 (time 1790951409.28 total 479.43 last 22.00) - open/close 9655 (time 1790951419.77 total 489.92 last 27.44) - open/close 9925 (time 1790951430.13 total 500.28 last 26.06) - open/close 10000 (time 1790951434.86 total 505.01 last 15.86) - open/close 10271 (time 1790951445.24 total 515.39 last 26.12) - open/close 10420 (time 1790951455.33 total 525.48 last 14.77) - open/close 10620 (time 1790951465.41 total 535.56 last 19.83) - open/close 10734 (time 1790951475.42 total 545.57 last 11.40) - open/close 10914 (time 1790951485.42 total 555.57 last 17.99) - open/close 11150 (time 1790951496.31 total 566.46 last 21.67) - open/close 11405 (time 1790951508.50 total 578.65 last 20.92) - open/close 11566 (time 1790951519.17 total 589.32 last 15.09) - open/close 11616 (time 1790951529.62 total 599.77 last 4.78) - open/close 11853 (time 1790951539.62 total 609.78 last 23.70) - open/close 12032 (time 1790951550.60 total 620.75 last 16.32) - open/close 12173 (time 1790951560.60 total 630.75 last 14.09) - open/close 12295 (time 1790951571.64 total 641.79 last 11.05) - open/close 12402 (time 1790951582.20 total 652.35 last 10.13) - open/close 12614 (time 1790951592.59 total 662.74 last 20.40) - open/close 12780 (time 1790951603.32 total 673.47 last 15.48) - open/close 12968 (time 1790951613.51 total 683.66 last 18.44) - open/close 13090 (time 1790951623.52 total 693.67 last 12.19) - open/close 13283 (time 1790951635.14 total 705.29 last 16.60) - open/close 13569 (time 1790951645.89 total 716.04 last 26.61) - open/close 13834 (time 1790951655.90 total 726.05 last 26.48) - open/close 13942 (time 1790951665.92 total 736.07 last 10.77) - open/close 14143 (time 1790951675.93 total 746.08 last 20.09) - open/close 14359 (time 1790951685.98 total 756.13 last 21.49) - open/close 14505 (time 1790951696.20 total 766.35 last 14.28) - open/close 14634 (time 1790951706.83 total 776.99 last 12.13) - open/close 14866 (time 1790951717.65 total 787.80 last 21.44) - open/close 15128 (time 1790951727.66 total 797.81 last 26.19) - open/close 15380 (time 1790951737.66 total 807.81 last 25.20) - open/close 15692 (time 1790951747.67 total 817.82 last 31.16) - open/close 15875 (time 1790951757.68 total 827.83 last 18.29) - open/close 15962 (time 1790951769.03 total 839.18 last 7.66) - open/close 16104 (time 1790951779.71 total 849.86 last 13.29) - open/close 16210 (time 1790951789.71 total 859.86 last 10.60) - open/close 16250 (time 1790951800.19 total 870.34 last 3.82) - open/close 16370 (time 1790951810.42 total 880.57 last 11.73) - open/close 16640 (time 1790951821.17 total 891.32 last 25.12) - open/close 16852 (time 1790951831.18 total 901.33 last 21.18) - open/close 17156 (time 1790951841.18 total 911.33 last 30.38) - open/close 17439 (time 1790951851.90 total 922.05 last 26.40) - open/close 17691 (time 1790951862.32 total 932.47 last 24.19) - open/close 17969 (time 1790951873.07 total 943.22 last 25.87) - open/close 18228 (time 1790951883.55 total 953.70 last 24.70) - open/close 18529 (time 1790951894.61 total 964.76 last 27.24) - open/close 18657 (time 1790951904.66 total 974.81 last 12.73) - open/close 18798 (time 1790951915.25 total 985.40 last 13.31) - open/close 18907 (time 1790951925.27 total 995.42 last 10.88) - open/close 19105 (time 1790951935.28 total 1005.43 last 19.78) - open/close 19267 (time 1790951945.54 total 1015.69 last 15.78) - open/close 19435 (time 1790951956.03 total 1026.18 last 16.01) - open/close 19707 (time 1790951966.05 total 1036.20 last 27.16) - open/close 20000 (time 1790951969.03 total 1039.18 last 98.18) - open/close 20876 (time 1790951979.04 total 1049.19 last 87.53) - open/close 21795 (time 1790951989.05 total 1059.20 last 91.84) - open/close 22721 (time 1790951999.06 total 1069.21 last 92.48) - open/close 23746 (time 1790952009.07 total 1079.22 last 102.43) - open/close 24643 (time 1790952019.07 total 1089.22 last 89.69) - open/close 25664 (time 1790952029.07 total 1099.22 last 102.10) - open/close 26719 (time 1790952039.08 total 1109.23 last 105.40) - open/close 27843 (time 1790952049.09 total 1119.24 last 112.23) - open/close 28818 (time 1790952059.10 total 1129.25 last 97.42) - open/close 29631 (time 1790952069.10 total 1139.25 last 81.29) - open/close 30000 (time 1790952073.26 total 1143.41 last 88.79) - open/close 30947 (time 1790952083.26 total 1153.41 last 94.67) - open/close 31888 (time 1790952093.27 total 1163.42 last 94.03) - open/close 32803 (time 1790952103.27 total 1173.42 last 91.48) - open/close 33803 (time 1790952113.28 total 1183.43 last 99.96) - open/close 34745 (time 1790952123.28 total 1193.43 last 94.14) - open/close 35757 (time 1790952133.29 total 1203.44 last 101.10) - open/close 36759 (time 1790952143.29 total 1213.44 last 100.19) - open/close 37589 (time 1790952153.29 total 1223.44 last 83.00) - open/close 38472 (time 1790952163.30 total 1233.45 last 88.24) - open/close 39286 (time 1790952173.31 total 1243.46 last 81.33) - open/close 40000 (time 1790952180.96 total 1251.11 last 93.34) - open/close 40880 (time 1790952190.98 total 1261.13 last 87.78) - open/close 41738 (time 1790952200.98 total 1271.13 last 85.79) - open/close 42749 (time 1790952210.99 total 1281.14 last 101.08) - open/close 43695 (time 1790952220.99 total 1291.14 last 94.57) - open/close 44525 (time 1790952230.99 total 1301.14 last 82.99) - open/close 45208 (time 1790952241.00 total 1311.15 last 68.26) - open/close 45791 (time 1790952251.00 total 1321.15 last 58.25) - open/close 46680 (time 1790952261.01 total 1331.16 last 88.84) - open/close 47591 (time 1790952271.01 total 1341.16 last 91.08) - open/close 48456 (time 1790952281.02 total 1351.17 last 86.49) - open/close 49257 (time 1790952291.02 total 1361.17 last 80.08) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1366.58 seconds: 36.32 ops/second - unlinked 0 (time 1790952302 ; total 0 ; last 0) - unlinked 10000 (time 1790952417 ; total 115 ; last 115) - unlinked 20000 (time 1790952517 ; total 215 ; last 100) - unlinked 30000 (time 1790952574 ; total 272 ; last 57) - unlinked 40000 (time 1790952631 ; total 329 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 394 seconds: 125.967003 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg139-server unloading modules via unload_modules_local on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: kmemleak disabled modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2