== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:23:14 (1788848594) start mds service on oleg101-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: kmemleak disabled oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: MODOPTS_LIBCFS= oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg101-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg101-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg101-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 173 (time 1788848718.86 total 10.01 last 17.29) - open/close 406 (time 1788848729.90 total 21.04 last 21.11) - open/close 664 (time 1788848739.90 total 31.05 last 25.79) - open/close 883 (time 1788848750.80 total 41.94 last 20.10) - open/close 1060 (time 1788848761.82 total 52.97 last 16.06) - open/close 1266 (time 1788848773.96 total 65.11 last 16.97) - open/close 1349 (time 1788848784.27 total 75.41 last 8.06) - open/close 1522 (time 1788848794.89 total 86.03 last 16.29) - open/close 1633 (time 1788848804.90 total 96.05 last 11.08) - open/close 1802 (time 1788848814.91 total 106.05 last 16.89) - open/close 1916 (time 1788848825.09 total 116.24 last 11.19) - open/close 2093 (time 1788848837.40 total 128.54 last 14.39) - open/close 2265 (time 1788848848.16 total 139.30 last 15.98) - open/close 2479 (time 1788848858.17 total 149.31 last 21.37) - open/close 2584 (time 1788848868.83 total 159.97 last 9.85) - open/close 2766 (time 1788848879.13 total 170.28 last 17.67) - open/close 2886 (time 1788848889.25 total 180.39 last 11.86) - open/close 2987 (time 1788848900.17 total 191.31 last 9.25) - open/close 3124 (time 1788848911.69 total 202.84 last 11.89) - open/close 3288 (time 1788848921.70 total 212.85 last 16.38) - open/close 3413 (time 1788848932.31 total 223.46 last 11.78) - open/close 3688 (time 1788848942.34 total 233.48 last 27.44) - open/close 3957 (time 1788848952.52 total 243.67 last 26.40) - open/close 4123 (time 1788848962.54 total 253.68 last 16.58) - open/close 4296 (time 1788848972.57 total 263.71 last 17.25) - open/close 4466 (time 1788848984.75 total 275.89 last 13.96) - open/close 4710 (time 1788848994.90 total 286.04 last 24.05) - open/close 4901 (time 1788849005.41 total 296.55 last 18.17) - open/close 5115 (time 1788849015.43 total 306.58 last 21.35) - open/close 5312 (time 1788849026.45 total 317.60 last 17.88) - open/close 5564 (time 1788849037.20 total 328.35 last 23.44) - open/close 5796 (time 1788849048.03 total 339.17 last 21.43) - open/close 6047 (time 1788849058.03 total 349.18 last 25.09) - open/close 6314 (time 1788849068.05 total 359.19 last 26.67) - open/close 6524 (time 1788849080.09 total 371.23 last 17.44) - open/close 6780 (time 1788849090.11 total 381.25 last 25.55) - open/close 7031 (time 1788849100.11 total 391.26 last 25.08) - open/close 7172 (time 1788849110.12 total 401.26 last 14.09) - open/close 7453 (time 1788849120.13 total 411.27 last 28.08) - open/close 7646 (time 1788849132.42 total 423.57 last 15.69) - open/close 7883 (time 1788849148.55 total 439.70 last 14.69) - open/close 8119 (time 1788849159.39 total 450.54 last 21.77) - open/close 8372 (time 1788849171.47 total 462.61 last 20.95) - open/close 8515 (time 1788849181.64 total 472.78 last 14.06) - open/close 8620 (time 1788849191.77 total 482.92 last 10.36) - open/close 8843 (time 1788849201.79 total 492.94 last 22.26) - open/close 9131 (time 1788849211.81 total 502.95 last 28.77) - open/close 9378 (time 1788849221.81 total 512.95 last 24.69) - open/close 9607 (time 1788849231.82 total 522.97 last 22.86) - open/close 9767 (time 1788849242.25 total 533.39 last 15.35) - open/close 9959 (time 1788849252.25 total 543.40 last 19.19) - open/close 10206 (time 1788849262.85 total 553.99 last 23.30) - open/close 10432 (time 1788849272.86 total 564.00 last 22.58) - open/close 10673 (time 1788849282.86 total 574.01 last 24.09) - open/close 10898 (time 1788849293.23 total 584.38 last 21.69) - open/close 11123 (time 1788849304.03 total 595.18 last 20.83) - open/close 11388 (time 1788849314.05 total 605.19 last 26.46) - open/close 11547 (time 1788849324.06 total 615.20 last 15.88) - open/close 11735 (time 1788849334.09 total 625.23 last 18.75) - open/close 11879 (time 1788849346.08 total 637.22 last 12.01) - open/close 12089 (time 1788849356.09 total 647.23 last 20.98) - open/close 12344 (time 1788849366.67 total 657.81 last 24.10) - open/close 12529 (time 1788849377.09 total 668.23 last 17.75) - open/close 12811 (time 1788849387.72 total 678.87 last 26.53) - open/close 13093 (time 1788849397.74 total 688.88 last 28.15) - open/close 13325 (time 1788849408.35 total 699.49 last 21.86) - open/close 13662 (time 1788849419.01 total 710.15 last 31.61) - open/close 13886 (time 1788849431.44 total 722.59 last 18.02) - open/close 14164 (time 1788849441.74 total 732.89 last 26.99) - open/close 14383 (time 1788849452.50 total 743.64 last 20.36) - open/close 14598 (time 1788849463.31 total 754.45 last 19.89) - open/close 14764 (time 1788849474.93 total 766.07 last 14.28) - open/close 15010 (time 1788849485.96 total 777.11 last 22.30) - open/close 15262 (time 1788849495.96 total 787.11 last 25.20) - open/close 15516 (time 1788849505.97 total 797.11 last 25.39) - open/close 15764 (time 1788849516.49 total 807.64 last 23.57) - open/close 16022 (time 1788849526.50 total 817.64 last 25.79) - open/close 16309 (time 1788849537.41 total 828.56 last 26.29) - open/close 16548 (time 1788849548.05 total 839.20 last 22.47) - open/close 16757 (time 1788849558.69 total 849.83 last 19.64) - open/close 16965 (time 1788849568.69 total 859.83 last 20.80) - open/close 17209 (time 1788849579.43 total 870.58 last 22.71) - open/close 17465 (time 1788849589.96 total 881.11 last 24.32) - open/close 17612 (time 1788849599.98 total 891.12 last 14.68) - open/close 17813 (time 1788849610.97 total 902.11 last 18.29) - open/close 18036 (time 1788849620.97 total 912.12 last 22.29) - open/close 18314 (time 1788849631.58 total 922.73 last 26.19) - open/close 18554 (time 1788849642.07 total 933.21 last 22.89) - open/close 18819 (time 1788849652.38 total 943.52 last 25.71) - open/close 19094 (time 1788849663.21 total 954.35 last 25.39) - open/close 19371 (time 1788849673.21 total 964.35 last 27.69) - open/close 19568 (time 1788849683.22 total 974.37 last 19.67) - open/close 20000 (time 1788849692.85 total 984.00 last 44.86) - open/close 20818 (time 1788849702.86 total 994.00 last 81.77) - open/close 21652 (time 1788849712.87 total 1004.01 last 83.33) - open/close 22581 (time 1788849722.87 total 1014.01 last 92.90) - open/close 23475 (time 1788849732.87 total 1024.01 last 89.38) - open/close 24312 (time 1788849742.87 total 1034.02 last 83.66) - open/close 25130 (time 1788849752.87 total 1044.02 last 81.78) - open/close 25934 (time 1788849762.88 total 1054.02 last 80.38) - open/close 26775 (time 1788849772.88 total 1064.02 last 84.08) - open/close 27651 (time 1788849782.89 total 1074.03 last 87.56) - open/close 28476 (time 1788849792.89 total 1084.03 last 82.46) - open/close 29292 (time 1788849802.90 total 1094.04 last 81.53) - open/close 30000 (time 1788849811.61 total 1102.76 last 81.25) - open/close 30726 (time 1788849821.62 total 1112.77 last 72.52) - open/close 31515 (time 1788849831.63 total 1122.77 last 78.89) - open/close 32333 (time 1788849841.63 total 1132.77 last 81.79) - open/close 33222 (time 1788849851.64 total 1142.79 last 88.74) - open/close 34049 (time 1788849861.66 total 1152.80 last 82.57) - open/close 34777 (time 1788849871.67 total 1162.81 last 72.76) - open/close 35543 (time 1788849881.67 total 1172.82 last 76.56) - open/close 36302 (time 1788849891.68 total 1182.82 last 75.87) - open/close 37038 (time 1788849901.68 total 1192.82 last 73.56) - open/close 37889 (time 1788849911.69 total 1202.83 last 85.05) - open/close 38736 (time 1788849921.70 total 1212.84 last 84.60) - open/close 39533 (time 1788849931.70 total 1222.84 last 79.70) - open/close 40000 (time 1788849937.69 total 1228.84 last 77.90) - open/close 40779 (time 1788849947.70 total 1238.84 last 77.86) - open/close 41634 (time 1788849957.71 total 1248.85 last 85.43) - open/close 42534 (time 1788849967.71 total 1258.85 last 89.98) - open/close 43402 (time 1788849977.71 total 1268.86 last 86.78) - open/close 44317 (time 1788849987.72 total 1278.86 last 91.45) - open/close 45240 (time 1788849997.72 total 1288.87 last 92.25) - open/close 46144 (time 1788850007.72 total 1298.87 last 90.38) - open/close 47173 (time 1788850017.73 total 1308.87 last 102.85) - open/close 48214 (time 1788850027.74 total 1318.88 last 104.04) - open/close 49218 (time 1788850037.74 total 1328.88 last 100.35) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1333.10 seconds: 37.23 ops/second - unlinked 0 (time 1788850045 ; total 0 ; last 0) - unlinked 10000 (time 1788850145 ; total 100 ; last 100) - unlinked 20000 (time 1788850238 ; total 193 ; last 93) - unlinked 30000 (time 1788850301 ; total 256 ; last 63) - unlinked 40000 (time 1788850365 ; total 320 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 374 seconds: 132.703201 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg101-server unloading modules via unload_modules_local on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local oleg101-server: kmemleak disabled modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2