== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:17:32 (1789406252) start mds service on oleg346-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: kmemleak disabled oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg346-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg346-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 210 (time 1789406331.56 total 10.48 last 20.03) - open/close 427 (time 1789406341.78 total 20.70 last 21.23) - open/close 631 (time 1789406351.82 total 30.75 last 20.31) - open/close 848 (time 1789406362.19 total 41.12 last 20.93) - open/close 1067 (time 1789406372.22 total 51.15 last 21.83) - open/close 1288 (time 1789406382.22 total 61.15 last 22.10) - open/close 1547 (time 1789406392.22 total 71.15 last 25.90) - open/close 1768 (time 1789406402.24 total 81.16 last 22.07) - open/close 2002 (time 1789406412.31 total 91.24 last 23.22) - open/close 2246 (time 1789406422.67 total 101.59 last 23.56) - open/close 2476 (time 1789406432.68 total 111.61 last 22.97) - open/close 2652 (time 1789406442.89 total 121.81 last 17.24) - open/close 2766 (time 1789406452.89 total 131.82 last 11.40) - open/close 2990 (time 1789406462.90 total 141.82 last 22.38) - open/close 3223 (time 1789406472.92 total 151.85 last 23.25) - open/close 3480 (time 1789406483.81 total 162.74 last 23.59) - open/close 3710 (time 1789406494.17 total 173.10 last 22.20) - open/close 3923 (time 1789406504.84 total 183.77 last 19.96) - open/close 4233 (time 1789406515.49 total 194.42 last 29.11) - open/close 4441 (time 1789406525.50 total 204.43 last 20.78) - open/close 4625 (time 1789406536.24 total 215.16 last 17.14) - open/close 4799 (time 1789406547.23 total 226.16 last 15.82) - open/close 4891 (time 1789406558.24 total 237.16 last 8.36) - open/close 5076 (time 1789406568.65 total 247.58 last 17.77) - open/close 5315 (time 1789406578.66 total 257.59 last 23.87) - open/close 5536 (time 1789406588.68 total 267.61 last 22.06) - open/close 5783 (time 1789406598.68 total 277.61 last 24.69) - open/close 6012 (time 1789406609.35 total 288.28 last 21.46) - open/close 6213 (time 1789406619.37 total 298.29 last 20.07) - open/close 6444 (time 1789406629.86 total 308.78 last 22.02) - open/close 6711 (time 1789406640.61 total 319.54 last 24.83) - open/close 6952 (time 1789406650.61 total 329.54 last 24.10) - open/close 7206 (time 1789406660.63 total 339.55 last 25.36) - open/close 7387 (time 1789406671.31 total 350.24 last 16.94) - open/close 7602 (time 1789406682.00 total 360.93 last 20.11) - open/close 7836 (time 1789406692.02 total 370.95 last 23.36) - open/close 8067 (time 1789406702.47 total 381.40 last 22.10) - open/close 8288 (time 1789406712.72 total 391.65 last 21.57) - open/close 8509 (time 1789406723.08 total 402.01 last 21.33) - open/close 8758 (time 1789406733.58 total 412.51 last 23.71) - open/close 8962 (time 1789406744.40 total 423.33 last 18.86) - open/close 9184 (time 1789406754.82 total 433.75 last 21.30) - open/close 9420 (time 1789406764.83 total 443.76 last 23.58) - open/close 9656 (time 1789406775.63 total 454.56 last 21.85) - open/close 9882 (time 1789406785.99 total 464.92 last 21.82) - open/close 10000 (time 1789406791.85 total 470.78 last 20.12) - open/close 10226 (time 1789406802.64 total 481.57 last 20.94) - open/close 10472 (time 1789406815.54 total 494.47 last 19.08) - open/close 10709 (time 1789406826.52 total 505.44 last 21.59) - open/close 10952 (time 1789406836.75 total 515.68 last 23.74) - open/close 11212 (time 1789406846.80 total 525.72 last 25.88) - open/close 11419 (time 1789406856.80 total 535.73 last 20.70) - open/close 11643 (time 1789406867.27 total 546.20 last 21.39) - open/close 11875 (time 1789406877.58 total 556.51 last 22.50) - open/close 12130 (time 1789406888.02 total 566.95 last 24.42) - open/close 12345 (time 1789406898.13 total 577.06 last 21.27) - open/close 12533 (time 1789406908.37 total 587.30 last 18.37) - open/close 12690 (time 1789406918.37 total 597.30 last 15.70) - open/close 12870 (time 1789406929.16 total 608.09 last 16.68) - open/close 13067 (time 1789406939.16 total 618.09 last 19.70) - open/close 13194 (time 1789406949.27 total 628.19 last 12.57) - open/close 13344 (time 1789406962.13 total 641.06 last 11.66) - open/close 13597 (time 1789406972.96 total 651.88 last 23.37) - open/close 13862 (time 1789406982.97 total 661.90 last 26.46) - open/close 14067 (time 1789406992.98 total 671.91 last 20.48) - open/close 14284 (time 1789407002.99 total 681.91 last 21.68) - open/close 14507 (time 1789407012.99 total 691.92 last 22.29) - open/close 14721 (time 1789407023.00 total 701.93 last 21.38) - open/close 14952 (time 1789407033.88 total 712.81 last 21.23) - open/close 15174 (time 1789407044.11 total 723.04 last 21.69) - open/close 15336 (time 1789407054.60 total 733.52 last 15.45) - open/close 15549 (time 1789407064.60 total 743.53 last 21.29) - open/close 15768 (time 1789407074.61 total 753.53 last 21.89) - open/close 15982 (time 1789407084.61 total 763.54 last 21.39) - open/close 16217 (time 1789407094.79 total 773.72 last 23.08) - open/close 16465 (time 1789407104.80 total 783.73 last 24.78) - open/close 16669 (time 1789407115.14 total 794.07 last 19.72) - open/close 16915 (time 1789407125.64 total 804.57 last 23.43) - open/close 17110 (time 1789407135.78 total 814.70 last 19.25) - open/close 17330 (time 1789407145.78 total 824.70 last 22.00) - open/close 17574 (time 1789407156.56 total 835.49 last 22.63) - open/close 17741 (time 1789407167.10 total 846.03 last 15.84) - open/close 17869 (time 1789407177.10 total 856.03 last 12.80) - open/close 18047 (time 1789407187.93 total 866.85 last 16.45) - open/close 18856 (time 1789407197.93 total 876.86 last 80.86) - open/close 19607 (time 1789407207.94 total 886.86 last 75.07) - open/close 20000 (time 1789407213.62 total 892.55 last 69.11) - open/close 20722 (time 1789407223.64 total 902.57 last 72.04) - open/close 21352 (time 1789407233.65 total 912.58 last 62.94) - open/close 22076 (time 1789407243.66 total 922.59 last 72.33) - open/close 22839 (time 1789407253.67 total 932.59 last 76.28) - open/close 23677 (time 1789407263.67 total 942.60 last 83.77) - open/close 24474 (time 1789407273.67 total 952.60 last 79.66) - open/close 25327 (time 1789407283.68 total 962.60 last 85.29) - open/close 26226 (time 1789407293.68 total 972.61 last 89.85) - open/close 27047 (time 1789407303.70 total 982.62 last 81.98) - open/close 27935 (time 1789407313.72 total 992.65 last 88.55) - open/close 28601 (time 1789407323.73 total 1002.65 last 66.59) - open/close 29334 (time 1789407333.73 total 1012.66 last 73.24) - open/close 29997 (time 1789407343.75 total 1022.67 last 66.20) - open/close 30711 (time 1789407353.76 total 1032.68 last 71.33) - open/close 31511 (time 1789407363.76 total 1042.69 last 79.97) - open/close 32372 (time 1789407373.76 total 1052.69 last 86.07) - open/close 33187 (time 1789407383.77 total 1062.70 last 81.46) - open/close 34033 (time 1789407393.78 total 1072.71 last 84.52) - open/close 34830 (time 1789407403.78 total 1082.71 last 79.65) - open/close 35634 (time 1789407413.80 total 1092.73 last 80.29) - open/close 36404 (time 1789407423.80 total 1102.73 last 76.99) - open/close 37168 (time 1789407433.82 total 1112.74 last 76.27) - open/close 37858 (time 1789407443.82 total 1122.75 last 68.99) - open/close 38667 (time 1789407453.82 total 1132.75 last 80.89) - open/close 39372 (time 1789407463.83 total 1142.75 last 70.46) - open/close 40000 (time 1789407471.92 total 1150.85 last 77.58) - open/close 40751 (time 1789407481.93 total 1160.85 last 75.06) - open/close 41472 (time 1789407491.94 total 1170.87 last 72.00) - open/close 42265 (time 1789407501.94 total 1180.87 last 79.30) - open/close 43008 (time 1789407511.94 total 1190.87 last 74.26) - open/close 43741 (time 1789407521.95 total 1200.87 last 73.29) - open/close 44541 (time 1789407531.95 total 1210.88 last 79.94) - open/close 45267 (time 1789407541.97 total 1220.90 last 72.46) - open/close 45992 (time 1789407551.98 total 1230.90 last 72.47) - open/close 46719 (time 1789407561.98 total 1240.91 last 72.68) - open/close 47397 (time 1789407572.01 total 1250.93 last 67.62) - open/close 48253 (time 1789407582.01 total 1260.94 last 85.58) - open/close 49016 (time 1789407592.02 total 1270.94 last 76.24) - open/close 49616 (time 1789407602.03 total 1280.95 last 59.95) total: 50000 open/close in 1285.99 seconds: 38.88 ops/second - unlinked 0 (time 1789407610 ; total 0 ; last 0) - unlinked 10000 (time 1789407746 ; total 136 ; last 136) - unlinked 20000 (time 1789407896 ; total 286 ; last 150) - unlinked 30000 (time 1789408033 ; total 423 ; last 137) - unlinked 40000 (time 1789408167 ; total 557 ; last 134) total: 50000 unlinks in 702 seconds: 71.225075 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server unloading modules via unload_modules_local on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: kmemleak disabled modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2