== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:52:53 (1790783573) start mds service on oleg615-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Sep 30 01:47 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing load_modules_local oleg615-server: kmemleak disabled oleg615-server: Loading modules from /home/green/git/lustre-release/lustre oleg615-server: detected 4 online CPUs by sysfs oleg615-server: MODOPTS_LIBCFS= oleg615-server: Force libcfs to create 2 CPU partitions oleg615-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg615-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg615-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg615-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg615-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0000 oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 69 (time 1790783664.88 total 10.14 last 6.80) - open/close 140 (time 1790783678.09 total 23.35 last 5.37) - open/close 288 (time 1790783689.35 total 34.61 last 13.15) - open/close 498 (time 1790783699.35 total 44.62 last 20.98) - open/close 579 (time 1790783709.36 total 54.63 last 8.09) - open/close 796 (time 1790783719.59 total 64.86 last 21.21) - open/close 982 (time 1790783732.72 total 77.98 last 14.17) - open/close 1120 (time 1790783742.72 total 87.98 last 13.80) - open/close 1224 (time 1790783752.74 total 98.00 last 10.38) - open/close 1479 (time 1790783763.70 total 108.96 last 23.26) - open/close 1706 (time 1790783774.39 total 119.66 last 21.23) - open/close 1925 (time 1790783784.41 total 129.67 last 21.87) - open/close 2114 (time 1790783795.05 total 140.31 last 17.76) - open/close 2421 (time 1790783806.12 total 151.38 last 27.73) - open/close 2633 (time 1790783817.23 total 162.50 last 19.08) - open/close 2932 (time 1790783828.20 total 173.46 last 27.27) - open/close 3200 (time 1790783838.30 total 183.57 last 26.52) - open/close 3457 (time 1790783849.01 total 194.28 last 24.00) - open/close 3660 (time 1790783859.02 total 204.29 last 20.28) - open/close 3870 (time 1790783869.03 total 214.30 last 20.98) - open/close 4052 (time 1790783879.99 total 225.26 last 16.61) - open/close 4315 (time 1790783890.14 total 235.41 last 25.91) - open/close 4563 (time 1790783900.78 total 246.04 last 23.31) - open/close 4750 (time 1790783911.74 total 257.00 last 17.06) - open/close 4948 (time 1790783921.75 total 267.01 last 19.78) - open/close 5121 (time 1790783932.53 total 277.80 last 16.04) - open/close 5365 (time 1790783942.56 total 287.82 last 24.34) - open/close 5573 (time 1790783952.68 total 297.94 last 20.56) - open/close 5843 (time 1790783963.39 total 308.65 last 25.20) - open/close 6093 (time 1790783974.08 total 319.34 last 23.38) - open/close 6357 (time 1790783984.94 total 330.21 last 24.30) - open/close 6546 (time 1790783995.62 total 340.89 last 17.70) - open/close 6808 (time 1790784006.38 total 351.64 last 24.35) - open/close 6960 (time 1790784016.94 total 362.21 last 14.39) - open/close 7057 (time 1790784027.43 total 372.69 last 9.25) - open/close 7206 (time 1790784037.94 total 383.20 last 14.18) - open/close 7404 (time 1790784047.94 total 393.21 last 19.79) - open/close 7635 (time 1790784058.16 total 403.43 last 22.60) - open/close 7852 (time 1790784068.35 total 413.61 last 21.31) - open/close 8047 (time 1790784078.45 total 423.71 last 19.31) - open/close 8222 (time 1790784088.59 total 433.86 last 17.25) - open/close 8366 (time 1790784098.61 total 443.87 last 14.38) - open/close 8577 (time 1790784110.77 total 456.03 last 17.35) - open/close 8782 (time 1790784120.77 total 466.03 last 20.50) - open/close 9010 (time 1790784131.30 total 476.56 last 21.65) - open/close 9192 (time 1790784141.36 total 486.62 last 18.09) - open/close 9343 (time 1790784151.36 total 496.63 last 15.10) - open/close 9576 (time 1790784161.95 total 507.22 last 22.01) - open/close 9697 (time 1790784172.71 total 517.98 last 11.24) - open/close 9852 (time 1790784183.72 total 528.98 last 14.08) - open/close 9934 (time 1790784193.73 total 538.99 last 8.19) - open/close 10000 (time 1790784196.04 total 541.30 last 28.60) - open/close 10120 (time 1790784206.95 total 552.21 last 11.00) - open/close 10363 (time 1790784216.95 total 562.22 last 24.29) - open/close 10597 (time 1790784228.00 total 573.26 last 21.18) - open/close 10844 (time 1790784238.39 total 583.65 last 23.77) - open/close 10968 (time 1790784248.39 total 593.66 last 12.40) - open/close 11069 (time 1790784258.86 total 604.13 last 9.65) - open/close 11251 (time 1790784268.92 total 614.18 last 18.10) - open/close 11470 (time 1790784278.92 total 624.19 last 21.89) - open/close 11659 (time 1790784290.53 total 635.80 last 16.28) - open/close 11789 (time 1790784302.44 total 647.70 last 10.92) - open/close 12051 (time 1790784312.45 total 657.71 last 26.18) - open/close 12287 (time 1790784322.46 total 667.73 last 23.56) - open/close 12541 (time 1790784332.47 total 677.73 last 25.39) - open/close 12771 (time 1790784343.16 total 688.42 last 21.52) - open/close 12961 (time 1790784354.03 total 699.30 last 17.47) - open/close 13144 (time 1790784364.79 total 710.05 last 17.01) - open/close 13242 (time 1790784375.50 total 720.76 last 9.15) - open/close 13298 (time 1790784388.84 total 734.11 last 4.20) - open/close 13443 (time 1790784399.91 total 745.18 last 13.10) - open/close 13612 (time 1790784410.10 total 755.36 last 16.59) - open/close 13845 (time 1790784420.71 total 765.98 last 21.95) - open/close 14078 (time 1790784431.14 total 776.41 last 22.34) - open/close 14277 (time 1790784441.20 total 786.47 last 19.78) - open/close 14477 (time 1790784451.21 total 796.48 last 19.98) - open/close 14678 (time 1790784462.12 total 807.38 last 18.43) - open/close 14819 (time 1790784472.49 total 817.75 last 13.59) - open/close 14919 (time 1790784483.06 total 828.32 last 9.46) - open/close 15013 (time 1790784493.73 total 839.00 last 8.81) - open/close 15181 (time 1790784504.61 total 849.87 last 15.44) - open/close 15278 (time 1790784514.62 total 859.88 last 9.69) - open/close 15397 (time 1790784524.62 total 869.89 last 11.89) - open/close 15522 (time 1790784534.66 total 879.92 last 12.46) - open/close 15694 (time 1790784545.00 total 890.26 last 16.63) - open/close 15853 (time 1790784555.02 total 900.28 last 15.88) - open/close 16017 (time 1790784565.56 total 910.82 last 15.56) - open/close 16150 (time 1790784575.57 total 920.83 last 13.29) - open/close 16361 (time 1790784585.58 total 930.85 last 21.06) - open/close 16604 (time 1790784598.11 total 943.37 last 19.40) - open/close 16858 (time 1790784608.63 total 953.89 last 24.14) - open/close 17083 (time 1790784621.18 total 966.44 last 17.93) - open/close 17323 (time 1790784631.85 total 977.11 last 22.50) - open/close 17584 (time 1790784641.89 total 987.15 last 26.00) - open/close 17801 (time 1790784652.23 total 997.49 last 20.99) - open/close 17921 (time 1790784662.75 total 1008.02 last 11.40) - open/close 18209 (time 1790784672.76 total 1018.03 last 28.77) - open/close 18721 (time 1790784682.78 total 1028.04 last 51.11) - open/close 19406 (time 1790784692.79 total 1038.06 last 68.41) - open/close 20000 (time 1790784701.47 total 1046.73 last 68.46) - open/close 20772 (time 1790784711.47 total 1056.74 last 77.17) - open/close 21463 (time 1790784721.48 total 1066.75 last 69.04) - open/close 22084 (time 1790784731.49 total 1076.75 last 62.05) - open/close 22777 (time 1790784741.50 total 1086.76 last 69.27) - open/close 23582 (time 1790784751.50 total 1096.76 last 80.47) - open/close 24310 (time 1790784761.50 total 1106.76 last 72.80) - open/close 25068 (time 1790784771.51 total 1116.78 last 75.69) - open/close 25794 (time 1790784781.52 total 1126.78 last 72.57) - open/close 26508 (time 1790784791.52 total 1136.79 last 71.36) - open/close 27251 (time 1790784801.54 total 1146.80 last 74.21) - open/close 27970 (time 1790784811.54 total 1156.80 last 71.89) - open/close 28756 (time 1790784821.54 total 1166.81 last 78.54) - open/close 29458 (time 1790784831.55 total 1176.81 last 70.15) - open/close 30000 (time 1790784838.43 total 1183.69 last 78.80) - open/close 30756 (time 1790784848.44 total 1193.70 last 75.51) - open/close 31403 (time 1790784858.45 total 1203.72 last 64.62) - open/close 32077 (time 1790784868.45 total 1213.72 last 67.39) - open/close 32824 (time 1790784878.46 total 1223.72 last 74.68) - open/close 33537 (time 1790784888.48 total 1233.74 last 71.14) - open/close 34326 (time 1790784898.49 total 1243.75 last 78.85) - open/close 35014 (time 1790784908.49 total 1253.75 last 68.78) - open/close 35816 (time 1790784918.49 total 1263.76 last 80.18) - open/close 36472 (time 1790784928.50 total 1273.76 last 65.58) - open/close 37161 (time 1790784938.51 total 1283.77 last 68.80) - open/close 37866 (time 1790784948.51 total 1293.78 last 70.48) - open/close 38689 (time 1790784958.52 total 1303.78 last 82.28) - open/close 39495 (time 1790784968.52 total 1313.78 last 80.57) - open/close 40000 (time 1790784975.20 total 1320.46 last 75.63) - open/close 40722 (time 1790784985.20 total 1330.47 last 72.15) - open/close 41489 (time 1790784995.21 total 1340.48 last 76.63) - open/close 42239 (time 1790785005.23 total 1350.49 last 74.88) - open/close 42998 (time 1790785015.23 total 1360.50 last 75.85) - open/close 43714 (time 1790785025.24 total 1370.51 last 71.53) - open/close 44399 (time 1790785035.24 total 1380.51 last 68.49) - open/close 45129 (time 1790785045.24 total 1390.51 last 72.99) - open/close 45808 (time 1790785055.25 total 1400.52 last 67.84) - open/close 46573 (time 1790785065.26 total 1410.52 last 76.48) - open/close 47361 (time 1790785075.26 total 1420.53 last 78.75) - open/close 48116 (time 1790785085.27 total 1430.53 last 75.45) - open/close 48880 (time 1790785095.28 total 1440.55 last 76.30) - open/close 49648 (time 1790785105.30 total 1450.56 last 76.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No space left on device total: 49729 open/close in 1451.80 seconds: 34.25 ops/second - unlinked 0 (time 1790785110 ; total 0 ; last 0) - unlinked 10000 (time 1790785249 ; total 139 ; last 139) - unlinked 20000 (time 1790785382 ; total 272 ; last 133) - unlinked 30000 (time 1790785528 ; total 418 ; last 146) - unlinked 40000 (time 1790785671 ; total 561 ; last 143) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No such file or directory total: 49729 unlinks in 699 seconds: 71.143059 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg615-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg615-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg615-server stop mds service on oleg615-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg615-server unloading modules via unload_modules_local on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing unload_modules_local oleg615-server: kmemleak disabled modules unloaded. pdsh@oleg615-client: oleg615-client: ssh exited with exit code 2 pdsh@oleg615-client: oleg615-server: ssh exited with exit code 2 pdsh@oleg615-client: oleg615-client: ssh exited with exit code 2 pdsh@oleg615-client: oleg615-server: ssh exited with exit code 2