== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:25:39 (1788528339) start mds service on oleg101-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1933656 Sep 3 13:58 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: kmemleak disabled oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: MODOPTS_LIBCFS= oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg101-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg101-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg101-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1788528479.25 total 10.37 last 17.46) - open/close 451 (time 1788528489.25 total 20.37 last 26.99) - open/close 597 (time 1788528499.67 total 30.79 last 14.01) - open/close 796 (time 1788528510.31 total 41.43 last 18.70) - open/close 964 (time 1788528520.68 total 51.79 last 16.21) - open/close 1201 (time 1788528531.43 total 62.55 last 22.04) - open/close 1350 (time 1788528541.57 total 72.69 last 14.69) - open/close 1510 (time 1788528552.36 total 83.48 last 14.83) - open/close 1770 (time 1788528563.29 total 94.41 last 23.79) - open/close 1977 (time 1788528574.32 total 105.43 last 18.78) - open/close 2108 (time 1788528584.47 total 115.59 last 12.90) - open/close 2151 (time 1788528594.65 total 125.77 last 4.22) - open/close 2307 (time 1788528604.66 total 135.77 last 15.59) - open/close 2452 (time 1788528617.02 total 148.13 last 11.73) - open/close 2694 (time 1788528628.08 total 159.20 last 21.87) - open/close 2773 (time 1788528639.13 total 170.24 last 7.15) - open/close 2870 (time 1788528649.75 total 180.87 last 9.13) - open/close 3052 (time 1788528659.76 total 190.87 last 18.19) - open/close 3278 (time 1788528671.71 total 202.83 last 18.91) - open/close 3361 (time 1788528682.17 total 213.29 last 7.93) - open/close 3550 (time 1788528692.63 total 223.75 last 18.06) - open/close 3634 (time 1788528702.81 total 233.92 last 8.26) - open/close 3760 (time 1788528712.81 total 243.93 last 12.60) - open/close 3990 (time 1788528723.81 total 254.93 last 20.90) - open/close 4200 (time 1788528734.24 total 265.36 last 20.13) - open/close 4286 (time 1788528744.45 total 275.56 last 8.43) - open/close 4394 (time 1788528754.47 total 285.58 last 10.78) - open/close 4419 (time 1788528764.64 total 295.75 last 2.46) - open/close 4559 (time 1788528774.65 total 305.77 last 13.98) - open/close 4697 (time 1788528785.00 total 316.12 last 13.33) - open/close 4784 (time 1788528795.31 total 326.43 last 8.44) - open/close 4956 (time 1788528805.82 total 336.94 last 16.36) - open/close 5025 (time 1788528815.84 total 346.95 last 6.89) - open/close 5227 (time 1788528826.80 total 357.92 last 18.43) - open/close 5422 (time 1788528837.44 total 368.56 last 18.32) - open/close 5574 (time 1788528847.45 total 378.57 last 15.18) - open/close 5701 (time 1788528858.20 total 389.32 last 11.81) - open/close 5831 (time 1788528868.83 total 399.95 last 12.23) - open/close 6000 (time 1788528878.84 total 409.95 last 16.89) - open/close 6088 (time 1788528890.38 total 421.50 last 7.62) - open/close 6151 (time 1788528903.47 total 434.59 last 4.81) - open/close 6271 (time 1788528914.04 total 445.16 last 11.35) - open/close 6401 (time 1788528924.55 total 455.67 last 12.37) - open/close 6441 (time 1788528935.10 total 466.21 last 3.79) - open/close 6603 (time 1788528945.65 total 476.77 last 15.34) - open/close 6768 (time 1788528955.68 total 486.79 last 16.47) - open/close 6996 (time 1788528967.43 total 498.54 last 19.40) - open/close 7133 (time 1788528977.79 total 508.91 last 13.22) - open/close 7237 (time 1788528988.15 total 519.26 last 10.04) - open/close 7340 (time 1788528998.18 total 529.29 last 10.27) - open/close 7499 (time 1788529008.19 total 539.31 last 15.88) - open/close 7624 (time 1788529018.91 total 550.03 last 11.66) - open/close 7803 (time 1788529029.94 total 561.06 last 16.23) - open/close 7849 (time 1788529040.44 total 571.56 last 4.38) - open/close 8099 (time 1788529050.45 total 581.56 last 24.99) - open/close 8382 (time 1788529061.07 total 592.19 last 26.65) - open/close 8486 (time 1788529071.08 total 602.20 last 10.39) - open/close 8622 (time 1788529081.83 total 612.95 last 12.65) - open/close 8832 (time 1788529092.44 total 623.55 last 19.79) - open/close 9002 (time 1788529102.83 total 633.95 last 16.35) - open/close 9096 (time 1788529113.64 total 644.76 last 8.70) - open/close 9170 (time 1788529124.06 total 655.18 last 7.10) - open/close 9287 (time 1788529134.38 total 665.50 last 11.34) - open/close 9468 (time 1788529144.40 total 675.51 last 18.08) - open/close 9586 (time 1788529155.31 total 686.43 last 10.81) - open/close 9643 (time 1788529166.19 total 697.30 last 5.24) - open/close 9776 (time 1788529176.64 total 707.75 last 12.73) - open/close 9943 (time 1788529186.79 total 717.90 last 16.45) - open/close 10000 (time 1788529196.46 total 727.58 last 5.89) - open/close 10136 (time 1788529206.88 total 738.00 last 13.05) - open/close 10253 (time 1788529216.95 total 748.07 last 11.63) - open/close 10341 (time 1788529227.61 total 758.73 last 8.25) - open/close 10470 (time 1788529238.18 total 769.30 last 12.20) - open/close 10569 (time 1788529248.93 total 780.05 last 9.21) - open/close 10641 (time 1788529259.16 total 790.28 last 7.04) - open/close 10797 (time 1788529269.87 total 800.98 last 14.57) - open/close 10926 (time 1788529280.37 total 811.48 last 12.29) - open/close 11034 (time 1788529291.17 total 822.29 last 9.99) - open/close 11207 (time 1788529302.24 total 833.35 last 15.64) - open/close 11275 (time 1788529312.42 total 843.54 last 6.68) - open/close 11476 (time 1788529323.00 total 854.12 last 19.00) - open/close 11551 (time 1788529333.99 total 865.11 last 6.82) - open/close 11786 (time 1788529344.21 total 875.33 last 22.98) - open/close 12070 (time 1788529355.19 total 886.31 last 25.88) - open/close 12193 (time 1788529365.20 total 896.31 last 12.29) - open/close 12339 (time 1788529375.22 total 906.34 last 14.57) - open/close 12460 (time 1788529385.88 total 916.99 last 11.35) - open/close 12660 (time 1788529396.88 total 928.00 last 18.17) - open/close 12752 (time 1788529407.04 total 938.16 last 9.06) - open/close 12919 (time 1788529417.18 total 948.30 last 16.46) - open/close 13079 (time 1788529430.01 total 961.12 last 12.48) - open/close 13157 (time 1788529440.09 total 971.21 last 7.73) - open/close 13268 (time 1788529450.10 total 981.21 last 11.09) - open/close 13534 (time 1788529460.11 total 991.23 last 26.56) - open/close 13667 (time 1788529471.01 total 1002.12 last 12.21) - open/close 13869 (time 1788529481.89 total 1013.01 last 18.56) - open/close 14042 (time 1788529491.93 total 1023.05 last 17.22) - open/close 14195 (time 1788529501.94 total 1033.06 last 15.29) - open/close 14317 (time 1788529511.96 total 1043.08 last 12.18) - open/close 14452 (time 1788529522.00 total 1053.12 last 13.45) - open/close 14572 (time 1788529532.00 total 1063.12 last 11.99) - open/close 14745 (time 1788529542.02 total 1073.13 last 17.28) - open/close 14938 (time 1788529552.60 total 1083.72 last 18.23) - open/close 15041 (time 1788529562.63 total 1093.75 last 10.27) - open/close 15124 (time 1788529573.26 total 1104.38 last 7.81) - open/close 15177 (time 1788529583.96 total 1115.08 last 4.95) - open/close 15289 (time 1788529593.97 total 1125.09 last 11.19) - open/close 15457 (time 1788529603.99 total 1135.11 last 16.77) - open/close 15725 (time 1788529614.00 total 1145.12 last 26.78) - open/close 16344 (time 1788529624.01 total 1155.13 last 61.83) - open/close 17037 (time 1788529634.02 total 1165.14 last 69.26) - open/close 17638 (time 1788529644.03 total 1175.14 last 60.06) - open/close 18260 (time 1788529654.03 total 1185.15 last 62.17) - open/close 18764 (time 1788529664.04 total 1195.15 last 50.37) - open/close 19392 (time 1788529674.05 total 1205.17 last 62.73) - open/close 20000 (time 1788529682.14 total 1213.26 last 75.10) - open/close 20625 (time 1788529692.18 total 1223.30 last 62.29) - open/close 21189 (time 1788529702.19 total 1233.31 last 56.34) - open/close 21718 (time 1788529712.19 total 1243.31 last 52.87) - open/close 22462 (time 1788529722.20 total 1253.31 last 74.39) - open/close 23167 (time 1788529732.20 total 1263.32 last 70.45) - open/close 23969 (time 1788529742.21 total 1273.33 last 80.10) - open/close 24576 (time 1788529752.24 total 1283.36 last 60.55) - open/close 25259 (time 1788529762.24 total 1293.36 last 68.29) - open/close 26078 (time 1788529772.24 total 1303.36 last 81.89) - open/close 26958 (time 1788529782.25 total 1313.37 last 87.94) - open/close 27744 (time 1788529792.26 total 1323.38 last 78.50) - open/close 28235 (time 1788529802.27 total 1333.39 last 49.05) - open/close 28784 (time 1788529812.37 total 1343.49 last 54.36) - open/close 29419 (time 1788529822.40 total 1353.52 last 63.32) - open/close 29937 (time 1788529832.40 total 1363.52 last 51.79) - open/close 30542 (time 1788529842.40 total 1373.52 last 60.49) - open/close 31284 (time 1788529852.42 total 1383.53 last 74.11) - open/close 31985 (time 1788529862.43 total 1393.54 last 70.03) - open/close 32595 (time 1788529872.43 total 1403.55 last 60.96) - open/close 33193 (time 1788529882.44 total 1413.56 last 59.76) - open/close 33926 (time 1788529892.44 total 1423.56 last 73.28) - open/close 34616 (time 1788529902.48 total 1433.60 last 68.72) - open/close 35333 (time 1788529912.50 total 1443.62 last 71.57) - open/close 36047 (time 1788529922.50 total 1453.62 last 71.40) - open/close 36574 (time 1788529932.50 total 1463.62 last 52.69) - open/close 37020 (time 1788529942.51 total 1473.63 last 44.57) - open/close 37557 (time 1788529952.53 total 1483.65 last 53.60) - open/close 38248 (time 1788529962.53 total 1493.65 last 69.10) - open/close 38860 (time 1788529972.53 total 1503.65 last 61.19) - open/close 39716 (time 1788529982.53 total 1513.65 last 85.59) - open/close 40000 (time 1788529986.32 total 1517.44 last 74.98) - open/close 40684 (time 1788529996.32 total 1527.44 last 68.38) - open/close 41258 (time 1788530006.34 total 1537.45 last 57.33) - open/close 41973 (time 1788530016.35 total 1547.47 last 71.40) - open/close 42740 (time 1788530026.35 total 1557.47 last 76.66) - open/close 43525 (time 1788530036.36 total 1567.48 last 78.45) - open/close 44070 (time 1788530046.40 total 1577.52 last 54.30) - open/close 44783 (time 1788530056.40 total 1587.52 last 71.26) - open/close 45448 (time 1788530066.41 total 1597.53 last 66.43) - open/close 46130 (time 1788530076.42 total 1607.54 last 68.13) - open/close 46765 (time 1788530086.43 total 1617.54 last 63.48) - open/close 47518 (time 1788530096.44 total 1627.55 last 75.23) - open/close 48240 (time 1788530106.44 total 1637.56 last 72.14) - open/close 48841 (time 1788530116.45 total 1647.57 last 60.07) - open/close 49416 (time 1788530126.45 total 1657.57 last 57.48) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1661.32 seconds: 29.87 ops/second - unlinked 0 (time 1788530136 ; total 0 ; last 0) - unlinked 10000 (time 1788530287 ; total 151 ; last 151) - unlinked 20000 (time 1788530426 ; total 290 ; last 139) - unlinked 30000 (time 1788530574 ; total 438 ; last 148) - unlinked 40000 (time 1788530702 ; total 566 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 690 seconds: 71.928986 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg101-server unloading modules via unload_modules_local on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local oleg101-server: kmemleak disabled modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2