== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:20:54 (1790976054) start mds service on oleg220-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 2 13:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: kmemleak disabled oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions oleg220-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg220-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg220-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg220-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg220-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 215 (time 1790976171.72 total 10.01 last 21.49) - open/close 422 (time 1790976181.72 total 20.01 last 20.69) - open/close 607 (time 1790976191.75 total 30.04 last 18.45) - open/close 743 (time 1790976202.83 total 41.12 last 12.27) - open/close 811 (time 1790976212.91 total 51.20 last 6.75) - open/close 949 (time 1790976223.19 total 61.47 last 13.43) - open/close 1117 (time 1790976233.84 total 72.12 last 15.78) - open/close 1299 (time 1790976244.28 total 82.57 last 17.42) - open/close 1512 (time 1790976254.28 total 92.57 last 21.30) - open/close 1772 (time 1790976264.32 total 102.61 last 25.91) - open/close 1939 (time 1790976274.32 total 112.61 last 16.70) - open/close 2163 (time 1790976284.66 total 122.94 last 21.67) - open/close 2278 (time 1790976294.68 total 132.97 last 11.47) - open/close 2550 (time 1790976305.48 total 143.77 last 25.19) - open/close 2837 (time 1790976315.48 total 153.77 last 28.70) - open/close 3078 (time 1790976325.49 total 163.78 last 24.09) - open/close 3280 (time 1790976335.51 total 173.80 last 20.15) - open/close 3442 (time 1790976346.25 total 184.54 last 15.09) - open/close 3566 (time 1790976357.26 total 195.55 last 11.26) - open/close 3833 (time 1790976368.13 total 206.42 last 24.55) - open/close 4018 (time 1790976378.74 total 217.03 last 17.44) - open/close 4251 (time 1790976388.76 total 227.05 last 23.26) - open/close 4413 (time 1790976399.70 total 237.99 last 14.81) - open/close 4552 (time 1790976410.61 total 248.90 last 12.74) - open/close 4720 (time 1790976421.27 total 259.56 last 15.76) - open/close 4980 (time 1790976432.24 total 270.53 last 23.70) - open/close 5266 (time 1790976442.24 total 280.53 last 28.59) - open/close 5438 (time 1790976452.40 total 290.69 last 16.93) - open/close 5558 (time 1790976462.91 total 301.20 last 11.42) - open/close 5881 (time 1790976473.47 total 311.75 last 30.59) - open/close 6132 (time 1790976483.47 total 321.76 last 25.10) - open/close 6420 (time 1790976493.47 total 331.76 last 28.78) - open/close 6698 (time 1790976503.68 total 341.97 last 27.23) - open/close 6910 (time 1790976514.43 total 352.71 last 19.73) - open/close 7153 (time 1790976524.45 total 362.74 last 24.25) - open/close 7393 (time 1790976534.47 total 372.76 last 23.95) - open/close 7687 (time 1790976544.47 total 382.76 last 29.40) - open/close 7949 (time 1790976554.89 total 393.18 last 25.14) - open/close 8204 (time 1790976565.37 total 403.66 last 24.34) - open/close 8488 (time 1790976575.40 total 413.69 last 28.32) - open/close 8690 (time 1790976586.43 total 424.72 last 18.30) - open/close 8821 (time 1790976596.44 total 434.73 last 13.09) - open/close 9077 (time 1790976607.03 total 445.32 last 24.17) - open/close 9330 (time 1790976617.65 total 455.94 last 23.83) - open/close 9615 (time 1790976627.65 total 465.94 last 28.50) - open/close 9919 (time 1790976638.26 total 476.55 last 28.66) - open/close 10000 (time 1790976646.50 total 484.79 last 9.83) - open/close 10216 (time 1790976656.88 total 495.17 last 20.81) - open/close 10442 (time 1790976669.05 total 507.34 last 18.57) - open/close 10608 (time 1790976679.29 total 517.58 last 16.21) - open/close 10791 (time 1790976689.72 total 528.01 last 17.54) - open/close 10978 (time 1790976699.76 total 538.05 last 18.62) - open/close 11054 (time 1790976710.57 total 548.86 last 7.03) - open/close 11173 (time 1790976721.54 total 559.83 last 10.85) - open/close 11382 (time 1790976731.75 total 570.04 last 20.47) - open/close 11412 (time 1790976741.77 total 580.06 last 3.00) - open/close 11634 (time 1790976752.39 total 590.68 last 20.89) - open/close 11831 (time 1790976762.93 total 601.22 last 18.69) - open/close 11979 (time 1790976773.00 total 611.29 last 14.70) - open/close 12177 (time 1790976783.97 total 622.26 last 18.05) - open/close 12387 (time 1790976794.17 total 632.46 last 20.60) - open/close 12576 (time 1790976806.58 total 644.87 last 15.22) - open/close 12830 (time 1790976817.60 total 655.89 last 23.06) - open/close 13107 (time 1790976827.60 total 665.89 last 27.69) - open/close 13368 (time 1790976837.61 total 675.90 last 26.08) - open/close 13573 (time 1790976847.93 total 686.22 last 19.86) - open/close 13866 (time 1790976857.95 total 696.24 last 29.25) - open/close 14140 (time 1790976868.21 total 706.50 last 26.71) - open/close 14277 (time 1790976878.76 total 717.05 last 12.99) - open/close 14398 (time 1790976889.61 total 727.90 last 11.15) - open/close 14483 (time 1790976899.63 total 737.92 last 8.48) - open/close 14705 (time 1790976910.39 total 748.68 last 20.62) - open/close 14967 (time 1790976921.08 total 759.37 last 24.51) - open/close 15140 (time 1790976933.08 total 771.37 last 14.42) - open/close 15266 (time 1790976943.77 total 782.06 last 11.79) - open/close 15522 (time 1790976954.54 total 792.83 last 23.77) - open/close 15748 (time 1790976965.27 total 803.56 last 21.06) - open/close 15926 (time 1790976975.94 total 814.22 last 16.69) - open/close 16107 (time 1790976986.12 total 824.41 last 17.77) - open/close 16289 (time 1790976996.29 total 834.58 last 17.89) - open/close 16377 (time 1790977008.14 total 846.43 last 7.43) - open/close 16537 (time 1790977018.17 total 856.46 last 15.96) - open/close 16708 (time 1790977028.42 total 866.71 last 16.67) - open/close 16844 (time 1790977038.43 total 876.72 last 13.60) - open/close 16880 (time 1790977048.64 total 886.92 last 3.53) - open/close 17064 (time 1790977060.79 total 899.07 last 15.14) - open/close 17320 (time 1790977070.79 total 909.08 last 25.59) - open/close 17580 (time 1790977083.64 total 921.93 last 20.23) - open/close 17764 (time 1790977093.66 total 931.95 last 18.37) - open/close 17974 (time 1790977103.68 total 941.97 last 20.95) - open/close 18195 (time 1790977114.37 total 952.66 last 20.69) - open/close 18394 (time 1790977124.37 total 962.66 last 19.89) - open/close 18666 (time 1790977134.59 total 972.88 last 26.61) - open/close 18817 (time 1790977145.35 total 983.64 last 14.04) - open/close 18997 (time 1790977156.35 total 994.64 last 16.36) - open/close 19080 (time 1790977166.37 total 1004.66 last 8.28) - open/close 19229 (time 1790977177.00 total 1015.29 last 14.02) - open/close 19501 (time 1790977187.27 total 1025.55 last 26.49) - open/close 19562 (time 1790977197.90 total 1036.19 last 5.74) - open/close 20000 (time 1790977202.18 total 1040.47 last 102.27) - open/close 20956 (time 1790977212.19 total 1050.48 last 95.50) - open/close 21832 (time 1790977222.23 total 1060.52 last 87.31) - open/close 22614 (time 1790977232.23 total 1070.52 last 78.16) - open/close 23377 (time 1790977242.23 total 1080.52 last 76.30) - open/close 24279 (time 1790977252.24 total 1090.53 last 90.14) - open/close 25200 (time 1790977262.25 total 1100.54 last 92.00) - open/close 25984 (time 1790977272.26 total 1110.55 last 78.34) - open/close 26785 (time 1790977282.27 total 1120.56 last 80.02) - open/close 27561 (time 1790977292.28 total 1130.57 last 77.51) - open/close 28471 (time 1790977302.29 total 1140.58 last 90.93) - open/close 29250 (time 1790977312.30 total 1150.59 last 77.77) - open/close 30000 (time 1790977320.31 total 1158.60 last 93.64) - open/close 30805 (time 1790977330.32 total 1168.61 last 80.45) - open/close 31436 (time 1790977340.35 total 1178.63 last 62.93) - open/close 32132 (time 1790977350.36 total 1188.65 last 69.50) - open/close 32764 (time 1790977360.37 total 1198.66 last 63.14) - open/close 33558 (time 1790977370.37 total 1208.66 last 79.37) - open/close 34247 (time 1790977380.38 total 1218.67 last 68.84) - open/close 35004 (time 1790977390.38 total 1228.67 last 75.69) - open/close 35792 (time 1790977400.40 total 1238.69 last 78.69) - open/close 36590 (time 1790977410.41 total 1248.70 last 79.69) - open/close 37177 (time 1790977420.44 total 1258.73 last 58.53) - open/close 37862 (time 1790977430.44 total 1268.73 last 68.50) - open/close 38738 (time 1790977440.45 total 1278.74 last 87.53) - open/close 39559 (time 1790977450.45 total 1288.74 last 82.09) - open/close 40000 (time 1790977455.34 total 1293.63 last 90.12) - open/close 40942 (time 1790977465.34 total 1303.63 last 94.17) - open/close 41935 (time 1790977475.36 total 1313.65 last 99.13) - open/close 42657 (time 1790977485.37 total 1323.65 last 72.17) - open/close 43501 (time 1790977495.37 total 1333.66 last 84.36) - open/close 44295 (time 1790977505.37 total 1343.66 last 79.38) - open/close 45237 (time 1790977515.38 total 1353.67 last 94.17) - open/close 46139 (time 1790977525.39 total 1363.68 last 90.04) - open/close 46838 (time 1790977535.40 total 1373.69 last 69.84) - open/close 47671 (time 1790977545.41 total 1383.69 last 83.28) - open/close 48478 (time 1790977555.41 total 1393.70 last 80.67) - open/close 49215 (time 1790977565.41 total 1403.70 last 73.67) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1409.29 seconds: 35.22 ops/second - unlinked 0 (time 1790977575 ; total 0 ; last 0) - unlinked 10000 (time 1790977702 ; total 127 ; last 127) - unlinked 20000 (time 1790977825 ; total 250 ; last 123) - unlinked 30000 (time 1790977951 ; total 376 ; last 126) - unlinked 40000 (time 1790978079 ; total 504 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 622 seconds: 79.792603 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg220-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server unloading modules via unload_modules_local on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing unload_modules_local oleg220-server: kmemleak disabled modules unloaded. pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2