== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:00:46 (1788393646) start mds service on oleg345-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 2 18:43 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: kmemleak disabled oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg345-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg345-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg345-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg345-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 273 (time 1788393747.82 total 10.38 last 26.29) - open/close 582 (time 1788393758.62 total 21.18 last 28.61) - open/close 856 (time 1788393769.06 total 31.62 last 26.25) - open/close 1068 (time 1788393779.76 total 42.32 last 19.81) - open/close 1370 (time 1788393789.76 total 52.33 last 30.18) - open/close 1628 (time 1788393800.41 total 62.98 last 24.22) - open/close 1776 (time 1788393813.42 total 75.98 last 11.38) - open/close 2066 (time 1788393823.42 total 85.98 last 28.99) - open/close 2327 (time 1788393835.56 total 98.12 last 21.50) - open/close 2480 (time 1788393845.58 total 108.14 last 15.27) - open/close 2690 (time 1788393855.58 total 118.14 last 20.99) - open/close 2992 (time 1788393865.60 total 128.16 last 30.14) - open/close 3260 (time 1788393876.17 total 138.73 last 25.36) - open/close 3453 (time 1788393886.95 total 149.51 last 17.90) - open/close 3594 (time 1788393896.96 total 159.52 last 14.09) - open/close 3868 (time 1788393908.02 total 170.58 last 24.78) - open/close 4008 (time 1788393919.80 total 182.36 last 11.88) - open/close 4265 (time 1788393930.22 total 192.78 last 24.66) - open/close 4445 (time 1788393940.23 total 202.79 last 17.98) - open/close 4673 (time 1788393950.25 total 212.81 last 22.76) - open/close 4931 (time 1788393960.25 total 222.82 last 25.78) - open/close 5138 (time 1788393971.55 total 234.12 last 18.32) - open/close 5430 (time 1788393981.56 total 244.12 last 29.19) - open/close 5719 (time 1788393992.56 total 255.12 last 26.27) - open/close 5896 (time 1788394002.90 total 265.46 last 17.12) - open/close 6066 (time 1788394014.22 total 276.78 last 15.02) - open/close 6310 (time 1788394024.67 total 287.24 last 23.34) - open/close 6525 (time 1788394034.74 total 297.31 last 21.35) - open/close 6646 (time 1788394045.29 total 307.86 last 11.47) - open/close 6923 (time 1788394055.30 total 317.86 last 27.69) - open/close 7091 (time 1788394065.35 total 327.91 last 16.72) - open/close 7255 (time 1788394076.07 total 338.64 last 15.29) - open/close 7484 (time 1788394086.44 total 349.01 last 22.08) - open/close 7717 (time 1788394097.51 total 360.08 last 21.05) - open/close 8053 (time 1788394108.02 total 370.58 last 31.99) - open/close 8287 (time 1788394119.08 total 381.65 last 21.14) - open/close 8520 (time 1788394129.27 total 391.84 last 22.86) - open/close 8814 (time 1788394140.21 total 402.77 last 26.88) - open/close 9087 (time 1788394150.21 total 412.77 last 27.30) - open/close 9369 (time 1788394162.86 total 425.42 last 22.30) - open/close 9669 (time 1788394173.87 total 436.43 last 27.24) - open/close 9869 (time 1788394184.24 total 446.80 last 19.29) - open/close 9982 (time 1788394194.28 total 456.84 last 11.25) - open/close 10238 (time 1788394204.77 total 467.33 last 24.40) - open/close 10448 (time 1788394215.15 total 477.71 last 20.24) - open/close 10757 (time 1788394225.82 total 488.38 last 28.95) - open/close 11047 (time 1788394236.49 total 499.06 last 27.17) - open/close 11302 (time 1788394246.54 total 509.11 last 25.37) - open/close 11494 (time 1788394256.78 total 519.35 last 18.75) - open/close 11766 (time 1788394267.45 total 530.01 last 25.51) - open/close 11978 (time 1788394278.05 total 540.61 last 20.00) - open/close 12267 (time 1788394288.70 total 551.26 last 27.14) - open/close 12522 (time 1788394298.87 total 561.43 last 25.07) - open/close 12813 (time 1788394309.48 total 572.05 last 27.41) - open/close 13020 (time 1788394320.03 total 582.59 last 19.63) - open/close 13256 (time 1788394330.03 total 592.59 last 23.60) - open/close 13548 (time 1788394340.98 total 603.54 last 26.68) - open/close 13809 (time 1788394351.07 total 613.64 last 25.85) - open/close 14095 (time 1788394361.71 total 624.27 last 26.89) - open/close 14382 (time 1788394371.71 total 634.27 last 28.70) - open/close 14536 (time 1788394381.75 total 644.31 last 15.34) - open/close 14801 (time 1788394392.17 total 654.74 last 25.42) - open/close 15100 (time 1788394402.17 total 664.74 last 29.90) - open/close 15373 (time 1788394412.97 total 675.53 last 25.30) - open/close 15657 (time 1788394424.00 total 686.56 last 25.75) - open/close 15894 (time 1788394434.03 total 696.59 last 23.63) - open/close 16164 (time 1788394444.03 total 706.59 last 26.98) - open/close 16277 (time 1788394454.38 total 716.94 last 10.92) - open/close 16555 (time 1788394464.38 total 726.95 last 27.78) - open/close 16810 (time 1788394474.40 total 736.96 last 25.47) - open/close 16991 (time 1788394484.79 total 747.36 last 17.41) - open/close 17112 (time 1788394495.10 total 757.66 last 11.74) - open/close 17334 (time 1788394505.14 total 767.70 last 22.10) - open/close 17527 (time 1788394515.19 total 777.75 last 19.21) - open/close 17560 (time 1788394525.31 total 787.87 last 3.26) - open/close 17722 (time 1788394535.32 total 797.89 last 16.17) - open/close 17960 (time 1788394545.46 total 808.03 last 23.48) - open/close 18196 (time 1788394556.26 total 818.82 last 21.87) - open/close 18464 (time 1788394566.76 total 829.33 last 25.51) - open/close 18727 (time 1788394576.77 total 839.33 last 26.28) - open/close 18966 (time 1788394586.78 total 849.34 last 23.88) - open/close 19262 (time 1788394596.79 total 859.36 last 29.55) - open/close 19520 (time 1788394606.81 total 869.38 last 25.75) - open/close 19762 (time 1788394616.84 total 879.41 last 24.13) - open/close 20000 (time 1788394625.19 total 887.75 last 28.53) - open/close 20189 (time 1788394635.62 total 898.18 last 18.11) - open/close 20472 (time 1788394645.93 total 908.49 last 27.46) - open/close 20734 (time 1788394656.93 total 919.50 last 23.80) - open/close 21445 (time 1788394666.94 total 929.50 last 71.08) - open/close 22259 (time 1788394676.94 total 939.51 last 81.34) - open/close 23283 (time 1788394686.95 total 949.51 last 102.33) - open/close 24274 (time 1788394696.97 total 959.53 last 98.95) - open/close 25168 (time 1788394706.98 total 969.54 last 89.29) - open/close 26066 (time 1788394716.98 total 979.55 last 89.75) - open/close 26936 (time 1788394726.99 total 989.56 last 86.93) - open/close 27832 (time 1788394737.00 total 999.56 last 89.58) - open/close 28684 (time 1788394747.01 total 1009.57 last 85.09) - open/close 29499 (time 1788394757.02 total 1019.58 last 81.40) - open/close 30000 (time 1788394763.00 total 1025.56 last 83.80) - open/close 30864 (time 1788394773.00 total 1035.56 last 86.38) - open/close 31703 (time 1788394783.01 total 1045.58 last 83.79) - open/close 32602 (time 1788394793.02 total 1055.59 last 89.83) - open/close 33494 (time 1788394803.03 total 1065.60 last 89.09) - open/close 34483 (time 1788394813.04 total 1075.60 last 98.84) - open/close 35427 (time 1788394823.05 total 1085.62 last 94.29) - open/close 36313 (time 1788394833.05 total 1095.62 last 88.60) - open/close 37157 (time 1788394843.09 total 1105.66 last 84.07) - open/close 38102 (time 1788394853.10 total 1115.66 last 94.43) - open/close 39035 (time 1788394863.10 total 1125.67 last 93.27) - open/close 40000 (time 1788394872.70 total 1135.26 last 100.59) - open/close 40905 (time 1788394882.70 total 1145.26 last 90.49) - open/close 41784 (time 1788394892.72 total 1155.28 last 87.73) - open/close 42730 (time 1788394902.72 total 1165.28 last 94.58) - open/close 43596 (time 1788394912.73 total 1175.29 last 86.49) - open/close 44488 (time 1788394922.74 total 1185.30 last 89.15) - open/close 45402 (time 1788394932.74 total 1195.30 last 91.38) - open/close 46347 (time 1788394942.74 total 1205.30 last 94.48) - open/close 47236 (time 1788394952.74 total 1215.31 last 88.88) - open/close 48076 (time 1788394962.75 total 1225.31 last 83.95) - open/close 48962 (time 1788394972.75 total 1235.32 last 88.56) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1242.74 seconds: 39.94 ops/second - unlinked 0 (time 1788394984 ; total 0 ; last 0) - unlinked 10000 (time 1788395095 ; total 111 ; last 111) - unlinked 20000 (time 1788395202 ; total 218 ; last 107) - unlinked 30000 (time 1788395263 ; total 279 ; last 61) - unlinked 40000 (time 1788395324 ; total 340 ; last 61) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 397 seconds: 125.015114 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: kmemleak disabled modules unloaded. pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2