== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:56:51 (1789639011) start mds service on oleg312-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 17 04:27 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing load_modules_local oleg312-server: kmemleak disabled oleg312-server: Loading modules from /home/green/git/lustre-release/lustre oleg312-server: detected 4 online CPUs by sysfs oleg312-server: MODOPTS_LIBCFS= oleg312-server: Force libcfs to create 2 CPU partitions oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg312-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg312-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg312-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg312-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1789639104.63 total 10.42 last 10.08) - open/close 279 (time 1789639114.63 total 20.42 last 17.40) - open/close 477 (time 1789639125.68 total 31.47 last 17.91) - open/close 745 (time 1789639136.63 total 42.42 last 24.49) - open/close 980 (time 1789639147.65 total 53.44 last 21.32) - open/close 1186 (time 1789639158.47 total 64.26 last 19.05) - open/close 1438 (time 1789639169.40 total 75.19 last 23.06) - open/close 1633 (time 1789639179.64 total 85.43 last 19.04) - open/close 1787 (time 1789639189.74 total 95.53 last 15.24) - open/close 2000 (time 1789639200.44 total 106.23 last 19.92) - open/close 2243 (time 1789639210.81 total 116.60 last 23.43) - open/close 2506 (time 1789639221.37 total 127.16 last 24.91) - open/close 2718 (time 1789639231.37 total 137.16 last 21.19) - open/close 2860 (time 1789639241.59 total 147.38 last 13.89) - open/close 3089 (time 1789639251.70 total 157.49 last 22.65) - open/close 3272 (time 1789639262.39 total 168.18 last 17.12) - open/close 3474 (time 1789639272.41 total 178.20 last 20.17) - open/close 3706 (time 1789639282.67 total 188.46 last 22.60) - open/close 3919 (time 1789639293.45 total 199.24 last 19.76) - open/close 4098 (time 1789639305.03 total 210.82 last 15.46) - open/close 4293 (time 1789639315.68 total 221.47 last 18.31) - open/close 4450 (time 1789639326.46 total 232.25 last 14.56) - open/close 4622 (time 1789639337.34 total 243.13 last 15.82) - open/close 4761 (time 1789639347.35 total 253.14 last 13.88) - open/close 4924 (time 1789639358.15 total 263.94 last 15.09) - open/close 5148 (time 1789639368.83 total 274.62 last 20.97) - open/close 5445 (time 1789639378.84 total 284.63 last 29.67) - open/close 5692 (time 1789639388.84 total 294.64 last 24.68) - open/close 5963 (time 1789639399.16 total 304.95 last 26.28) - open/close 6176 (time 1789639409.66 total 315.45 last 20.28) - open/close 6473 (time 1789639419.67 total 325.46 last 29.67) - open/close 6556 (time 1789639429.68 total 335.47 last 8.29) - open/close 6753 (time 1789639440.27 total 346.06 last 18.61) - open/close 6889 (time 1789639451.33 total 357.12 last 12.29) - open/close 7065 (time 1789639462.25 total 368.04 last 16.12) - open/close 7237 (time 1789639472.76 total 378.55 last 16.35) - open/close 7364 (time 1789639482.95 total 388.74 last 12.47) - open/close 7482 (time 1789639493.26 total 399.05 last 11.45) - open/close 7665 (time 1789639504.27 total 410.06 last 16.62) - open/close 7934 (time 1789639514.81 total 420.60 last 25.50) - open/close 8143 (time 1789639525.43 total 431.22 last 19.69) - open/close 8336 (time 1789639535.81 total 441.60 last 18.60) - open/close 8401 (time 1789639546.08 total 451.87 last 6.33) - open/close 8623 (time 1789639556.21 total 462.00 last 21.91) - open/close 8938 (time 1789639566.24 total 472.03 last 31.42) - open/close 9226 (time 1789639576.84 total 482.63 last 27.16) - open/close 9474 (time 1789639587.14 total 492.93 last 24.08) - open/close 9694 (time 1789639597.15 total 502.94 last 21.98) - open/close 9990 (time 1789639607.35 total 513.14 last 29.01) - open/close 10219 (time 1789639617.35 total 523.14 last 22.90) - open/close 10382 (time 1789639628.16 total 533.95 last 15.08) - open/close 10553 (time 1789639638.32 total 544.11 last 16.82) - open/close 10769 (time 1789639648.34 total 554.13 last 21.57) - open/close 10957 (time 1789639658.37 total 564.16 last 18.73) - open/close 11219 (time 1789639668.43 total 574.22 last 26.06) - open/close 11332 (time 1789639678.44 total 584.23 last 11.29) - open/close 11563 (time 1789639692.82 total 598.61 last 16.06) - open/close 11794 (time 1789639705.24 total 611.03 last 18.61) - open/close 11890 (time 1789639715.72 total 621.51 last 9.16) - open/close 12160 (time 1789639726.46 total 632.25 last 25.13) - open/close 12410 (time 1789639737.01 total 642.80 last 23.69) - open/close 12696 (time 1789639747.87 total 653.66 last 26.36) - open/close 12990 (time 1789639757.87 total 663.66 last 29.39) - open/close 13244 (time 1789639768.06 total 673.85 last 24.94) - open/close 13519 (time 1789639778.24 total 684.03 last 26.99) - open/close 13744 (time 1789639788.26 total 694.05 last 22.47) - open/close 13964 (time 1789639800.01 total 705.80 last 18.72) - open/close 14205 (time 1789639810.07 total 715.86 last 23.93) - open/close 14498 (time 1789639820.10 total 725.89 last 29.21) - open/close 14743 (time 1789639831.17 total 736.96 last 22.15) - open/close 15047 (time 1789639842.04 total 747.83 last 27.97) - open/close 15281 (time 1789639852.04 total 757.83 last 23.39) - open/close 15527 (time 1789639863.06 total 768.85 last 22.32) - open/close 15731 (time 1789639873.11 total 778.90 last 20.30) - open/close 15950 (time 1789639883.12 total 788.91 last 21.88) - open/close 16223 (time 1789639893.13 total 798.92 last 27.27) - open/close 16484 (time 1789639903.14 total 808.93 last 26.08) - open/close 16773 (time 1789639913.16 total 818.95 last 28.85) - open/close 17013 (time 1789639923.17 total 828.96 last 23.98) - open/close 17259 (time 1789639933.68 total 839.47 last 23.39) - open/close 17511 (time 1789639943.69 total 849.48 last 25.19) - open/close 17789 (time 1789639954.63 total 860.42 last 25.41) - open/close 18049 (time 1789639964.64 total 870.43 last 25.98) - open/close 18320 (time 1789639974.65 total 880.44 last 27.07) - open/close 19137 (time 1789639984.67 total 890.46 last 81.54) - open/close 20000 (time 1789639994.24 total 900.03 last 90.22) - open/close 20954 (time 1789640004.25 total 910.04 last 95.29) - open/close 21927 (time 1789640014.26 total 920.05 last 97.18) - open/close 22843 (time 1789640024.27 total 930.06 last 91.54) - open/close 23600 (time 1789640034.27 total 940.06 last 75.70) - open/close 24364 (time 1789640044.27 total 950.06 last 76.34) - open/close 25236 (time 1789640054.28 total 960.07 last 87.11) - open/close 26105 (time 1789640064.29 total 970.08 last 86.86) - open/close 26968 (time 1789640074.29 total 980.08 last 86.30) - open/close 27803 (time 1789640084.29 total 990.08 last 83.50) - open/close 28674 (time 1789640094.29 total 1000.08 last 87.04) - open/close 29570 (time 1789640104.31 total 1010.10 last 89.47) - open/close 30000 (time 1789640108.93 total 1014.72 last 92.98) - open/close 30877 (time 1789640118.94 total 1024.73 last 87.62) - open/close 31618 (time 1789640128.95 total 1034.74 last 74.02) - open/close 32390 (time 1789640138.96 total 1044.75 last 77.18) - open/close 33229 (time 1789640148.97 total 1054.76 last 83.79) - open/close 34123 (time 1789640158.97 total 1064.76 last 89.40) - open/close 35024 (time 1789640168.98 total 1074.77 last 90.04) - open/close 35969 (time 1789640178.98 total 1084.77 last 94.50) - open/close 36866 (time 1789640189.00 total 1094.79 last 89.47) - open/close 37724 (time 1789640199.01 total 1104.80 last 85.75) - open/close 38531 (time 1789640209.01 total 1114.80 last 80.68) - open/close 39436 (time 1789640219.01 total 1124.80 last 90.50) - open/close 40000 (time 1789640225.52 total 1131.31 last 86.65) - open/close 40884 (time 1789640235.53 total 1141.32 last 88.30) - open/close 41753 (time 1789640245.53 total 1151.32 last 86.89) - open/close 42582 (time 1789640255.54 total 1161.33 last 82.86) - open/close 43516 (time 1789640265.54 total 1171.33 last 93.37) - open/close 44403 (time 1789640275.55 total 1181.34 last 88.63) - open/close 45274 (time 1789640285.56 total 1191.35 last 86.98) - open/close 46151 (time 1789640295.56 total 1201.35 last 87.68) - open/close 47136 (time 1789640305.57 total 1211.36 last 98.44) - open/close 48005 (time 1789640315.57 total 1221.36 last 86.88) - open/close 48904 (time 1789640325.58 total 1231.37 last 89.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1240.24 seconds: 40.02 ops/second - unlinked 0 (time 1789640339 ; total 0 ; last 0) - unlinked 10000 (time 1789640453 ; total 114 ; last 114) - unlinked 20000 (time 1789640555 ; total 216 ; last 102) - unlinked 30000 (time 1789640618 ; total 279 ; last 63) - unlinked 40000 (time 1789640683 ; total 344 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 403 seconds: 123.153847 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg312-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg312-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server unloading modules via unload_modules_local on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing unload_modules_local oleg312-server: kmemleak disabled modules unloaded. pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2