== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:58:37 (1788803917) start mds service on oleg646-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 7 12:38 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing load_modules_local oleg646-server: kmemleak disabled oleg646-server: Loading modules from /home/green/git/lustre-release/lustre oleg646-server: detected 4 online CPUs by sysfs oleg646-server: MODOPTS_LIBCFS= oleg646-server: Force libcfs to create 2 CPU partitions oleg646-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg646-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg646-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg646-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg646-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg646-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-OST0000 oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg646-client.virtnet: -o user_xattr,flock 192.168.206.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 236 (time 1788804017.03 total 10.22 last 23.10) - open/close 490 (time 1788804028.04 total 21.23 last 23.06) - open/close 769 (time 1788804040.96 total 34.15 last 21.60) - open/close 997 (time 1788804051.99 total 45.18 last 20.66) - open/close 1183 (time 1788804062.89 total 56.08 last 17.08) - open/close 1432 (time 1788804073.67 total 66.86 last 23.10) - open/close 1606 (time 1788804083.90 total 77.09 last 17.01) - open/close 1942 (time 1788804094.69 total 87.88 last 31.14) - open/close 2236 (time 1788804105.56 total 98.75 last 27.04) - open/close 2365 (time 1788804116.17 total 109.36 last 12.16) - open/close 2474 (time 1788804126.18 total 119.37 last 10.89) - open/close 2641 (time 1788804139.04 total 132.23 last 12.99) - open/close 2822 (time 1788804149.83 total 143.02 last 16.78) - open/close 3049 (time 1788804159.84 total 153.03 last 22.68) - open/close 3307 (time 1788804169.84 total 163.03 last 25.79) - open/close 3458 (time 1788804179.91 total 173.10 last 15.00) - open/close 3621 (time 1788804190.21 total 183.40 last 15.82) - open/close 3833 (time 1788804200.22 total 193.41 last 21.19) - open/close 4063 (time 1788804210.73 total 203.92 last 21.88) - open/close 4264 (time 1788804220.74 total 213.93 last 20.08) - open/close 4552 (time 1788804230.91 total 224.10 last 28.30) - open/close 4818 (time 1788804241.47 total 234.66 last 25.20) - open/close 5055 (time 1788804251.48 total 244.66 last 23.69) - open/close 5283 (time 1788804261.49 total 254.68 last 22.76) - open/close 5556 (time 1788804271.50 total 264.69 last 27.29) - open/close 5830 (time 1788804282.12 total 275.31 last 25.80) - open/close 5993 (time 1788804294.18 total 287.37 last 13.51) - open/close 6179 (time 1788804304.19 total 297.38 last 18.58) - open/close 6419 (time 1788804314.19 total 307.38 last 24.00) - open/close 6632 (time 1788804325.13 total 318.32 last 19.48) - open/close 6838 (time 1788804343.81 total 337.00 last 11.03) - open/close 7075 (time 1788804353.84 total 347.03 last 23.64) - open/close 7271 (time 1788804364.19 total 357.38 last 18.94) - open/close 7484 (time 1788804376.04 total 369.23 last 17.97) - open/close 7785 (time 1788804386.90 total 380.09 last 27.72) - open/close 8093 (time 1788804397.76 total 390.95 last 28.34) - open/close 8352 (time 1788804408.71 total 401.90 last 23.65) - open/close 8570 (time 1788804418.71 total 411.90 last 21.80) - open/close 8853 (time 1788804429.70 total 422.89 last 25.76) - open/close 9127 (time 1788804440.33 total 433.52 last 25.77) - open/close 9383 (time 1788804450.55 total 443.74 last 25.05) - open/close 9669 (time 1788804460.86 total 454.05 last 27.75) - open/close 9940 (time 1788804471.04 total 464.23 last 26.62) - open/close 10000 (time 1788804473.20 total 466.39 last 27.73) - open/close 10278 (time 1788804483.78 total 476.97 last 26.29) - open/close 10582 (time 1788804494.59 total 487.78 last 28.12) - open/close 10856 (time 1788804504.77 total 497.96 last 26.92) - open/close 11054 (time 1788804515.67 total 508.86 last 18.16) - open/close 11305 (time 1788804525.86 total 519.05 last 24.62) - open/close 11615 (time 1788804536.26 total 529.45 last 29.82) - open/close 11846 (time 1788804546.30 total 539.49 last 23.01) - open/close 12070 (time 1788804556.30 total 549.49 last 22.39) - open/close 12357 (time 1788804567.18 total 560.37 last 26.40) - open/close 12628 (time 1788804577.86 total 571.05 last 25.37) - open/close 12923 (time 1788804587.86 total 581.05 last 29.49) - open/close 13101 (time 1788804598.30 total 591.48 last 17.06) - open/close 13367 (time 1788804608.65 total 601.84 last 25.68) - open/close 13595 (time 1788804619.09 total 612.28 last 21.85) - open/close 13883 (time 1788804629.89 total 623.08 last 26.66) - open/close 14151 (time 1788804639.89 total 633.08 last 26.79) - open/close 14395 (time 1788804649.90 total 643.09 last 24.39) - open/close 14563 (time 1788804660.86 total 654.05 last 15.32) - open/close 14754 (time 1788804670.87 total 664.06 last 19.09) - open/close 14864 (time 1788804681.30 total 674.49 last 10.55) - open/close 15005 (time 1788804691.40 total 684.59 last 13.95) - open/close 15298 (time 1788804701.40 total 694.59 last 29.30) - open/close 15558 (time 1788804711.87 total 705.06 last 24.85) - open/close 15828 (time 1788804722.88 total 716.07 last 24.52) - open/close 16105 (time 1788804732.90 total 726.09 last 27.65) - open/close 16353 (time 1788804742.91 total 736.10 last 24.76) - open/close 16652 (time 1788804752.93 total 746.12 last 29.86) - open/close 16957 (time 1788804763.39 total 756.58 last 29.14) - open/close 17237 (time 1788804773.85 total 767.04 last 26.78) - open/close 17445 (time 1788804784.20 total 777.39 last 20.09) - open/close 17707 (time 1788804794.75 total 787.94 last 24.82) - open/close 17943 (time 1788804805.01 total 798.20 last 23.01) - open/close 18126 (time 1788804815.37 total 808.56 last 17.66) - open/close 18386 (time 1788804826.37 total 819.56 last 23.64) - open/close 18672 (time 1788804836.87 total 830.06 last 27.23) - open/close 18942 (time 1788804846.87 total 840.06 last 27.00) - open/close 19202 (time 1788804857.73 total 850.92 last 23.95) - open/close 19479 (time 1788804867.84 total 861.03 last 27.39) - open/close 19735 (time 1788804878.14 total 871.33 last 24.87) - open/close 20000 (time 1788804887.73 total 880.92 last 27.61) - open/close 20215 (time 1788804898.38 total 891.57 last 20.20) - open/close 20422 (time 1788804908.81 total 902.00 last 19.85) - open/close 20701 (time 1788804918.81 total 912.00 last 27.88) - open/close 21003 (time 1788804929.85 total 923.04 last 27.36) - open/close 21582 (time 1788804939.85 total 933.04 last 57.88) - open/close 22503 (time 1788804949.86 total 943.05 last 92.03) - open/close 23505 (time 1788804959.86 total 953.05 last 100.18) - open/close 24524 (time 1788804969.87 total 963.06 last 101.85) - open/close 25523 (time 1788804979.87 total 973.06 last 99.87) - open/close 26550 (time 1788804989.87 total 983.06 last 102.67) - open/close 27533 (time 1788804999.88 total 993.07 last 98.29) - open/close 28544 (time 1788805009.88 total 1003.07 last 101.09) - open/close 29511 (time 1788805019.88 total 1013.07 last 96.66) - open/close 30000 (time 1788805024.91 total 1018.10 last 97.32) - open/close 30953 (time 1788805034.91 total 1028.10 last 95.30) - open/close 31881 (time 1788805044.91 total 1038.10 last 92.75) - open/close 32750 (time 1788805054.91 total 1048.10 last 86.89) - open/close 33777 (time 1788805064.92 total 1058.11 last 102.60) - open/close 34757 (time 1788805074.93 total 1068.12 last 97.93) - open/close 35733 (time 1788805084.94 total 1078.13 last 97.51) - open/close 36723 (time 1788805094.94 total 1088.13 last 98.96) - open/close 37693 (time 1788805104.95 total 1098.14 last 96.88) - open/close 38681 (time 1788805114.96 total 1108.15 last 98.79) - open/close 39624 (time 1788805124.97 total 1118.16 last 94.21) - open/close 40000 (time 1788805128.62 total 1121.81 last 102.88) - open/close 40890 (time 1788805138.63 total 1131.82 last 88.90) - open/close 41864 (time 1788805148.64 total 1141.82 last 97.36) - open/close 42824 (time 1788805158.64 total 1151.83 last 95.94) - open/close 43762 (time 1788805168.64 total 1161.83 last 93.79) - open/close 44735 (time 1788805178.64 total 1171.83 last 97.28) - open/close 45742 (time 1788805188.65 total 1181.84 last 100.65) - open/close 46702 (time 1788805198.67 total 1191.86 last 95.85) - open/close 47654 (time 1788805208.67 total 1201.86 last 95.19) - open/close 48641 (time 1788805218.68 total 1211.87 last 98.62) - open/close 49609 (time 1788805228.68 total 1221.87 last 96.76) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1222.07 seconds: 40.61 ops/second - unlinked 0 (time 1788805232 ; total 0 ; last 0) - unlinked 10000 (time 1788805333 ; total 101 ; last 101) - unlinked 20000 (time 1788805420 ; total 188 ; last 87) - unlinked 30000 (time 1788805469 ; total 237 ; last 49) - unlinked 40000 (time 1788805521 ; total 289 ; last 52) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 339 seconds: 146.404129 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg646-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg646-server unloading modules via unload_modules_local on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing unload_modules_local oleg646-server: kmemleak disabled modules unloaded. pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2