== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:11:42 (1788203502) start mds service on oleg612-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Aug 31 03:29 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing load_modules_local oleg612-server: kmemleak disabled oleg612-server: Loading modules from /home/green/git/lustre-release/lustre oleg612-server: detected 4 online CPUs by sysfs oleg612-server: MODOPTS_LIBCFS= oleg612-server: Force libcfs to create 2 CPU partitions oleg612-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg612-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg612-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg612-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg612-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg612-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-OST0000 oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg612-client.virtnet: -o user_xattr,flock 192.168.206.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 120 (time 1788203610.80 total 10.75 last 11.16) - open/close 299 (time 1788203620.81 total 20.77 last 17.87) - open/close 386 (time 1788203630.83 total 30.78 last 8.69) - open/close 478 (time 1788203640.99 total 40.94 last 9.05) - open/close 651 (time 1788203651.00 total 50.96 last 17.28) - open/close 834 (time 1788203661.01 total 60.97 last 18.28) - open/close 999 (time 1788203671.37 total 71.33 last 15.92) - open/close 1089 (time 1788203681.75 total 81.71 last 8.67) - open/close 1241 (time 1788203691.79 total 91.75 last 15.14) - open/close 1495 (time 1788203702.82 total 102.78 last 23.02) - open/close 1716 (time 1788203713.25 total 113.21 last 21.19) - open/close 1947 (time 1788203723.60 total 123.55 last 22.33) - open/close 2144 (time 1788203733.60 total 133.55 last 19.70) - open/close 2333 (time 1788203743.64 total 143.59 last 18.83) - open/close 2563 (time 1788203756.10 total 156.06 last 18.44) - open/close 2756 (time 1788203766.18 total 166.13 last 19.16) - open/close 2785 (time 1788203777.10 total 177.06 last 2.66) - open/close 2980 (time 1788203787.79 total 187.75 last 18.24) - open/close 3180 (time 1788203797.79 total 197.75 last 20.00) - open/close 3402 (time 1788203808.29 total 208.25 last 21.15) - open/close 3598 (time 1788203818.32 total 218.27 last 19.55) - open/close 3781 (time 1788203828.33 total 228.29 last 18.27) - open/close 3913 (time 1788203840.74 total 240.70 last 10.64) - open/close 4204 (time 1788203850.82 total 250.78 last 28.87) - open/close 4341 (time 1788203861.33 total 261.29 last 13.04) - open/close 4582 (time 1788203871.51 total 271.47 last 23.67) - open/close 4882 (time 1788203882.22 total 282.18 last 28.01) - open/close 5137 (time 1788203893.06 total 293.01 last 23.53) - open/close 5387 (time 1788203903.31 total 303.27 last 24.39) - open/close 5686 (time 1788203913.32 total 313.28 last 29.87) - open/close 5949 (time 1788203923.92 total 323.87 last 24.82) - open/close 6196 (time 1788203934.23 total 334.19 last 23.94) - open/close 6452 (time 1788203944.53 total 344.49 last 24.87) - open/close 6738 (time 1788203954.54 total 354.50 last 28.57) - open/close 6973 (time 1788203964.54 total 364.50 last 23.49) - open/close 7235 (time 1788203975.08 total 375.04 last 24.87) - open/close 7530 (time 1788203986.07 total 386.03 last 26.84) - open/close 7827 (time 1788203998.61 total 398.56 last 23.69) - open/close 8095 (time 1788204009.56 total 409.52 last 24.46) - open/close 8232 (time 1788204019.58 total 419.53 last 13.68) - open/close 8335 (time 1788204029.85 total 429.81 last 10.03) - open/close 8522 (time 1788204039.88 total 439.84 last 18.64) - open/close 8802 (time 1788204049.89 total 449.85 last 27.99) - open/close 9002 (time 1788204060.49 total 460.45 last 18.87) - open/close 9275 (time 1788204070.49 total 470.45 last 27.29) - open/close 9506 (time 1788204080.51 total 480.46 last 23.07) - open/close 9739 (time 1788204090.73 total 490.69 last 22.79) - open/close 9953 (time 1788204100.90 total 500.86 last 21.03) - open/close 10191 (time 1788204110.90 total 510.86 last 23.80) - open/close 10464 (time 1788204120.91 total 520.87 last 27.27) - open/close 10692 (time 1788204130.92 total 530.87 last 22.80) - open/close 11024 (time 1788204141.32 total 541.28 last 31.91) - open/close 11235 (time 1788204151.34 total 551.30 last 21.05) - open/close 11374 (time 1788204162.14 total 562.10 last 12.87) - open/close 11597 (time 1788204172.18 total 572.13 last 22.22) - open/close 11892 (time 1788204182.53 total 582.49 last 28.49) - open/close 12213 (time 1788204193.03 total 592.99 last 30.59) - open/close 12458 (time 1788204203.03 total 602.99 last 24.50) - open/close 12709 (time 1788204213.04 total 612.99 last 25.08) - open/close 12983 (time 1788204223.95 total 623.91 last 25.09) - open/close 13228 (time 1788204233.97 total 633.93 last 24.46) - open/close 13446 (time 1788204243.98 total 643.93 last 21.78) - open/close 13659 (time 1788204253.99 total 653.95 last 21.28) - open/close 13860 (time 1788204264.78 total 664.74 last 18.63) - open/close 14051 (time 1788204275.72 total 675.67 last 17.46) - open/close 14297 (time 1788204286.43 total 686.39 last 22.95) - open/close 14571 (time 1788204297.44 total 697.40 last 24.89) - open/close 14817 (time 1788204307.44 total 707.40 last 24.60) - open/close 14901 (time 1788204318.31 total 718.27 last 7.73) - open/close 15106 (time 1788204328.40 total 728.36 last 20.31) - open/close 15331 (time 1788204338.42 total 738.38 last 22.47) - open/close 15552 (time 1788204348.83 total 748.79 last 21.22) - open/close 15828 (time 1788204359.82 total 759.78 last 25.12) - open/close 16066 (time 1788204370.44 total 770.40 last 22.40) - open/close 16257 (time 1788204380.61 total 780.57 last 18.78) - open/close 16468 (time 1788204390.93 total 790.89 last 20.45) - open/close 16731 (time 1788204401.30 total 801.25 last 25.38) - open/close 16982 (time 1788204411.89 total 811.84 last 23.70) - open/close 17231 (time 1788204422.38 total 822.33 last 23.74) - open/close 17339 (time 1788204433.23 total 833.19 last 9.95) - open/close 17458 (time 1788204444.27 total 844.23 last 10.78) - open/close 17624 (time 1788204454.80 total 854.75 last 15.77) - open/close 17870 (time 1788204464.80 total 864.75 last 24.60) - open/close 18029 (time 1788204474.82 total 874.78 last 15.87) - open/close 18239 (time 1788204484.84 total 884.80 last 20.96) - open/close 18429 (time 1788204496.34 total 896.29 last 16.53) - open/close 18698 (time 1788204506.34 total 906.29 last 26.90) - open/close 18873 (time 1788204516.52 total 916.47 last 17.19) - open/close 19175 (time 1788204526.52 total 926.47 last 30.20) - open/close 19434 (time 1788204537.06 total 937.02 last 24.56) - open/close 19717 (time 1788204547.08 total 947.03 last 28.26) - open/close 19866 (time 1788204557.10 total 957.05 last 14.87) - open/close 20637 (time 1788204567.10 total 967.05 last 77.09) - open/close 21265 (time 1788204577.10 total 977.06 last 62.78) - open/close 21792 (time 1788204587.11 total 987.07 last 52.63) - open/close 22500 (time 1788204597.12 total 997.08 last 70.72) - open/close 23328 (time 1788204607.13 total 1007.08 last 82.78) - open/close 24134 (time 1788204617.13 total 1017.08 last 80.60) - open/close 24859 (time 1788204627.13 total 1027.09 last 72.47) - open/close 25673 (time 1788204637.14 total 1037.10 last 81.32) - open/close 26488 (time 1788204647.14 total 1047.10 last 81.48) - open/close 27316 (time 1788204657.15 total 1057.11 last 82.73) - open/close 28111 (time 1788204667.15 total 1067.11 last 79.50) - open/close 28983 (time 1788204677.16 total 1077.12 last 87.12) - open/close 29845 (time 1788204687.16 total 1087.12 last 86.19) - open/close 30000 (time 1788204689.34 total 1089.29 last 71.32) - open/close 30799 (time 1788204699.34 total 1099.29 last 79.89) - open/close 31588 (time 1788204709.34 total 1109.30 last 78.87) - open/close 32282 (time 1788204719.35 total 1119.30 last 69.36) - open/close 32973 (time 1788204729.35 total 1129.31 last 69.06) - open/close 33721 (time 1788204739.35 total 1139.31 last 74.79) - open/close 34486 (time 1788204749.36 total 1149.32 last 76.46) - open/close 35338 (time 1788204759.38 total 1159.33 last 85.05) - open/close 36115 (time 1788204769.38 total 1169.34 last 77.67) - open/close 36881 (time 1788204779.39 total 1179.35 last 76.51) - open/close 37602 (time 1788204789.40 total 1189.35 last 72.06) - open/close 38244 (time 1788204799.40 total 1199.36 last 64.19) - open/close 38981 (time 1788204809.41 total 1209.37 last 73.61) - open/close 39647 (time 1788204819.42 total 1219.38 last 66.55) - open/close 40000 (time 1788204824.80 total 1224.76 last 65.54) - open/close 40745 (time 1788204834.81 total 1234.77 last 74.47) - open/close 41517 (time 1788204844.81 total 1244.77 last 77.19) - open/close 42276 (time 1788204854.81 total 1254.77 last 75.88) - open/close 42924 (time 1788204864.83 total 1264.78 last 64.72) - open/close 43591 (time 1788204874.84 total 1274.79 last 66.63) - open/close 44318 (time 1788204884.85 total 1284.81 last 72.61) - open/close 45072 (time 1788204894.86 total 1294.81 last 75.34) - open/close 45815 (time 1788204904.86 total 1304.81 last 74.29) - open/close 46592 (time 1788204914.86 total 1314.82 last 77.65) - open/close 47398 (time 1788204924.86 total 1324.82 last 80.59) - open/close 48212 (time 1788204934.87 total 1334.83 last 81.33) - open/close 49018 (time 1788204944.88 total 1344.83 last 80.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1352.73 seconds: 36.69 ops/second - unlinked 0 (time 1788204957 ; total 0 ; last 0) - unlinked 10000 (time 1788205086 ; total 129 ; last 129) - unlinked 20000 (time 1788205212 ; total 255 ; last 126) - unlinked 30000 (time 1788205319 ; total 362 ; last 107) - unlinked 40000 (time 1788205441 ; total 484 ; last 122) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 606 seconds: 81.899338 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg612-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg612-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg612-server stop mds service on oleg612-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg612-server stop mds service on oleg612-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg612-server unloading modules via unload_modules_local on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing unload_modules_local oleg612-server: kmemleak disabled modules unloaded. pdsh@oleg612-client: oleg612-server: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-client: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-client: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-server: ssh exited with exit code 2