== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:55:07 (1788836107) start mds service on oleg120-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:47 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: kmemleak disabled oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg120-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg120-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1788836196.71 total 10.00 last 23.79) - open/close 457 (time 1788836208.44 total 21.73 last 18.67) - open/close 595 (time 1788836218.60 total 31.89 last 13.59) - open/close 846 (time 1788836229.55 total 42.85 last 22.91) - open/close 1036 (time 1788836239.70 total 52.99 last 18.73) - open/close 1217 (time 1788836249.96 total 63.26 last 17.63) - open/close 1405 (time 1788836259.97 total 73.26 last 18.79) - open/close 1523 (time 1788836269.98 total 83.27 last 11.79) - open/close 1702 (time 1788836280.62 total 93.92 last 16.82) - open/close 1919 (time 1788836291.04 total 104.33 last 20.83) - open/close 1975 (time 1788836302.01 total 115.30 last 5.10) - open/close 2100 (time 1788836312.09 total 125.38 last 12.41) - open/close 2269 (time 1788836322.10 total 135.39 last 16.89) - open/close 2476 (time 1788836332.10 total 145.39 last 20.69) - open/close 2660 (time 1788836342.11 total 155.40 last 18.40) - open/close 2887 (time 1788836352.29 total 165.58 last 22.30) - open/close 2946 (time 1788836362.31 total 175.60 last 5.89) - open/close 3040 (time 1788836372.66 total 185.95 last 9.08) - open/close 3121 (time 1788836382.97 total 196.26 last 7.86) - open/close 3181 (time 1788836393.85 total 207.14 last 5.52) - open/close 3330 (time 1788836404.36 total 217.65 last 14.17) - open/close 3480 (time 1788836415.02 total 228.31 last 14.08) - open/close 3604 (time 1788836425.03 total 238.32 last 12.39) - open/close 3728 (time 1788836438.26 total 251.55 last 9.37) - open/close 3942 (time 1788836448.26 total 261.56 last 21.38) - open/close 4146 (time 1788836458.29 total 271.58 last 20.35) - open/close 4356 (time 1788836468.29 total 281.59 last 20.99) - open/close 4442 (time 1788836478.32 total 291.61 last 8.58) - open/close 4618 (time 1788836488.48 total 301.77 last 17.32) - open/close 4777 (time 1788836499.10 total 312.39 last 14.97) - open/close 4964 (time 1788836509.11 total 322.41 last 18.67) - open/close 5187 (time 1788836519.12 total 332.41 last 22.30) - open/close 5478 (time 1788836529.16 total 342.45 last 28.97) - open/close 5597 (time 1788836539.21 total 352.50 last 11.84) - open/close 5700 (time 1788836549.31 total 362.60 last 10.20) - open/close 5935 (time 1788836559.86 total 373.15 last 22.27) - open/close 6107 (time 1788836569.95 total 383.24 last 17.05) - open/close 6287 (time 1788836582.90 total 396.19 last 13.90) - open/close 6384 (time 1788836593.93 total 407.22 last 8.80) - open/close 6531 (time 1788836604.91 total 418.20 last 13.39) - open/close 6607 (time 1788836615.29 total 428.58 last 7.32) - open/close 6707 (time 1788836626.28 total 439.57 last 9.10) - open/close 6819 (time 1788836637.30 total 450.59 last 10.16) - open/close 6980 (time 1788836648.04 total 461.33 last 14.99) - open/close 7101 (time 1788836658.05 total 471.34 last 12.09) - open/close 7326 (time 1788836669.02 total 482.32 last 20.50) - open/close 7502 (time 1788836679.03 total 492.32 last 17.59) - open/close 7721 (time 1788836689.05 total 502.34 last 21.86) - open/close 7946 (time 1788836700.06 total 513.35 last 20.44) - open/close 8126 (time 1788836712.04 total 525.33 last 15.02) - open/close 8239 (time 1788836722.05 total 535.34 last 11.29) - open/close 8341 (time 1788836733.68 total 546.97 last 8.77) - open/close 8558 (time 1788836743.69 total 556.98 last 21.66) - open/close 8767 (time 1788836753.88 total 567.18 last 20.50) - open/close 9017 (time 1788836763.91 total 577.20 last 24.94) - open/close 9127 (time 1788836774.71 total 588.00 last 10.18) - open/close 9350 (time 1788836784.94 total 598.23 last 21.81) - open/close 9532 (time 1788836795.72 total 609.01 last 16.88) - open/close 9673 (time 1788836806.06 total 619.35 last 13.64) - open/close 9933 (time 1788836816.08 total 629.37 last 25.95) - open/close 10000 (time 1788836820.89 total 634.18 last 13.91) - open/close 10235 (time 1788836831.91 total 645.20 last 21.33) - open/close 10401 (time 1788836842.80 total 656.09 last 15.24) - open/close 10626 (time 1788836852.81 total 666.10 last 22.49) - open/close 10878 (time 1788836862.81 total 676.10 last 25.19) - open/close 11108 (time 1788836873.70 total 686.99 last 21.12) - open/close 11210 (time 1788836885.62 total 698.91 last 8.56) - open/close 11418 (time 1788836895.67 total 708.96 last 20.69) - open/close 11612 (time 1788836908.42 total 721.71 last 15.22) - open/close 11790 (time 1788836918.43 total 731.73 last 17.77) - open/close 11982 (time 1788836930.20 total 743.49 last 16.32) - open/close 12193 (time 1788836940.24 total 753.53 last 21.02) - open/close 12420 (time 1788836951.14 total 764.43 last 20.82) - open/close 12569 (time 1788836961.76 total 775.05 last 14.03) - open/close 12735 (time 1788836972.07 total 785.36 last 16.10) - open/close 12864 (time 1788836982.08 total 795.37 last 12.89) - open/close 13029 (time 1788836992.08 total 805.38 last 16.49) - open/close 13165 (time 1788837002.28 total 815.58 last 13.33) - open/close 13343 (time 1788837012.30 total 825.59 last 17.78) - open/close 13492 (time 1788837022.69 total 835.98 last 14.34) - open/close 13539 (time 1788837033.31 total 846.60 last 4.43) - open/close 13613 (time 1788837043.31 total 856.60 last 7.40) - open/close 13745 (time 1788837053.32 total 866.61 last 13.19) - open/close 13823 (time 1788837063.52 total 876.81 last 7.65) - open/close 13986 (time 1788837076.58 total 889.87 last 12.49) - open/close 14216 (time 1788837086.58 total 899.87 last 23.00) - open/close 14287 (time 1788837097.57 total 910.86 last 6.46) - open/close 14460 (time 1788837108.13 total 921.43 last 16.37) - open/close 14611 (time 1788837119.07 total 932.36 last 13.81) - open/close 14778 (time 1788837130.66 total 943.95 last 14.41) - open/close 14893 (time 1788837141.66 total 954.95 last 10.45) - open/close 15027 (time 1788837152.07 total 965.36 last 12.88) - open/close 15184 (time 1788837162.97 total 976.26 last 14.39) - open/close 15316 (time 1788837172.99 total 986.28 last 13.17) - open/close 15535 (time 1788837183.00 total 996.29 last 21.89) - open/close 15803 (time 1788837193.02 total 1006.31 last 26.75) - open/close 16091 (time 1788837203.03 total 1016.32 last 28.78) - open/close 16324 (time 1788837213.04 total 1026.33 last 23.27) - open/close 16556 (time 1788837223.05 total 1036.34 last 23.18) - open/close 16849 (time 1788837233.06 total 1046.35 last 29.26) - open/close 17430 (time 1788837243.10 total 1056.39 last 57.88) - open/close 18045 (time 1788837253.10 total 1066.39 last 61.49) - open/close 18569 (time 1788837263.10 total 1076.40 last 52.39) - open/close 19227 (time 1788837273.12 total 1086.41 last 65.68) - open/close 19899 (time 1788837283.13 total 1096.42 last 67.15) - open/close 20483 (time 1788837293.13 total 1106.42 last 58.39) - open/close 21109 (time 1788837303.15 total 1116.44 last 62.49) - open/close 21624 (time 1788837313.16 total 1126.45 last 51.43) - open/close 22240 (time 1788837323.17 total 1136.46 last 61.54) - open/close 22916 (time 1788837333.18 total 1146.47 last 67.56) - open/close 23636 (time 1788837343.18 total 1156.47 last 71.97) - open/close 24304 (time 1788837353.19 total 1166.48 last 66.77) - open/close 24984 (time 1788837363.20 total 1176.49 last 67.93) - open/close 25726 (time 1788837373.21 total 1186.50 last 74.12) - open/close 26270 (time 1788837383.22 total 1196.51 last 54.34) - open/close 26834 (time 1788837393.23 total 1206.52 last 56.35) - open/close 27343 (time 1788837403.24 total 1216.53 last 50.85) - open/close 27950 (time 1788837413.24 total 1226.53 last 60.68) - open/close 28643 (time 1788837423.24 total 1236.53 last 69.27) - open/close 29408 (time 1788837433.25 total 1246.54 last 76.46) - open/close 30000 (time 1788837440.96 total 1254.25 last 76.76) - open/close 30685 (time 1788837450.97 total 1264.26 last 68.46) - open/close 31298 (time 1788837460.97 total 1274.26 last 61.28) - open/close 31953 (time 1788837470.97 total 1284.26 last 65.47) - open/close 32610 (time 1788837480.98 total 1294.27 last 65.64) - open/close 33252 (time 1788837491.00 total 1304.29 last 64.11) - open/close 33949 (time 1788837501.01 total 1314.30 last 69.64) - open/close 34608 (time 1788837511.02 total 1324.31 last 65.79) - open/close 35219 (time 1788837521.03 total 1334.33 last 61.03) - open/close 35881 (time 1788837531.04 total 1344.33 last 66.16) - open/close 36611 (time 1788837541.05 total 1354.34 last 72.95) - open/close 37253 (time 1788837551.09 total 1364.38 last 63.93) - open/close 37930 (time 1788837561.09 total 1374.39 last 67.66) - open/close 38527 (time 1788837571.12 total 1384.41 last 59.56) - open/close 39271 (time 1788837581.13 total 1394.42 last 74.31) - open/close 40000 (time 1788837591.11 total 1404.40 last 73.05) - open/close 40659 (time 1788837601.11 total 1414.40 last 65.87) - open/close 41371 (time 1788837611.12 total 1424.41 last 71.18) - open/close 42126 (time 1788837621.12 total 1434.41 last 75.46) - open/close 42861 (time 1788837631.13 total 1444.42 last 73.47) - open/close 43596 (time 1788837641.13 total 1454.42 last 73.49) - open/close 44376 (time 1788837651.13 total 1464.42 last 77.97) - open/close 45186 (time 1788837661.14 total 1474.43 last 80.91) - open/close 45911 (time 1788837671.15 total 1484.44 last 72.46) - open/close 46700 (time 1788837681.16 total 1494.45 last 78.83) - open/close 47346 (time 1788837691.16 total 1504.45 last 64.59) - open/close 48086 (time 1788837701.17 total 1514.46 last 73.92) - open/close 48813 (time 1788837711.18 total 1524.47 last 72.64) - open/close 49384 (time 1788837721.21 total 1534.50 last 56.90) total: 50000 open/close in 1542.98 seconds: 32.40 ops/second - unlinked 0 (time 1788837734 ; total 0 ; last 0) - unlinked 10000 (time 1788837887 ; total 153 ; last 153) - unlinked 20000 (time 1788838033 ; total 299 ; last 146) - unlinked 30000 (time 1788838181 ; total 447 ; last 148) - unlinked 40000 (time 1788838338 ; total 604 ; last 157) total: 50000 unlinks in 751 seconds: 66.577896 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: kmemleak disabled modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2