== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:50:36 (1789044636) start mds service on oleg630-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 10 00:34 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: kmemleak disabled oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg630-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg630-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg630-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 137 (time 1789044728.16 total 10.26 last 13.35) - open/close 374 (time 1789044738.18 total 20.27 last 23.67) - open/close 581 (time 1789044748.66 total 30.75 last 19.75) - open/close 770 (time 1789044758.67 total 40.76 last 18.88) - open/close 939 (time 1789044769.20 total 51.29 last 16.05) - open/close 1151 (time 1789044779.37 total 61.46 last 20.84) - open/close 1396 (time 1789044789.38 total 71.47 last 24.48) - open/close 1528 (time 1789044799.39 total 81.48 last 13.19) - open/close 1777 (time 1789044810.23 total 92.33 last 22.96) - open/close 1952 (time 1789044820.23 total 102.33 last 17.50) - open/close 2074 (time 1789044833.20 total 115.29 last 9.41) - open/close 2253 (time 1789044843.75 total 125.85 last 16.96) - open/close 2436 (time 1789044854.11 total 136.20 last 17.67) - open/close 2675 (time 1789044864.11 total 146.21 last 23.88) - open/close 2851 (time 1789044874.15 total 156.24 last 17.54) - open/close 3033 (time 1789044884.84 total 166.93 last 17.02) - open/close 3241 (time 1789044894.84 total 176.94 last 20.79) - open/close 3426 (time 1789044904.87 total 186.96 last 18.46) - open/close 3669 (time 1789044914.87 total 196.96 last 24.29) - open/close 3885 (time 1789044924.88 total 206.98 last 21.58) - open/close 4116 (time 1789044935.67 total 217.76 last 21.42) - open/close 4352 (time 1789044945.91 total 228.00 last 23.04) - open/close 4608 (time 1789044956.46 total 238.56 last 24.25) - open/close 4870 (time 1789044966.47 total 248.57 last 26.18) - open/close 5030 (time 1789044977.85 total 259.95 last 14.06) - open/close 5321 (time 1789044987.86 total 269.95 last 29.09) - open/close 5441 (time 1789044998.76 total 280.85 last 11.01) - open/close 5619 (time 1789045008.76 total 290.85 last 17.80) - open/close 5794 (time 1789045018.76 total 300.86 last 17.50) - open/close 5988 (time 1789045028.77 total 310.86 last 19.39) - open/close 6278 (time 1789045039.48 total 321.57 last 27.07) - open/close 6517 (time 1789045049.50 total 331.60 last 23.84) - open/close 6709 (time 1789045060.46 total 342.56 last 17.52) - open/close 6957 (time 1789045071.48 total 353.58 last 22.50) - open/close 7190 (time 1789045082.39 total 364.49 last 21.36) - open/close 7421 (time 1789045092.40 total 374.49 last 23.09) - open/close 7562 (time 1789045102.56 total 384.65 last 13.88) - open/close 7864 (time 1789045112.57 total 394.66 last 30.16) - open/close 8021 (time 1789045123.08 total 405.18 last 14.94) - open/close 8131 (time 1789045133.12 total 415.21 last 10.96) - open/close 8250 (time 1789045143.13 total 425.22 last 11.89) - open/close 8445 (time 1789045153.15 total 435.24 last 19.46) - open/close 8619 (time 1789045163.16 total 445.26 last 17.38) - open/close 8871 (time 1789045173.80 total 455.89 last 23.69) - open/close 9084 (time 1789045184.02 total 466.11 last 20.84) - open/close 9333 (time 1789045194.85 total 476.94 last 23.00) - open/close 9531 (time 1789045205.09 total 487.18 last 19.34) - open/close 9668 (time 1789045217.23 total 499.32 last 11.29) - open/close 9845 (time 1789045228.06 total 510.16 last 16.33) - open/close 10000 (time 1789045235.07 total 517.17 last 22.10) - open/close 10137 (time 1789045245.24 total 527.33 last 13.48) - open/close 10385 (time 1789045255.25 total 537.35 last 24.76) - open/close 10578 (time 1789045265.26 total 547.36 last 19.28) - open/close 10738 (time 1789045275.56 total 557.65 last 15.53) - open/close 10914 (time 1789045285.58 total 567.68 last 17.56) - open/close 11093 (time 1789045295.60 total 577.69 last 17.87) - open/close 11306 (time 1789045306.34 total 588.43 last 19.84) - open/close 11380 (time 1789045316.64 total 598.74 last 7.18) - open/close 11523 (time 1789045327.23 total 609.32 last 13.51) - open/close 11689 (time 1789045340.76 total 622.86 last 12.26) - open/close 11907 (time 1789045351.34 total 633.43 last 20.62) - open/close 12091 (time 1789045361.76 total 643.86 last 17.65) - open/close 12343 (time 1789045371.76 total 653.86 last 25.20) - open/close 12551 (time 1789045381.77 total 663.87 last 20.78) - open/close 12774 (time 1789045392.03 total 674.13 last 21.74) - open/close 13047 (time 1789045402.98 total 685.07 last 24.95) - open/close 13293 (time 1789045413.61 total 695.70 last 23.14) - open/close 13539 (time 1789045424.31 total 706.41 last 22.98) - open/close 13802 (time 1789045436.92 total 719.02 last 20.85) - open/close 14047 (time 1789045449.09 total 731.18 last 20.15) - open/close 14174 (time 1789045460.00 total 742.10 last 11.63) - open/close 14392 (time 1789045470.01 total 752.11 last 21.77) - open/close 14617 (time 1789045480.01 total 762.11 last 22.50) - open/close 14808 (time 1789045490.85 total 772.95 last 17.62) - open/close 14977 (time 1789045501.52 total 783.62 last 15.84) - open/close 15163 (time 1789045511.55 total 793.64 last 18.55) - open/close 15364 (time 1789045522.47 total 804.56 last 18.41) - open/close 15547 (time 1789045532.83 total 814.93 last 17.65) - open/close 15777 (time 1789045543.09 total 825.19 last 22.42) - open/close 16022 (time 1789045553.11 total 835.20 last 24.47) - open/close 16183 (time 1789045563.12 total 845.21 last 16.08) - open/close 16365 (time 1789045573.15 total 855.24 last 18.15) - open/close 16608 (time 1789045583.16 total 865.25 last 24.27) - open/close 16873 (time 1789045593.17 total 875.26 last 26.47) - open/close 17120 (time 1789045603.26 total 885.35 last 24.48) - open/close 17395 (time 1789045613.27 total 895.36 last 27.47) - open/close 17512 (time 1789045623.32 total 905.41 last 11.65) - open/close 17651 (time 1789045633.35 total 915.45 last 13.85) - open/close 17827 (time 1789045643.36 total 925.45 last 17.59) - open/close 18078 (time 1789045653.37 total 935.46 last 25.08) - open/close 18296 (time 1789045663.53 total 945.62 last 21.45) - open/close 18514 (time 1789045673.53 total 955.63 last 21.79) - open/close 18735 (time 1789045683.81 total 965.91 last 21.50) - open/close 18973 (time 1789045694.00 total 976.10 last 23.35) - open/close 19612 (time 1789045704.01 total 986.10 last 63.89) - open/close 20000 (time 1789045709.49 total 991.58 last 70.79) - open/close 20675 (time 1789045719.50 total 1001.59 last 67.43) - open/close 21330 (time 1789045729.51 total 1011.60 last 65.44) - open/close 22044 (time 1789045739.51 total 1021.60 last 71.40) - open/close 22703 (time 1789045749.51 total 1031.61 last 65.88) - open/close 23381 (time 1789045759.51 total 1041.61 last 67.78) - open/close 24070 (time 1789045769.53 total 1051.62 last 68.82) - open/close 24820 (time 1789045779.53 total 1061.62 last 75.00) - open/close 25544 (time 1789045789.53 total 1071.63 last 72.35) - open/close 26247 (time 1789045799.55 total 1081.64 last 70.20) - open/close 26950 (time 1789045809.57 total 1091.67 last 70.13) - open/close 27630 (time 1789045819.58 total 1101.68 last 67.92) - open/close 28342 (time 1789045829.59 total 1111.68 last 71.17) - open/close 29095 (time 1789045839.59 total 1121.68 last 75.30) - open/close 29778 (time 1789045849.59 total 1131.68 last 68.28) - open/close 30000 (time 1789045852.60 total 1134.69 last 73.88) - open/close 30754 (time 1789045862.61 total 1144.70 last 75.32) - open/close 31469 (time 1789045872.61 total 1154.70 last 71.49) - open/close 32207 (time 1789045882.61 total 1164.70 last 73.78) - open/close 32900 (time 1789045892.62 total 1174.72 last 69.22) - open/close 33637 (time 1789045902.63 total 1184.73 last 73.61) - open/close 34229 (time 1789045912.64 total 1194.74 last 59.15) - open/close 34777 (time 1789045922.66 total 1204.75 last 54.71) - open/close 35361 (time 1789045932.66 total 1214.75 last 58.38) - open/close 36066 (time 1789045942.67 total 1224.76 last 70.45) - open/close 36726 (time 1789045952.67 total 1234.76 last 65.98) - open/close 37455 (time 1789045962.67 total 1244.77 last 72.88) - open/close 38202 (time 1789045972.68 total 1254.78 last 74.62) - open/close 38920 (time 1789045982.70 total 1264.79 last 71.72) - open/close 39670 (time 1789045992.70 total 1274.80 last 74.94) - open/close 40000 (time 1789045997.23 total 1279.32 last 72.96) - open/close 40704 (time 1789046007.23 total 1289.33 last 70.37) - open/close 41442 (time 1789046017.25 total 1299.34 last 73.67) - open/close 42161 (time 1789046027.27 total 1309.36 last 71.76) - open/close 42881 (time 1789046037.27 total 1319.37 last 71.95) - open/close 43620 (time 1789046047.28 total 1329.38 last 73.84) - open/close 44272 (time 1789046057.31 total 1339.40 last 65.05) - open/close 44932 (time 1789046067.32 total 1349.41 last 65.92) - open/close 45608 (time 1789046077.32 total 1359.42 last 67.56) - open/close 46282 (time 1789046087.33 total 1369.43 last 67.33) - open/close 46906 (time 1789046097.34 total 1379.43 last 62.36) - open/close 47641 (time 1789046107.35 total 1389.44 last 73.42) - open/close 48344 (time 1789046117.35 total 1399.45 last 70.27) - open/close 49101 (time 1789046127.35 total 1409.45 last 75.70) - open/close 49825 (time 1789046137.36 total 1419.46 last 72.33) total: 50000 open/close in 1421.75 seconds: 35.17 ops/second - unlinked 0 (time 1789046143 ; total 0 ; last 0) - unlinked 10000 (time 1789046290 ; total 147 ; last 147) - unlinked 20000 (time 1789046442 ; total 299 ; last 152) - unlinked 30000 (time 1789046600 ; total 457 ; last 158) - unlinked 40000 (time 1789046751 ; total 608 ; last 151) total: 50000 unlinks in 753 seconds: 66.401062 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server unloading modules via unload_modules_local on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: kmemleak disabled modules unloaded. pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2