== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:38:06 (1788575886) start mds service on oleg229-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1944872 Sep 4 04:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 483 (time 1788575945.00 total 11.80 last 40.93) - open/close 709 (time 1788575955.05 total 21.85 last 22.49) - open/close 1153 (time 1788575966.00 total 32.80 last 40.55) - open/close 1491 (time 1788575976.48 total 43.29 last 32.24) - open/close 2099 (time 1788575986.66 total 53.46 last 59.75) - open/close 3317 (time 1788575996.66 total 63.46 last 121.78) - open/close 3615 (time 1788576007.07 total 73.88 last 28.62) - open/close 4556 (time 1788576017.07 total 83.88 last 94.09) - open/close 5120 (time 1788576027.08 total 93.88 last 56.39) - open/close 5323 (time 1788576037.66 total 104.47 last 19.17) - open/close 5522 (time 1788576048.36 total 115.16 last 18.61) - open/close 6117 (time 1788576058.37 total 125.17 last 59.47) - open/close 6318 (time 1788576069.22 total 136.02 last 18.52) - open/close 6791 (time 1788576079.98 total 146.78 last 43.95) - open/close 7016 (time 1788576090.34 total 157.15 last 21.71) - open/close 7213 (time 1788576100.40 total 167.20 last 19.60) - open/close 7334 (time 1788576111.15 total 177.95 last 11.25) - open/close 7556 (time 1788576121.16 total 187.96 last 22.19) - open/close 7813 (time 1788576131.75 total 198.56 last 24.25) - open/close 8016 (time 1788576142.63 total 209.43 last 18.66) - open/close 8221 (time 1788576152.95 total 219.75 last 19.86) - open/close 8365 (time 1788576163.06 total 229.86 last 14.24) - open/close 8892 (time 1788576173.07 total 239.87 last 52.68) - open/close 9227 (time 1788576183.08 total 249.88 last 33.46) - open/close 9899 (time 1788576193.10 total 259.90 last 67.09) - open/close 10000 (time 1788576197.01 total 263.81 last 25.81) - open/close 10133 (time 1788576207.03 total 273.84 last 13.27) - open/close 10302 (time 1788576217.04 total 283.84 last 16.90) - open/close 10490 (time 1788576227.27 total 294.07 last 18.38) - open/close 10671 (time 1788576237.74 total 304.54 last 17.28) - open/close 10842 (time 1788576249.97 total 316.77 last 13.99) - open/close 11010 (time 1788576261.83 total 328.63 last 14.16) - open/close 11163 (time 1788576271.85 total 338.65 last 15.27) - open/close 11316 (time 1788576281.87 total 348.67 last 15.27) - open/close 11446 (time 1788576292.94 total 359.74 last 11.75) - open/close 11567 (time 1788576303.66 total 370.47 last 11.28) - open/close 11685 (time 1788576314.55 total 381.35 last 10.84) - open/close 11874 (time 1788576327.55 total 394.35 last 14.54) - open/close 12071 (time 1788576340.62 total 407.42 last 15.07) - open/close 12258 (time 1788576351.40 total 418.20 last 17.35) - open/close 12510 (time 1788576361.59 total 428.39 last 24.74) - open/close 12777 (time 1788576371.59 total 438.39 last 26.69) - open/close 13065 (time 1788576381.60 total 448.40 last 28.77) - open/close 13281 (time 1788576392.17 total 458.97 last 20.44) - open/close 13438 (time 1788576402.37 total 469.17 last 15.40) - open/close 13616 (time 1788576412.37 total 479.18 last 17.79) - open/close 13835 (time 1788576422.87 total 489.67 last 20.87) - open/close 14040 (time 1788576433.07 total 499.88 last 20.09) - open/close 14122 (time 1788576443.09 total 509.89 last 8.19) - open/close 14342 (time 1788576453.10 total 519.91 last 21.97) - open/close 14463 (time 1788576463.23 total 530.03 last 11.95) - open/close 14692 (time 1788576473.23 total 540.03 last 22.90) - open/close 14878 (time 1788576483.23 total 550.04 last 18.59) - open/close 15120 (time 1788576493.24 total 560.04 last 24.19) - open/close 15282 (time 1788576504.78 total 571.58 last 14.04) - open/close 15488 (time 1788576515.12 total 581.92 last 19.92) - open/close 15587 (time 1788576525.58 total 592.38 last 9.46) - open/close 15713 (time 1788576535.59 total 602.39 last 12.59) - open/close 15824 (time 1788576545.71 total 612.52 last 10.96) - open/close 15921 (time 1788576555.90 total 622.71 last 9.52) - open/close 16033 (time 1788576566.65 total 633.45 last 10.42) - open/close 16180 (time 1788576576.66 total 643.46 last 14.69) - open/close 16508 (time 1788576586.66 total 653.46 last 32.80) - open/close 16699 (time 1788576597.29 total 664.10 last 17.96) - open/close 16801 (time 1788576607.30 total 674.10 last 10.20) - open/close 16931 (time 1788576617.31 total 684.11 last 12.99) - open/close 17124 (time 1788576627.31 total 694.12 last 19.29) - open/close 17244 (time 1788576637.58 total 704.39 last 11.68) - open/close 17457 (time 1788576647.60 total 714.40 last 21.28) - open/close 17545 (time 1788576657.90 total 724.71 last 8.54) - open/close 17743 (time 1788576668.36 total 735.16 last 18.94) - open/close 17922 (time 1788576678.37 total 745.17 last 17.89) - open/close 18070 (time 1788576690.09 total 756.89 last 12.63) - open/close 18183 (time 1788576700.09 total 766.89 last 11.29) - open/close 18307 (time 1788576710.10 total 776.91 last 12.39) - open/close 18425 (time 1788576720.83 total 787.63 last 11.00) - open/close 19174 (time 1788576730.84 total 797.64 last 74.82) - open/close 19886 (time 1788576740.85 total 807.65 last 71.16) - open/close 20585 (time 1788576750.86 total 817.66 last 69.82) - open/close 21288 (time 1788576760.86 total 827.66 last 70.28) - open/close 22016 (time 1788576770.86 total 837.67 last 72.77) - open/close 22708 (time 1788576780.87 total 847.67 last 69.19) - open/close 23446 (time 1788576790.87 total 857.67 last 73.78) - open/close 24036 (time 1788576800.87 total 867.67 last 58.98) - open/close 24684 (time 1788576810.88 total 877.69 last 64.72) - open/close 25363 (time 1788576820.89 total 887.69 last 67.87) - open/close 25964 (time 1788576830.89 total 897.69 last 60.10) - open/close 26535 (time 1788576840.92 total 907.72 last 56.94) - open/close 27106 (time 1788576850.92 total 917.72 last 57.07) - open/close 27777 (time 1788576860.92 total 927.72 last 67.09) - open/close 28271 (time 1788576870.95 total 937.75 last 49.25) - open/close 28861 (time 1788576880.96 total 947.76 last 58.97) - open/close 29626 (time 1788576890.97 total 957.78 last 76.38) - open/close 30000 (time 1788576895.97 total 962.78 last 74.80) - open/close 30624 (time 1788576905.97 total 972.78 last 62.39) - open/close 31267 (time 1788576915.98 total 982.79 last 64.23) - open/close 31943 (time 1788576925.99 total 992.80 last 67.54) - open/close 32545 (time 1788576936.00 total 1002.80 last 60.17) - open/close 33124 (time 1788576946.02 total 1012.82 last 57.77) - open/close 33688 (time 1788576956.03 total 1022.83 last 56.34) - open/close 34235 (time 1788576966.04 total 1032.85 last 54.63) - open/close 34803 (time 1788576976.05 total 1042.85 last 56.75) - open/close 35528 (time 1788576986.05 total 1052.86 last 72.49) - open/close 36159 (time 1788576996.05 total 1062.86 last 63.09) - open/close 36816 (time 1788577006.06 total 1072.87 last 65.63) - open/close 37494 (time 1788577016.08 total 1082.88 last 67.73) - open/close 38165 (time 1788577026.09 total 1092.89 last 67.02) - open/close 38957 (time 1788577036.10 total 1102.90 last 79.11) - open/close 39696 (time 1788577046.11 total 1112.91 last 73.81) - open/close 40000 (time 1788577050.32 total 1117.12 last 72.26) - open/close 40750 (time 1788577060.33 total 1127.13 last 74.92) - open/close 41425 (time 1788577070.34 total 1137.14 last 67.42) - open/close 42130 (time 1788577080.35 total 1147.15 last 70.43) - open/close 42756 (time 1788577090.36 total 1157.16 last 62.55) - open/close 43441 (time 1788577100.37 total 1167.17 last 68.45) - open/close 44083 (time 1788577110.38 total 1177.18 last 64.13) - open/close 44730 (time 1788577120.38 total 1187.19 last 64.65) - open/close 45405 (time 1788577130.39 total 1197.20 last 67.45) - open/close 46067 (time 1788577140.40 total 1207.20 last 66.14) - open/close 46709 (time 1788577150.43 total 1217.23 last 64.01) - open/close 47277 (time 1788577160.44 total 1227.24 last 56.74) - open/close 47918 (time 1788577170.45 total 1237.25 last 64.05) - open/close 48568 (time 1788577180.45 total 1247.25 last 64.99) - open/close 49262 (time 1788577190.46 total 1257.27 last 69.32) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1262.18 seconds: 39.32 ops/second - unlinked 0 (time 1788577200 ; total 0 ; last 0) - unlinked 10000 (time 1788577341 ; total 141 ; last 141) - unlinked 20000 (time 1788577471 ; total 271 ; last 130) - unlinked 30000 (time 1788577604 ; total 404 ; last 133) - unlinked 40000 (time 1788577732 ; total 532 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 661 seconds: 75.084717 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2