== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:39:25 (1789677565) start mds service on oleg607-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 17 12:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing load_modules_local oleg607-server: kmemleak disabled oleg607-server: Loading modules from /home/green/git/lustre-release/lustre oleg607-server: detected 4 online CPUs by sysfs oleg607-server: MODOPTS_LIBCFS= oleg607-server: Force libcfs to create 2 CPU partitions oleg607-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg607-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg607-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg607-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg607-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg607-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-OST0000 oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg607-client.virtnet: -o user_xattr,flock 192.168.206.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 157 (time 1789677680.96 total 10.60 last 14.82) - open/close 369 (time 1789677690.96 total 20.60 last 21.19) - open/close 614 (time 1789677702.85 total 32.49 last 20.61) - open/close 866 (time 1789677713.53 total 43.17 last 23.60) - open/close 1070 (time 1789677723.53 total 53.17 last 20.40) - open/close 1210 (time 1789677734.59 total 64.22 last 12.66) - open/close 1483 (time 1789677745.59 total 75.22 last 24.82) - open/close 1749 (time 1789677756.29 total 85.92 last 24.86) - open/close 2023 (time 1789677766.29 total 95.92 last 27.40) - open/close 2264 (time 1789677776.81 total 106.45 last 22.90) - open/close 2555 (time 1789677787.66 total 117.30 last 26.82) - open/close 2625 (time 1789677799.34 total 128.98 last 5.99) - open/close 2848 (time 1789677809.87 total 139.50 last 21.19) - open/close 3115 (time 1789677820.66 total 150.30 last 24.73) - open/close 3428 (time 1789677830.67 total 160.31 last 31.27) - open/close 3630 (time 1789677841.22 total 170.86 last 19.14) - open/close 3939 (time 1789677851.23 total 180.86 last 30.88) - open/close 4175 (time 1789677861.31 total 190.95 last 23.41) - open/close 4428 (time 1789677871.73 total 201.37 last 24.28) - open/close 4704 (time 1789677881.74 total 211.37 last 27.59) - open/close 5014 (time 1789677891.74 total 221.38 last 30.99) - open/close 5237 (time 1789677902.07 total 231.71 last 21.58) - open/close 5538 (time 1789677912.08 total 241.71 last 30.09) - open/close 5763 (time 1789677922.08 total 251.71 last 22.49) - open/close 6057 (time 1789677932.08 total 261.72 last 29.38) - open/close 6299 (time 1789677943.18 total 272.81 last 21.82) - open/close 6512 (time 1789677953.18 total 282.82 last 21.29) - open/close 6772 (time 1789677963.19 total 292.83 last 25.97) - open/close 7025 (time 1789677973.20 total 302.84 last 25.27) - open/close 7309 (time 1789677983.93 total 313.57 last 26.48) - open/close 7568 (time 1789677994.82 total 324.45 last 23.80) - open/close 7832 (time 1789678005.84 total 335.47 last 23.95) - open/close 8101 (time 1789678016.69 total 346.33 last 24.78) - open/close 8370 (time 1789678026.70 total 356.34 last 26.87) - open/close 8615 (time 1789678037.44 total 367.07 last 22.83) - open/close 8864 (time 1789678048.32 total 377.95 last 22.88) - open/close 9130 (time 1789678058.55 total 388.18 last 26.00) - open/close 9367 (time 1789678069.06 total 398.69 last 22.56) - open/close 9631 (time 1789678079.22 total 408.86 last 25.96) - open/close 9912 (time 1789678089.78 total 419.41 last 26.63) - open/close 10000 (time 1789678093.24 total 422.87 last 25.44) - open/close 10233 (time 1789678103.25 total 432.89 last 23.25) - open/close 10547 (time 1789678114.22 total 443.86 last 28.63) - open/close 10744 (time 1789678124.23 total 453.87 last 19.68) - open/close 11003 (time 1789678134.32 total 463.96 last 25.67) - open/close 11262 (time 1789678144.33 total 473.96 last 25.89) - open/close 11473 (time 1789678154.65 total 484.29 last 20.44) - open/close 11720 (time 1789678165.11 total 494.74 last 23.62) - open/close 11931 (time 1789678175.12 total 504.76 last 21.07) - open/close 12174 (time 1789678185.13 total 514.77 last 24.27) - open/close 12370 (time 1789678195.72 total 525.36 last 18.51) - open/close 12612 (time 1789678205.82 total 535.45 last 23.98) - open/close 12897 (time 1789678215.83 total 545.47 last 28.45) - open/close 13152 (time 1789678225.85 total 555.48 last 25.47) - open/close 13361 (time 1789678235.97 total 565.61 last 20.65) - open/close 13605 (time 1789678245.99 total 575.63 last 24.35) - open/close 13882 (time 1789678256.76 total 586.39 last 25.73) - open/close 14088 (time 1789678267.28 total 596.92 last 19.57) - open/close 14286 (time 1789678277.70 total 607.34 last 19.00) - open/close 14491 (time 1789678288.65 total 618.28 last 18.73) - open/close 14720 (time 1789678298.66 total 628.30 last 22.87) - open/close 14982 (time 1789678309.06 total 638.69 last 25.21) - open/close 15262 (time 1789678319.36 total 649.00 last 27.18) - open/close 15529 (time 1789678329.61 total 659.24 last 26.05) - open/close 15730 (time 1789678339.70 total 669.34 last 19.91) - open/close 15993 (time 1789678350.64 total 680.28 last 24.04) - open/close 16235 (time 1789678360.84 total 690.48 last 23.72) - open/close 16457 (time 1789678370.85 total 700.48 last 22.19) - open/close 16698 (time 1789678381.45 total 711.08 last 22.74) - open/close 16933 (time 1789678391.61 total 721.25 last 23.11) - open/close 17193 (time 1789678401.78 total 731.41 last 25.58) - open/close 17462 (time 1789678412.22 total 741.86 last 25.76) - open/close 17697 (time 1789678422.34 total 751.98 last 23.22) - open/close 17934 (time 1789678432.90 total 762.54 last 22.45) - open/close 18211 (time 1789678442.91 total 772.54 last 27.68) - open/close 18353 (time 1789678452.91 total 782.55 last 14.19) - open/close 18517 (time 1789678465.11 total 794.75 last 13.45) - open/close 18717 (time 1789678475.84 total 805.47 last 18.64) - open/close 18946 (time 1789678485.85 total 815.48 last 22.88) - open/close 19127 (time 1789678495.85 total 825.49 last 18.09) - open/close 19382 (time 1789678506.10 total 835.73 last 24.89) - open/close 19685 (time 1789678516.11 total 845.75 last 30.26) - open/close 19926 (time 1789678526.12 total 855.76 last 24.08) - open/close 20000 (time 1789678529.47 total 859.11 last 22.06) - open/close 20296 (time 1789678539.50 total 869.14 last 29.52) - open/close 20561 (time 1789678550.56 total 880.19 last 23.96) - open/close 21076 (time 1789678560.56 total 890.20 last 51.47) - open/close 21779 (time 1789678570.58 total 900.22 last 70.20) - open/close 22503 (time 1789678580.58 total 910.22 last 72.37) - open/close 23336 (time 1789678590.58 total 920.22 last 83.29) - open/close 24095 (time 1789678600.58 total 930.22 last 75.89) - open/close 24899 (time 1789678610.59 total 940.22 last 80.38) - open/close 25680 (time 1789678620.59 total 950.23 last 78.04) - open/close 26489 (time 1789678630.60 total 960.24 last 80.87) - open/close 27179 (time 1789678640.61 total 970.24 last 68.94) - open/close 27887 (time 1789678650.61 total 980.25 last 70.76) - open/close 28677 (time 1789678660.62 total 990.26 last 78.93) - open/close 29540 (time 1789678670.63 total 1000.26 last 86.25) - open/close 30000 (time 1789678676.83 total 1006.47 last 74.11) - open/close 30794 (time 1789678686.84 total 1016.48 last 79.33) - open/close 31581 (time 1789678696.86 total 1026.49 last 78.60) - open/close 32374 (time 1789678706.86 total 1036.49 last 79.30) - open/close 33162 (time 1789678716.86 total 1046.50 last 78.78) - open/close 34035 (time 1789678726.87 total 1056.51 last 87.20) - open/close 34802 (time 1789678736.88 total 1066.52 last 76.60) - open/close 35533 (time 1789678746.89 total 1076.52 last 73.08) - open/close 36356 (time 1789678756.89 total 1086.52 last 82.29) - open/close 37157 (time 1789678766.90 total 1096.53 last 80.05) - open/close 37950 (time 1789678776.90 total 1106.53 last 79.28) - open/close 38721 (time 1789678786.91 total 1116.55 last 76.99) - open/close 39508 (time 1789678796.93 total 1126.56 last 78.58) - open/close 40000 (time 1789678804.24 total 1133.87 last 67.33) - open/close 40676 (time 1789678814.25 total 1143.89 last 67.51) - open/close 41262 (time 1789678824.25 total 1153.89 last 58.60) - open/close 42010 (time 1789678834.25 total 1163.89 last 74.80) - open/close 42799 (time 1789678844.27 total 1173.90 last 78.77) - open/close 43616 (time 1789678854.27 total 1183.91 last 81.69) - open/close 44411 (time 1789678864.28 total 1193.91 last 79.44) - open/close 45190 (time 1789678874.28 total 1203.92 last 77.84) - open/close 45884 (time 1789678884.31 total 1213.94 last 69.23) - open/close 46689 (time 1789678894.31 total 1223.95 last 80.49) - open/close 47438 (time 1789678904.32 total 1233.96 last 74.83) - open/close 48219 (time 1789678914.33 total 1243.96 last 78.04) - open/close 48960 (time 1789678924.33 total 1253.96 last 74.09) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1262.83 seconds: 39.30 ops/second - unlinked 0 (time 1789678937 ; total 0 ; last 0) - unlinked 10000 (time 1789679067 ; total 130 ; last 130) - unlinked 20000 (time 1789679197 ; total 260 ; last 130) - unlinked 30000 (time 1789679329 ; total 392 ; last 132) - unlinked 40000 (time 1789679457 ; total 520 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 643 seconds: 77.186623 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg607-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg607-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg607-server unloading modules via unload_modules_local on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing unload_modules_local oleg607-server: kmemleak disabled modules unloaded. pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2