== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:24:23 (1788441863) start mds service on oleg615-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing load_modules_local oleg615-server: kmemleak disabled oleg615-server: Loading modules from /home/green/git/lustre-release/lustre oleg615-server: detected 4 online CPUs by sysfs oleg615-server: MODOPTS_LIBCFS= oleg615-server: Force libcfs to create 2 CPU partitions oleg615-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg615-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg615-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg615-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg615-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg615-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0000 oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 277 (time 1788441966.88 total 10.38 last 26.69) - open/close 575 (time 1788441977.58 total 21.07 last 27.86) - open/close 793 (time 1788441988.02 total 31.52 last 20.87) - open/close 1072 (time 1788441998.02 total 41.52 last 27.90) - open/close 1383 (time 1788442008.03 total 51.53 last 31.07) - open/close 1643 (time 1788442018.72 total 62.21 last 24.33) - open/close 1931 (time 1788442028.73 total 72.23 last 28.77) - open/close 2212 (time 1788442038.75 total 82.24 last 28.06) - open/close 2527 (time 1788442049.19 total 92.68 last 30.17) - open/close 2806 (time 1788442060.01 total 103.50 last 25.78) - open/close 3094 (time 1788442070.68 total 114.18 last 26.98) - open/close 3372 (time 1788442080.76 total 124.26 last 27.58) - open/close 3662 (time 1788442091.04 total 134.54 last 28.22) - open/close 3913 (time 1788442101.87 total 145.37 last 23.17) - open/close 4248 (time 1788442111.88 total 155.38 last 33.47) - open/close 4511 (time 1788442123.96 total 167.46 last 21.77) - open/close 4801 (time 1788442136.48 total 179.98 last 23.16) - open/close 5057 (time 1788442146.50 total 190.00 last 25.55) - open/close 5357 (time 1788442156.51 total 200.00 last 29.99) - open/close 5614 (time 1788442167.22 total 210.72 last 23.98) - open/close 5858 (time 1788442177.23 total 220.73 last 24.37) - open/close 6122 (time 1788442187.25 total 230.74 last 26.37) - open/close 6399 (time 1788442197.25 total 240.75 last 27.68) - open/close 6661 (time 1788442207.26 total 250.75 last 26.19) - open/close 6934 (time 1788442218.08 total 261.57 last 25.23) - open/close 7199 (time 1788442229.03 total 272.53 last 24.20) - open/close 7460 (time 1788442239.04 total 282.54 last 26.07) - open/close 7729 (time 1788442249.95 total 293.45 last 24.65) - open/close 8008 (time 1788442259.96 total 303.46 last 27.87) - open/close 8290 (time 1788442269.97 total 313.47 last 28.17) - open/close 8559 (time 1788442280.87 total 324.37 last 24.68) - open/close 8845 (time 1788442291.87 total 335.37 last 26.00) - open/close 9124 (time 1788442301.88 total 345.38 last 27.88) - open/close 9396 (time 1788442312.74 total 356.23 last 25.06) - open/close 9677 (time 1788442323.32 total 366.81 last 26.56) - open/close 9962 (time 1788442333.47 total 376.97 last 28.07) - open/close 10228 (time 1788442343.48 total 386.98 last 26.57) - open/close 10527 (time 1788442353.70 total 397.20 last 29.26) - open/close 10793 (time 1788442363.87 total 407.37 last 26.16) - open/close 11034 (time 1788442374.05 total 417.55 last 23.67) - open/close 11314 (time 1788442384.68 total 428.17 last 26.35) - open/close 11540 (time 1788442395.24 total 438.73 last 21.40) - open/close 11819 (time 1788442405.60 total 449.10 last 26.92) - open/close 12100 (time 1788442415.65 total 459.14 last 27.97) - open/close 12382 (time 1788442426.01 total 469.51 last 27.21) - open/close 12701 (time 1788442436.02 total 479.51 last 31.89) - open/close 12986 (time 1788442446.76 total 490.26 last 26.52) - open/close 13261 (time 1788442457.19 total 500.68 last 26.38) - open/close 13541 (time 1788442467.51 total 511.00 last 27.14) - open/close 13756 (time 1788442477.73 total 521.23 last 21.03) - open/close 14047 (time 1788442488.29 total 531.79 last 27.55) - open/close 14307 (time 1788442498.72 total 542.22 last 24.92) - open/close 14574 (time 1788442508.75 total 552.24 last 26.63) - open/close 14927 (time 1788442518.76 total 562.25 last 35.27) - open/close 15189 (time 1788442528.97 total 572.46 last 25.67) - open/close 15488 (time 1788442538.97 total 582.47 last 29.87) - open/close 15772 (time 1788442549.02 total 592.52 last 28.27) - open/close 16106 (time 1788442559.21 total 602.70 last 32.79) - open/close 16401 (time 1788442569.64 total 613.13 last 28.29) - open/close 16693 (time 1788442580.27 total 623.76 last 27.46) - open/close 16969 (time 1788442590.36 total 633.85 last 27.36) - open/close 17260 (time 1788442600.55 total 644.04 last 28.56) - open/close 17573 (time 1788442611.23 total 654.73 last 29.29) - open/close 17887 (time 1788442622.30 total 665.79 last 28.38) - open/close 18146 (time 1788442632.61 total 676.11 last 25.11) - open/close 18442 (time 1788442643.05 total 686.54 last 28.36) - open/close 18708 (time 1788442653.49 total 696.99 last 25.48) - open/close 18886 (time 1788442663.50 total 706.99 last 17.79) - open/close 19135 (time 1788442673.55 total 717.04 last 24.77) - open/close 19417 (time 1788442683.56 total 727.05 last 28.17) - open/close 19693 (time 1788442694.05 total 737.55 last 26.31) - open/close 19910 (time 1788442704.47 total 747.97 last 20.82) - open/close 20000 (time 1788442707.75 total 751.25 last 27.43) - open/close 20249 (time 1788442717.99 total 761.49 last 24.31) - open/close 20478 (time 1788442728.67 total 772.16 last 21.45) - open/close 20712 (time 1788442738.80 total 782.29 last 23.10) - open/close 20930 (time 1788442749.54 total 793.03 last 20.30) - open/close 21229 (time 1788442759.97 total 803.47 last 28.65) - open/close 21528 (time 1788442770.66 total 814.16 last 27.96) - open/close 21832 (time 1788442780.67 total 824.17 last 30.37) - open/close 22098 (time 1788442791.32 total 834.82 last 24.98) - open/close 22369 (time 1788442801.83 total 845.32 last 25.80) - open/close 22661 (time 1788442811.83 total 855.32 last 29.19) - open/close 23270 (time 1788442821.83 total 865.32 last 60.90) - open/close 24162 (time 1788442831.84 total 875.33 last 89.12) - open/close 25033 (time 1788442841.84 total 885.34 last 87.07) - open/close 25966 (time 1788442851.84 total 895.34 last 93.29) - open/close 26873 (time 1788442861.86 total 905.35 last 90.56) - open/close 27837 (time 1788442871.86 total 915.36 last 96.35) - open/close 28778 (time 1788442881.87 total 925.36 last 94.06) - open/close 29616 (time 1788442891.88 total 935.37 last 83.72) - open/close 30000 (time 1788442896.06 total 939.55 last 91.90) - open/close 30914 (time 1788442906.06 total 949.56 last 91.36) - open/close 31846 (time 1788442916.07 total 959.56 last 93.15) - open/close 32756 (time 1788442926.07 total 969.56 last 90.97) - open/close 33760 (time 1788442936.08 total 979.57 last 100.34) - open/close 34697 (time 1788442946.08 total 989.58 last 93.65) - open/close 35607 (time 1788442956.08 total 999.58 last 90.99) - open/close 36547 (time 1788442966.10 total 1009.59 last 93.84) - open/close 37395 (time 1788442976.11 total 1019.61 last 84.67) - open/close 38351 (time 1788442986.12 total 1029.62 last 95.52) - open/close 39305 (time 1788442996.13 total 1039.63 last 95.29) - open/close 40000 (time 1788443003.32 total 1046.81 last 96.74) - open/close 40959 (time 1788443013.32 total 1056.82 last 95.85) - open/close 41888 (time 1788443023.33 total 1066.83 last 92.83) - open/close 42702 (time 1788443033.33 total 1076.83 last 81.39) - open/close 43629 (time 1788443043.33 total 1086.83 last 92.68) - open/close 44557 (time 1788443053.33 total 1096.83 last 92.79) - open/close 45449 (time 1788443063.34 total 1106.83 last 89.16) - open/close 46373 (time 1788443073.34 total 1116.84 last 92.34) - open/close 47318 (time 1788443083.37 total 1126.86 last 94.31) - open/close 48168 (time 1788443093.37 total 1136.86 last 84.99) - open/close 49015 (time 1788443103.37 total 1146.86 last 84.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1154.11 seconds: 43.00 ops/second - unlinked 0 (time 1788443114 ; total 0 ; last 0) - unlinked 10000 (time 1788443224 ; total 110 ; last 110) - unlinked 20000 (time 1788443329 ; total 215 ; last 105) - unlinked 30000 (time 1788443393 ; total 279 ; last 64) - unlinked 40000 (time 1788443452 ; total 338 ; last 59) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 394 seconds: 125.967003 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg615-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg615-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg615-server stop mds service on oleg615-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg615-server stop mds service on oleg615-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg615-server unloading modules via unload_modules_local on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing unload_modules_local oleg615-server: kmemleak disabled modules unloaded. pdsh@oleg615-client: oleg615-client: ssh exited with exit code 2 pdsh@oleg615-client: oleg615-server: ssh exited with exit code 2 pdsh@oleg615-client: oleg615-client: ssh exited with exit code 2 pdsh@oleg615-client: oleg615-server: ssh exited with exit code 2