== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:36:44 (1780342604) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg339-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 141 (time 1780342690.73 total 10.00 last 14.10) - open/close 327 (time 1780342701.24 total 20.51 last 17.70) - open/close 511 (time 1780342711.92 total 31.19 last 17.23) - open/close 686 (time 1780342722.48 total 41.75 last 16.57) - open/close 835 (time 1780342732.59 total 51.86 last 14.74) - open/close 942 (time 1780342743.21 total 62.48 last 10.08) - open/close 1146 (time 1780342753.97 total 73.24 last 18.96) - open/close 1305 (time 1780342764.33 total 83.60 last 15.34) - open/close 1501 (time 1780342774.44 total 93.71 last 19.39) - open/close 1602 (time 1780342784.88 total 104.15 last 9.68) - open/close 1775 (time 1780342794.88 total 114.15 last 17.30) - open/close 2062 (time 1780342804.88 total 124.15 last 28.68) - open/close 2115 (time 1780342815.67 total 134.94 last 4.91) - open/close 2240 (time 1780342826.48 total 145.75 last 11.56) - open/close 2376 (time 1780342836.78 total 156.05 last 13.21) - open/close 2644 (time 1780342847.82 total 167.09 last 24.27) - open/close 2826 (time 1780342858.87 total 178.14 last 16.47) - open/close 3006 (time 1780342869.70 total 188.97 last 16.63) - open/close 3215 (time 1780342879.70 total 198.97 last 20.90) - open/close 3361 (time 1780342890.09 total 209.36 last 14.05) - open/close 3497 (time 1780342900.09 total 219.36 last 13.59) - open/close 3623 (time 1780342910.11 total 229.38 last 12.58) - open/close 3827 (time 1780342920.63 total 239.90 last 19.39) - open/close 3992 (time 1780342931.08 total 250.35 last 15.79) - open/close 4137 (time 1780342941.10 total 260.37 last 14.47) - open/close 4317 (time 1780342951.47 total 270.74 last 17.35) - open/close 4540 (time 1780342961.48 total 280.75 last 22.29) - open/close 4792 (time 1780342972.34 total 291.61 last 23.20) - open/close 5008 (time 1780342982.41 total 301.68 last 21.46) - open/close 5198 (time 1780342992.42 total 311.69 last 18.98) - open/close 5389 (time 1780343002.93 total 322.20 last 18.17) - open/close 5643 (time 1780343012.93 total 332.20 last 25.39) - open/close 5908 (time 1780343022.94 total 342.21 last 26.47) - open/close 6130 (time 1780343032.96 total 352.23 last 22.17) - open/close 6257 (time 1780343046.76 total 366.03 last 9.20) - open/close 6512 (time 1780343057.86 total 377.13 last 22.98) - open/close 6715 (time 1780343067.86 total 387.13 last 20.30) - open/close 6937 (time 1780343078.07 total 397.34 last 21.74) - open/close 7228 (time 1780343088.56 total 407.83 last 27.75) - open/close 7504 (time 1780343098.70 total 417.97 last 27.21) - open/close 7715 (time 1780343108.72 total 427.99 last 21.07) - open/close 7927 (time 1780343120.25 total 439.52 last 18.39) - open/close 8199 (time 1780343131.12 total 450.39 last 25.02) - open/close 8470 (time 1780343142.00 total 461.27 last 24.90) - open/close 8748 (time 1780343152.50 total 471.77 last 26.47) - open/close 8990 (time 1780343162.51 total 481.78 last 24.19) - open/close 9250 (time 1780343173.11 total 492.38 last 24.52) - open/close 9493 (time 1780343185.15 total 504.42 last 20.18) - open/close 9779 (time 1780343195.77 total 515.04 last 26.93) - open/close 10000 (time 1780343203.52 total 522.79 last 28.50) - open/close 10221 (time 1780343213.53 total 532.80 last 22.08) - open/close 10476 (time 1780343224.31 total 543.58 last 23.66) - open/close 10663 (time 1780343235.07 total 554.34 last 17.38) - open/close 10853 (time 1780343248.75 total 568.02 last 13.88) - open/close 11146 (time 1780343258.94 total 578.21 last 28.78) - open/close 11400 (time 1780343269.49 total 588.77 last 24.06) - open/close 11682 (time 1780343279.50 total 598.77 last 28.20) - open/close 11905 (time 1780343290.48 total 609.75 last 20.30) - open/close 12187 (time 1780343301.37 total 620.64 last 25.91) - open/close 12428 (time 1780343311.66 total 630.93 last 23.41) - open/close 12628 (time 1780343322.16 total 641.43 last 19.04) - open/close 12883 (time 1780343332.78 total 652.05 last 24.02) - open/close 13132 (time 1780343342.80 total 662.07 last 24.85) - open/close 13383 (time 1780343353.60 total 672.88 last 23.24) - open/close 13622 (time 1780343364.03 total 683.30 last 22.93) - open/close 13874 (time 1780343374.65 total 693.92 last 23.73) - open/close 14115 (time 1780343384.88 total 704.15 last 23.55) - open/close 14365 (time 1780343394.90 total 714.17 last 24.96) - open/close 14591 (time 1780343404.92 total 724.19 last 22.55) - open/close 14822 (time 1780343414.93 total 734.20 last 23.08) - open/close 15059 (time 1780343425.28 total 744.55 last 22.90) - open/close 15324 (time 1780343436.02 total 755.29 last 24.66) - open/close 15559 (time 1780343446.02 total 765.29 last 23.50) - open/close 15814 (time 1780343456.04 total 775.31 last 25.45) - open/close 16068 (time 1780343466.22 total 785.49 last 24.95) - open/close 16337 (time 1780343476.23 total 795.50 last 26.89) - open/close 16572 (time 1780343486.24 total 805.51 last 23.47) - open/close 16809 (time 1780343496.25 total 815.52 last 23.68) - open/close 17045 (time 1780343507.01 total 826.28 last 21.92) - open/close 17244 (time 1780343517.03 total 836.30 last 19.86) - open/close 17560 (time 1780343527.03 total 846.30 last 31.60) - open/close 18375 (time 1780343537.04 total 856.31 last 81.43) - open/close 19157 (time 1780343547.04 total 866.31 last 78.18) - open/close 20000 (time 1780343556.67 total 875.94 last 87.61) - open/close 20967 (time 1780343566.67 total 885.94 last 96.66) - open/close 21901 (time 1780343576.68 total 895.95 last 93.32) - open/close 22752 (time 1780343586.69 total 905.96 last 85.05) - open/close 23651 (time 1780343596.69 total 915.96 last 89.88) - open/close 24424 (time 1780343606.69 total 925.96 last 77.26) - open/close 25246 (time 1780343616.69 total 935.96 last 82.19) - open/close 26081 (time 1780343626.70 total 945.97 last 83.47) - open/close 26899 (time 1780343636.71 total 955.98 last 81.73) - open/close 27749 (time 1780343646.71 total 965.98 last 84.95) - open/close 28580 (time 1780343656.72 total 975.99 last 83.03) - open/close 29449 (time 1780343666.74 total 986.01 last 86.77) - open/close 30000 (time 1780343672.52 total 991.79 last 95.21) - open/close 30880 (time 1780343682.52 total 1001.80 last 88.00) - open/close 31792 (time 1780343692.53 total 1011.80 last 91.17) - open/close 32777 (time 1780343702.53 total 1021.81 last 98.43) - open/close 33725 (time 1780343712.54 total 1031.81 last 94.75) - open/close 34688 (time 1780343722.55 total 1041.82 last 96.17) - open/close 35520 (time 1780343732.56 total 1051.83 last 83.17) - open/close 36327 (time 1780343742.56 total 1061.83 last 80.69) - open/close 37121 (time 1780343752.56 total 1071.83 last 79.35) - open/close 37963 (time 1780343762.57 total 1081.84 last 84.15) - open/close 38750 (time 1780343772.58 total 1091.85 last 78.64) - open/close 39581 (time 1780343782.58 total 1101.85 last 83.09) - open/close 40000 (time 1780343787.57 total 1106.84 last 84.03) - open/close 40805 (time 1780343797.57 total 1116.84 last 80.49) - open/close 41604 (time 1780343807.58 total 1126.85 last 79.77) - open/close 42430 (time 1780343817.60 total 1136.87 last 82.50) - open/close 43270 (time 1780343827.60 total 1146.87 last 83.98) - open/close 44086 (time 1780343837.62 total 1156.89 last 81.45) - open/close 44845 (time 1780343847.62 total 1166.89 last 75.88) - open/close 45667 (time 1780343857.62 total 1176.89 last 82.17) - open/close 46525 (time 1780343867.63 total 1186.90 last 85.74) - open/close 47456 (time 1780343877.63 total 1196.90 last 93.07) - open/close 48339 (time 1780343887.64 total 1206.91 last 88.25) - open/close 49199 (time 1780343897.64 total 1216.91 last 85.95) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1221.60 seconds: 40.63 ops/second - unlinked 0 (time 1780343905 ; total 0 ; last 0) - unlinked 10000 (time 1780344018 ; total 113 ; last 113) - unlinked 20000 (time 1780344110 ; total 205 ; last 92) - unlinked 30000 (time 1780344169 ; total 264 ; last 59) - unlinked 40000 (time 1780344226 ; total 321 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 378 seconds: 131.298935 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2