== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:42:21 (1788169341) start mds service on oleg214-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg214-server: detected 4 online CPUs by sysfs oleg214-server: MODOPTS_LIBCFS= oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg214-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg214-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 312 (time 1788169447.67 total 10.01 last 31.17) - open/close 402 (time 1788169457.69 total 20.03 last 8.99) - open/close 637 (time 1788169467.93 total 30.27 last 22.94) - open/close 926 (time 1788169478.50 total 40.84 last 27.34) - open/close 1086 (time 1788169488.52 total 50.86 last 15.96) - open/close 1219 (time 1788169498.96 total 61.30 last 12.74) - open/close 1336 (time 1788169509.08 total 71.42 last 11.57) - open/close 1468 (time 1788169519.71 total 82.05 last 12.41) - open/close 1743 (time 1788169529.72 total 92.06 last 27.48) - open/close 2003 (time 1788169540.25 total 102.59 last 24.68) - open/close 2234 (time 1788169550.26 total 112.60 last 23.09) - open/close 2450 (time 1788169560.73 total 123.06 last 20.64) - open/close 2709 (time 1788169571.49 total 133.83 last 24.06) - open/close 3000 (time 1788169581.50 total 143.84 last 29.08) - open/close 3238 (time 1788169591.50 total 153.84 last 23.79) - open/close 3470 (time 1788169601.51 total 163.85 last 23.18) - open/close 3697 (time 1788169611.60 total 173.94 last 22.50) - open/close 3969 (time 1788169621.86 total 184.19 last 26.53) - open/close 4259 (time 1788169631.87 total 194.21 last 28.96) - open/close 4466 (time 1788169641.90 total 204.24 last 20.63) - open/close 4757 (time 1788169652.56 total 214.90 last 27.30) - open/close 5018 (time 1788169662.80 total 225.14 last 25.49) - open/close 5306 (time 1788169673.89 total 236.23 last 25.98) - open/close 5453 (time 1788169683.99 total 246.33 last 14.56) - open/close 5715 (time 1788169694.00 total 256.34 last 26.18) - open/close 5963 (time 1788169704.14 total 266.48 last 24.44) - open/close 6201 (time 1788169714.44 total 276.78 last 23.12) - open/close 6510 (time 1788169724.45 total 286.79 last 30.87) - open/close 6773 (time 1788169734.46 total 296.80 last 26.26) - open/close 7013 (time 1788169744.53 total 306.87 last 23.83) - open/close 7239 (time 1788169754.58 total 316.92 last 22.48) - open/close 7511 (time 1788169765.32 total 327.66 last 25.33) - open/close 7786 (time 1788169775.55 total 337.89 last 26.89) - open/close 8001 (time 1788169788.35 total 350.69 last 16.80) - open/close 8257 (time 1788169798.46 total 360.80 last 25.34) - open/close 8436 (time 1788169808.47 total 370.81 last 17.88) - open/close 8589 (time 1788169818.71 total 381.05 last 14.94) - open/close 8835 (time 1788169829.61 total 391.95 last 22.57) - open/close 9084 (time 1788169839.69 total 402.02 last 24.71) - open/close 9169 (time 1788169850.65 total 412.99 last 7.75) - open/close 9310 (time 1788169861.04 total 423.38 last 13.57) - open/close 9617 (time 1788169872.02 total 434.36 last 27.97) - open/close 9815 (time 1788169882.60 total 444.94 last 18.71) - open/close 10000 (time 1788169889.83 total 452.17 last 25.59) - open/close 10219 (time 1788169902.90 total 465.24 last 16.75) - open/close 10366 (time 1788169913.50 total 475.83 last 13.87) - open/close 10571 (time 1788169923.50 total 485.84 last 20.49) - open/close 10789 (time 1788169933.61 total 495.95 last 21.55) - open/close 11011 (time 1788169943.62 total 505.96 last 22.19) - open/close 11271 (time 1788169954.45 total 516.79 last 24.00) - open/close 11635 (time 1788169964.47 total 526.81 last 36.32) - open/close 11911 (time 1788169975.19 total 537.53 last 25.75) - open/close 12192 (time 1788169985.40 total 547.74 last 27.52) - open/close 12517 (time 1788169995.41 total 557.75 last 32.46) - open/close 12729 (time 1788170005.43 total 567.77 last 21.16) - open/close 13051 (time 1788170016.43 total 578.77 last 29.29) - open/close 13263 (time 1788170027.45 total 589.79 last 19.24) - open/close 13529 (time 1788170037.45 total 599.79 last 26.58) - open/close 13779 (time 1788170048.09 total 610.43 last 23.50) - open/close 14038 (time 1788170058.64 total 620.98 last 24.55) - open/close 14287 (time 1788170068.89 total 631.23 last 24.29) - open/close 14551 (time 1788170079.25 total 641.59 last 25.47) - open/close 14809 (time 1788170090.21 total 652.55 last 23.55) - open/close 15089 (time 1788170100.95 total 663.29 last 26.06) - open/close 15368 (time 1788170110.97 total 673.31 last 27.84) - open/close 15481 (time 1788170123.70 total 686.04 last 8.88) - open/close 15716 (time 1788170134.25 total 696.58 last 22.28) - open/close 15949 (time 1788170144.71 total 707.05 last 22.26) - open/close 16197 (time 1788170157.77 total 720.11 last 19.00) - open/close 16460 (time 1788170168.75 total 731.09 last 23.95) - open/close 16734 (time 1788170180.69 total 743.03 last 22.94) - open/close 17019 (time 1788170191.31 total 753.65 last 26.83) - open/close 17297 (time 1788170202.14 total 764.48 last 25.68) - open/close 17590 (time 1788170213.10 total 775.44 last 26.72) - open/close 17819 (time 1788170223.11 total 785.45 last 22.88) - open/close 18037 (time 1788170233.20 total 795.54 last 21.59) - open/close 18165 (time 1788170243.22 total 805.56 last 12.77) - open/close 18394 (time 1788170253.23 total 815.57 last 22.89) - open/close 18592 (time 1788170263.24 total 825.58 last 19.77) - open/close 18817 (time 1788170273.68 total 836.02 last 21.55) - open/close 18942 (time 1788170284.23 total 846.57 last 11.86) - open/close 19185 (time 1788170296.79 total 859.13 last 19.34) - open/close 19387 (time 1788170306.82 total 869.16 last 20.15) - open/close 19642 (time 1788170317.65 total 879.99 last 23.53) - open/close 19779 (time 1788170328.55 total 890.89 last 12.57) - open/close 20000 (time 1788170337.08 total 899.42 last 25.91) - open/close 20263 (time 1788170347.10 total 909.44 last 26.24) - open/close 20493 (time 1788170357.28 total 919.62 last 22.60) - open/close 20721 (time 1788170367.37 total 929.71 last 22.58) - open/close 21011 (time 1788170378.20 total 940.54 last 26.77) - open/close 21510 (time 1788170388.21 total 950.55 last 49.89) - open/close 22187 (time 1788170398.21 total 960.55 last 67.65) - open/close 22943 (time 1788170408.24 total 970.58 last 75.42) - open/close 23578 (time 1788170418.24 total 980.58 last 63.48) - open/close 24397 (time 1788170428.24 total 990.58 last 81.88) - open/close 25238 (time 1788170438.26 total 1000.60 last 83.98) - open/close 26295 (time 1788170448.26 total 1010.60 last 105.67) - open/close 27435 (time 1788170458.26 total 1020.60 last 113.97) - open/close 28284 (time 1788170468.27 total 1030.61 last 84.88) - open/close 29103 (time 1788170478.27 total 1040.61 last 81.84) - open/close 30000 (time 1788170488.03 total 1050.37 last 91.93) - open/close 31027 (time 1788170498.03 total 1060.37 last 102.67) - open/close 32095 (time 1788170508.04 total 1070.38 last 106.78) - open/close 33162 (time 1788170518.04 total 1080.38 last 106.65) - open/close 34205 (time 1788170528.05 total 1090.39 last 104.20) - open/close 35058 (time 1788170538.06 total 1100.40 last 85.22) - open/close 35984 (time 1788170548.07 total 1110.41 last 92.50) - open/close 36877 (time 1788170558.08 total 1120.42 last 89.25) - open/close 37510 (time 1788170568.08 total 1130.42 last 63.28) - open/close 37962 (time 1788170578.09 total 1140.43 last 45.17) - open/close 38709 (time 1788170588.10 total 1150.44 last 74.64) - open/close 39424 (time 1788170598.10 total 1160.44 last 71.44) - open/close 40000 (time 1788170604.04 total 1166.38 last 96.97) - open/close 40843 (time 1788170614.07 total 1176.41 last 84.09) - open/close 41769 (time 1788170624.07 total 1186.41 last 92.58) - open/close 42631 (time 1788170634.08 total 1196.42 last 86.17) - open/close 43299 (time 1788170644.09 total 1206.43 last 66.73) - open/close 44240 (time 1788170654.09 total 1216.43 last 94.03) - open/close 45150 (time 1788170664.09 total 1226.43 last 91.00) - open/close 45922 (time 1788170674.10 total 1236.43 last 77.19) - open/close 46556 (time 1788170684.10 total 1246.44 last 63.39) - open/close 47179 (time 1788170694.12 total 1256.46 last 62.18) - open/close 47964 (time 1788170704.13 total 1266.47 last 78.43) - open/close 48907 (time 1788170714.13 total 1276.47 last 94.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1284.56 seconds: 38.64 ops/second - unlinked 0 (time 1788170726 ; total 0 ; last 0) - unlinked 10000 (time 1788170847 ; total 121 ; last 121) - unlinked 20000 (time 1788170970 ; total 244 ; last 123) - unlinked 30000 (time 1788171046 ; total 320 ; last 76) - unlinked 40000 (time 1788171111 ; total 385 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 453 seconds: 109.560707 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: modules unloaded. pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2