== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:40:57 (1786952457) start mds service on oleg120-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg120-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 144 (time 1786952573.71 total 10.00 last 14.40) - open/close 274 (time 1786952584.12 total 20.41 last 12.49) - open/close 375 (time 1786952594.66 total 30.95 last 9.58) - open/close 551 (time 1786952604.67 total 40.96 last 17.59) - open/close 715 (time 1786952614.89 total 51.18 last 16.04) - open/close 970 (time 1786952625.54 total 61.83 last 23.94) - open/close 1078 (time 1786952636.37 total 72.66 last 9.97) - open/close 1330 (time 1786952646.39 total 82.68 last 25.16) - open/close 1504 (time 1786952656.42 total 92.71 last 17.34) - open/close 1591 (time 1786952666.67 total 102.96 last 8.49) - open/close 1659 (time 1786952676.96 total 113.25 last 6.61) - open/close 1835 (time 1786952687.14 total 123.43 last 17.29) - open/close 1986 (time 1786952697.68 total 133.97 last 14.32) - open/close 2085 (time 1786952708.43 total 144.72 last 9.21) - open/close 2210 (time 1786952718.44 total 154.72 last 12.49) - open/close 2415 (time 1786952728.48 total 164.77 last 20.42) - open/close 2503 (time 1786952738.93 total 175.22 last 8.41) - open/close 2632 (time 1786952749.16 total 185.45 last 12.62) - open/close 2892 (time 1786952761.36 total 197.65 last 21.31) - open/close 2983 (time 1786952771.85 total 208.14 last 8.68) - open/close 3244 (time 1786952781.86 total 218.15 last 26.07) - open/close 3406 (time 1786952792.53 total 228.82 last 15.18) - open/close 3513 (time 1786952803.10 total 239.39 last 10.12) - open/close 3701 (time 1786952813.98 total 250.27 last 17.28) - open/close 3817 (time 1786952823.99 total 260.28 last 11.59) - open/close 3903 (time 1786952834.40 total 270.69 last 8.26) - open/close 3954 (time 1786952845.14 total 281.43 last 4.75) - open/close 4168 (time 1786952855.84 total 292.13 last 20.00) - open/close 4376 (time 1786952866.00 total 302.28 last 20.48) - open/close 4554 (time 1786952876.82 total 313.11 last 16.44) - open/close 4760 (time 1786952886.83 total 323.12 last 20.58) - open/close 4907 (time 1786952897.44 total 333.73 last 13.86) - open/close 5054 (time 1786952908.09 total 344.38 last 13.80) - open/close 5156 (time 1786952918.12 total 354.41 last 10.17) - open/close 5370 (time 1786952928.53 total 364.81 last 20.57) - open/close 5466 (time 1786952938.91 total 375.20 last 9.25) - open/close 5745 (time 1786952948.92 total 385.21 last 27.87) - open/close 6008 (time 1786952959.01 total 395.30 last 26.08) - open/close 6181 (time 1786952969.78 total 406.06 last 16.06) - open/close 6336 (time 1786952980.30 total 416.59 last 14.72) - open/close 6513 (time 1786952991.13 total 427.42 last 16.34) - open/close 6544 (time 1786953003.10 total 439.39 last 2.59) - open/close 6660 (time 1786953015.69 total 451.98 last 9.22) - open/close 6870 (time 1786953025.70 total 461.99 last 20.97) - open/close 7077 (time 1786953035.72 total 472.01 last 20.66) - open/close 7241 (time 1786953046.30 total 482.59 last 15.50) - open/close 7513 (time 1786953056.30 total 492.59 last 27.19) - open/close 7706 (time 1786953066.31 total 502.60 last 19.28) - open/close 7785 (time 1786953077.35 total 513.64 last 7.16) - open/close 7971 (time 1786953087.37 total 523.66 last 18.56) - open/close 8044 (time 1786953097.39 total 533.68 last 7.29) - open/close 8192 (time 1786953107.39 total 543.68 last 14.79) - open/close 8300 (time 1786953117.85 total 554.14 last 10.33) - open/close 8506 (time 1786953128.48 total 564.77 last 19.39) - open/close 8703 (time 1786953139.53 total 575.82 last 17.83) - open/close 8805 (time 1786953150.15 total 586.44 last 9.60) - open/close 8978 (time 1786953160.22 total 596.51 last 17.19) - open/close 9097 (time 1786953170.23 total 606.52 last 11.89) - open/close 9328 (time 1786953180.88 total 617.17 last 21.70) - open/close 9419 (time 1786953191.14 total 627.43 last 8.87) - open/close 9612 (time 1786953201.15 total 637.44 last 19.27) - open/close 9751 (time 1786953211.16 total 647.45 last 13.89) - open/close 9882 (time 1786953222.17 total 658.46 last 11.90) - open/close 10000 (time 1786953226.66 total 662.95 last 26.26) - open/close 10190 (time 1786953239.67 total 675.96 last 14.61) - open/close 10363 (time 1786953251.13 total 687.42 last 15.10) - open/close 10515 (time 1786953261.66 total 697.95 last 14.43) - open/close 10649 (time 1786953273.58 total 709.87 last 11.24) - open/close 10879 (time 1786953283.59 total 719.88 last 22.97) - open/close 10900 (time 1786953294.62 total 730.91 last 1.90) - open/close 10994 (time 1786953304.65 total 740.94 last 9.37) - open/close 11216 (time 1786953314.92 total 751.21 last 21.61) - open/close 11500 (time 1786953325.26 total 761.54 last 27.49) - open/close 11734 (time 1786953335.45 total 771.74 last 22.94) - open/close 11885 (time 1786953345.46 total 781.75 last 15.09) - open/close 12128 (time 1786953356.26 total 792.55 last 22.51) - open/close 12216 (time 1786953368.73 total 805.02 last 7.06) - open/close 12315 (time 1786953379.81 total 816.10 last 8.94) - open/close 12496 (time 1786953390.68 total 826.97 last 16.65) - open/close 12695 (time 1786953401.17 total 837.46 last 18.96) - open/close 12880 (time 1786953411.18 total 847.46 last 18.49) - open/close 13023 (time 1786953421.20 total 857.49 last 14.27) - open/close 13246 (time 1786953431.50 total 867.79 last 21.64) - open/close 13490 (time 1786953441.51 total 877.80 last 24.38) - open/close 13694 (time 1786953451.53 total 887.82 last 20.36) - open/close 13778 (time 1786953462.22 total 898.51 last 7.85) - open/close 14003 (time 1786953472.24 total 908.53 last 22.47) - open/close 14239 (time 1786953482.83 total 919.12 last 22.28) - open/close 14510 (time 1786953493.84 total 930.13 last 24.62) - open/close 14631 (time 1786953504.52 total 940.81 last 11.33) - open/close 14739 (time 1786953514.52 total 950.81 last 10.80) - open/close 14853 (time 1786953526.80 total 963.09 last 9.28) - open/close 15001 (time 1786953536.81 total 973.09 last 14.80) - open/close 15202 (time 1786953548.38 total 984.67 last 17.37) - open/close 15420 (time 1786953559.31 total 995.60 last 19.93) - open/close 15561 (time 1786953569.50 total 1005.79 last 13.85) - open/close 15728 (time 1786953579.80 total 1016.09 last 16.21) - open/close 15919 (time 1786953590.04 total 1026.33 last 18.64) - open/close 16145 (time 1786953600.04 total 1036.33 last 22.60) - open/close 16339 (time 1786953610.06 total 1046.35 last 19.37) - open/close 16570 (time 1786953620.07 total 1056.36 last 23.07) - open/close 17251 (time 1786953630.08 total 1066.37 last 68.06) - open/close 18144 (time 1786953640.08 total 1076.37 last 89.25) - open/close 18922 (time 1786953650.09 total 1086.38 last 77.74) - open/close 19946 (time 1786953660.09 total 1096.38 last 102.39) - open/close 20927 (time 1786953670.10 total 1106.39 last 98.07) - open/close 21881 (time 1786953680.10 total 1116.39 last 95.36) - open/close 22970 (time 1786953690.10 total 1126.39 last 108.86) - open/close 24014 (time 1786953700.11 total 1136.40 last 104.34) - open/close 24921 (time 1786953710.12 total 1146.41 last 90.62) - open/close 25921 (time 1786953720.12 total 1156.41 last 99.96) - open/close 26681 (time 1786953730.13 total 1166.42 last 75.95) - open/close 27425 (time 1786953740.14 total 1176.43 last 74.31) - open/close 28199 (time 1786953750.16 total 1186.45 last 77.26) - open/close 28927 (time 1786953760.17 total 1196.46 last 72.76) - open/close 29799 (time 1786953770.17 total 1206.46 last 87.13) - open/close 30704 (time 1786953780.18 total 1216.47 last 90.49) - open/close 31696 (time 1786953790.19 total 1226.48 last 99.06) - open/close 32614 (time 1786953800.19 total 1236.48 last 91.79) - open/close 33596 (time 1786953810.19 total 1246.48 last 98.18) - open/close 34446 (time 1786953820.22 total 1256.51 last 84.82) - open/close 35120 (time 1786953830.22 total 1266.51 last 67.39) - open/close 35944 (time 1786953840.22 total 1276.51 last 82.35) - open/close 36886 (time 1786953850.22 total 1286.51 last 94.19) - open/close 37834 (time 1786953860.23 total 1296.52 last 94.78) - open/close 38529 (time 1786953870.23 total 1306.52 last 69.45) - open/close 39303 (time 1786953880.24 total 1316.53 last 77.37) - open/close 40000 (time 1786953889.07 total 1325.36 last 78.95) - open/close 40884 (time 1786953899.07 total 1335.36 last 88.38) - open/close 41727 (time 1786953909.07 total 1345.36 last 84.29) - open/close 42875 (time 1786953919.08 total 1355.37 last 114.75) - open/close 43922 (time 1786953929.09 total 1365.38 last 104.57) - open/close 44671 (time 1786953939.11 total 1375.40 last 74.74) - open/close 45116 (time 1786953949.13 total 1385.42 last 44.40) - open/close 45889 (time 1786953959.14 total 1395.43 last 77.26) - open/close 47053 (time 1786953969.14 total 1405.43 last 116.38) - open/close 48085 (time 1786953979.15 total 1415.44 last 103.12) - open/close 48981 (time 1786953989.15 total 1425.44 last 89.53) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1432.38 seconds: 34.65 ops/second - unlinked 0 (time 1786954000 ; total 0 ; last 0) - unlinked 10000 (time 1786954113 ; total 113 ; last 113) - unlinked 20000 (time 1786954201 ; total 201 ; last 88) - unlinked 30000 (time 1786954271 ; total 271 ; last 70) - unlinked 40000 (time 1786954340 ; total 340 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 396 seconds: 125.330811 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2