== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:24:20 (1787444660) start mds service on oleg601-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg601-server' oleg601-server: oleg601-server.virtnet: executing load_modules_local oleg601-server: Loading modules from /home/green/git/lustre-release/lustre oleg601-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg601-server: detected 4 online CPUs by sysfs oleg601-server: MODOPTS_LIBCFS= oleg601-server: Force libcfs to create 2 CPU partitions oleg601-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg601-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg601-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg601-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg601-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 276 (time 1787444755.40 total 10.23 last 26.97) - open/close 556 (time 1787444765.55 total 20.39 last 27.58) - open/close 805 (time 1787444775.57 total 30.41 last 24.85) - open/close 1022 (time 1787444785.57 total 40.41 last 21.70) - open/close 1289 (time 1787444796.26 total 51.09 last 24.98) - open/close 1589 (time 1787444807.16 total 62.00 last 27.51) - open/close 1744 (time 1787444819.75 total 74.59 last 12.31) - open/close 1969 (time 1787444829.77 total 84.60 last 22.47) - open/close 2322 (time 1787444840.24 total 95.08 last 33.70) - open/close 2594 (time 1787444850.80 total 105.64 last 25.76) - open/close 2815 (time 1787444860.81 total 115.65 last 22.08) - open/close 3067 (time 1787444870.81 total 125.65 last 25.19) - open/close 3345 (time 1787444880.82 total 135.65 last 27.78) - open/close 3553 (time 1787444891.58 total 146.42 last 19.33) - open/close 3752 (time 1787444901.59 total 156.42 last 19.89) - open/close 3957 (time 1787444912.12 total 166.95 last 19.47) - open/close 4233 (time 1787444922.92 total 177.75 last 25.56) - open/close 4527 (time 1787444933.66 total 188.50 last 27.35) - open/close 4836 (time 1787444944.69 total 199.52 last 28.04) - open/close 5026 (time 1787444955.24 total 210.08 last 18.00) - open/close 5313 (time 1787444965.81 total 220.64 last 27.17) - open/close 5617 (time 1787444975.81 total 230.64 last 30.40) - open/close 5851 (time 1787444986.29 total 241.13 last 22.32) - open/close 6095 (time 1787444998.19 total 253.02 last 20.52) - open/close 6242 (time 1787445008.56 total 263.39 last 14.17) - open/close 6428 (time 1787445019.38 total 274.21 last 17.19) - open/close 6571 (time 1787445029.38 total 284.22 last 14.29) - open/close 6867 (time 1787445040.05 total 294.88 last 27.75) - open/close 7088 (time 1787445050.27 total 305.11 last 21.61) - open/close 7374 (time 1787445060.28 total 315.11 last 28.59) - open/close 7542 (time 1787445071.01 total 325.85 last 15.65) - open/close 7687 (time 1787445081.46 total 336.29 last 13.89) - open/close 7946 (time 1787445093.80 total 348.64 last 20.98) - open/close 8155 (time 1787445104.55 total 359.39 last 19.45) - open/close 8353 (time 1787445114.56 total 369.39 last 19.78) - open/close 8521 (time 1787445124.97 total 379.80 last 16.14) - open/close 8780 (time 1787445135.36 total 390.20 last 24.91) - open/close 9028 (time 1787445146.53 total 401.37 last 22.20) - open/close 9332 (time 1787445156.79 total 411.63 last 29.63) - open/close 9600 (time 1787445168.87 total 423.71 last 22.19) - open/close 9761 (time 1787445178.92 total 433.76 last 16.02) - open/close 10000 (time 1787445187.38 total 442.22 last 28.26) - open/close 10211 (time 1787445197.39 total 452.22 last 21.09) - open/close 10443 (time 1787445207.41 total 462.25 last 23.14) - open/close 10755 (time 1787445217.43 total 472.27 last 31.14) - open/close 11020 (time 1787445228.22 total 483.06 last 24.55) - open/close 11220 (time 1787445238.88 total 493.72 last 18.77) - open/close 11564 (time 1787445249.84 total 504.68 last 31.38) - open/close 11875 (time 1787445260.53 total 515.37 last 29.09) - open/close 12160 (time 1787445271.60 total 526.43 last 25.76) - open/close 12433 (time 1787445282.30 total 537.14 last 25.51) - open/close 12694 (time 1787445292.77 total 547.61 last 24.92) - open/close 12942 (time 1787445302.77 total 557.61 last 24.80) - open/close 13225 (time 1787445313.03 total 567.87 last 27.58) - open/close 13472 (time 1787445324.72 total 579.56 last 21.14) - open/close 13775 (time 1787445335.47 total 590.30 last 28.19) - open/close 14019 (time 1787445345.47 total 600.31 last 24.39) - open/close 14285 (time 1787445355.94 total 610.78 last 25.40) - open/close 14583 (time 1787445366.76 total 621.60 last 27.54) - open/close 14902 (time 1787445377.46 total 632.29 last 29.83) - open/close 15191 (time 1787445388.46 total 643.29 last 26.28) - open/close 15443 (time 1787445398.70 total 653.53 last 24.61) - open/close 15743 (time 1787445408.70 total 663.53 last 29.99) - open/close 15912 (time 1787445419.63 total 674.47 last 15.45) - open/close 16074 (time 1787445429.80 total 684.64 last 15.93) - open/close 16334 (time 1787445440.61 total 695.44 last 24.07) - open/close 16625 (time 1787445451.20 total 706.03 last 27.48) - open/close 16850 (time 1787445461.21 total 716.05 last 22.47) - open/close 17126 (time 1787445472.23 total 727.06 last 25.06) - open/close 17393 (time 1787445482.80 total 737.64 last 25.24) - open/close 17735 (time 1787445495.62 total 750.46 last 26.68) - open/close 17811 (time 1787445505.64 total 760.47 last 7.59) - open/close 18017 (time 1787445515.89 total 770.72 last 20.09) - open/close 18226 (time 1787445526.70 total 781.54 last 19.33) - open/close 18458 (time 1787445536.75 total 791.59 last 23.09) - open/close 18531 (time 1787445546.77 total 801.61 last 7.29) - open/close 18705 (time 1787445556.79 total 811.62 last 17.37) - open/close 18883 (time 1787445566.79 total 821.62 last 17.80) - open/close 19117 (time 1787445577.13 total 831.97 last 22.62) - open/close 19442 (time 1787445588.19 total 843.03 last 29.38) - open/close 19650 (time 1787445599.01 total 853.85 last 19.22) - open/close 19911 (time 1787445609.61 total 864.44 last 24.64) - open/close 20000 (time 1787445613.23 total 868.07 last 24.54) - open/close 20189 (time 1787445623.34 total 878.17 last 18.70) - open/close 20416 (time 1787445634.09 total 888.93 last 21.11) - open/close 20713 (time 1787445644.72 total 899.55 last 27.95) - open/close 20969 (time 1787445654.84 total 909.68 last 25.28) - open/close 21254 (time 1787445665.60 total 920.43 last 26.50) - open/close 21535 (time 1787445675.60 total 930.44 last 28.09) - open/close 22539 (time 1787445685.61 total 940.44 last 100.35) - open/close 23533 (time 1787445695.61 total 950.44 last 99.38) - open/close 24600 (time 1787445705.61 total 960.45 last 106.67) - open/close 25585 (time 1787445715.62 total 970.45 last 98.45) - open/close 26586 (time 1787445725.62 total 980.46 last 100.06) - open/close 27524 (time 1787445735.62 total 990.46 last 93.78) - open/close 28517 (time 1787445745.63 total 1000.47 last 99.21) - open/close 29446 (time 1787445755.63 total 1010.47 last 92.86) - open/close 30000 (time 1787445761.44 total 1016.28 last 95.39) - open/close 30937 (time 1787445771.44 total 1026.28 last 93.70) - open/close 32009 (time 1787445781.45 total 1036.29 last 107.11) - open/close 32970 (time 1787445791.46 total 1046.30 last 96.02) - open/close 33665 (time 1787445801.46 total 1056.30 last 69.50) - open/close 34578 (time 1787445811.46 total 1066.30 last 91.29) - open/close 35486 (time 1787445821.46 total 1076.30 last 90.76) - open/close 36472 (time 1787445831.47 total 1086.30 last 98.58) - open/close 37411 (time 1787445841.47 total 1096.31 last 93.88) - open/close 38399 (time 1787445851.47 total 1106.31 last 98.79) - open/close 39167 (time 1787445861.47 total 1116.31 last 76.77) - open/close 40000 (time 1787445869.66 total 1124.50 last 101.71) - open/close 41001 (time 1787445879.67 total 1134.51 last 100.00) - open/close 41942 (time 1787445889.68 total 1144.51 last 94.09) - open/close 42963 (time 1787445899.68 total 1154.52 last 102.02) - open/close 43946 (time 1787445909.68 total 1164.52 last 98.28) - open/close 44972 (time 1787445919.69 total 1174.53 last 102.54) - open/close 45953 (time 1787445929.70 total 1184.53 last 98.06) - open/close 46859 (time 1787445939.70 total 1194.54 last 90.54) - open/close 47766 (time 1787445949.71 total 1204.55 last 90.61) - open/close 48773 (time 1787445959.71 total 1214.55 last 100.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1223.64 seconds: 40.56 ops/second - unlinked 0 (time 1787445972 ; total 0 ; last 0) - unlinked 10000 (time 1787446083 ; total 111 ; last 111) - unlinked 20000 (time 1787446190 ; total 218 ; last 107) - unlinked 30000 (time 1787446247 ; total 275 ; last 57) - unlinked 40000 (time 1787446304 ; total 332 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 389 seconds: 127.586121 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg601-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg601-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg601-server stop mds service on oleg601-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg601-server stop mds service on oleg601-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg601-server unloading modules via unload_modules_local on: 'oleg601-server' oleg601-server: oleg601-server.virtnet: executing unload_modules_local oleg601-server: modules unloaded. pdsh@oleg601-client: oleg601-server: ssh exited with exit code 2 pdsh@oleg601-client: oleg601-client: ssh exited with exit code 2 pdsh@oleg601-client: oleg601-client: ssh exited with exit code 2 pdsh@oleg601-client: oleg601-server: ssh exited with exit code 2