== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:39:59 (1763347199) start mds service on oleg158-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1763347300.95 total 10.84 last 22.41) - open/close 467 (time 1763347310.97 total 20.86 last 22.35) - open/close 684 (time 1763347321.84 total 31.73 last 19.97) - open/close 902 (time 1763347332.07 total 41.96 last 21.31) - open/close 1081 (time 1763347342.08 total 51.97 last 17.88) - open/close 1291 (time 1763347352.50 total 62.38 last 20.16) - open/close 1549 (time 1763347362.51 total 72.39 last 25.78) - open/close 1730 (time 1763347372.52 total 82.41 last 18.08) - open/close 1886 (time 1763347382.52 total 92.41 last 15.59) - open/close 1929 (time 1763347392.83 total 102.72 last 4.17) - open/close 2108 (time 1763347403.52 total 113.41 last 16.75) - open/close 2328 (time 1763347413.52 total 123.41 last 21.99) - open/close 2553 (time 1763347425.38 total 135.27 last 18.97) - open/close 2785 (time 1763347436.40 total 146.29 last 21.06) - open/close 3031 (time 1763347447.40 total 157.29 last 22.37) - open/close 3272 (time 1763347458.08 total 167.97 last 22.56) - open/close 3401 (time 1763347468.99 total 178.88 last 11.82) - open/close 3584 (time 1763347478.99 total 188.88 last 18.30) - open/close 3796 (time 1763347489.39 total 199.28 last 20.39) - open/close 4059 (time 1763347499.68 total 209.57 last 25.56) - open/close 4263 (time 1763347509.68 total 219.57 last 20.40) - open/close 4487 (time 1763347520.03 total 229.92 last 21.65) - open/close 4726 (time 1763347530.03 total 239.92 last 23.89) - open/close 4947 (time 1763347540.38 total 250.27 last 21.35) - open/close 5179 (time 1763347550.83 total 260.71 last 22.22) - open/close 5412 (time 1763347560.84 total 270.73 last 23.27) - open/close 5576 (time 1763347571.49 total 281.38 last 15.40) - open/close 5617 (time 1763347582.10 total 291.99 last 3.86) - open/close 5830 (time 1763347592.83 total 302.72 last 19.85) - open/close 6076 (time 1763347603.55 total 313.44 last 22.95) - open/close 6239 (time 1763347613.58 total 323.47 last 16.25) - open/close 6384 (time 1763347623.59 total 333.48 last 14.48) - open/close 6481 (time 1763347634.38 total 344.27 last 8.99) - open/close 6610 (time 1763347644.61 total 354.50 last 12.62) - open/close 6747 (time 1763347654.75 total 364.64 last 13.50) - open/close 6998 (time 1763347664.76 total 374.65 last 25.09) - open/close 7277 (time 1763347675.36 total 385.25 last 26.31) - open/close 7489 (time 1763347685.38 total 395.26 last 21.17) - open/close 7729 (time 1763347695.78 total 405.67 last 23.06) - open/close 7963 (time 1763347708.08 total 417.97 last 19.03) - open/close 8040 (time 1763347719.04 total 428.93 last 7.02) - open/close 8322 (time 1763347730.04 total 439.93 last 25.63) - open/close 8575 (time 1763347740.04 total 449.93 last 25.29) - open/close 8820 (time 1763347750.96 total 460.85 last 22.44) - open/close 9072 (time 1763347761.46 total 471.35 last 24.00) - open/close 9247 (time 1763347771.47 total 481.36 last 17.49) - open/close 9492 (time 1763347783.48 total 493.37 last 20.39) - open/close 9752 (time 1763347793.49 total 503.38 last 25.97) - open/close 9933 (time 1763347804.35 total 514.24 last 16.67) - open/close 10095 (time 1763347814.37 total 524.26 last 16.17) - open/close 10125 (time 1763347824.38 total 534.26 last 3.00) - open/close 10239 (time 1763347834.38 total 544.27 last 11.39) - open/close 10458 (time 1763347844.75 total 554.64 last 21.13) - open/close 10620 (time 1763347854.75 total 564.64 last 16.19) - open/close 10764 (time 1763347865.13 total 575.02 last 13.87) - open/close 10969 (time 1763347875.14 total 585.03 last 20.47) - open/close 11207 (time 1763347885.15 total 595.04 last 23.79) - open/close 11446 (time 1763347898.02 total 607.91 last 18.56) - open/close 11589 (time 1763347908.39 total 618.28 last 13.79) - open/close 11805 (time 1763347918.83 total 628.72 last 20.70) - open/close 12065 (time 1763347929.46 total 639.34 last 24.46) - open/close 12247 (time 1763347940.36 total 650.25 last 16.69) - open/close 12420 (time 1763347950.89 total 660.78 last 16.43) - open/close 12655 (time 1763347961.65 total 671.54 last 21.84) - open/close 12896 (time 1763347971.66 total 681.55 last 24.08) - open/close 13163 (time 1763347982.33 total 692.22 last 25.02) - open/close 13418 (time 1763347992.34 total 702.22 last 25.49) - open/close 13569 (time 1763348002.40 total 712.29 last 15.01) - open/close 13788 (time 1763348013.29 total 723.18 last 20.10) - open/close 14048 (time 1763348024.26 total 734.15 last 23.71) - open/close 14309 (time 1763348034.28 total 744.17 last 26.03) - open/close 14485 (time 1763348044.88 total 754.77 last 16.61) - open/close 14714 (time 1763348054.90 total 764.79 last 22.85) - open/close 14835 (time 1763348064.93 total 774.82 last 12.06) - open/close 15036 (time 1763348075.23 total 785.12 last 19.51) - open/close 15280 (time 1763348086.12 total 796.01 last 22.40) - open/close 15552 (time 1763348096.58 total 806.47 last 26.02) - open/close 15793 (time 1763348107.37 total 817.26 last 22.32) - open/close 16031 (time 1763348117.38 total 827.26 last 23.80) - open/close 16249 (time 1763348127.39 total 837.27 last 21.78) - open/close 16447 (time 1763348138.28 total 848.17 last 18.17) - open/close 16683 (time 1763348148.29 total 858.18 last 23.57) - open/close 16890 (time 1763348158.30 total 868.19 last 20.68) - open/close 17104 (time 1763348169.22 total 879.10 last 19.61) - open/close 17304 (time 1763348179.37 total 889.25 last 19.70) - open/close 17527 (time 1763348189.37 total 899.26 last 22.30) - open/close 17783 (time 1763348199.91 total 909.80 last 24.29) - open/close 17985 (time 1763348210.02 total 919.90 last 19.98) - open/close 18258 (time 1763348220.02 total 929.91 last 27.29) - open/close 18503 (time 1763348230.57 total 940.46 last 23.22) - open/close 19138 (time 1763348240.58 total 950.47 last 63.41) - open/close 19870 (time 1763348250.59 total 960.48 last 73.13) - open/close 20655 (time 1763348260.59 total 970.48 last 78.50) - open/close 21466 (time 1763348270.60 total 980.49 last 81.04) - open/close 22245 (time 1763348280.60 total 990.49 last 77.87) - open/close 22947 (time 1763348290.62 total 1000.51 last 70.09) - open/close 23674 (time 1763348300.63 total 1010.52 last 72.62) - open/close 24394 (time 1763348310.64 total 1020.53 last 71.95) - open/close 25105 (time 1763348320.64 total 1030.53 last 71.05) - open/close 25820 (time 1763348330.65 total 1040.54 last 71.48) - open/close 26509 (time 1763348340.66 total 1050.55 last 68.79) - open/close 27110 (time 1763348350.67 total 1060.55 last 60.09) - open/close 27831 (time 1763348360.68 total 1070.57 last 72.01) - open/close 28560 (time 1763348370.68 total 1080.57 last 72.86) - open/close 29275 (time 1763348380.68 total 1090.57 last 71.50) - open/close 29919 (time 1763348390.69 total 1100.58 last 64.35) - open/close 30557 (time 1763348400.69 total 1110.58 last 63.77) - open/close 31329 (time 1763348410.71 total 1120.60 last 77.05) - open/close 32072 (time 1763348420.72 total 1130.61 last 74.27) - open/close 32801 (time 1763348430.72 total 1140.61 last 72.87) - open/close 33560 (time 1763348440.72 total 1150.61 last 75.89) - open/close 34323 (time 1763348450.73 total 1160.62 last 76.25) - open/close 35102 (time 1763348460.73 total 1170.62 last 77.86) - open/close 35781 (time 1763348470.74 total 1180.63 last 67.85) - open/close 36466 (time 1763348480.74 total 1190.63 last 68.48) - open/close 37192 (time 1763348490.76 total 1200.65 last 72.51) - open/close 37945 (time 1763348500.77 total 1210.66 last 75.18) - open/close 38645 (time 1763348510.79 total 1220.68 last 69.88) - open/close 39307 (time 1763348520.80 total 1230.69 last 66.14) - open/close 39997 (time 1763348530.81 total 1240.70 last 68.91) - open/close 40599 (time 1763348540.82 total 1250.71 last 60.17) - open/close 41335 (time 1763348550.83 total 1260.72 last 73.52) - open/close 42047 (time 1763348560.83 total 1270.72 last 71.20) - open/close 42739 (time 1763348570.84 total 1280.73 last 69.11) - open/close 43475 (time 1763348580.85 total 1290.74 last 73.50) - open/close 44167 (time 1763348590.87 total 1300.76 last 69.07) - open/close 44897 (time 1763348600.88 total 1310.77 last 72.92) - open/close 45664 (time 1763348610.89 total 1320.77 last 76.69) - open/close 46438 (time 1763348620.89 total 1330.78 last 77.39) - open/close 47173 (time 1763348630.89 total 1340.78 last 73.48) - open/close 47878 (time 1763348640.92 total 1350.81 last 70.30) - open/close 48604 (time 1763348650.92 total 1360.81 last 72.56) - open/close 49242 (time 1763348660.93 total 1370.82 last 63.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1376.51 seconds: 36.06 ops/second - unlinked 0 (time 1763348670 ; total 0 ; last 0) - unlinked 10000 (time 1763348806 ; total 136 ; last 136) - unlinked 20000 (time 1763348947 ; total 277 ; last 141) - unlinked 30000 (time 1763349081 ; total 411 ; last 134) - unlinked 40000 (time 1763349213 ; total 543 ; last 132) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 676 seconds: 73.420120 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2