== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:56:18 (1787864178) start mds service on oleg140-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg140-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 143 (time 1787864296.01 total 10.68 last 13.38) - open/close 261 (time 1787864307.12 total 21.79 last 10.62) - open/close 394 (time 1787864317.60 total 32.28 last 12.68) - open/close 565 (time 1787864327.98 total 42.66 last 16.47) - open/close 712 (time 1787864337.99 total 52.67 last 14.69) - open/close 865 (time 1787864348.30 total 62.98 last 14.84) - open/close 1030 (time 1787864359.28 total 73.95 last 15.04) - open/close 1297 (time 1787864369.35 total 84.02 last 26.51) - open/close 1503 (time 1787864379.35 total 94.03 last 20.58) - open/close 1675 (time 1787864389.89 total 104.57 last 16.32) - open/close 1939 (time 1787864400.89 total 115.57 last 23.99) - open/close 2200 (time 1787864411.81 total 126.49 last 23.90) - open/close 2396 (time 1787864422.22 total 136.89 last 18.84) - open/close 2667 (time 1787864432.22 total 146.89 last 27.10) - open/close 2796 (time 1787864444.61 total 159.28 last 10.41) - open/close 2919 (time 1787864454.82 total 169.49 last 12.05) - open/close 3079 (time 1787864465.22 total 179.89 last 15.39) - open/close 3192 (time 1787864476.17 total 190.84 last 10.32) - open/close 3429 (time 1787864486.17 total 200.85 last 23.69) - open/close 3584 (time 1787864496.18 total 210.85 last 15.49) - open/close 3847 (time 1787864506.71 total 221.39 last 24.96) - open/close 4036 (time 1787864516.87 total 231.54 last 18.61) - open/close 4253 (time 1787864526.88 total 241.55 last 21.68) - open/close 4477 (time 1787864537.29 total 251.96 last 21.52) - open/close 4737 (time 1787864547.29 total 261.96 last 25.99) - open/close 4873 (time 1787864557.38 total 272.05 last 13.48) - open/close 5122 (time 1787864567.57 total 282.25 last 24.43) - open/close 5357 (time 1787864578.51 total 293.18 last 21.48) - open/close 5496 (time 1787864589.46 total 304.13 last 12.70) - open/close 5634 (time 1787864599.47 total 314.14 last 13.79) - open/close 5892 (time 1787864609.47 total 324.15 last 25.77) - open/close 6009 (time 1787864620.24 total 334.92 last 10.87) - open/close 6201 (time 1787864630.42 total 345.09 last 18.86) - open/close 6445 (time 1787864655.58 total 370.25 last 9.70) - open/close 6600 (time 1787864666.49 total 381.17 last 14.20) - open/close 6817 (time 1787864677.14 total 391.81 last 20.39) - open/close 7057 (time 1787864687.14 total 401.82 last 23.99) - open/close 7232 (time 1787864697.56 total 412.23 last 16.80) - open/close 7438 (time 1787864707.60 total 422.27 last 20.52) - open/close 7598 (time 1787864717.89 total 432.57 last 15.55) - open/close 7874 (time 1787864727.90 total 442.57 last 27.59) - open/close 8145 (time 1787864737.91 total 452.58 last 27.07) - open/close 8350 (time 1787864748.93 total 463.60 last 18.60) - open/close 8627 (time 1787864758.93 total 473.61 last 27.68) - open/close 8921 (time 1787864769.75 total 484.42 last 27.18) - open/close 9185 (time 1787864779.75 total 494.43 last 26.39) - open/close 9368 (time 1787864790.35 total 505.03 last 17.27) - open/close 9549 (time 1787864801.44 total 516.11 last 16.33) - open/close 9730 (time 1787864811.99 total 526.67 last 17.15) - open/close 9851 (time 1787864822.60 total 537.28 last 11.40) - open/close 9989 (time 1787864832.91 total 547.58 last 13.39) - open/close 10242 (time 1787864842.92 total 557.60 last 25.27) - open/close 10330 (time 1787864852.93 total 567.61 last 8.79) - open/close 10570 (time 1787864863.43 total 578.11 last 22.86) - open/close 10852 (time 1787864873.94 total 588.61 last 26.84) - open/close 11112 (time 1787864884.03 total 598.70 last 25.77) - open/close 11364 (time 1787864894.72 total 609.39 last 23.57) - open/close 11465 (time 1787864904.72 total 619.39 last 10.10) - open/close 11716 (time 1787864914.74 total 629.41 last 25.05) - open/close 11863 (time 1787864925.05 total 639.72 last 14.26) - open/close 12125 (time 1787864936.10 total 650.78 last 23.69) - open/close 12346 (time 1787864946.16 total 660.83 last 21.98) - open/close 12490 (time 1787864956.59 total 671.26 last 13.81) - open/close 12740 (time 1787864967.65 total 682.32 last 22.60) - open/close 12985 (time 1787864977.66 total 692.33 last 24.47) - open/close 13202 (time 1787864987.66 total 702.33 last 21.70) - open/close 13458 (time 1787864998.34 total 713.02 last 23.96) - open/close 13708 (time 1787865009.16 total 723.84 last 23.11) - open/close 13948 (time 1787865019.17 total 733.84 last 23.99) - open/close 14149 (time 1787865029.46 total 744.13 last 19.54) - open/close 14369 (time 1787865039.57 total 754.24 last 21.76) - open/close 14555 (time 1787865051.30 total 765.97 last 15.85) - open/close 14731 (time 1787865066.37 total 781.05 last 11.68) - open/close 14872 (time 1787865078.53 total 793.20 last 11.60) - open/close 15011 (time 1787865089.62 total 804.29 last 12.54) - open/close 15189 (time 1787865101.78 total 816.46 last 14.63) - open/close 15318 (time 1787865111.81 total 826.49 last 12.86) - open/close 15413 (time 1787865122.44 total 837.11 last 8.94) - open/close 15541 (time 1787865132.75 total 847.42 last 12.42) - open/close 15718 (time 1787865145.33 total 860.00 last 14.07) - open/close 15915 (time 1787865156.13 total 870.80 last 18.25) - open/close 16034 (time 1787865166.27 total 880.94 last 11.73) - open/close 16163 (time 1787865177.18 total 891.85 last 11.82) - open/close 16317 (time 1787865189.46 total 904.13 last 12.54) - open/close 16469 (time 1787865199.71 total 914.39 last 14.83) - open/close 16676 (time 1787865210.15 total 924.82 last 19.83) - open/close 16888 (time 1787865220.99 total 935.66 last 19.57) - open/close 16998 (time 1787865232.04 total 946.71 last 9.95) - open/close 17191 (time 1787865242.05 total 956.72 last 19.29) - open/close 17308 (time 1787865252.05 total 966.73 last 11.69) - open/close 17516 (time 1787865262.07 total 976.74 last 20.77) - open/close 17755 (time 1787865272.08 total 986.76 last 23.86) - open/close 17966 (time 1787865282.10 total 996.77 last 21.07) - open/close 18109 (time 1787865292.25 total 1006.93 last 14.08) - open/close 18242 (time 1787865302.27 total 1016.94 last 13.28) - open/close 18940 (time 1787865312.27 total 1026.94 last 69.78) - open/close 19950 (time 1787865322.27 total 1036.95 last 100.97) - open/close 20869 (time 1787865332.28 total 1046.95 last 91.88) - open/close 21648 (time 1787865342.29 total 1056.96 last 77.82) - open/close 22553 (time 1787865352.29 total 1066.96 last 90.49) - open/close 23413 (time 1787865362.29 total 1076.97 last 85.95) - open/close 24233 (time 1787865372.30 total 1086.97 last 81.97) - open/close 25212 (time 1787865382.30 total 1096.98 last 97.84) - open/close 26047 (time 1787865392.31 total 1106.98 last 83.43) - open/close 26867 (time 1787865402.33 total 1117.00 last 81.88) - open/close 27613 (time 1787865412.34 total 1127.01 last 74.51) - open/close 28334 (time 1787865422.34 total 1137.02 last 72.06) - open/close 29324 (time 1787865432.35 total 1147.02 last 98.97) - open/close 30000 (time 1787865442.11 total 1156.79 last 69.20) - open/close 30767 (time 1787865452.12 total 1166.79 last 76.65) - open/close 31537 (time 1787865462.12 total 1176.80 last 76.98) - open/close 32271 (time 1787865472.13 total 1186.80 last 73.36) - open/close 32884 (time 1787865482.13 total 1196.81 last 61.26) - open/close 33584 (time 1787865492.14 total 1206.81 last 69.99) - open/close 34526 (time 1787865502.14 total 1216.81 last 94.16) - open/close 35378 (time 1787865512.14 total 1226.82 last 85.18) - open/close 36287 (time 1787865522.15 total 1236.82 last 90.88) - open/close 37078 (time 1787865532.15 total 1246.83 last 79.03) - open/close 37978 (time 1787865542.17 total 1256.85 last 89.84) - open/close 38854 (time 1787865552.18 total 1266.85 last 87.57) - open/close 39542 (time 1787865562.19 total 1276.87 last 68.68) - open/close 40000 (time 1787865567.29 total 1281.96 last 89.85) - open/close 40811 (time 1787865577.29 total 1291.96 last 81.09) - open/close 41602 (time 1787865587.29 total 1301.97 last 79.07) - open/close 42439 (time 1787865597.30 total 1311.98 last 83.64) - open/close 43183 (time 1787865607.30 total 1321.98 last 74.38) - open/close 43739 (time 1787865617.33 total 1332.00 last 55.48) - open/close 44266 (time 1787865627.35 total 1342.02 last 52.57) - open/close 45035 (time 1787865637.35 total 1352.03 last 76.87) - open/close 45875 (time 1787865647.36 total 1362.03 last 83.95) - open/close 46564 (time 1787865657.37 total 1372.04 last 68.84) - open/close 47324 (time 1787865667.37 total 1382.05 last 75.95) - open/close 48254 (time 1787865677.39 total 1392.06 last 92.88) - open/close 49135 (time 1787865687.39 total 1402.06 last 88.10) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1409.16 seconds: 35.22 ops/second - unlinked 0 (time 1787865699 ; total 0 ; last 0) - unlinked 10000 (time 1787865818 ; total 119 ; last 119) - unlinked 20000 (time 1787865939 ; total 240 ; last 121) - unlinked 30000 (time 1787866062 ; total 363 ; last 123) - unlinked 40000 (time 1787866186 ; total 487 ; last 124) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 609 seconds: 81.494255 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: modules unloaded. pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2