== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:53:39 (1787378019) start mds service on oleg118-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg118-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 265 (time 1787378139.68 total 11.01 last 24.08) - open/close 439 (time 1787378150.31 total 21.64 last 16.37) - open/close 699 (time 1787378160.32 total 31.64 last 25.99) - open/close 842 (time 1787378170.37 total 41.70 last 14.22) - open/close 1043 (time 1787378180.38 total 51.70 last 20.09) - open/close 1288 (time 1787378191.14 total 62.46 last 22.76) - open/close 1550 (time 1787378201.15 total 72.47 last 26.18) - open/close 1799 (time 1787378212.20 total 83.52 last 22.53) - open/close 2053 (time 1787378223.08 total 94.41 last 23.34) - open/close 2338 (time 1787378233.65 total 104.97 last 26.98) - open/close 2613 (time 1787378244.47 total 115.79 last 25.41) - open/close 2858 (time 1787378254.53 total 125.85 last 24.36) - open/close 3033 (time 1787378264.86 total 136.19 last 16.93) - open/close 3237 (time 1787378275.18 total 146.51 last 19.77) - open/close 3360 (time 1787378286.06 total 157.38 last 11.31) - open/close 3606 (time 1787378296.08 total 167.40 last 24.56) - open/close 3834 (time 1787378306.11 total 177.44 last 22.72) - open/close 4093 (time 1787378316.12 total 187.45 last 25.88) - open/close 4312 (time 1787378326.12 total 197.45 last 21.89) - open/close 4524 (time 1787378336.30 total 207.63 last 20.83) - open/close 4744 (time 1787378346.31 total 217.64 last 21.98) - open/close 5036 (time 1787378356.51 total 227.83 last 28.64) - open/close 5143 (time 1787378367.77 total 239.09 last 9.50) - open/close 5394 (time 1787378378.66 total 249.98 last 23.05) - open/close 5505 (time 1787378389.35 total 260.68 last 10.38) - open/close 5758 (time 1787378400.23 total 271.55 last 23.26) - open/close 5965 (time 1787378410.24 total 281.57 last 20.67) - open/close 6170 (time 1787378420.90 total 292.22 last 19.24) - open/close 6464 (time 1787378430.90 total 302.23 last 29.39) - open/close 6736 (time 1787378441.91 total 313.24 last 24.70) - open/close 6948 (time 1787378452.01 total 323.33 last 21.00) - open/close 7184 (time 1787378462.18 total 333.50 last 23.20) - open/close 7412 (time 1787378472.36 total 343.69 last 22.39) - open/close 7646 (time 1787378482.82 total 354.15 last 22.37) - open/close 7880 (time 1787378492.86 total 364.19 last 23.31) - open/close 8143 (time 1787378503.55 total 374.88 last 24.59) - open/close 8379 (time 1787378513.56 total 384.89 last 23.58) - open/close 8623 (time 1787378525.90 total 397.23 last 19.77) - open/close 8839 (time 1787378536.38 total 407.70 last 20.62) - open/close 9061 (time 1787378546.39 total 417.71 last 22.18) - open/close 9285 (time 1787378556.92 total 428.24 last 21.27) - open/close 9550 (time 1787378566.93 total 438.25 last 26.48) - open/close 9729 (time 1787378577.46 total 448.78 last 17.00) - open/close 9995 (time 1787378587.74 total 459.07 last 25.86) - open/close 10093 (time 1787378598.58 total 469.90 last 9.04) - open/close 10125 (time 1787378608.59 total 479.91 last 3.20) - open/close 10290 (time 1787378619.23 total 490.56 last 15.50) - open/close 10512 (time 1787378629.25 total 500.58 last 22.16) - open/close 10771 (time 1787378639.25 total 510.58 last 25.90) - open/close 10997 (time 1787378649.65 total 520.98 last 21.73) - open/close 11262 (time 1787378659.66 total 530.99 last 26.47) - open/close 11520 (time 1787378669.67 total 541.00 last 25.76) - open/close 11801 (time 1787378679.68 total 551.01 last 28.07) - open/close 12030 (time 1787378689.84 total 561.17 last 22.54) - open/close 12241 (time 1787378700.85 total 572.17 last 19.17) - open/close 12458 (time 1787378711.13 total 582.45 last 21.11) - open/close 12646 (time 1787378721.38 total 592.71 last 18.33) - open/close 12952 (time 1787378731.40 total 602.72 last 30.55) - open/close 13268 (time 1787378742.17 total 613.50 last 29.33) - open/close 13514 (time 1787378752.18 total 623.50 last 24.59) - open/close 13746 (time 1787378763.06 total 634.38 last 21.32) - open/close 13982 (time 1787378773.06 total 644.39 last 23.59) - open/close 14222 (time 1787378783.07 total 654.40 last 23.98) - open/close 14408 (time 1787378793.57 total 664.89 last 17.72) - open/close 14664 (time 1787378804.22 total 675.55 last 24.02) - open/close 14960 (time 1787378814.70 total 686.03 last 28.24) - open/close 15230 (time 1787378824.76 total 696.08 last 26.86) - open/close 15473 (time 1787378834.80 total 706.12 last 24.20) - open/close 15687 (time 1787378845.41 total 716.74 last 20.16) - open/close 15953 (time 1787378856.23 total 727.56 last 24.59) - open/close 16194 (time 1787378867.69 total 739.02 last 21.03) - open/close 16453 (time 1787378877.71 total 749.03 last 25.86) - open/close 16725 (time 1787378888.04 total 759.37 last 26.32) - open/close 16983 (time 1787378898.35 total 769.67 last 25.04) - open/close 17253 (time 1787378911.20 total 782.52 last 21.01) - open/close 17466 (time 1787378921.20 total 792.53 last 21.29) - open/close 17729 (time 1787378931.23 total 802.55 last 26.24) - open/close 17918 (time 1787378944.11 total 815.43 last 14.67) - open/close 18160 (time 1787378954.11 total 825.43 last 24.20) - open/close 18377 (time 1787378964.25 total 835.57 last 21.40) - open/close 18607 (time 1787378974.57 total 845.90 last 22.28) - open/close 18857 (time 1787378984.58 total 855.91 last 24.97) - open/close 19121 (time 1787378994.59 total 865.91 last 26.38) - open/close 19340 (time 1787379004.61 total 875.94 last 21.85) - open/close 19706 (time 1787379014.62 total 885.95 last 36.55) - open/close 19958 (time 1787379025.18 total 896.51 last 23.86) - open/close 20167 (time 1787379035.19 total 906.52 last 20.88) - open/close 20360 (time 1787379045.76 total 917.08 last 18.26) - open/close 20589 (time 1787379055.78 total 927.10 last 22.86) - open/close 21095 (time 1787379065.78 total 937.11 last 50.59) - open/close 22013 (time 1787379075.78 total 947.11 last 91.79) - open/close 22716 (time 1787379085.81 total 957.13 last 70.13) - open/close 23549 (time 1787379095.82 total 967.15 last 83.17) - open/close 24353 (time 1787379105.82 total 977.15 last 80.40) - open/close 25200 (time 1787379115.82 total 987.15 last 84.70) - open/close 26132 (time 1787379125.83 total 997.15 last 93.15) - open/close 27014 (time 1787379135.83 total 1007.16 last 88.16) - open/close 27884 (time 1787379145.83 total 1017.16 last 86.99) - open/close 28737 (time 1787379155.84 total 1027.17 last 85.21) - open/close 29487 (time 1787379165.86 total 1037.18 last 74.91) - open/close 30000 (time 1787379173.57 total 1044.90 last 66.47) - open/close 30759 (time 1787379183.60 total 1054.92 last 75.73) - open/close 31620 (time 1787379193.62 total 1064.94 last 85.90) - open/close 32373 (time 1787379203.62 total 1074.95 last 75.27) - open/close 33100 (time 1787379213.64 total 1084.97 last 72.57) - open/close 33763 (time 1787379223.64 total 1094.97 last 66.30) - open/close 34588 (time 1787379233.65 total 1104.97 last 82.43) - open/close 35356 (time 1787379243.66 total 1114.99 last 76.69) - open/close 36040 (time 1787379253.67 total 1125.00 last 68.35) - open/close 36819 (time 1787379263.68 total 1135.00 last 77.84) - open/close 37632 (time 1787379273.68 total 1145.01 last 81.28) - open/close 38449 (time 1787379283.68 total 1155.01 last 81.68) - open/close 39294 (time 1787379293.69 total 1165.02 last 84.44) - open/close 40000 (time 1787379303.66 total 1174.98 last 70.84) - open/close 40806 (time 1787379313.66 total 1184.99 last 80.54) - open/close 41649 (time 1787379323.67 total 1195.00 last 84.24) - open/close 42502 (time 1787379333.67 total 1205.00 last 85.29) - open/close 43453 (time 1787379343.68 total 1215.00 last 95.03) - open/close 44326 (time 1787379353.68 total 1225.01 last 87.26) - open/close 45168 (time 1787379363.68 total 1235.01 last 84.20) - open/close 45978 (time 1787379373.69 total 1245.01 last 80.98) - open/close 46618 (time 1787379383.70 total 1255.02 last 63.93) - open/close 47077 (time 1787379393.71 total 1265.04 last 45.82) - open/close 47864 (time 1787379403.72 total 1275.05 last 78.63) - open/close 48527 (time 1787379413.73 total 1285.05 last 66.27) - open/close 49346 (time 1787379423.74 total 1295.07 last 81.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1299.17 seconds: 38.20 ops/second - unlinked 0 (time 1787379432 ; total 0 ; last 0) - unlinked 10000 (time 1787379558 ; total 126 ; last 126) - unlinked 20000 (time 1787379685 ; total 253 ; last 127) - unlinked 30000 (time 1787379823 ; total 391 ; last 138) - unlinked 40000 (time 1787379959 ; total 527 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 660 seconds: 75.198486 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: modules unloaded. pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2