== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:41:26 (1779702086) start mds service on oleg420-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg420-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 224 (time 1779702183.10 total 10.47 last 21.39) - open/close 474 (time 1779702194.14 total 21.51 last 22.65) - open/close 790 (time 1779702204.14 total 31.52 last 31.58) - open/close 1033 (time 1779702214.16 total 41.54 last 24.25) - open/close 1293 (time 1779702224.46 total 51.84 last 25.25) - open/close 1503 (time 1779702234.47 total 61.84 last 20.98) - open/close 1751 (time 1779702244.49 total 71.86 last 24.76) - open/close 2033 (time 1779702257.02 total 84.40 last 22.49) - open/close 2142 (time 1779702267.66 total 95.03 last 10.25) - open/close 2372 (time 1779702278.49 total 105.86 last 21.23) - open/close 2583 (time 1779702288.50 total 115.87 last 21.09) - open/close 2841 (time 1779702298.50 total 125.88 last 25.78) - open/close 3070 (time 1779702308.51 total 135.88 last 22.88) - open/close 3312 (time 1779702318.53 total 145.90 last 24.15) - open/close 3555 (time 1779702328.53 total 155.91 last 24.28) - open/close 3793 (time 1779702338.55 total 165.92 last 23.77) - open/close 4081 (time 1779702349.13 total 176.51 last 27.21) - open/close 4394 (time 1779702359.13 total 186.51 last 31.29) - open/close 4594 (time 1779702372.04 total 199.41 last 15.50) - open/close 4718 (time 1779702382.73 total 210.11 last 11.59) - open/close 4878 (time 1779702393.28 total 220.66 last 15.17) - open/close 5042 (time 1779702405.84 total 233.21 last 13.06) - open/close 5277 (time 1779702416.46 total 243.84 last 22.12) - open/close 5437 (time 1779702426.83 total 254.20 last 15.43) - open/close 5598 (time 1779702436.89 total 264.26 last 16.01) - open/close 5724 (time 1779702446.99 total 274.36 last 12.47) - open/close 5981 (time 1779702457.74 total 285.11 last 23.91) - open/close 6294 (time 1779702468.42 total 295.79 last 29.31) - open/close 6487 (time 1779702478.42 total 305.79 last 19.30) - open/close 6758 (time 1779702488.65 total 316.02 last 26.49) - open/close 7015 (time 1779702499.09 total 326.46 last 24.62) - open/close 7261 (time 1779702509.57 total 336.94 last 23.47) - open/close 7574 (time 1779702519.58 total 346.96 last 31.26) - open/close 7818 (time 1779702530.51 total 357.88 last 22.33) - open/close 7969 (time 1779702540.61 total 367.99 last 14.95) - open/close 8164 (time 1779702550.78 total 378.16 last 19.17) - open/close 8371 (time 1779702561.16 total 388.54 last 19.94) - open/close 8492 (time 1779702571.19 total 398.56 last 12.07) - open/close 8710 (time 1779702581.31 total 408.68 last 21.54) - open/close 8956 (time 1779702592.91 total 420.28 last 21.21) - open/close 9100 (time 1779702603.33 total 430.70 last 13.82) - open/close 9278 (time 1779702614.16 total 441.54 last 16.43) - open/close 9507 (time 1779702624.17 total 451.54 last 22.88) - open/close 9713 (time 1779702634.58 total 461.95 last 19.80) - open/close 9935 (time 1779702644.62 total 471.99 last 22.11) - open/close 10000 (time 1779702648.77 total 476.15 last 15.64) - open/close 10177 (time 1779702658.78 total 486.15 last 17.70) - open/close 10228 (time 1779702669.25 total 496.62 last 4.87) - open/close 10490 (time 1779702679.25 total 506.62 last 26.20) - open/close 10642 (time 1779702690.25 total 517.62 last 13.82) - open/close 10838 (time 1779702701.15 total 528.52 last 17.98) - open/close 11074 (time 1779702711.15 total 538.53 last 23.59) - open/close 11342 (time 1779702721.79 total 549.16 last 25.19) - open/close 11571 (time 1779702734.85 total 562.22 last 17.53) - open/close 11756 (time 1779702745.83 total 573.20 last 16.85) - open/close 11978 (time 1779702756.81 total 584.18 last 20.22) - open/close 12241 (time 1779702767.30 total 594.68 last 25.05) - open/close 12488 (time 1779702777.74 total 605.11 last 23.68) - open/close 12714 (time 1779702788.29 total 615.66 last 21.42) - open/close 12909 (time 1779702798.98 total 626.36 last 18.23) - open/close 13107 (time 1779702808.99 total 636.36 last 19.80) - open/close 13182 (time 1779702818.99 total 646.37 last 7.49) - open/close 13289 (time 1779702829.52 total 656.89 last 10.17) - open/close 13565 (time 1779702839.55 total 666.93 last 27.51) - open/close 13770 (time 1779702852.49 total 679.86 last 15.85) - open/close 13983 (time 1779702862.72 total 690.09 last 20.82) - open/close 14194 (time 1779702875.01 total 702.39 last 17.16) - open/close 14407 (time 1779702886.11 total 713.48 last 19.19) - open/close 14613 (time 1779702896.12 total 723.49 last 20.58) - open/close 14908 (time 1779702906.12 total 733.50 last 29.49) - open/close 15116 (time 1779702916.46 total 743.84 last 20.11) - open/close 15259 (time 1779702926.48 total 753.86 last 14.27) - open/close 15409 (time 1779702937.46 total 764.83 last 13.67) - open/close 15637 (time 1779702948.44 total 775.82 last 20.76) - open/close 15728 (time 1779702959.26 total 786.64 last 8.41) - open/close 15907 (time 1779702971.81 total 799.18 last 14.27) - open/close 16138 (time 1779702982.35 total 809.73 last 21.90) - open/close 16385 (time 1779702992.65 total 820.02 last 24.00) - open/close 16660 (time 1779703002.65 total 830.02 last 27.50) - open/close 16842 (time 1779703012.65 total 840.03 last 18.19) - open/close 16967 (time 1779703022.72 total 850.10 last 12.42) - open/close 17196 (time 1779703033.81 total 861.18 last 20.66) - open/close 17476 (time 1779703043.82 total 871.20 last 27.95) - open/close 17658 (time 1779703056.29 total 883.66 last 14.60) - open/close 17821 (time 1779703066.56 total 893.94 last 15.86) - open/close 17989 (time 1779703076.87 total 904.25 last 16.30) - open/close 18221 (time 1779703087.50 total 914.87 last 21.83) - open/close 18450 (time 1779703097.51 total 924.89 last 22.86) - open/close 18677 (time 1779703108.42 total 935.80 last 20.80) - open/close 18880 (time 1779703118.43 total 945.80 last 20.29) - open/close 19041 (time 1779703128.43 total 955.80 last 16.10) - open/close 19192 (time 1779703138.51 total 965.89 last 14.97) - open/close 19696 (time 1779703148.52 total 975.90 last 50.37) - open/close 20000 (time 1779703152.15 total 979.52 last 83.86) - open/close 20811 (time 1779703162.16 total 989.53 last 81.01) - open/close 21563 (time 1779703172.16 total 999.53 last 75.20) - open/close 22434 (time 1779703182.16 total 1009.53 last 87.09) - open/close 23322 (time 1779703192.16 total 1019.54 last 88.75) - open/close 24145 (time 1779703202.17 total 1029.54 last 82.26) - open/close 24925 (time 1779703212.17 total 1039.54 last 77.99) - open/close 25788 (time 1779703222.18 total 1049.55 last 86.22) - open/close 26636 (time 1779703232.18 total 1059.56 last 84.77) - open/close 27532 (time 1779703242.19 total 1069.56 last 89.52) - open/close 28331 (time 1779703252.19 total 1079.57 last 79.86) - open/close 29106 (time 1779703262.20 total 1089.58 last 77.43) - open/close 29738 (time 1779703272.22 total 1099.60 last 63.09) - open/close 30000 (time 1779703276.02 total 1103.40 last 68.90) - open/close 30726 (time 1779703286.03 total 1113.40 last 72.55) - open/close 31462 (time 1779703296.04 total 1123.42 last 73.52) - open/close 32190 (time 1779703306.05 total 1133.43 last 72.71) - open/close 32870 (time 1779703316.05 total 1143.43 last 67.99) - open/close 33721 (time 1779703326.07 total 1153.45 last 84.96) - open/close 34580 (time 1779703336.10 total 1163.48 last 85.63) - open/close 35460 (time 1779703346.11 total 1173.48 last 87.95) - open/close 36360 (time 1779703356.12 total 1183.50 last 89.88) - open/close 37293 (time 1779703366.13 total 1193.51 last 93.21) - open/close 38154 (time 1779703376.15 total 1203.52 last 85.96) - open/close 39032 (time 1779703386.15 total 1213.53 last 87.76) - open/close 39850 (time 1779703396.15 total 1223.53 last 81.80) - open/close 40701 (time 1779703406.18 total 1233.55 last 84.87) - open/close 41596 (time 1779703416.18 total 1243.56 last 89.45) - open/close 42457 (time 1779703426.19 total 1253.57 last 86.02) - open/close 43306 (time 1779703436.19 total 1263.57 last 84.89) - open/close 44131 (time 1779703446.20 total 1273.58 last 82.42) - open/close 44976 (time 1779703456.21 total 1283.58 last 84.48) - open/close 45858 (time 1779703466.22 total 1293.59 last 88.10) - open/close 46595 (time 1779703476.22 total 1303.60 last 73.68) - open/close 47422 (time 1779703486.23 total 1313.60 last 82.65) - open/close 48229 (time 1779703496.24 total 1323.61 last 80.63) - open/close 49076 (time 1779703506.24 total 1333.61 last 84.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1340.81 seconds: 37.02 ops/second - unlinked 0 (time 1779703517 ; total 0 ; last 0) - unlinked 10000 (time 1779703649 ; total 132 ; last 132) - unlinked 20000 (time 1779703777 ; total 260 ; last 128) - unlinked 30000 (time 1779703905 ; total 388 ; last 128) - unlinked 40000 (time 1779704039 ; total 522 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 648 seconds: 76.591049 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2