== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:28:23 (1778657303) start mds service on oleg143-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg143-server: detected 4 online CPUs by sysfs oleg143-server: MODOPTS_LIBCFS= oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg143-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 102 (time 1778657440.04 total 10.36 last 9.85) - open/close 211 (time 1778657450.06 total 20.38 last 10.88) - open/close 337 (time 1778657460.18 total 30.50 last 12.45) - open/close 437 (time 1778657471.15 total 41.46 last 9.12) - open/close 548 (time 1778657481.19 total 51.51 last 11.05) - open/close 730 (time 1778657491.58 total 61.90 last 17.52) - open/close 864 (time 1778657501.97 total 72.29 last 12.90) - open/close 977 (time 1778657512.02 total 82.34 last 11.24) - open/close 1157 (time 1778657522.53 total 92.85 last 17.13) - open/close 1396 (time 1778657532.66 total 102.97 last 23.61) - open/close 1610 (time 1778657543.48 total 113.79 last 19.78) - open/close 1832 (time 1778657553.48 total 123.80 last 22.19) - open/close 1943 (time 1778657563.49 total 133.81 last 11.09) - open/close 2085 (time 1778657574.06 total 144.38 last 13.43) - open/close 2183 (time 1778657584.08 total 154.39 last 9.79) - open/close 2272 (time 1778657594.33 total 164.65 last 8.68) - open/close 2295 (time 1778657607.15 total 177.47 last 1.79) - open/close 2473 (time 1778657618.11 total 188.43 last 16.24) - open/close 2646 (time 1778657628.13 total 198.45 last 17.26) - open/close 2764 (time 1778657638.16 total 208.47 last 11.77) - open/close 2958 (time 1778657648.23 total 218.54 last 19.26) - open/close 3126 (time 1778657658.24 total 228.55 last 16.79) - open/close 3267 (time 1778657669.33 total 239.64 last 12.72) - open/close 3374 (time 1778657680.23 total 250.55 last 9.81) - open/close 3443 (time 1778657690.34 total 260.66 last 6.83) - open/close 3574 (time 1778657700.91 total 271.22 last 12.39) - open/close 3716 (time 1778657710.92 total 281.24 last 14.18) - open/close 3875 (time 1778657721.77 total 292.08 last 14.66) - open/close 3989 (time 1778657731.82 total 302.14 last 11.34) - open/close 4102 (time 1778657741.94 total 312.25 last 11.17) - open/close 4259 (time 1778657753.00 total 323.31 last 14.20) - open/close 4381 (time 1778657764.07 total 334.38 last 11.02) - open/close 4540 (time 1778657774.14 total 344.45 last 15.79) - open/close 4572 (time 1778657785.44 total 355.75 last 2.83) - open/close 4775 (time 1778657796.13 total 366.44 last 18.99) - open/close 4892 (time 1778657806.43 total 376.74 last 11.36) - open/close 4965 (time 1778657816.75 total 387.07 last 7.07) - open/close 5140 (time 1778657827.75 total 398.06 last 15.92) - open/close 5298 (time 1778657838.55 total 408.87 last 14.62) - open/close 5518 (time 1778657849.50 total 419.82 last 20.10) - open/close 5558 (time 1778657859.94 total 430.25 last 3.83) - open/close 5745 (time 1778657870.49 total 440.81 last 17.71) - open/close 5899 (time 1778657882.40 total 452.71 last 12.94) - open/close 6116 (time 1778657892.92 total 463.23 last 20.63) - open/close 6243 (time 1778657905.96 total 476.27 last 9.74) - open/close 6326 (time 1778657916.91 total 487.22 last 7.58) - open/close 6499 (time 1778657927.26 total 497.58 last 16.71) - open/close 6552 (time 1778657938.21 total 508.52 last 4.84) - open/close 6678 (time 1778657948.91 total 519.22 last 11.78) - open/close 6937 (time 1778657959.26 total 529.57 last 25.01) - open/close 7151 (time 1778657970.10 total 540.42 last 19.74) - open/close 7306 (time 1778657980.10 total 550.42 last 15.50) - open/close 7548 (time 1778657990.89 total 561.20 last 22.44) - open/close 7672 (time 1778658001.75 total 572.06 last 11.41) - open/close 7848 (time 1778658011.75 total 582.07 last 17.60) - open/close 8067 (time 1778658022.00 total 592.32 last 21.37) - open/close 8205 (time 1778658034.14 total 604.45 last 11.37) - open/close 8330 (time 1778658045.09 total 615.40 last 11.42) - open/close 8442 (time 1778658056.05 total 626.37 last 10.21) - open/close 8496 (time 1778658066.06 total 636.38 last 5.40) - open/close 8624 (time 1778658076.34 total 646.66 last 12.45) - open/close 8793 (time 1778658086.59 total 656.91 last 16.49) - open/close 8961 (time 1778658096.60 total 666.91 last 16.79) - open/close 9078 (time 1778658106.63 total 676.95 last 11.66) - open/close 9158 (time 1778658116.64 total 686.96 last 7.99) - open/close 9346 (time 1778658127.72 total 698.04 last 16.96) - open/close 9502 (time 1778658139.31 total 709.63 last 13.46) - open/close 9688 (time 1778658149.60 total 719.91 last 18.09) - open/close 9817 (time 1778658159.60 total 729.92 last 12.89) - open/close 9919 (time 1778658169.64 total 739.95 last 10.17) - open/close 10000 (time 1778658177.13 total 747.44 last 10.81) - open/close 10171 (time 1778658187.15 total 757.46 last 17.07) - open/close 10295 (time 1778658198.24 total 768.55 last 11.18) - open/close 10451 (time 1778658210.00 total 780.31 last 13.26) - open/close 10603 (time 1778658220.75 total 791.06 last 14.14) - open/close 10739 (time 1778658231.07 total 801.39 last 13.17) - open/close 10882 (time 1778658241.73 total 812.05 last 13.41) - open/close 11041 (time 1778658252.60 total 822.92 last 14.63) - open/close 11148 (time 1778658263.23 total 833.54 last 10.07) - open/close 11343 (time 1778658273.24 total 843.56 last 19.47) - open/close 11530 (time 1778658283.52 total 853.83 last 18.20) - open/close 11734 (time 1778658293.85 total 864.16 last 19.75) - open/close 11883 (time 1778658304.37 total 874.68 last 14.16) - open/close 12076 (time 1778658314.39 total 884.71 last 19.25) - open/close 12288 (time 1778658324.99 total 895.30 last 20.01) - open/close 12502 (time 1778658336.94 total 907.26 last 17.91) - open/close 12690 (time 1778658347.82 total 918.14 last 17.27) - open/close 12823 (time 1778658357.98 total 928.30 last 13.09) - open/close 13022 (time 1778658367.99 total 938.31 last 19.88) - open/close 13184 (time 1778658378.74 total 949.05 last 15.08) - open/close 13266 (time 1778658389.66 total 959.98 last 7.50) - open/close 13466 (time 1778658400.08 total 970.40 last 19.20) - open/close 13572 (time 1778658410.83 total 981.15 last 9.86) - open/close 13804 (time 1778658421.76 total 992.08 last 21.23) - open/close 13989 (time 1778658432.80 total 1003.12 last 16.75) - open/close 14218 (time 1778658443.55 total 1013.86 last 21.32) - open/close 14363 (time 1778658454.23 total 1024.55 last 13.57) - open/close 14514 (time 1778658464.26 total 1034.58 last 15.05) - open/close 14710 (time 1778658474.78 total 1045.10 last 18.63) - open/close 14958 (time 1778658485.17 total 1055.48 last 23.88) - open/close 15116 (time 1778658495.24 total 1065.55 last 15.70) - open/close 15376 (time 1778658505.52 total 1075.84 last 25.28) - open/close 15593 (time 1778658516.25 total 1086.56 last 20.23) - open/close 15733 (time 1778658526.25 total 1096.57 last 14.00) - open/close 15806 (time 1778658536.25 total 1106.57 last 7.30) - open/close 16598 (time 1778658546.26 total 1116.57 last 79.17) - open/close 17247 (time 1778658556.26 total 1126.58 last 64.89) - open/close 17994 (time 1778658566.27 total 1136.58 last 74.66) - open/close 18796 (time 1778658576.27 total 1146.58 last 80.18) - open/close 19621 (time 1778658586.27 total 1156.59 last 82.48) - open/close 20000 (time 1778658590.27 total 1160.59 last 94.73) - open/close 20817 (time 1778658600.28 total 1170.60 last 81.63) - open/close 21683 (time 1778658610.29 total 1180.60 last 86.55) - open/close 22520 (time 1778658620.31 total 1190.62 last 83.51) - open/close 23337 (time 1778658630.31 total 1200.62 last 81.69) - open/close 24082 (time 1778658640.32 total 1210.63 last 74.45) - open/close 24917 (time 1778658650.32 total 1220.64 last 83.45) - open/close 25661 (time 1778658660.33 total 1230.64 last 74.37) - open/close 26346 (time 1778658670.33 total 1240.65 last 68.47) - open/close 26939 (time 1778658680.36 total 1250.67 last 59.14) - open/close 27604 (time 1778658690.36 total 1260.68 last 66.47) - open/close 28370 (time 1778658700.37 total 1270.69 last 76.51) - open/close 29007 (time 1778658710.38 total 1280.69 last 63.68) - open/close 29769 (time 1778658720.39 total 1290.71 last 76.08) - open/close 30000 (time 1778658723.84 total 1294.16 last 66.94) - open/close 30693 (time 1778658733.85 total 1304.16 last 69.25) - open/close 31365 (time 1778658743.86 total 1314.17 last 67.13) - open/close 32178 (time 1778658753.86 total 1324.17 last 81.30) - open/close 33044 (time 1778658763.86 total 1334.18 last 86.59) - open/close 33924 (time 1778658773.86 total 1344.18 last 87.99) - open/close 34650 (time 1778658783.86 total 1354.18 last 72.59) - open/close 35326 (time 1778658793.87 total 1364.18 last 67.57) - open/close 36103 (time 1778658803.88 total 1374.19 last 77.62) - open/close 36763 (time 1778658813.89 total 1384.21 last 65.88) - open/close 37215 (time 1778658823.91 total 1394.22 last 45.15) - open/close 38090 (time 1778658833.91 total 1404.23 last 87.44) - open/close 38901 (time 1778658843.91 total 1414.23 last 81.09) - open/close 39610 (time 1778658853.92 total 1424.24 last 70.84) - open/close 40000 (time 1778658858.86 total 1429.17 last 79.03) - open/close 40646 (time 1778658868.86 total 1439.18 last 64.58) - open/close 41353 (time 1778658878.86 total 1449.18 last 70.69) - open/close 42001 (time 1778658888.87 total 1459.18 last 64.78) - open/close 42707 (time 1778658898.87 total 1469.18 last 70.57) - open/close 43385 (time 1778658908.87 total 1479.19 last 67.77) - open/close 44107 (time 1778658918.88 total 1489.19 last 72.17) - open/close 44853 (time 1778658928.89 total 1499.20 last 74.52) - open/close 45519 (time 1778658938.89 total 1509.20 last 66.59) - open/close 46352 (time 1778658948.89 total 1519.21 last 83.27) - open/close 47158 (time 1778658958.90 total 1529.21 last 80.57) - open/close 47981 (time 1778658968.91 total 1539.22 last 82.20) - open/close 48767 (time 1778658978.93 total 1549.25 last 78.40) - open/close 49571 (time 1778658988.94 total 1559.26 last 80.32) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1559.96 seconds: 31.82 ops/second - unlinked 0 (time 1778658994 ; total 0 ; last 0) - unlinked 10000 (time 1778659132 ; total 138 ; last 138) - unlinked 20000 (time 1778659291 ; total 297 ; last 159) - unlinked 30000 (time 1778659427 ; total 433 ; last 136) - unlinked 40000 (time 1778659575 ; total 581 ; last 148) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 708 seconds: 70.100281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules via unload_modules_local on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local oleg143-server: modules unloaded. pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2