== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:11:53 (1787832713) start mds service on oleg656-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing load_modules_local oleg656-server: Loading modules from /home/green/git/lustre-release/lustre oleg656-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg656-server: detected 4 online CPUs by sysfs oleg656-server: MODOPTS_LIBCFS= oleg656-server: Force libcfs to create 2 CPU partitions oleg656-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg656-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg656-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg656-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg656-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 253 (time 1787832812.13 total 10.56 last 23.96) - open/close 388 (time 1787832822.14 total 20.56 last 13.49) - open/close 513 (time 1787832832.15 total 30.58 last 12.48) - open/close 641 (time 1787832842.53 total 40.96 last 12.33) - open/close 927 (time 1787832852.61 total 51.04 last 28.36) - open/close 1155 (time 1787832862.62 total 61.05 last 22.78) - open/close 1386 (time 1787832872.63 total 71.06 last 23.08) - open/close 1625 (time 1787832883.35 total 81.77 last 22.31) - open/close 1844 (time 1787832893.37 total 91.79 last 21.86) - open/close 2097 (time 1787832903.59 total 102.01 last 24.75) - open/close 2328 (time 1787832913.60 total 112.02 last 23.08) - open/close 2547 (time 1787832923.79 total 122.22 last 21.48) - open/close 2791 (time 1787832933.83 total 132.25 last 24.32) - open/close 3037 (time 1787832944.20 total 142.63 last 23.71) - open/close 3307 (time 1787832954.53 total 152.96 last 26.13) - open/close 3546 (time 1787832964.93 total 163.36 last 22.98) - open/close 3793 (time 1787832975.37 total 173.79 last 23.67) - open/close 4038 (time 1787832985.37 total 183.80 last 24.48) - open/close 4269 (time 1787832995.40 total 193.83 last 23.04) - open/close 4459 (time 1787833005.41 total 203.83 last 18.98) - open/close 4643 (time 1787833016.02 total 214.45 last 17.33) - open/close 4875 (time 1787833026.03 total 224.46 last 23.18) - open/close 5119 (time 1787833036.18 total 234.61 last 24.04) - open/close 5410 (time 1787833046.20 total 244.63 last 29.05) - open/close 5663 (time 1787833057.03 total 255.46 last 23.36) - open/close 5908 (time 1787833067.04 total 265.47 last 24.47) - open/close 6103 (time 1787833077.91 total 276.34 last 17.95) - open/close 6351 (time 1787833087.92 total 286.35 last 24.77) - open/close 6520 (time 1787833097.92 total 296.35 last 16.90) - open/close 6771 (time 1787833107.92 total 306.35 last 25.10) - open/close 6976 (time 1787833118.11 total 316.53 last 20.13) - open/close 7178 (time 1787833130.66 total 329.09 last 16.09) - open/close 7383 (time 1787833140.90 total 339.33 last 20.02) - open/close 7578 (time 1787833151.91 total 350.34 last 17.71) - open/close 7889 (time 1787833162.58 total 361.00 last 29.16) - open/close 8166 (time 1787833173.51 total 371.94 last 25.33) - open/close 8402 (time 1787833183.51 total 381.94 last 23.60) - open/close 8642 (time 1787833193.51 total 391.94 last 24.00) - open/close 8899 (time 1787833203.79 total 402.22 last 25.00) - open/close 9155 (time 1787833214.16 total 412.59 last 24.69) - open/close 9429 (time 1787833224.51 total 422.94 last 26.48) - open/close 9672 (time 1787833234.52 total 432.95 last 24.28) - open/close 9932 (time 1787833245.06 total 443.49 last 24.66) - open/close 10224 (time 1787833255.07 total 453.50 last 29.18) - open/close 10426 (time 1787833265.43 total 463.85 last 19.50) - open/close 10659 (time 1787833275.43 total 473.85 last 23.30) - open/close 10904 (time 1787833285.75 total 484.18 last 23.72) - open/close 11179 (time 1787833296.59 total 495.02 last 25.37) - open/close 11338 (time 1787833306.64 total 505.07 last 15.82) - open/close 11580 (time 1787833317.33 total 515.75 last 22.66) - open/close 11826 (time 1787833327.33 total 525.76 last 24.59) - open/close 12052 (time 1787833337.69 total 536.12 last 21.81) - open/close 12319 (time 1787833348.12 total 546.55 last 25.60) - open/close 12557 (time 1787833358.25 total 556.68 last 23.50) - open/close 12827 (time 1787833368.78 total 567.21 last 25.64) - open/close 13074 (time 1787833378.95 total 577.38 last 24.29) - open/close 13322 (time 1787833389.20 total 587.63 last 24.19) - open/close 13552 (time 1787833399.21 total 597.64 last 22.99) - open/close 13802 (time 1787833409.31 total 607.74 last 24.75) - open/close 14084 (time 1787833420.17 total 618.60 last 25.97) - open/close 14373 (time 1787833430.18 total 628.60 last 28.87) - open/close 14624 (time 1787833440.52 total 638.95 last 24.27) - open/close 14894 (time 1787833451.01 total 649.44 last 25.74) - open/close 15139 (time 1787833461.33 total 659.76 last 23.74) - open/close 15272 (time 1787833472.27 total 670.70 last 12.16) - open/close 15563 (time 1787833483.29 total 681.72 last 26.41) - open/close 15856 (time 1787833494.35 total 692.78 last 26.49) - open/close 16060 (time 1787833504.85 total 703.28 last 19.42) - open/close 16332 (time 1787833515.57 total 713.99 last 25.39) - open/close 16593 (time 1787833525.58 total 724.01 last 26.06) - open/close 16900 (time 1787833536.54 total 734.96 last 28.02) - open/close 17133 (time 1787833547.27 total 745.70 last 21.70) - open/close 17360 (time 1787833557.48 total 755.91 last 22.23) - open/close 17613 (time 1787833567.75 total 766.18 last 24.63) - open/close 17822 (time 1787833580.58 total 779.00 last 16.30) - open/close 18045 (time 1787833590.59 total 789.01 last 22.28) - open/close 18292 (time 1787833600.86 total 799.29 last 24.03) - open/close 18565 (time 1787833611.76 total 810.19 last 25.06) - open/close 18760 (time 1787833622.41 total 820.84 last 18.31) - open/close 18983 (time 1787833633.11 total 831.54 last 20.84) - open/close 19219 (time 1787833643.47 total 841.90 last 22.79) - open/close 19491 (time 1787833653.96 total 852.39 last 25.92) - open/close 19763 (time 1787833664.92 total 863.34 last 24.84) - open/close 20000 (time 1787833674.18 total 872.60 last 25.59) - open/close 20725 (time 1787833684.18 total 882.61 last 72.47) - open/close 21462 (time 1787833694.18 total 892.61 last 73.68) - open/close 22266 (time 1787833704.19 total 902.62 last 80.34) - open/close 23000 (time 1787833714.19 total 912.62 last 73.38) - open/close 23819 (time 1787833724.20 total 922.63 last 81.85) - open/close 24536 (time 1787833734.21 total 932.64 last 71.62) - open/close 25356 (time 1787833744.21 total 942.64 last 81.97) - open/close 26206 (time 1787833754.23 total 952.65 last 84.90) - open/close 27004 (time 1787833764.24 total 962.67 last 79.69) - open/close 27763 (time 1787833774.25 total 972.67 last 75.85) - open/close 28618 (time 1787833784.25 total 982.68 last 85.48) - open/close 29364 (time 1787833794.26 total 992.68 last 74.55) - open/close 30000 (time 1787833802.09 total 1000.52 last 81.18) - open/close 30711 (time 1787833812.10 total 1010.52 last 71.06) - open/close 31414 (time 1787833822.11 total 1020.54 last 70.19) - open/close 32173 (time 1787833832.12 total 1030.55 last 75.83) - open/close 32905 (time 1787833842.13 total 1040.56 last 73.14) - open/close 33599 (time 1787833852.15 total 1050.58 last 69.24) - open/close 34286 (time 1787833862.20 total 1060.62 last 68.39) - open/close 35023 (time 1787833872.21 total 1070.64 last 73.62) - open/close 35794 (time 1787833882.21 total 1080.64 last 77.08) - open/close 36594 (time 1787833892.21 total 1090.64 last 79.98) - open/close 37396 (time 1787833902.23 total 1100.65 last 80.09) - open/close 38210 (time 1787833912.23 total 1110.66 last 81.39) - open/close 39009 (time 1787833922.23 total 1120.66 last 79.90) - open/close 39808 (time 1787833932.24 total 1130.66 last 79.83) - open/close 40000 (time 1787833934.66 total 1133.08 last 79.33) - open/close 40782 (time 1787833944.66 total 1143.09 last 78.15) - open/close 41567 (time 1787833954.67 total 1153.10 last 78.45) - open/close 42332 (time 1787833964.68 total 1163.10 last 76.46) - open/close 43092 (time 1787833974.70 total 1173.12 last 75.84) - open/close 43836 (time 1787833984.71 total 1183.13 last 74.32) - open/close 44563 (time 1787833994.72 total 1193.15 last 72.61) - open/close 45400 (time 1787834004.73 total 1203.16 last 83.61) - open/close 46187 (time 1787834014.74 total 1213.17 last 78.63) - open/close 46991 (time 1787834024.75 total 1223.18 last 80.32) - open/close 47813 (time 1787834034.76 total 1233.18 last 82.13) - open/close 48542 (time 1787834044.77 total 1243.20 last 72.78) - open/close 49241 (time 1787834054.78 total 1253.20 last 69.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1258.41 seconds: 39.44 ops/second - unlinked 0 (time 1787834064 ; total 0 ; last 0) - unlinked 10000 (time 1787834190 ; total 126 ; last 126) - unlinked 20000 (time 1787834316 ; total 252 ; last 126) - unlinked 30000 (time 1787834444 ; total 380 ; last 128) - unlinked 40000 (time 1787834571 ; total 507 ; last 127) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 627 seconds: 79.156303 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg656-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg656-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg656-server unloading modules via unload_modules_local on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing unload_modules_local oleg656-server: modules unloaded. pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2