== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:20:18 (1787851218) start mds service on oleg650-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg650-server' oleg650-server: oleg650-server.virtnet: executing load_modules_local oleg650-server: Loading modules from /home/green/git/lustre-release/lustre oleg650-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg650-server: detected 4 online CPUs by sysfs oleg650-server: MODOPTS_LIBCFS= oleg650-server: Force libcfs to create 2 CPU partitions oleg650-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg650-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg650-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg650-server: oleg650-server.virtnet: executing set_default_debug -1 all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg650-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg650-server: oleg650-server.virtnet: executing set_default_debug -1 all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg650-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug -1 all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0000 oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1787851330.67 total 10.90 last 18.72) - open/close 396 (time 1787851341.32 total 21.55 last 18.03) - open/close 633 (time 1787851351.63 total 31.85 last 23.00) - open/close 889 (time 1787851364.31 total 44.54 last 20.18) - open/close 1125 (time 1787851374.43 total 54.65 last 23.34) - open/close 1402 (time 1787851384.43 total 64.66 last 27.68) - open/close 1658 (time 1787851394.77 total 75.00 last 24.76) - open/close 1931 (time 1787851405.33 total 85.56 last 25.86) - open/close 2151 (time 1787851415.36 total 95.59 last 21.93) - open/close 2387 (time 1787851425.55 total 105.78 last 23.16) - open/close 2635 (time 1787851435.56 total 115.79 last 24.77) - open/close 2879 (time 1787851445.96 total 126.19 last 23.47) - open/close 3157 (time 1787851455.97 total 136.20 last 27.77) - open/close 3410 (time 1787851465.97 total 146.20 last 25.29) - open/close 3723 (time 1787851476.71 total 156.93 last 29.17) - open/close 3847 (time 1787851487.69 total 167.92 last 11.29) - open/close 4032 (time 1787851498.36 total 178.59 last 17.34) - open/close 4286 (time 1787851509.19 total 189.42 last 23.45) - open/close 4449 (time 1787851521.11 total 201.33 last 13.68) - open/close 4702 (time 1787851531.40 total 211.62 last 24.59) - open/close 4921 (time 1787851541.60 total 221.82 last 21.47) - open/close 5022 (time 1787851552.09 total 232.31 last 9.63) - open/close 5265 (time 1787851562.71 total 242.94 last 22.86) - open/close 5478 (time 1787851572.72 total 252.94 last 21.29) - open/close 5714 (time 1787851583.25 total 263.47 last 22.41) - open/close 5949 (time 1787851593.68 total 273.90 last 22.53) - open/close 6131 (time 1787851604.17 total 284.40 last 17.34) - open/close 6337 (time 1787851614.69 total 294.91 last 19.60) - open/close 6574 (time 1787851624.69 total 304.92 last 23.69) - open/close 6785 (time 1787851634.70 total 314.92 last 21.08) - open/close 6941 (time 1787851644.88 total 325.11 last 15.32) - open/close 7261 (time 1787851655.00 total 335.23 last 31.61) - open/close 7431 (time 1787851666.51 total 346.73 last 14.78) - open/close 7580 (time 1787851676.51 total 356.73 last 14.90) - open/close 7721 (time 1787851686.55 total 366.78 last 14.04) - open/close 7837 (time 1787851696.66 total 376.88 last 11.48) - open/close 8044 (time 1787851709.00 total 389.23 last 16.77) - open/close 8192 (time 1787851719.70 total 399.92 last 13.84) - open/close 8290 (time 1787851729.81 total 410.04 last 9.69) - open/close 8422 (time 1787851739.84 total 420.06 last 13.17) - open/close 8630 (time 1787851749.86 total 430.08 last 20.76) - open/close 8887 (time 1787851759.88 total 440.11 last 25.64) - open/close 9038 (time 1787851770.90 total 451.12 last 13.71) - open/close 9160 (time 1787851781.15 total 461.38 last 11.90) - open/close 9278 (time 1787851791.77 total 472.00 last 11.11) - open/close 9419 (time 1787851802.00 total 482.22 last 13.79) - open/close 9702 (time 1787851813.00 total 493.23 last 25.72) - open/close 9921 (time 1787851823.02 total 503.24 last 21.87) - open/close 10000 (time 1787851825.53 total 505.76 last 31.43) - open/close 10208 (time 1787851835.54 total 515.77 last 20.78) - open/close 10393 (time 1787851846.16 total 526.38 last 17.43) - open/close 10553 (time 1787851856.59 total 536.81 last 15.34) - open/close 10698 (time 1787851866.85 total 547.07 last 14.13) - open/close 10955 (time 1787851877.27 total 557.50 last 24.65) - open/close 11155 (time 1787851887.28 total 567.50 last 19.99) - open/close 11403 (time 1787851897.88 total 578.10 last 23.39) - open/close 11665 (time 1787851908.50 total 588.72 last 24.67) - open/close 11947 (time 1787851919.44 total 599.67 last 25.77) - open/close 12184 (time 1787851930.40 total 610.62 last 21.63) - open/close 12458 (time 1787851940.42 total 620.64 last 27.35) - open/close 12675 (time 1787851950.42 total 630.64 last 21.70) - open/close 12887 (time 1787851962.94 total 643.17 last 16.92) - open/close 12925 (time 1787851975.40 total 655.62 last 3.05) - open/close 13024 (time 1787851985.81 total 666.04 last 9.50) - open/close 13245 (time 1787851996.45 total 676.67 last 20.79) - open/close 13419 (time 1787852007.12 total 687.35 last 16.29) - open/close 13681 (time 1787852018.96 total 699.19 last 22.14) - open/close 13884 (time 1787852029.00 total 709.23 last 20.21) - open/close 14035 (time 1787852039.02 total 719.25 last 15.07) - open/close 14234 (time 1787852049.05 total 729.28 last 19.84) - open/close 14469 (time 1787852059.07 total 739.30 last 23.45) - open/close 14662 (time 1787852069.78 total 750.00 last 18.03) - open/close 14900 (time 1787852079.78 total 760.01 last 23.78) - open/close 15103 (time 1787852091.55 total 771.77 last 17.25) - open/close 15339 (time 1787852101.92 total 782.14 last 22.76) - open/close 15477 (time 1787852111.95 total 792.17 last 13.76) - open/close 15700 (time 1787852122.73 total 802.96 last 20.67) - open/close 15909 (time 1787852133.14 total 813.36 last 20.09) - open/close 16047 (time 1787852143.14 total 823.36 last 13.79) - open/close 16189 (time 1787852153.89 total 834.12 last 13.21) - open/close 16362 (time 1787852163.91 total 844.13 last 17.27) - open/close 16458 (time 1787852174.05 total 854.28 last 9.46) - open/close 16602 (time 1787852185.05 total 865.28 last 13.09) - open/close 16801 (time 1787852196.05 total 876.28 last 18.09) - open/close 16935 (time 1787852206.06 total 886.29 last 13.39) - open/close 17170 (time 1787852216.08 total 896.31 last 23.46) - open/close 17443 (time 1787852227.09 total 907.32 last 24.80) - open/close 17772 (time 1787852238.10 total 918.33 last 29.88) - open/close 17827 (time 1787852248.11 total 928.33 last 5.50) - open/close 17999 (time 1787852258.11 total 938.34 last 17.18) - open/close 18260 (time 1787852268.85 total 949.07 last 24.32) - open/close 18520 (time 1787852279.27 total 959.50 last 24.95) - open/close 18795 (time 1787852289.28 total 969.50 last 27.48) - open/close 18963 (time 1787852299.29 total 979.51 last 16.79) - open/close 19142 (time 1787852309.55 total 989.78 last 17.43) - open/close 19381 (time 1787852321.37 total 1001.59 last 20.23) - open/close 20000 (time 1787852330.34 total 1010.56 last 69.02) - open/close 20823 (time 1787852340.34 total 1020.57 last 82.25) - open/close 21470 (time 1787852350.36 total 1030.58 last 64.62) - open/close 22153 (time 1787852360.36 total 1040.59 last 68.27) - open/close 22816 (time 1787852370.37 total 1050.60 last 66.22) - open/close 23526 (time 1787852380.38 total 1060.61 last 70.95) - open/close 24158 (time 1787852390.40 total 1070.63 last 63.06) - open/close 24847 (time 1787852400.42 total 1080.64 last 68.80) - open/close 25492 (time 1787852410.44 total 1090.67 last 64.34) - open/close 26115 (time 1787852420.45 total 1100.67 last 62.28) - open/close 26816 (time 1787852430.46 total 1110.68 last 70.04) - open/close 27505 (time 1787852440.48 total 1120.70 last 68.74) - open/close 28305 (time 1787852450.48 total 1130.71 last 79.99) - open/close 29031 (time 1787852460.48 total 1140.71 last 72.57) - open/close 29689 (time 1787852470.49 total 1150.71 last 65.79) - open/close 30000 (time 1787852474.91 total 1155.14 last 70.25) - open/close 30667 (time 1787852484.92 total 1165.14 last 66.66) - open/close 31319 (time 1787852494.93 total 1175.16 last 65.12) - open/close 31956 (time 1787852504.95 total 1185.18 last 63.56) - open/close 32456 (time 1787852514.96 total 1195.18 last 49.98) - open/close 33016 (time 1787852524.98 total 1205.20 last 55.89) - open/close 33586 (time 1787852534.99 total 1215.22 last 56.92) - open/close 34246 (time 1787852545.00 total 1225.22 last 65.96) - open/close 34896 (time 1787852555.02 total 1235.25 last 64.86) - open/close 35486 (time 1787852565.03 total 1245.26 last 58.93) - open/close 36085 (time 1787852575.07 total 1255.29 last 59.69) - open/close 36683 (time 1787852585.08 total 1265.30 last 59.72) - open/close 37356 (time 1787852595.09 total 1275.31 last 67.24) - open/close 38140 (time 1787852605.09 total 1285.32 last 78.39) - open/close 38861 (time 1787852615.10 total 1295.32 last 72.06) - open/close 39503 (time 1787852625.10 total 1305.32 last 64.18) - open/close 40000 (time 1787852634.53 total 1314.76 last 52.70) - open/close 40600 (time 1787852644.54 total 1324.76 last 59.96) - open/close 41186 (time 1787852654.54 total 1334.76 last 58.60) - open/close 41852 (time 1787852664.54 total 1344.76 last 66.59) - open/close 42462 (time 1787852674.55 total 1354.77 last 60.93) - open/close 43244 (time 1787852684.58 total 1364.81 last 77.94) - open/close 43931 (time 1787852694.58 total 1374.81 last 68.70) - open/close 44625 (time 1787852704.58 total 1384.81 last 69.39) - open/close 45160 (time 1787852714.64 total 1394.86 last 53.22) - open/close 45801 (time 1787852724.65 total 1404.87 last 64.04) - open/close 46571 (time 1787852734.65 total 1414.88 last 76.96) - open/close 47354 (time 1787852744.66 total 1424.88 last 78.24) - open/close 48087 (time 1787852754.66 total 1434.89 last 73.28) - open/close 48860 (time 1787852764.66 total 1444.89 last 77.28) - open/close 49609 (time 1787852774.67 total 1454.89 last 74.89) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1455.16 seconds: 34.11 ops/second - unlinked 0 (time 1787852779 ; total 0 ; last 0) - unlinked 10000 (time 1787852914 ; total 135 ; last 135) - unlinked 20000 (time 1787853045 ; total 266 ; last 131) - unlinked 30000 (time 1787853177 ; total 398 ; last 132) - unlinked 40000 (time 1787853308 ; total 529 ; last 131) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 652 seconds: 76.121162 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg650-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg650-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg650-server stop mds service on oleg650-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg650-server stop mds service on oleg650-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg650-server unloading modules via unload_modules_local on: 'oleg650-server' oleg650-server: oleg650-server.virtnet: executing unload_modules_local oleg650-server: modules unloaded. pdsh@oleg650-client: oleg650-client: ssh exited with exit code 2 pdsh@oleg650-client: oleg650-server: ssh exited with exit code 2 pdsh@oleg650-client: oleg650-client: ssh exited with exit code 2 pdsh@oleg650-client: oleg650-server: ssh exited with exit code 2