== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:57:05 (1781233025) start mds service on oleg102-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing load_modules_local oleg102-server: Loading modules from /home/green/git/lustre-release/lustre oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg102-server: detected 4 online CPUs by sysfs oleg102-server: MODOPTS_LIBCFS= oleg102-server: Force libcfs to create 2 CPU partitions oleg102-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg102-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg102-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg102-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 294 (time 1781233123.03 total 10.00 last 29.40) - open/close 533 (time 1781233133.05 total 20.01 last 23.87) - open/close 861 (time 1781233143.08 total 30.05 last 32.68) - open/close 1111 (time 1781233153.26 total 40.23 last 24.57) - open/close 1338 (time 1781233163.27 total 50.24 last 22.67) - open/close 1579 (time 1781233173.73 total 60.70 last 23.05) - open/close 1787 (time 1781233183.73 total 70.70 last 20.80) - open/close 1979 (time 1781233193.76 total 80.73 last 19.14) - open/close 2133 (time 1781233203.76 total 90.73 last 15.40) - open/close 2234 (time 1781233214.31 total 101.28 last 9.57) - open/close 2485 (time 1781233224.31 total 111.28 last 25.10) - open/close 2611 (time 1781233235.48 total 122.45 last 11.27) - open/close 2751 (time 1781233246.07 total 133.04 last 13.23) - open/close 2885 (time 1781233256.08 total 143.05 last 13.39) - open/close 3143 (time 1781233266.22 total 153.19 last 25.43) - open/close 3414 (time 1781233276.25 total 163.22 last 27.04) - open/close 3671 (time 1781233286.36 total 173.33 last 25.42) - open/close 3984 (time 1781233296.68 total 183.65 last 30.31) - open/close 4224 (time 1781233306.68 total 193.65 last 24.00) - open/close 4492 (time 1781233316.69 total 203.66 last 26.79) - open/close 4666 (time 1781233326.98 total 213.95 last 16.91) - open/close 4837 (time 1781233337.76 total 224.73 last 15.87) - open/close 5001 (time 1781233347.76 total 234.73 last 16.39) - open/close 5161 (time 1781233358.11 total 245.08 last 15.46) - open/close 5406 (time 1781233368.14 total 255.11 last 24.44) - open/close 5688 (time 1781233378.65 total 265.62 last 26.81) - open/close 5936 (time 1781233388.66 total 275.63 last 24.79) - open/close 6134 (time 1781233398.66 total 285.63 last 19.79) - open/close 6290 (time 1781233408.68 total 295.65 last 15.57) - open/close 6496 (time 1781233419.62 total 306.59 last 18.84) - open/close 6749 (time 1781233429.68 total 316.65 last 25.14) - open/close 6887 (time 1781233439.69 total 326.66 last 13.79) - open/close 7081 (time 1781233449.70 total 336.67 last 19.38) - open/close 7239 (time 1781233460.45 total 347.42 last 14.70) - open/close 7437 (time 1781233470.46 total 357.43 last 19.79) - open/close 7656 (time 1781233480.48 total 367.45 last 21.85) - open/close 7971 (time 1781233490.88 total 377.85 last 30.27) - open/close 8168 (time 1781233501.79 total 388.76 last 18.06) - open/close 8441 (time 1781233512.87 total 399.84 last 24.64) - open/close 8648 (time 1781233523.95 total 410.92 last 18.68) - open/close 8829 (time 1781233533.97 total 420.94 last 18.06) - open/close 8905 (time 1781233544.00 total 430.97 last 7.58) - open/close 9191 (time 1781233554.97 total 441.94 last 26.06) - open/close 9391 (time 1781233564.99 total 451.96 last 19.98) - open/close 9664 (time 1781233575.70 total 462.67 last 25.49) - open/close 9905 (time 1781233585.71 total 472.68 last 24.06) - open/close 10212 (time 1781233595.72 total 482.69 last 30.69) - open/close 10476 (time 1781233606.36 total 493.33 last 24.80) - open/close 10710 (time 1781233616.99 total 503.96 last 22.02) - open/close 10984 (time 1781233627.37 total 514.34 last 26.41) - open/close 11284 (time 1781233638.08 total 525.05 last 27.99) - open/close 11595 (time 1781233648.98 total 535.95 last 28.54) - open/close 11962 (time 1781233659.54 total 546.51 last 34.74) - open/close 12237 (time 1781233670.56 total 557.53 last 24.96) - open/close 12611 (time 1781233681.50 total 568.46 last 34.21) - open/close 12886 (time 1781233691.50 total 578.47 last 27.50) - open/close 13101 (time 1781233702.18 total 589.15 last 20.12) - open/close 13323 (time 1781233712.19 total 599.16 last 22.17) - open/close 13474 (time 1781233722.21 total 609.18 last 15.08) - open/close 13705 (time 1781233732.28 total 619.25 last 22.94) - open/close 13974 (time 1781233743.08 total 630.05 last 24.90) - open/close 14283 (time 1781233753.08 total 640.05 last 30.90) - open/close 14543 (time 1781233763.46 total 650.43 last 25.06) - open/close 14678 (time 1781233774.20 total 661.17 last 12.57) - open/close 14779 (time 1781233785.77 total 672.74 last 8.72) - open/close 14940 (time 1781233797.42 total 684.39 last 13.82) - open/close 15190 (time 1781233807.91 total 694.88 last 23.84) - open/close 15417 (time 1781233818.27 total 705.24 last 21.90) - open/close 15695 (time 1781233828.71 total 715.68 last 26.65) - open/close 15883 (time 1781233839.48 total 726.45 last 17.45) - open/close 16174 (time 1781233849.50 total 736.47 last 29.05) - open/close 16280 (time 1781233860.99 total 747.96 last 9.22) - open/close 16566 (time 1781233871.00 total 757.97 last 28.59) - open/close 16783 (time 1781233881.00 total 767.97 last 21.70) - open/close 17032 (time 1781233891.02 total 777.99 last 24.85) - open/close 17279 (time 1781233901.02 total 787.99 last 24.68) - open/close 17517 (time 1781233913.71 total 800.68 last 18.76) - open/close 17728 (time 1781233924.40 total 811.37 last 19.74) - open/close 17896 (time 1781233934.40 total 821.37 last 16.79) - open/close 18075 (time 1781233945.23 total 832.20 last 16.54) - open/close 18303 (time 1781233955.23 total 842.20 last 22.80) - open/close 18587 (time 1781233965.23 total 852.20 last 28.39) - open/close 18806 (time 1781233975.25 total 862.22 last 21.85) - open/close 19019 (time 1781233986.04 total 873.01 last 19.74) - open/close 19233 (time 1781233996.70 total 883.67 last 20.08) - open/close 19332 (time 1781234007.46 total 894.43 last 9.21) - open/close 19687 (time 1781234017.47 total 904.44 last 35.47) - open/close 19842 (time 1781234027.49 total 914.46 last 15.46) - open/close 20000 (time 1781234034.69 total 921.66 last 21.95) - open/close 20120 (time 1781234044.69 total 931.66 last 12.00) - open/close 20387 (time 1781234054.72 total 941.69 last 26.62) - open/close 20561 (time 1781234064.74 total 951.71 last 17.38) - open/close 20686 (time 1781234075.11 total 962.08 last 12.05) - open/close 20892 (time 1781234085.11 total 972.08 last 20.60) - open/close 21986 (time 1781234095.12 total 982.09 last 109.31) - open/close 22923 (time 1781234105.13 total 992.10 last 93.65) - open/close 23668 (time 1781234115.13 total 1002.10 last 74.45) - open/close 24477 (time 1781234125.14 total 1012.11 last 80.83) - open/close 25382 (time 1781234135.15 total 1022.12 last 90.45) - open/close 26285 (time 1781234145.15 total 1032.12 last 90.27) - open/close 27084 (time 1781234155.15 total 1042.12 last 79.88) - open/close 28121 (time 1781234165.16 total 1052.13 last 103.67) - open/close 28978 (time 1781234175.16 total 1062.13 last 85.68) - open/close 29948 (time 1781234185.16 total 1072.13 last 96.97) - open/close 30884 (time 1781234195.18 total 1082.15 last 93.42) - open/close 31587 (time 1781234205.19 total 1092.16 last 70.26) - open/close 32585 (time 1781234215.20 total 1102.17 last 99.73) - open/close 33602 (time 1781234225.20 total 1112.17 last 101.65) - open/close 34601 (time 1781234235.21 total 1122.18 last 99.81) - open/close 35655 (time 1781234245.21 total 1132.18 last 105.38) - open/close 36727 (time 1781234255.22 total 1142.19 last 107.15) - open/close 37634 (time 1781234265.22 total 1152.19 last 90.67) - open/close 38585 (time 1781234275.23 total 1162.20 last 95.04) - open/close 39390 (time 1781234285.23 total 1172.20 last 80.43) - open/close 40000 (time 1781234294.35 total 1181.32 last 66.93) - open/close 40876 (time 1781234304.37 total 1191.34 last 87.45) - open/close 41747 (time 1781234314.37 total 1201.34 last 87.07) - open/close 42842 (time 1781234324.37 total 1211.34 last 109.44) - open/close 43794 (time 1781234334.38 total 1221.35 last 95.17) - open/close 44710 (time 1781234344.39 total 1231.36 last 91.48) - open/close 45630 (time 1781234354.39 total 1241.36 last 91.95) - open/close 46345 (time 1781234364.40 total 1251.37 last 71.44) - open/close 47241 (time 1781234374.40 total 1261.37 last 89.59) - open/close 48116 (time 1781234384.41 total 1271.38 last 87.45) - open/close 49079 (time 1781234394.42 total 1281.39 last 96.17) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1287.07 seconds: 38.56 ops/second - unlinked 0 (time 1781234405 ; total 0 ; last 0) - unlinked 10000 (time 1781234519 ; total 114 ; last 114) - unlinked 20000 (time 1781234633 ; total 228 ; last 114) - unlinked 30000 (time 1781234697 ; total 292 ; last 64) - unlinked 40000 (time 1781234754 ; total 349 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 405 seconds: 122.545677 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg102-server unloading modules via unload_modules_local on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing unload_modules_local oleg102-server: modules unloaded. pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2