== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:38:14 (1788446294) start mds service on oleg230-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions oleg230-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg230-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg230-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg230-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 107 (time 1788446411.23 total 10.01 last 10.69) - open/close 370 (time 1788446422.15 total 20.94 last 24.08) - open/close 605 (time 1788446432.17 total 30.96 last 23.45) - open/close 803 (time 1788446442.18 total 40.96 last 19.79) - open/close 1055 (time 1788446452.49 total 51.28 last 24.42) - open/close 1270 (time 1788446462.87 total 61.66 last 20.71) - open/close 1456 (time 1788446473.12 total 71.90 last 18.15) - open/close 1610 (time 1788446484.28 total 83.07 last 13.80) - open/close 1875 (time 1788446495.19 total 93.98 last 24.28) - open/close 2051 (time 1788446505.67 total 104.46 last 16.80) - open/close 2179 (time 1788446517.90 total 116.69 last 10.47) - open/close 2399 (time 1788446527.92 total 126.70 last 21.96) - open/close 2543 (time 1788446538.63 total 137.42 last 13.44) - open/close 2788 (time 1788446549.38 total 148.16 last 22.79) - open/close 2951 (time 1788446559.50 total 158.29 last 16.11) - open/close 3192 (time 1788446569.61 total 168.40 last 23.83) - open/close 3319 (time 1788446580.26 total 179.05 last 11.93) - open/close 3536 (time 1788446591.05 total 189.84 last 20.11) - open/close 3817 (time 1788446601.06 total 199.84 last 28.09) - open/close 4072 (time 1788446613.26 total 212.04 last 20.90) - open/close 4278 (time 1788446624.95 total 223.74 last 17.62) - open/close 4500 (time 1788446635.14 total 233.93 last 21.78) - open/close 4818 (time 1788446645.15 total 243.93 last 31.79) - open/close 5061 (time 1788446655.77 total 254.56 last 22.87) - open/close 5306 (time 1788446665.77 total 264.56 last 24.50) - open/close 5435 (time 1788446676.75 total 275.54 last 11.75) - open/close 5494 (time 1788446687.32 total 286.11 last 5.58) - open/close 5707 (time 1788446697.33 total 296.12 last 21.29) - open/close 5992 (time 1788446710.01 total 308.80 last 22.48) - open/close 6151 (time 1788446720.40 total 319.18 last 15.31) - open/close 6299 (time 1788446730.56 total 329.34 last 14.57) - open/close 6540 (time 1788446741.37 total 340.16 last 22.29) - open/close 6796 (time 1788446752.33 total 351.12 last 23.36) - open/close 7113 (time 1788446762.34 total 361.12 last 31.69) - open/close 7366 (time 1788446772.72 total 371.50 last 24.37) - open/close 7635 (time 1788446783.11 total 381.90 last 25.89) - open/close 7872 (time 1788446793.48 total 392.26 last 22.87) - open/close 8110 (time 1788446805.90 total 404.68 last 19.16) - open/close 8381 (time 1788446816.59 total 415.37 last 25.35) - open/close 8663 (time 1788446827.16 total 425.95 last 26.67) - open/close 8910 (time 1788446837.17 total 435.95 last 24.69) - open/close 9033 (time 1788446847.17 total 445.95 last 12.30) - open/close 9201 (time 1788446861.46 total 460.24 last 11.76) - open/close 9468 (time 1788446871.47 total 470.26 last 26.66) - open/close 9646 (time 1788446881.48 total 480.26 last 17.79) - open/close 9896 (time 1788446891.59 total 490.38 last 24.72) - open/close 10000 (time 1788446895.44 total 494.23 last 27.00) - open/close 10157 (time 1788446905.45 total 504.23 last 15.70) - open/close 10352 (time 1788446916.44 total 515.23 last 17.73) - open/close 10506 (time 1788446927.29 total 526.07 last 14.20) - open/close 10702 (time 1788446937.93 total 536.71 last 18.42) - open/close 10988 (time 1788446948.09 total 546.88 last 28.13) - open/close 11267 (time 1788446958.73 total 557.52 last 26.22) - open/close 11545 (time 1788446968.74 total 567.53 last 27.78) - open/close 11735 (time 1788446979.26 total 578.04 last 18.07) - open/close 12001 (time 1788446990.28 total 589.07 last 24.13) - open/close 12290 (time 1788447001.21 total 599.99 last 26.45) - open/close 12566 (time 1788447011.91 total 610.69 last 25.80) - open/close 12839 (time 1788447022.02 total 620.81 last 26.99) - open/close 13094 (time 1788447033.90 total 632.68 last 21.48) - open/close 13419 (time 1788447043.90 total 642.69 last 32.49) - open/close 13683 (time 1788447053.90 total 652.69 last 26.39) - open/close 13841 (time 1788447064.02 total 662.80 last 15.62) - open/close 13985 (time 1788447074.60 total 673.39 last 13.61) - open/close 14207 (time 1788447084.65 total 683.44 last 22.09) - open/close 14366 (time 1788447094.74 total 693.53 last 15.75) - open/close 14559 (time 1788447105.36 total 704.15 last 18.18) - open/close 14697 (time 1788447116.43 total 715.21 last 12.47) - open/close 14861 (time 1788447127.42 total 726.20 last 14.92) - open/close 15109 (time 1788447137.87 total 736.65 last 23.73) - open/close 15377 (time 1788447147.91 total 746.70 last 26.69) - open/close 15652 (time 1788447157.92 total 756.70 last 27.48) - open/close 15832 (time 1788447167.93 total 766.71 last 17.98) - open/close 16086 (time 1788447177.94 total 776.73 last 25.36) - open/close 16354 (time 1788447187.97 total 786.75 last 26.74) - open/close 16630 (time 1788447198.98 total 797.77 last 25.06) - open/close 16767 (time 1788447209.16 total 807.94 last 13.46) - open/close 17022 (time 1788447219.22 total 818.00 last 25.35) - open/close 17180 (time 1788447230.03 total 828.82 last 14.61) - open/close 17386 (time 1788447241.22 total 840.00 last 18.41) - open/close 17530 (time 1788447253.84 total 852.62 last 11.41) - open/close 17789 (time 1788447264.65 total 863.44 last 23.95) - open/close 18071 (time 1788447275.28 total 874.07 last 26.53) - open/close 18290 (time 1788447285.29 total 884.07 last 21.89) - open/close 18514 (time 1788447296.06 total 894.85 last 20.79) - open/close 18847 (time 1788447306.18 total 904.96 last 32.93) - open/close 19091 (time 1788447316.18 total 914.97 last 24.39) - open/close 19305 (time 1788447326.19 total 924.97 last 21.39) - open/close 19567 (time 1788447336.53 total 935.31 last 25.34) - open/close 19803 (time 1788447346.76 total 945.54 last 23.06) - open/close 20000 (time 1788447353.95 total 952.73 last 27.39) - open/close 20578 (time 1788447363.95 total 962.74 last 57.79) - open/close 21408 (time 1788447373.96 total 972.75 last 82.91) - open/close 22327 (time 1788447383.99 total 982.77 last 91.66) - open/close 23110 (time 1788447394.02 total 992.80 last 78.05) - open/close 23949 (time 1788447404.03 total 1002.82 last 83.79) - open/close 24726 (time 1788447414.07 total 1012.85 last 77.45) - open/close 25473 (time 1788447424.09 total 1022.87 last 74.54) - open/close 26364 (time 1788447434.09 total 1032.88 last 89.07) - open/close 27282 (time 1788447444.09 total 1042.88 last 91.78) - open/close 28096 (time 1788447454.09 total 1052.88 last 81.40) - open/close 29012 (time 1788447464.10 total 1062.88 last 91.56) - open/close 29915 (time 1788447474.10 total 1072.89 last 90.26) - open/close 30867 (time 1788447484.11 total 1082.89 last 95.14) - open/close 31787 (time 1788447494.11 total 1092.89 last 91.99) - open/close 32466 (time 1788447504.14 total 1102.93 last 67.67) - open/close 33213 (time 1788447514.15 total 1112.94 last 74.62) - open/close 34135 (time 1788447524.16 total 1122.94 last 92.19) - open/close 34961 (time 1788447534.16 total 1132.95 last 82.54) - open/close 35714 (time 1788447544.17 total 1142.95 last 75.26) - open/close 36345 (time 1788447554.18 total 1152.96 last 63.03) - open/close 36990 (time 1788447564.18 total 1162.97 last 64.46) - open/close 37675 (time 1788447574.19 total 1172.98 last 68.44) - open/close 38507 (time 1788447584.20 total 1182.99 last 83.13) - open/close 39433 (time 1788447594.20 total 1192.99 last 92.57) - open/close 40000 (time 1788447601.95 total 1200.74 last 73.17) - open/close 40709 (time 1788447611.97 total 1210.76 last 70.74) - open/close 41290 (time 1788447622.01 total 1220.80 last 57.87) - open/close 42063 (time 1788447632.01 total 1230.80 last 77.30) - open/close 42881 (time 1788447642.02 total 1240.80 last 81.79) - open/close 43654 (time 1788447652.02 total 1250.80 last 77.30) - open/close 44389 (time 1788447662.02 total 1260.80 last 73.48) - open/close 44904 (time 1788447672.04 total 1270.82 last 51.39) - open/close 45798 (time 1788447682.05 total 1280.83 last 89.33) - open/close 46694 (time 1788447692.05 total 1290.84 last 89.56) - open/close 47383 (time 1788447702.06 total 1300.84 last 68.88) - open/close 48368 (time 1788447712.06 total 1310.85 last 98.46) - open/close 49239 (time 1788447722.07 total 1320.85 last 87.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1325.73 seconds: 37.44 ops/second - unlinked 0 (time 1788447731 ; total 0 ; last 0) - unlinked 10000 (time 1788447852 ; total 121 ; last 121) - unlinked 20000 (time 1788447991 ; total 260 ; last 139) - unlinked 30000 (time 1788448113 ; total 382 ; last 122) - unlinked 40000 (time 1788448235 ; total 504 ; last 122) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 622 seconds: 79.792603 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg230-server unloading modules via unload_modules_local on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local oleg230-server: modules unloaded. pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2