== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:38:52 (1783139932) start mds service on oleg204-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 266 (time 1783140048.82 total 10.51 last 25.30) - open/close 514 (time 1783140059.16 total 20.86 last 23.98) - open/close 761 (time 1783140069.16 total 30.86 last 24.69) - open/close 994 (time 1783140079.17 total 40.87 last 23.28) - open/close 1249 (time 1783140089.51 total 51.21 last 24.66) - open/close 1466 (time 1783140100.41 total 62.10 last 19.92) - open/close 1659 (time 1783140110.89 total 72.58 last 18.42) - open/close 1935 (time 1783140120.89 total 82.58 last 27.60) - open/close 2108 (time 1783140130.90 total 92.59 last 17.29) - open/close 2229 (time 1783140141.52 total 103.22 last 11.38) - open/close 2397 (time 1783140151.54 total 113.23 last 16.78) - open/close 2595 (time 1783140161.54 total 123.24 last 19.79) - open/close 2750 (time 1783140171.56 total 133.25 last 15.48) - open/close 2935 (time 1783140181.61 total 143.30 last 18.41) - open/close 3114 (time 1783140192.35 total 154.04 last 16.67) - open/close 3388 (time 1783140203.36 total 165.05 last 24.89) - open/close 3629 (time 1783140214.36 total 176.05 last 21.91) - open/close 3861 (time 1783140225.38 total 187.07 last 21.05) - open/close 4044 (time 1783140236.03 total 197.72 last 17.18) - open/close 4319 (time 1783140246.75 total 208.45 last 25.65) - open/close 4514 (time 1783140257.52 total 219.22 last 18.10) - open/close 4667 (time 1783140268.79 total 230.48 last 13.59) - open/close 4912 (time 1783140279.00 total 240.70 last 23.98) - open/close 5213 (time 1783140289.01 total 250.70 last 30.09) - open/close 5353 (time 1783140299.69 total 261.39 last 13.10) - open/close 5627 (time 1783140309.70 total 271.39 last 27.39) - open/close 5876 (time 1783140319.90 total 281.60 last 24.40) - open/close 6054 (time 1783140330.60 total 292.29 last 16.64) - open/close 6163 (time 1783140341.21 total 302.90 last 10.27) - open/close 6393 (time 1783140351.57 total 313.26 last 22.19) - open/close 6602 (time 1783140361.58 total 323.27 last 20.89) - open/close 6668 (time 1783140372.51 total 334.21 last 6.04) - open/close 6854 (time 1783140382.89 total 344.59 last 17.92) - open/close 7027 (time 1783140392.91 total 354.61 last 17.27) - open/close 7328 (time 1783140402.91 total 364.61 last 30.09) - open/close 7531 (time 1783140413.15 total 374.84 last 19.83) - open/close 7727 (time 1783140423.73 total 385.43 last 18.52) - open/close 7957 (time 1783140433.90 total 395.59 last 22.62) - open/close 8223 (time 1783140444.45 total 406.15 last 25.20) - open/close 8420 (time 1783140454.69 total 416.38 last 19.25) - open/close 8631 (time 1783140465.44 total 427.14 last 19.62) - open/close 8873 (time 1783140476.32 total 438.02 last 22.25) - open/close 9068 (time 1783140486.55 total 448.25 last 19.06) - open/close 9329 (time 1783140497.47 total 459.16 last 23.91) - open/close 9570 (time 1783140507.54 total 469.24 last 23.92) - open/close 9833 (time 1783140518.38 total 480.07 last 24.27) - open/close 9987 (time 1783140529.25 total 490.94 last 14.17) - open/close 10228 (time 1783140539.29 total 500.99 last 24.00) - open/close 10500 (time 1783140549.30 total 511.00 last 27.17) - open/close 10706 (time 1783140559.66 total 521.36 last 19.88) - open/close 10941 (time 1783140570.02 total 531.71 last 22.70) - open/close 11184 (time 1783140580.24 total 541.94 last 23.76) - open/close 11419 (time 1783140590.25 total 551.94 last 23.49) - open/close 11705 (time 1783140600.26 total 561.96 last 28.56) - open/close 11917 (time 1783140610.26 total 571.96 last 21.19) - open/close 12173 (time 1783140620.27 total 581.96 last 25.59) - open/close 12409 (time 1783140630.28 total 591.98 last 23.56) - open/close 12627 (time 1783140640.63 total 602.32 last 21.07) - open/close 12836 (time 1783140651.09 total 612.79 last 19.98) - open/close 13111 (time 1783140661.10 total 622.80 last 27.48) - open/close 13322 (time 1783140671.22 total 632.92 last 20.84) - open/close 13558 (time 1783140681.24 total 642.93 last 23.57) - open/close 13809 (time 1783140691.25 total 652.94 last 25.07) - open/close 13980 (time 1783140701.54 total 663.24 last 16.61) - open/close 14179 (time 1783140712.53 total 674.23 last 18.11) - open/close 14461 (time 1783140725.35 total 687.04 last 22.00) - open/close 14689 (time 1783140735.41 total 697.10 last 22.66) - open/close 14941 (time 1783140746.40 total 708.10 last 22.92) - open/close 15262 (time 1783140757.28 total 718.98 last 29.51) - open/close 15447 (time 1783140767.52 total 729.21 last 18.07) - open/close 15672 (time 1783140778.04 total 739.73 last 21.39) - open/close 15936 (time 1783140788.72 total 750.42 last 24.70) - open/close 16178 (time 1783140799.07 total 760.76 last 23.39) - open/close 16433 (time 1783140809.07 total 770.76 last 25.50) - open/close 16677 (time 1783140820.15 total 781.85 last 22.02) - open/close 16900 (time 1783140832.04 total 793.74 last 18.76) - open/close 17092 (time 1783140842.62 total 804.32 last 18.14) - open/close 17307 (time 1783140852.78 total 814.47 last 21.17) - open/close 17468 (time 1783140865.65 total 827.34 last 12.51) - open/close 17755 (time 1783140876.20 total 837.90 last 27.19) - open/close 18106 (time 1783140886.21 total 847.90 last 35.08) - open/close 18391 (time 1783140897.13 total 858.83 last 26.08) - open/close 18697 (time 1783140907.16 total 868.85 last 30.53) - open/close 18955 (time 1783140917.75 total 879.45 last 24.35) - open/close 19235 (time 1783140927.77 total 889.47 last 27.94) - open/close 19612 (time 1783140937.78 total 899.47 last 37.68) - open/close 20000 (time 1783140941.61 total 903.30 last 101.29) - open/close 20893 (time 1783140951.62 total 913.32 last 89.16) - open/close 21827 (time 1783140961.63 total 923.33 last 93.35) - open/close 22820 (time 1783140971.64 total 933.33 last 99.23) - open/close 23810 (time 1783140981.64 total 943.34 last 98.93) - open/close 24813 (time 1783140991.65 total 953.34 last 100.25) - open/close 25854 (time 1783141001.65 total 963.34 last 104.10) - open/close 26791 (time 1783141011.66 total 973.36 last 93.59) - open/close 27647 (time 1783141021.66 total 983.36 last 85.57) - open/close 28673 (time 1783141031.68 total 993.37 last 102.49) - open/close 29692 (time 1783141041.68 total 1003.38 last 101.84) - open/close 30000 (time 1783141044.28 total 1005.98 last 118.50) - open/close 31110 (time 1783141054.28 total 1015.98 last 110.96) - open/close 32252 (time 1783141064.29 total 1025.98 last 114.19) - open/close 33325 (time 1783141074.29 total 1035.98 last 107.27) - open/close 34222 (time 1783141084.30 total 1045.99 last 89.61) - open/close 35051 (time 1783141094.31 total 1056.00 last 82.84) - open/close 35811 (time 1783141104.31 total 1066.00 last 75.98) - open/close 36822 (time 1783141114.31 total 1076.01 last 101.04) - open/close 37702 (time 1783141124.32 total 1086.01 last 87.95) - open/close 38836 (time 1783141134.32 total 1096.02 last 113.39) - open/close 39856 (time 1783141144.34 total 1106.03 last 101.84) - open/close 40797 (time 1783141154.34 total 1116.04 last 94.03) - open/close 41783 (time 1783141164.34 total 1126.04 last 98.60) - open/close 42965 (time 1783141174.34 total 1136.04 last 118.19) - open/close 43889 (time 1783141184.35 total 1146.04 last 92.37) - open/close 44951 (time 1783141194.35 total 1156.04 last 106.20) - open/close 45968 (time 1783141204.37 total 1166.06 last 101.51) - open/close 46904 (time 1783141214.37 total 1176.07 last 93.54) - open/close 47867 (time 1783141224.37 total 1186.07 last 96.29) - open/close 48884 (time 1783141234.38 total 1196.07 last 101.65) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1203.55 seconds: 41.24 ops/second - unlinked 0 (time 1783141245 ; total 0 ; last 0) - unlinked 10000 (time 1783141332 ; total 87 ; last 87) - unlinked 20000 (time 1783141399 ; total 154 ; last 67) - unlinked 30000 (time 1783141447 ; total 202 ; last 48) - unlinked 40000 (time 1783141514 ; total 269 ; last 67) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 327 seconds: 151.776764 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2