== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:02:27 (1781240547) start mds service on oleg359-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg359-server: detected 4 online CPUs by sysfs oleg359-server: MODOPTS_LIBCFS= oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg359-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 197 (time 1781240627.00 total 10.01 last 19.69) - open/close 422 (time 1781240638.02 total 21.02 last 20.42) - open/close 640 (time 1781240648.02 total 31.02 last 21.80) - open/close 848 (time 1781240658.25 total 41.26 last 20.32) - open/close 987 (time 1781240668.74 total 51.75 last 13.25) - open/close 1147 (time 1781240679.64 total 62.64 last 14.68) - open/close 1328 (time 1781240690.08 total 73.08 last 17.34) - open/close 1472 (time 1781240701.52 total 84.52 last 12.59) - open/close 1715 (time 1781240712.39 total 95.40 last 22.34) - open/close 1947 (time 1781240722.63 total 105.63 last 22.67) - open/close 2185 (time 1781240732.63 total 115.64 last 23.79) - open/close 2399 (time 1781240742.64 total 125.65 last 21.37) - open/close 2523 (time 1781240753.49 total 136.50 last 11.43) - open/close 2683 (time 1781240763.52 total 146.53 last 15.96) - open/close 2858 (time 1781240775.30 total 158.31 last 14.86) - open/close 3055 (time 1781240785.30 total 168.31 last 19.69) - open/close 3267 (time 1781240795.31 total 178.32 last 21.19) - open/close 3512 (time 1781240807.57 total 190.58 last 19.98) - open/close 3649 (time 1781240817.73 total 200.74 last 13.48) - open/close 3856 (time 1781240827.75 total 210.75 last 20.67) - open/close 4093 (time 1781240837.75 total 220.76 last 23.69) - open/close 4262 (time 1781240847.76 total 230.77 last 16.88) - open/close 4497 (time 1781240857.76 total 240.77 last 23.49) - open/close 4708 (time 1781240869.04 total 252.05 last 18.71) - open/close 4894 (time 1781240879.05 total 262.06 last 18.59) - open/close 5029 (time 1781240889.06 total 272.06 last 13.49) - open/close 5220 (time 1781240899.71 total 282.72 last 17.92) - open/close 5395 (time 1781240909.71 total 292.72 last 17.50) - open/close 5611 (time 1781240919.72 total 302.73 last 21.59) - open/close 5772 (time 1781240930.11 total 313.12 last 15.49) - open/close 5980 (time 1781240941.12 total 324.13 last 18.90) - open/close 6206 (time 1781240951.45 total 334.45 last 21.88) - open/close 6386 (time 1781240961.46 total 344.47 last 17.98) - open/close 6460 (time 1781240971.53 total 354.53 last 7.35) - open/close 6692 (time 1781240981.77 total 364.78 last 22.65) - open/close 6838 (time 1781240994.45 total 377.46 last 11.51) - open/close 7073 (time 1781241005.39 total 388.40 last 21.48) - open/close 7338 (time 1781241015.88 total 398.89 last 25.27) - open/close 7531 (time 1781241026.94 total 409.95 last 17.45) - open/close 7756 (time 1781241037.08 total 420.08 last 22.20) - open/close 7949 (time 1781241048.26 total 431.27 last 17.26) - open/close 8136 (time 1781241058.57 total 441.58 last 18.14) - open/close 8377 (time 1781241070.37 total 453.37 last 20.43) - open/close 8487 (time 1781241081.35 total 464.36 last 10.02) - open/close 8587 (time 1781241092.25 total 475.26 last 9.18) - open/close 8719 (time 1781241102.78 total 485.78 last 12.54) - open/close 8783 (time 1781241112.78 total 495.79 last 6.40) - open/close 8928 (time 1781241123.03 total 506.04 last 14.14) - open/close 9152 (time 1781241133.04 total 516.05 last 22.39) - open/close 9283 (time 1781241144.14 total 527.15 last 11.80) - open/close 9483 (time 1781241154.15 total 537.16 last 19.98) - open/close 9650 (time 1781241164.16 total 547.17 last 16.68) - open/close 9819 (time 1781241174.18 total 557.19 last 16.87) - open/close 10000 (time 1781241182.43 total 565.44 last 21.94) - open/close 10136 (time 1781241193.30 total 576.31 last 12.51) - open/close 10244 (time 1781241204.22 total 587.23 last 9.89) - open/close 10382 (time 1781241216.52 total 599.53 last 11.22) - open/close 10494 (time 1781241226.62 total 609.63 last 11.08) - open/close 10684 (time 1781241237.25 total 620.26 last 17.88) - open/close 10745 (time 1781241247.97 total 630.98 last 5.69) - open/close 10894 (time 1781241257.97 total 640.98 last 14.89) - open/close 10980 (time 1781241268.05 total 651.06 last 8.53) - open/close 11160 (time 1781241278.14 total 661.14 last 17.85) - open/close 11385 (time 1781241288.14 total 671.14 last 22.50) - open/close 11659 (time 1781241298.87 total 681.88 last 25.52) - open/close 11899 (time 1781241309.32 total 692.33 last 22.97) - open/close 12029 (time 1781241319.57 total 702.58 last 12.68) - open/close 12274 (time 1781241330.02 total 713.03 last 23.45) - open/close 12448 (time 1781241340.05 total 723.06 last 17.35) - open/close 12718 (time 1781241350.53 total 733.54 last 25.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12849) error: No space left on device total: 12849 open/close in 743.52 seconds: 17.28 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8987: kill: (74340) - No such process - unlinked 0 (time 1781241371 ; total 0 ; last 0) - unlinked 10000 (time 1781241495 ; total 124 ; last 124) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12849) error: No such file or directory total: 12849 unlinks in 151 seconds: 85.092712 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server unloading modules via unload_modules_local on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local oleg359-server: modules unloaded. pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2