== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:14:31 (1781820871) start mds service on oleg142-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: detected 4 online CPUs by sysfs oleg142-server: MODOPTS_LIBCFS= oleg142-server: Force libcfs to create 2 CPU partitions oleg142-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg142-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg142-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg142-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 127 (time 1781820992.81 total 10.96 last 11.59) - open/close 255 (time 1781821003.02 total 21.17 last 12.54) - open/close 388 (time 1781821013.12 total 31.27 last 13.17) - open/close 537 (time 1781821023.18 total 41.33 last 14.81) - open/close 669 (time 1781821033.22 total 51.37 last 13.15) - open/close 771 (time 1781821043.94 total 62.09 last 9.51) - open/close 826 (time 1781821054.63 total 72.78 last 5.14) - open/close 951 (time 1781821065.05 total 83.20 last 12.00) - open/close 1028 (time 1781821075.61 total 93.76 last 7.29) - open/close 1106 (time 1781821085.62 total 103.77 last 7.79) - open/close 1195 (time 1781821096.22 total 114.37 last 8.40) - open/close 1273 (time 1781821106.59 total 124.74 last 7.52) - open/close 1360 (time 1781821117.48 total 135.64 last 7.99) - open/close 1452 (time 1781821127.49 total 145.64 last 9.20) - open/close 1549 (time 1781821137.51 total 155.66 last 9.68) - open/close 1676 (time 1781821147.52 total 165.67 last 12.68) - open/close 1794 (time 1781821158.51 total 176.66 last 10.74) - open/close 1876 (time 1781821169.19 total 187.34 last 7.68) - open/close 1969 (time 1781821179.96 total 198.11 last 8.63) - open/close 2136 (time 1781821190.20 total 208.35 last 16.32) - open/close 2225 (time 1781821201.12 total 219.27 last 8.15) - open/close 2313 (time 1781821211.93 total 230.08 last 8.14) - open/close 2422 (time 1781821221.94 total 240.10 last 10.88) - open/close 2606 (time 1781821232.87 total 251.02 last 16.84) - open/close 2747 (time 1781821242.89 total 261.04 last 14.07) - open/close 2916 (time 1781821252.90 total 271.05 last 16.89) - open/close 3032 (time 1781821263.60 total 281.76 last 10.83) - open/close 3160 (time 1781821273.64 total 291.79 last 12.75) - open/close 3320 (time 1781821284.19 total 302.34 last 15.17) - open/close 3391 (time 1781821294.51 total 312.66 last 6.88) - open/close 3467 (time 1781821304.55 total 322.70 last 7.57) - open/close 3576 (time 1781821315.48 total 333.63 last 9.97) - open/close 3707 (time 1781821326.30 total 344.45 last 12.11) - open/close 3806 (time 1781821336.31 total 354.46 last 9.89) - open/close 3935 (time 1781821346.48 total 364.63 last 12.69) - open/close 4045 (time 1781821356.88 total 375.03 last 10.57) - open/close 4156 (time 1781821367.52 total 385.68 last 10.43) - open/close 4264 (time 1781821377.57 total 395.73 last 10.75) - open/close 4389 (time 1781821388.51 total 406.66 last 11.44) - open/close 4502 (time 1781821398.54 total 416.69 last 11.26) - open/close 4620 (time 1781821408.93 total 427.08 last 11.36) - open/close 4720 (time 1781821419.43 total 437.58 last 9.52) - open/close 4849 (time 1781821429.60 total 447.75 last 12.68) - open/close 4955 (time 1781821439.97 total 458.12 last 10.22) - open/close 5061 (time 1781821449.98 total 468.13 last 10.59) - open/close 5191 (time 1781821460.26 total 478.41 last 12.65) - open/close 5281 (time 1781821470.27 total 488.42 last 8.99) - open/close 5373 (time 1781821480.28 total 498.43 last 9.19) - open/close 5445 (time 1781821491.87 total 510.02 last 6.21) - open/close 5580 (time 1781821501.92 total 520.07 last 13.44) - open/close 5696 (time 1781821512.50 total 530.65 last 10.96) - open/close 5799 (time 1781821522.64 total 540.79 last 10.16) - open/close 5914 (time 1781821532.71 total 550.86 last 11.42) - open/close 6006 (time 1781821542.72 total 560.87 last 9.19) - open/close 6135 (time 1781821552.74 total 570.89 last 12.88) - open/close 6211 (time 1781821563.14 total 581.29 last 7.31) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6322) error: No space left on device total: 6322 open/close in 589.49 seconds: 10.72 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8991: kill: (75129) - No such process - unlinked 0 (time 1781821581 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6322) error: No such file or directory total: 6322 unlinks in 93 seconds: 67.978493 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg142-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server unloading modules via unload_modules_local on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing unload_modules_local oleg142-server: modules unloaded. pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2