== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:52:39 (1786315959) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 164 (time 1786316095.33 total 10.01 last 16.39) - open/close 315 (time 1786316105.33 total 20.01 last 15.10) - open/close 416 (time 1786316115.88 total 30.56 last 9.58) - open/close 591 (time 1786316126.88 total 41.56 last 15.91) - open/close 758 (time 1786316136.94 total 51.62 last 16.61) - open/close 961 (time 1786316147.41 total 62.10 last 19.38) - open/close 1078 (time 1786316157.59 total 72.27 last 11.50) - open/close 1273 (time 1786316167.90 total 82.58 last 18.91) - open/close 1456 (time 1786316178.21 total 92.90 last 17.74) - open/close 1585 (time 1786316188.21 total 102.90 last 12.90) - open/close 1830 (time 1786316198.95 total 113.63 last 22.82) - open/close 1901 (time 1786316208.95 total 123.64 last 7.10) - open/close 2067 (time 1786316218.99 total 133.67 last 16.54) - open/close 2148 (time 1786316229.21 total 143.89 last 7.93) - open/close 2321 (time 1786316239.80 total 154.48 last 16.34) - open/close 2508 (time 1786316250.22 total 164.90 last 17.94) - open/close 2742 (time 1786316260.25 total 174.93 last 23.33) - open/close 2895 (time 1786316271.03 total 185.71 last 14.20) - open/close 3104 (time 1786316281.76 total 196.44 last 19.47) - open/close 3301 (time 1786316292.37 total 207.05 last 18.57) - open/close 3486 (time 1786316302.37 total 217.05 last 18.50) - open/close 3564 (time 1786316312.39 total 227.07 last 7.78) - open/close 3787 (time 1786316323.43 total 238.12 last 20.19) - open/close 3921 (time 1786316333.44 total 248.13 last 13.39) - open/close 4155 (time 1786316344.07 total 258.75 last 22.02) - open/close 4309 (time 1786316354.21 total 268.89 last 15.19) - open/close 4433 (time 1786316364.22 total 278.91 last 12.38) - open/close 4565 (time 1786316374.26 total 288.94 last 13.15) - open/close 4764 (time 1786316384.27 total 298.95 last 19.88) - open/close 5005 (time 1786316394.97 total 309.66 last 22.51) - open/close 5206 (time 1786316405.77 total 320.46 last 18.61) - open/close 5404 (time 1786316416.31 total 330.99 last 18.79) - open/close 5544 (time 1786316426.71 total 341.39 last 13.46) - open/close 5746 (time 1786316436.87 total 351.56 last 19.88) - open/close 5921 (time 1786316447.88 total 362.56 last 15.90) - open/close 6138 (time 1786316458.47 total 373.15 last 20.49) - open/close 6264 (time 1786316468.82 total 383.51 last 12.17) - open/close 6342 (time 1786316479.07 total 393.76 last 7.61) - open/close 6470 (time 1786316489.10 total 403.79 last 12.76) - open/close 6724 (time 1786316499.16 total 413.85 last 25.25) - open/close 6983 (time 1786316509.60 total 424.29 last 24.82) - open/close 7153 (time 1786316520.29 total 434.98 last 15.90) - open/close 7352 (time 1786316533.04 total 447.72 last 15.61) - open/close 7593 (time 1786316543.38 total 458.07 last 23.30) - open/close 7797 (time 1786316553.39 total 468.07 last 20.39) - open/close 7984 (time 1786316564.27 total 478.96 last 17.18) - open/close 8189 (time 1786316574.50 total 489.18 last 20.05) - open/close 8382 (time 1786316585.30 total 499.99 last 17.86) - open/close 8568 (time 1786316595.31 total 509.99 last 18.59) - open/close 8732 (time 1786316605.32 total 520.00 last 16.38) - open/close 8901 (time 1786316615.70 total 530.38 last 16.28) - open/close 9096 (time 1786316626.00 total 540.68 last 18.94) - open/close 9308 (time 1786316636.91 total 551.60 last 19.42) - open/close 9463 (time 1786316647.39 total 562.07 last 14.80) - open/close 9625 (time 1786316657.44 total 572.12 last 16.12) - open/close 9796 (time 1786316668.32 total 583.00 last 15.71) - open/close 10000 (time 1786316677.03 total 591.72 last 23.41) - open/close 10203 (time 1786316687.07 total 601.75 last 20.23) - open/close 10283 (time 1786316697.08 total 611.77 last 7.99) - open/close 10441 (time 1786316707.87 total 622.55 last 14.65) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-10482) error: No space left on device total: 10482 open/close in 623.34 seconds: 16.82 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (114858) - No such process - unlinked 0 (time 1786316717 ; total 0 ; last 0) - unlinked 10000 (time 1786316796 ; total 79 ; last 79) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-10482) error: No such file or directory total: 10482 unlinks in 82 seconds: 127.829269 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2