== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:09:28 (1771441768) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 660 (time 1771441824.88 total 10.84 last 60.89) - open/close 1200 (time 1771441835.31 total 21.27 last 51.77) - open/close 1841 (time 1771441845.94 total 31.89 last 60.33) - open/close 2448 (time 1771441856.69 total 42.65 last 56.46) - open/close 2938 (time 1771441867.00 total 52.95 last 47.55) - open/close 3544 (time 1771441877.68 total 63.64 last 56.70) - open/close 4180 (time 1771441888.43 total 74.39 last 59.16) - open/close 4788 (time 1771441899.19 total 85.14 last 56.54) - open/close 5304 (time 1771441909.55 total 95.51 last 49.77) - open/close 5962 (time 1771441919.55 total 105.51 last 65.79) - open/close 6628 (time 1771441930.48 total 116.44 last 60.95) - open/close 7193 (time 1771441941.17 total 127.13 last 52.86) - open/close 7941 (time 1771441952.18 total 138.13 last 67.95) - open/close 8623 (time 1771441963.00 total 148.95 last 63.05) - open/close 9263 (time 1771441973.62 total 159.57 last 60.25) - open/close 9867 (time 1771441984.05 total 170.01 last 57.91) - open/close 10000 (time 1771441986.45 total 172.41 last 55.28) - open/close 10751 (time 1771441997.43 total 183.38 last 68.46) - open/close 11479 (time 1771442008.37 total 194.33 last 66.52) - open/close 12279 (time 1771442018.37 total 204.33 last 79.98) - open/close 12950 (time 1771442029.24 total 215.19 last 61.75) - open/close 13596 (time 1771442039.99 total 225.94 last 60.10) - open/close 14477 (time 1771442050.35 total 236.31 last 84.97) - open/close 15373 (time 1771442060.86 total 246.81 last 85.32) - open/close 16322 (time 1771442071.35 total 257.31 last 90.40) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-16979) error: No space left on device total: 16979 open/close in 264.17 seconds: 64.27 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8955: kill: (111890) - No such process - unlinked 0 (time 1771442081 ; total 0 ; last 0) - unlinked 10000 (time 1771442093 ; total 12 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-16979) error: No such file or directory total: 16979 unlinks in 21 seconds: 808.523804 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2