== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:36:28 (1767036988) start mds service on oleg429-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg429-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 678 (time 1767037043.21 total 10.83 last 62.63) - open/close 1571 (time 1767037053.52 total 21.13 last 86.67) - open/close 2332 (time 1767037063.52 total 31.13 last 76.09) - open/close 3072 (time 1767037073.55 total 41.16 last 73.77) - open/close 3621 (time 1767037084.11 total 51.72 last 51.99) - open/close 4343 (time 1767037094.92 total 62.54 last 66.75) - open/close 5137 (time 1767037104.97 total 72.59 last 79.01) - open/close 5973 (time 1767037115.09 total 82.70 last 82.68) - open/close 6741 (time 1767037125.09 total 92.70 last 76.80) - open/close 7522 (time 1767037135.09 total 102.70 last 78.10) - open/close 8277 (time 1767037145.09 total 112.70 last 75.49) - open/close 9069 (time 1767037155.15 total 122.76 last 78.71) - open/close 9807 (time 1767037165.20 total 132.81 last 73.45) - open/close 10671 (time 1767037175.44 total 143.05 last 84.35) - open/close 11534 (time 1767037185.62 total 153.23 last 84.81) - open/close 12274 (time 1767037195.66 total 163.28 last 73.65) - open/close 13064 (time 1767037205.66 total 173.28 last 79.00) - open/close 13842 (time 1767037215.66 total 183.28 last 77.79) - open/close 14572 (time 1767037226.64 total 194.25 last 66.53) - open/close 15465 (time 1767037236.88 total 204.49 last 87.21) - open/close 16331 (time 1767037247.12 total 214.73 last 84.57) - open/close 17115 (time 1767037257.36 total 224.97 last 76.53) - open/close 17892 (time 1767037269.58 total 237.19 last 63.58) - open/close 18612 (time 1767037279.69 total 247.31 last 71.21) - open/close 19252 (time 1767037290.44 total 258.06 last 59.52) - open/close 20000 (time 1767037300.40 total 268.01 last 75.14) - open/close 20594 (time 1767037310.93 total 278.54 last 56.42) - open/close 21360 (time 1767037321.93 total 289.55 last 69.60) - open/close 22096 (time 1767037334.16 total 301.78 last 60.18) - open/close 22985 (time 1767037344.72 total 312.33 last 84.22) - open/close 26925 (time 1767037354.72 total 322.33 last 393.96) - open/close 30000 (time 1767037361.82 total 329.43 last 433.11) - open/close 34437 (time 1767037371.82 total 339.43 last 443.66) - open/close 38915 (time 1767037381.82 total 349.43 last 447.75) - open/close 40000 (time 1767037384.39 total 352.01 last 421.87) - open/close 44523 (time 1767037394.39 total 362.01 last 452.28) - open/close 49011 (time 1767037404.40 total 372.01 last 448.71) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 373.35 seconds: 132.94 ops/second - unlinked 0 (time 1767037407 ; total 0 ; last 0) - unlinked 10000 (time 1767037420 ; total 13 ; last 13) - unlinked 20000 (time 1767037432 ; total 25 ; last 12) - unlinked 30000 (time 1767037443 ; total 36 ; last 11) - unlinked 40000 (time 1767037454 ; total 47 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 59 seconds: 841.220337 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg429-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local oleg429-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg429-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2