== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:27:52 (1777426072) start mds service on oleg237-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg237-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 758 (time 1777426131.26 total 10.94 last 69.28) - open/close 1492 (time 1777426142.14 total 21.82 last 67.46) - open/close 2223 (time 1777426153.08 total 32.77 last 66.79) - open/close 2934 (time 1777426163.08 total 42.77 last 71.10) - open/close 3577 (time 1777426173.88 total 53.57 last 59.55) - open/close 4307 (time 1777426184.70 total 64.38 last 67.49) - open/close 5076 (time 1777426195.58 total 75.26 last 70.66) - open/close 5894 (time 1777426206.58 total 86.27 last 74.33) - open/close 6648 (time 1777426217.59 total 97.28 last 68.48) - open/close 7422 (time 1777426228.47 total 108.16 last 71.15) - open/close 8136 (time 1777426239.29 total 118.98 last 65.98) - open/close 8876 (time 1777426250.30 total 129.98 last 67.25) - open/close 9604 (time 1777426261.31 total 140.99 last 66.13) - open/close 10000 (time 1777426266.35 total 146.03 last 78.57) - open/close 10750 (time 1777426276.35 total 156.03 last 74.99) - open/close 11534 (time 1777426286.35 total 166.04 last 78.37) - open/close 12237 (time 1777426297.34 total 177.02 last 63.99) - open/close 13003 (time 1777426307.34 total 187.02 last 76.60) - open/close 13764 (time 1777426317.37 total 197.05 last 75.86) - open/close 14554 (time 1777426328.38 total 208.07 last 71.75) - open/close 15334 (time 1777426338.38 total 218.07 last 77.99) - open/close 15989 (time 1777426349.18 total 228.86 last 60.67) - open/close 16483 (time 1777426359.55 total 239.23 last 47.66) - open/close 17247 (time 1777426369.55 total 249.23 last 76.40) - open/close 17943 (time 1777426380.47 total 260.16 last 63.69) - open/close 18642 (time 1777426391.36 total 271.04 last 64.22) - open/close 19348 (time 1777426402.23 total 281.92 last 64.92) - open/close 20000 (time 1777426410.96 total 290.64 last 74.73) - open/close 20789 (time 1777426421.95 total 301.63 last 71.79) - open/close 24052 (time 1777426431.95 total 311.63 last 326.26) - open/close 28869 (time 1777426441.95 total 321.63 last 481.65) - open/close 30000 (time 1777426444.28 total 323.96 last 486.19) - open/close 34670 (time 1777426454.28 total 333.96 last 466.91) - open/close 39345 (time 1777426464.28 total 343.96 last 467.42) - open/close 44105 (time 1777426474.28 total 353.97 last 475.94) - open/close 48843 (time 1777426484.28 total 363.97 last 473.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 365.64 seconds: 135.74 ops/second - unlinked 0 (time 1777426487 ; total 0 ; last 0) - unlinked 10000 (time 1777426499 ; total 12 ; last 12) - unlinked 20000 (time 1777426512 ; total 25 ; last 13) - unlinked 30000 (time 1777426524 ; total 37 ; last 12) - unlinked 40000 (time 1777426536 ; total 49 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 60 seconds: 827.200012 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2