== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:22:33 (1786371753) start mds service on oleg354-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing load_modules_local oleg354-server: Loading modules from /home/green/git/lustre-release/lustre oleg354-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg354-server: detected 4 online CPUs by sysfs oleg354-server: MODOPTS_LIBCFS= oleg354-server: Force libcfs to create 2 CPU partitions oleg354-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg354-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg354-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 242 (time 1786371820.94 total 10.94 last 22.12) - open/close 427 (time 1786371830.99 total 20.99 last 18.42) - open/close 656 (time 1786371841.99 total 31.99 last 20.82) - open/close 873 (time 1786371852.37 total 42.37 last 20.90) - open/close 1080 (time 1786371862.92 total 52.91 last 19.63) - open/close 1306 (time 1786371873.40 total 63.40 last 21.56) - open/close 1579 (time 1786371884.16 total 74.16 last 25.36) - open/close 1726 (time 1786371894.47 total 84.46 last 14.27) - open/close 1919 (time 1786371905.28 total 95.28 last 17.85) - open/close 2060 (time 1786371915.47 total 105.46 last 13.84) - open/close 2255 (time 1786371925.76 total 115.76 last 18.94) - open/close 2450 (time 1786371935.76 total 125.76 last 19.50) - open/close 2683 (time 1786371945.76 total 135.76 last 23.30) - open/close 2856 (time 1786371955.76 total 145.76 last 17.30) - open/close 3029 (time 1786371966.66 total 156.66 last 15.87) - open/close 3168 (time 1786371977.16 total 167.15 last 13.25) - open/close 3326 (time 1786371987.62 total 177.62 last 15.10) - open/close 3618 (time 1786371997.63 total 187.63 last 29.18) - open/close 3764 (time 1786372007.76 total 197.76 last 14.41) - open/close 3974 (time 1786372018.50 total 208.50 last 19.55) - open/close 4117 (time 1786372030.81 total 220.80 last 11.62) - open/close 4261 (time 1786372041.80 total 231.79 last 13.10) - open/close 4350 (time 1786372052.11 total 242.11 last 8.63) - open/close 4495 (time 1786372062.91 total 252.91 last 13.42) - open/close 4566 (time 1786372073.03 total 263.02 last 7.02) - open/close 4703 (time 1786372084.07 total 274.06 last 12.41) - open/close 4871 (time 1786372094.53 total 284.53 last 16.06) - open/close 5069 (time 1786372105.22 total 295.22 last 18.52) - open/close 5204 (time 1786372115.22 total 305.22 last 13.50) - open/close 5370 (time 1786372125.58 total 315.58 last 16.03) - open/close 5615 (time 1786372136.08 total 326.08 last 23.33) - open/close 5834 (time 1786372146.08 total 336.08 last 21.89) - open/close 5998 (time 1786372156.76 total 346.75 last 15.36) - open/close 6240 (time 1786372167.06 total 357.05 last 23.50) - open/close 6588 (time 1786372177.06 total 367.06 last 34.79) - open/close 6897 (time 1786372187.06 total 377.06 last 30.88) - open/close 7246 (time 1786372198.08 total 388.08 last 31.68) - open/close 7593 (time 1786372208.08 total 398.08 last 34.70) - open/close 7931 (time 1786372218.08 total 408.08 last 33.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8305) error: No space left on device total: 8305 open/close in 418.05 seconds: 19.87 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (75900) - No such process - unlinked 0 (time 1786372233 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8305) error: No such file or directory total: 8305 unlinks in 31 seconds: 267.903229 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server unloading modules via unload_modules_local on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local oleg354-server: modules unloaded. pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2