== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:54:44 (1785401684) start mds service on oleg335-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg335-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 64 (time 1785401795.88 total 10.29 last 6.22) - open/close 193 (time 1785401805.88 total 20.29 last 12.90) - open/close 283 (time 1785401815.89 total 30.29 last 9.00) - open/close 406 (time 1785401825.92 total 40.32 last 12.27) - open/close 542 (time 1785401836.10 total 50.51 last 13.35) - open/close 636 (time 1785401846.11 total 60.52 last 9.39) - open/close 712 (time 1785401857.27 total 71.67 last 6.81) - open/close 891 (time 1785401867.27 total 81.67 last 17.90) - open/close 966 (time 1785401877.68 total 92.08 last 7.20) - open/close 1085 (time 1785401888.29 total 102.69 last 11.21) - open/close 1212 (time 1785401898.44 total 112.85 last 12.51) - open/close 1344 (time 1785401908.97 total 123.38 last 12.54) - open/close 1486 (time 1785401918.98 total 133.38 last 14.19) - open/close 1578 (time 1785401929.78 total 144.18 last 8.52) - open/close 1678 (time 1785401940.77 total 155.18 last 9.09) - open/close 1830 (time 1785401951.80 total 166.21 last 13.78) - open/close 1983 (time 1785401962.82 total 177.22 last 13.89) - open/close 2090 (time 1785401973.18 total 187.59 last 10.32) - open/close 2203 (time 1785401983.73 total 198.13 last 10.72) - open/close 2335 (time 1785401994.21 total 208.61 last 12.60) - open/close 2495 (time 1785402004.79 total 219.20 last 15.11) - open/close 2664 (time 1785402015.44 total 229.85 last 15.87) - open/close 2773 (time 1785402025.98 total 240.38 last 10.35) - open/close 2803 (time 1785402036.22 total 250.63 last 2.93) - open/close 2877 (time 1785402046.64 total 261.05 last 7.10) - open/close 2993 (time 1785402057.06 total 271.47 last 11.13) - open/close 3042 (time 1785402067.83 total 282.23 last 4.55) - open/close 3134 (time 1785402077.86 total 292.27 last 9.17) - open/close 3268 (time 1785402088.18 total 302.59 last 12.99) - open/close 3410 (time 1785402098.35 total 312.75 last 13.97) - open/close 3503 (time 1785402108.89 total 323.30 last 8.82) - open/close 3664 (time 1785402119.85 total 334.26 last 14.69) - open/close 3814 (time 1785402129.91 total 344.31 last 14.92) - open/close 3947 (time 1785402139.91 total 354.32 last 13.29) - open/close 4057 (time 1785402150.06 total 364.47 last 10.84) - open/close 4124 (time 1785402160.44 total 374.84 last 6.46) - open/close 4200 (time 1785402170.99 total 385.39 last 7.20) - open/close 4292 (time 1785402181.06 total 395.47 last 9.13) - open/close 4360 (time 1785402191.14 total 405.55 last 6.74) - open/close 4491 (time 1785402201.17 total 415.57 last 13.07) - open/close 4614 (time 1785402211.64 total 426.05 last 11.74) - open/close 4749 (time 1785402221.84 total 436.24 last 13.24) - open/close 4855 (time 1785402232.24 total 446.64 last 10.19) - open/close 5010 (time 1785402242.25 total 456.65 last 15.48) - open/close 5126 (time 1785402252.78 total 467.18 last 11.02) - open/close 5279 (time 1785402262.80 total 477.21 last 15.26) - open/close 5405 (time 1785402273.54 total 487.94 last 11.73) - open/close 5514 (time 1785402283.70 total 498.10 last 10.73) - open/close 5608 (time 1785402293.73 total 508.13 last 9.38) - open/close 5783 (time 1785402303.74 total 518.14 last 17.48) - open/close 5926 (time 1785402313.74 total 528.15 last 14.29) - open/close 6086 (time 1785402323.75 total 538.16 last 15.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6162) error: No space left on device total: 6162 open/close in 544.44 seconds: 11.32 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (75778) - No such process - unlinked 0 (time 1785402342 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6162) error: No such file or directory total: 6162 unlinks in 84 seconds: 73.357140 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2