== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:18:36 (1775168316) start mds service on oleg116-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg116-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 227 (time 1775168363.91 total 10.50 last 21.63) - open/close 573 (time 1775168374.80 total 21.38 last 31.79) - open/close 883 (time 1775168385.44 total 32.02 last 29.14) - open/close 1215 (time 1775168396.31 total 42.90 last 30.52) - open/close 1526 (time 1775168408.38 total 54.96 last 25.78) - open/close 1862 (time 1775168418.60 total 65.18 last 32.89) - open/close 2080 (time 1775168429.27 total 75.85 last 20.42) - open/close 2287 (time 1775168439.28 total 85.86 last 20.69) - open/close 2462 (time 1775168449.28 total 95.86 last 17.50) - open/close 2673 (time 1775168459.41 total 105.99 last 20.83) - open/close 2866 (time 1775168469.96 total 116.54 last 18.28) - open/close 3015 (time 1775168480.21 total 126.79 last 14.54) - open/close 3315 (time 1775168490.71 total 137.30 last 28.56) - open/close 3564 (time 1775168501.58 total 148.16 last 22.92) - open/close 3714 (time 1775168512.54 total 159.12 last 13.69) - open/close 3991 (time 1775168522.54 total 169.12 last 27.70) - open/close 4292 (time 1775168533.58 total 180.16 last 27.25) - open/close 4482 (time 1775168543.76 total 190.34 last 18.67) - open/close 4747 (time 1775168554.51 total 201.09 last 24.65) - open/close 4997 (time 1775168565.45 total 212.04 last 22.84) - open/close 5315 (time 1775168575.76 total 222.34 last 30.85) - open/close 5475 (time 1775168586.77 total 233.35 last 14.54) - open/close 5766 (time 1775168597.78 total 244.36 last 26.42) - open/close 5979 (time 1775168607.83 total 254.42 last 21.19) - open/close 6324 (time 1775168618.13 total 264.71 last 33.50) - open/close 6602 (time 1775168628.95 total 275.53 last 25.69) - open/close 6827 (time 1775168639.12 total 285.70 last 22.13) - open/close 7174 (time 1775168649.94 total 296.52 last 32.07) - open/close 7478 (time 1775168660.37 total 306.95 last 29.15) - open/close 7813 (time 1775168671.32 total 317.90 last 30.58) - open/close 8087 (time 1775168681.93 total 328.52 last 25.82) - open/close 8403 (time 1775168692.82 total 339.40 last 29.04) - open/close 8759 (time 1775168702.94 total 349.52 last 35.18) - open/close 8970 (time 1775168713.68 total 360.26 last 19.64) - open/close 9321 (time 1775168723.68 total 370.27 last 35.09) - open/close 9522 (time 1775168733.89 total 380.48 last 19.69) - open/close 9855 (time 1775168743.90 total 390.48 last 33.29) - open/close 10000 (time 1775168751.01 total 397.59 last 20.39) - open/close 10197 (time 1775168761.42 total 408.00 last 18.92) - open/close 10444 (time 1775168771.42 total 418.00 last 24.70) - open/close 10719 (time 1775168781.91 total 428.49 last 26.23) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-10865) error: No space left on device total: 10865 open/close in 432.55 seconds: 25.12 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8955: kill: (74202) - No such process - unlinked 0 (time 1775168791 ; total 0 ; last 0) - unlinked 10000 (time 1775168816 ; total 25 ; last 25) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-10865) error: No such file or directory total: 10865 unlinks in 26 seconds: 417.884613 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local oleg116-server: modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2