== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:39:37 (1762493977) start mds service on oleg403-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1762494072.54 total 10.00 last 20.19) - open/close 406 (time 1762494083.28 total 20.74 last 18.99) - open/close 554 (time 1762494093.90 total 31.37 last 13.93) - open/close 758 (time 1762494104.28 total 41.74 last 19.67) - open/close 987 (time 1762494114.58 total 52.05 last 22.22) - open/close 1118 (time 1762494124.94 total 62.40 last 12.65) - open/close 1330 (time 1762494135.31 total 72.77 last 20.45) - open/close 1516 (time 1762494145.55 total 83.02 last 18.15) - open/close 1686 (time 1762494156.58 total 94.04 last 15.42) - open/close 1941 (time 1762494166.59 total 104.05 last 25.48) - open/close 2167 (time 1762494176.81 total 114.27 last 22.11) - open/close 2397 (time 1762494187.08 total 124.54 last 22.39) - open/close 2611 (time 1762494197.10 total 134.56 last 21.37) - open/close 2848 (time 1762494207.57 total 145.03 last 22.62) - open/close 3070 (time 1762494217.71 total 155.17 last 21.90) - open/close 3234 (time 1762494227.99 total 165.45 last 15.95) - open/close 3400 (time 1762494238.63 total 176.09 last 15.60) - open/close 3504 (time 1762494248.73 total 186.19 last 10.29) - open/close 3727 (time 1762494259.42 total 196.88 last 20.86) - open/close 3934 (time 1762494269.43 total 206.89 last 20.69) - open/close 4153 (time 1762494279.52 total 216.98 last 21.70) - open/close 4405 (time 1762494292.30 total 229.77 last 19.71) - open/close 4638 (time 1762494302.60 total 240.07 last 22.62) - open/close 4895 (time 1762494313.49 total 250.95 last 23.62) - open/close 5117 (time 1762494323.64 total 261.10 last 21.87) - open/close 5358 (time 1762494334.62 total 272.08 last 21.95) - open/close 5594 (time 1762494344.64 total 282.10 last 23.56) - open/close 5792 (time 1762494354.64 total 292.11 last 19.79) - open/close 5993 (time 1762494364.66 total 302.12 last 20.07) - open/close 6276 (time 1762494377.10 total 314.56 last 22.75) - open/close 6500 (time 1762494387.31 total 324.77 last 21.94) - open/close 6674 (time 1762494397.85 total 335.31 last 16.51) - open/close 6896 (time 1762494408.28 total 345.74 last 21.29) - open/close 7173 (time 1762494418.30 total 355.76 last 27.64) - open/close 7388 (time 1762494429.27 total 366.73 last 19.60) - open/close 7591 (time 1762494439.28 total 376.74 last 20.28) - open/close 7709 (time 1762494449.93 total 387.40 last 11.07) - open/close 7947 (time 1762494459.94 total 397.40 last 23.79) - open/close 8161 (time 1762494470.11 total 407.57 last 21.04) - open/close 8352 (time 1762494480.11 total 417.58 last 19.09) - open/close 8518 (time 1762494490.83 total 428.30 last 15.48) - open/close 8809 (time 1762494500.85 total 438.32 last 29.04) - open/close 8990 (time 1762494511.84 total 449.31 last 16.47) - open/close 9164 (time 1762494522.71 total 460.18 last 16.00) - open/close 9343 (time 1762494533.59 total 471.05 last 16.46) - open/close 9512 (time 1762494543.65 total 481.11 last 16.81) - open/close 9617 (time 1762494554.71 total 492.17 last 9.49) - open/close 9800 (time 1762494565.58 total 503.04 last 16.83) - open/close 10000 (time 1762494573.51 total 510.97 last 25.24) - open/close 10194 (time 1762494583.51 total 520.97 last 19.39) - open/close 10406 (time 1762494594.44 total 531.90 last 19.39) - open/close 10623 (time 1762494605.19 total 542.65 last 20.19) - open/close 10864 (time 1762494615.20 total 552.66 last 24.07) - open/close 11126 (time 1762494625.20 total 562.67 last 26.19) - open/close 11336 (time 1762494637.65 total 575.12 last 16.87) - open/close 11547 (time 1762494648.09 total 585.55 last 20.22) - open/close 11765 (time 1762494658.51 total 595.97 last 20.92) - open/close 11929 (time 1762494668.64 total 606.10 last 16.19) - open/close 12160 (time 1762494679.00 total 616.46 last 22.29) - open/close 12339 (time 1762494690.64 total 628.10 last 15.38) - open/close 12565 (time 1762494701.32 total 638.78 last 21.16) - open/close 12800 (time 1762494711.95 total 649.42 last 22.09) - open/close 13072 (time 1762494723.02 total 660.48 last 24.59) - open/close 13234 (time 1762494733.03 total 670.49 last 16.18) - open/close 13443 (time 1762494743.65 total 681.11 last 19.68) - open/close 13671 (time 1762494753.66 total 691.13 last 22.76) - open/close 13866 (time 1762494764.33 total 701.79 last 18.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13905) error: No space left on device total: 13905 open/close in 702.56 seconds: 19.79 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8861: kill: (111826) - No such process - unlinked 0 (time 1762494773 ; total 0 ; last 0) - unlinked 10000 (time 1762494864 ; total 91 ; last 91) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13905) error: No such file or directory total: 13905 unlinks in 119 seconds: 116.848740 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2