== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:37:12 (1777912632) start mds service on oleg333-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg333-server: detected 4 online CPUs by sysfs oleg333-server: MODOPTS_LIBCFS= oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 575 (time 1777912688.53 total 10.51 last 54.68) - open/close 1023 (time 1777912698.70 total 20.69 last 44.03) - open/close 1411 (time 1777912708.75 total 30.74 last 38.61) - open/close 1891 (time 1777912719.12 total 41.11 last 46.30) - open/close 2328 (time 1777912729.30 total 51.28 last 42.94) - open/close 2779 (time 1777912739.54 total 61.52 last 44.05) - open/close 3372 (time 1777912750.03 total 72.02 last 56.50) - open/close 3920 (time 1777912760.66 total 82.64 last 51.57) - open/close 4460 (time 1777912771.15 total 93.14 last 51.45) - open/close 5149 (time 1777912781.84 total 103.83 last 64.45) - open/close 5791 (time 1777912792.59 total 114.58 last 59.71) - open/close 6312 (time 1777912802.90 total 124.88 last 50.57) - open/close 6758 (time 1777912813.14 total 135.13 last 43.54) - open/close 7336 (time 1777912823.76 total 145.75 last 54.42) - open/close 7894 (time 1777912834.39 total 156.37 last 52.51) - open/close 8352 (time 1777912844.82 total 166.80 last 43.91) - open/close 8908 (time 1777912855.44 total 177.43 last 52.34) - open/close 9366 (time 1777912865.75 total 187.73 last 44.44) - open/close 9942 (time 1777912876.18 total 198.16 last 55.22) - open/close 10571 (time 1777912886.22 total 208.21 last 62.61) - open/close 11184 (time 1777912896.78 total 218.77 last 58.05) - open/close 11871 (time 1777912907.67 total 229.65 last 63.12) - open/close 12476 (time 1777912918.35 total 240.34 last 56.61) - open/close 13077 (time 1777912929.04 total 251.03 last 56.23) - open/close 13790 (time 1777912939.04 total 261.03 last 71.29) - open/close 14417 (time 1777912949.97 total 271.96 last 57.38) - open/close 15064 (time 1777912960.72 total 282.71 last 60.18) - open/close 15739 (time 1777912971.54 total 293.53 last 62.40) - open/close 20000 (time 1777912980.80 total 302.79 last 460.04) - open/close 24942 (time 1777912990.80 total 312.79 last 494.19) - open/close 29837 (time 1777913000.80 total 322.79 last 489.37) - open/close 34255 (time 1777913010.81 total 332.79 last 441.76) - open/close 38674 (time 1777913020.81 total 342.79 last 441.84) - open/close 40000 (time 1777913023.84 total 345.82 last 437.89) - open/close 44409 (time 1777913033.84 total 355.83 last 440.66) - open/close 48723 (time 1777913043.84 total 365.83 last 431.33) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 368.23 seconds: 134.79 ops/second - unlinked 0 (time 1777913047 ; total 0 ; last 0) - unlinked 10000 (time 1777913059 ; total 12 ; last 12) - unlinked 20000 (time 1777913072 ; total 25 ; last 13) - unlinked 30000 (time 1777913084 ; total 37 ; last 12) - unlinked 40000 (time 1777913095 ; total 48 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 58 seconds: 855.724121 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server unloading modules via unload_modules_local on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local oleg333-server: modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2