== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:26:48 (1753367208) start mds service on oleg218-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg218-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 53 (time 1753367275.05 total 10.09 last 5.26) - open/close 106 (time 1753367285.05 total 20.09 last 5.30) - open/close 392 (time 1753367295.72 total 30.76 last 26.81) - open/close 511 (time 1753367306.21 total 41.25 last 11.35) - open/close 816 (time 1753367316.64 total 51.68 last 29.24) - open/close 1126 (time 1753367327.01 total 62.05 last 29.90) - open/close 1437 (time 1753367337.31 total 72.35 last 30.20) - open/close 1733 (time 1753367348.34 total 83.37 last 26.84) - open/close 1887 (time 1753367358.35 total 93.38 last 15.39) - open/close 2058 (time 1753367369.05 total 104.09 last 15.97) - open/close 2245 (time 1753367380.52 total 115.55 last 16.31) - open/close 2420 (time 1753367390.56 total 125.60 last 17.42) - open/close 2499 (time 1753367400.92 total 135.96 last 7.62) - open/close 2742 (time 1753367411.17 total 146.21 last 23.71) - open/close 2984 (time 1753367421.34 total 156.38 last 23.79) - open/close 3264 (time 1753367432.04 total 167.08 last 26.18) - open/close 3558 (time 1753367442.05 total 177.08 last 29.38) - open/close 3842 (time 1753367452.19 total 187.23 last 27.99) - open/close 3988 (time 1753367462.19 total 197.23 last 14.60) - open/close 4135 (time 1753367472.29 total 207.33 last 14.56) - open/close 4233 (time 1753367482.86 total 217.89 last 9.28) - open/close 4298 (time 1753367493.03 total 228.06 last 6.39) - open/close 4387 (time 1753367503.45 total 238.49 last 8.54) - open/close 4573 (time 1753367513.46 total 248.50 last 18.59) - open/close 4863 (time 1753367523.47 total 258.50 last 28.98) - open/close 5122 (time 1753367533.73 total 268.76 last 25.24) - open/close 5381 (time 1753367543.73 total 278.77 last 25.88) - open/close 5647 (time 1753367553.96 total 288.99 last 26.02) - open/close 5940 (time 1753367564.45 total 299.48 last 27.93) - open/close 6249 (time 1753367575.20 total 310.24 last 28.74) - open/close 6540 (time 1753367585.43 total 320.47 last 28.44) - open/close 6823 (time 1753367596.19 total 331.23 last 26.32) - open/close 7013 (time 1753367606.55 total 341.59 last 18.33) - open/close 7289 (time 1753367617.25 total 352.28 last 25.81) - open/close 7544 (time 1753367627.80 total 362.84 last 24.16) - open/close 7882 (time 1753367638.49 total 373.52 last 31.63) - open/close 8165 (time 1753367648.80 total 383.83 last 27.45) - open/close 8657 (time 1753367659.80 total 394.84 last 44.71) - open/close 8794 (time 1753367670.56 total 405.59 last 12.74) - open/close 8896 (time 1753367681.43 total 416.47 last 9.38) - open/close 9236 (time 1753367692.25 total 427.29 last 31.42) - open/close 9342 (time 1753367705.18 total 440.21 last 8.20) - open/close 9986 (time 1753367715.19 total 450.22 last 64.34) - open/close 10782 (time 1753367725.19 total 460.23 last 79.55) - open/close 11596 (time 1753367735.19 total 470.23 last 81.40) - open/close 12319 (time 1753367745.19 total 480.23 last 72.29) - open/close 13199 (time 1753367755.20 total 490.24 last 87.96) - open/close 14166 (time 1753367765.20 total 500.24 last 96.66) - open/close 15386 (time 1753367775.21 total 510.24 last 121.93) - open/close 16776 (time 1753367785.21 total 520.25 last 138.95) - open/close 18550 (time 1753367795.22 total 530.26 last 177.28) - open/close 19861 (time 1753367805.22 total 540.26 last 131.05) - open/close 21047 (time 1753367815.23 total 550.26 last 118.56) - open/close 22587 (time 1753367825.23 total 560.26 last 154.00) - open/close 24271 (time 1753367835.23 total 570.27 last 168.34) - open/close 25826 (time 1753367845.23 total 580.27 last 155.50) - open/close 27251 (time 1753367855.23 total 590.27 last 142.46) - open/close 28885 (time 1753367865.23 total 600.27 last 163.38) - open/close 30000 (time 1753367871.96 total 607.00 last 165.68) - open/close 31331 (time 1753367881.97 total 617.00 last 133.05) - open/close 32668 (time 1753367891.97 total 627.01 last 133.64) - open/close 34332 (time 1753367901.97 total 637.01 last 166.39) - open/close 35839 (time 1753367911.97 total 647.01 last 150.67) - open/close 37750 (time 1753367921.98 total 657.02 last 190.98) - open/close 39231 (time 1753367931.99 total 667.02 last 148.00) - open/close 40000 (time 1753367935.92 total 670.96 last 195.32) - open/close 42048 (time 1753367945.93 total 680.96 last 204.75) - open/close 44153 (time 1753367955.93 total 690.97 last 210.46) - open/close 46166 (time 1753367965.93 total 700.97 last 201.28) - open/close 48266 (time 1753367975.93 total 710.97 last 209.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 716.43 seconds: 69.28 ops/second - unlinked 0 (time 1753367983 ; total 1 ; last 1) - unlinked 10000 (time 1753368010 ; total 28 ; last 27) - unlinked 20000 (time 1753368029 ; total 47 ; last 19) - unlinked 30000 (time 1753368052 ; total 70 ; last 23) - unlinked 40000 (time 1753368074 ; total 92 ; last 22) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 115 seconds: 431.582611 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2