== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:33:27 (1771968807) Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:-f) oleg341-server: oleg341-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg341-server: detected 4 online CPUs by sysfs oleg341-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg341-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg341-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg341-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock oleg341-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1745 (time 1771968872.29 total 10.00 last 174.47) - open/close 4022 (time 1771968882.29 total 20.00 last 227.69) - open/close 6865 (time 1771968892.29 total 30.00 last 284.25) - open/close 10000 (time 1771968901.73 total 39.44 last 332.14) - open/close 13195 (time 1771968911.73 total 49.44 last 319.47) - open/close 15434 (time 1771968921.74 total 59.44 last 223.86) - open/close 16984 (time 1771968931.74 total 69.45 last 154.98) - open/close 18816 (time 1771968941.74 total 79.45 last 183.17) - open/close 20000 (time 1771968945.97 total 83.68 last 279.48) - open/close 22865 (time 1771968955.98 total 93.69 last 286.47) - open/close 25455 (time 1771968965.98 total 103.69 last 258.93) - open/close 28007 (time 1771968975.98 total 113.69 last 255.19) - open/close 30000 (time 1771968982.77 total 120.47 last 293.67) - open/close 32587 (time 1771968992.77 total 130.48 last 258.60) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 130.53 seconds: 249.71 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5791 (time 1771969004.52 total 10.00 last 578.91) - /unlink 10000 (time 1771969012.43 total 17.91 last 532.16) - /unlink 16181 (time 1771969022.43 total 27.91 last 618.09) - /unlink 20000 (time 1771969028.02 total 33.51 last 682.39) - /unlink 26634 (time 1771969038.03 total 43.51 last 663.34) - /unlink 30000 (time 1771969043.64 total 49.13 last 599.26) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 52.99 seconds: 615.15 ops/second Waiting for MDT destroys to complete - open/close 2436 (time 1771969068.43 total 10.00 last 243.55) - open/close 5966 (time 1771969078.44 total 20.01 last 352.76) - open/close 8686 (time 1771969088.44 total 30.01 last 272.00) - open/close 10000 (time 1771969092.59 total 34.16 last 316.64) - open/close 13220 (time 1771969102.59 total 44.16 last 321.93) - open/close 15860 (time 1771969112.60 total 54.17 last 263.87) - open/close 18441 (time 1771969122.60 total 64.17 last 258.04) - open/close 20000 (time 1771969129.39 total 70.96 last 229.50) - open/close 22890 (time 1771969139.39 total 80.96 last 288.93) - open/close 25180 (time 1771969149.40 total 90.96 last 228.98) - open/close 27893 (time 1771969159.40 total 100.96 last 271.29) - open/close 30000 (time 1771969167.89 total 109.46 last 248.13) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 117.82 seconds: 276.64 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6587 (time 1771969187.81 total 10.00 last 658.61) - /unlink 10000 (time 1771969192.48 total 14.67 last 731.76) - /unlink 17093 (time 1771969202.48 total 24.67 last 709.27) - /unlink 20000 (time 1771969206.37 total 28.55 last 747.67) - /unlink 27538 (time 1771969216.37 total 38.55 last 753.73) - /unlink 30000 (time 1771969219.90 total 42.08 last 697.74) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 45.34 seconds: 718.87 ops/second Waiting for MDT destroys to complete - open/close 3601 (time 1771969246.92 total 10.00 last 360.08) - open/close 6354 (time 1771969256.92 total 20.00 last 275.22) - open/close 9232 (time 1771969266.92 total 30.00 last 287.77) - open/close 10000 (time 1771969268.96 total 32.04 last 376.51) - open/close 13249 (time 1771969278.96 total 42.05 last 324.84) - open/close 16117 (time 1771969288.96 total 52.05 last 286.77) - open/close 19797 (time 1771969298.97 total 62.05 last 367.86) - open/close 23336 (time 1771969308.97 total 72.05 last 353.85) - open/close 26445 (time 1771969318.97 total 82.06 last 310.73) - open/close 29563 (time 1771969328.97 total 92.06 last 311.80) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 99.11 seconds: 328.88 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7152 (time 1771969347.38 total 10.00 last 715.13) - /unlink 10000 (time 1771969352.19 total 14.81 last 592.23) - /unlink 17033 (time 1771969362.19 total 24.81 last 703.27) - /unlink 20000 (time 1771969365.99 total 28.61 last 780.80) - /unlink 27017 (time 1771969375.99 total 38.61 last 701.64) - /unlink 30000 (time 1771969380.06 total 42.68 last 732.67) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 46.75 seconds: 697.13 ops/second Waiting for MDT destroys to complete - open/close 4581 (time 1771969424.94 total 10.00 last 457.99) - open/close 8236 (time 1771969434.94 total 20.00 last 365.41) - open/close 10000 (time 1771969439.67 total 24.73 last 372.96) - open/close 13983 (time 1771969449.67 total 34.74 last 398.25) - open/close 17709 (time 1771969459.67 total 44.74 last 372.58) - open/close 20000 (time 1771969464.65 total 49.71 last 460.18) - open/close 24718 (time 1771969474.65 total 59.72 last 471.74) - open/close 29141 (time 1771969484.65 total 69.72 last 442.21) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 76.33 seconds: 427.03 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8066 (time 1771969502.79 total 10.00 last 806.49) - /unlink 10000 (time 1771969505.29 total 12.51 last 771.78) - /unlink 18542 (time 1771969515.29 total 22.51 last 854.09) - /unlink 27330 (time 1771969525.29 total 32.51 last 878.79) - /unlink 30000 (time 1771969528.16 total 35.38 last 930.62) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 37.89 seconds: 860.12 ops/second Waiting for MDT destroys to complete - open/close 5137 (time 1771969590.15 total 10.00 last 513.68) - open/close 10000 (time 1771969599.06 total 18.91 last 546.12) - open/close 14863 (time 1771969609.06 total 28.91 last 486.24) - open/close 19730 (time 1771969619.06 total 38.91 last 486.64) - open/close 25142 (time 1771969629.06 total 48.91 last 541.13) - open/close 30000 (time 1771969638.80 total 58.65 last 498.74) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 63.54 seconds: 512.98 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9966 (time 1771969654.94 total 10.00 last 996.56) - /unlink 19446 (time 1771969664.94 total 20.00 last 947.98) - /unlink 28801 (time 1771969674.94 total 30.00 last 935.47) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 33.59 seconds: 970.21 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg341-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg341-server: LNET ready to unload oleg341-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2