== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:41:57 (1734259317) Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 oleg424-server: oleg424-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg424-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg424-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg424-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5779 (time 1734259370.61 total 10.00 last 577.80) - open/close 10000 (time 1734259379.61 total 19.00 last 468.89) - open/close 14110 (time 1734259389.61 total 29.00 last 410.98) - open/close 18596 (time 1734259399.61 total 39.00 last 448.58) - open/close 20000 (time 1734259402.87 total 42.26 last 431.28) - open/close 24440 (time 1734259412.87 total 52.26 last 443.93) - open/close 28838 (time 1734259422.87 total 62.26 last 439.78) - open/close 30000 (time 1734259425.47 total 64.86 last 447.08) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 70.23 seconds: 463.63 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8625 (time 1734259441.76 total 10.00 last 862.48) - /unlink 16840 (time 1734259451.76 total 20.00 last 821.42) - /unlink 20000 (time 1734259455.08 total 23.32 last 952.41) - /unlink 28600 (time 1734259465.08 total 33.32 last 860.00) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 38.13 seconds: 853.86 ops/second Waiting for MDT destroys to complete oleg424-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg424-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 - open/close 6028 (time 1734259508.19 total 10.00 last 602.77) - open/close 10000 (time 1734259514.89 total 16.70 last 592.69) - open/close 16009 (time 1734259524.89 total 26.70 last 600.86) - open/close 20000 (time 1734259532.12 total 33.93 last 551.95) - open/close 26089 (time 1734259542.12 total 43.93 last 608.83) - open/close 30000 (time 1734259548.31 total 50.13 last 631.44) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 54.51 seconds: 597.35 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259562.57 total 9.06 last 1103.49) - /unlink 20000 (time 1734259571.21 total 17.70 last 1157.85) - /unlink 30000 (time 1734259580.06 total 26.55 last 1129.70) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 28.91 seconds: 1126.19 ops/second Waiting for MDT destroys to complete oleg424-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg424-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 - open/close 5158 (time 1734259623.66 total 10.00 last 515.72) - open/close 10000 (time 1734259632.87 total 19.22 last 525.52) - open/close 15850 (time 1734259642.87 total 29.22 last 584.97) - open/close 20000 (time 1734259650.54 total 36.88 last 541.39) - open/close 25279 (time 1734259660.54 total 46.88 last 527.88) - open/close 30000 (time 1734259669.54 total 55.88 last 524.74) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 60.85 seconds: 535.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8425 (time 1734259685.42 total 10.00 last 842.43) - /unlink 17584 (time 1734259695.43 total 20.00 last 915.77) - /unlink 20000 (time 1734259697.94 total 22.51 last 962.15) - /unlink 30000 (time 1734259707.89 total 32.47 last 1004.71) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 34.87 seconds: 933.85 ops/second Waiting for MDT destroys to complete oleg424-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg424-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 - open/close 6502 (time 1734259750.91 total 10.00 last 650.15) - open/close 10000 (time 1734259756.51 total 15.60 last 624.62) - open/close 15451 (time 1734259766.51 total 25.60 last 545.10) - open/close 20000 (time 1734259774.85 total 33.95 last 544.98) - open/close 25092 (time 1734259784.86 total 43.95 last 509.19) - open/close 30000 (time 1734259793.96 total 53.06 last 538.88) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 57.51 seconds: 566.20 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259809.03 total 9.77 last 1023.76) - /unlink 19914 (time 1734259819.03 total 19.77 last 991.34) - /unlink 29944 (time 1734259829.03 total 29.77 last 1002.92) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.36 seconds: 1006.21 ops/second Waiting for MDT destroys to complete oleg424-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg424-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 - open/close 5607 (time 1734259868.92 total 10.00 last 560.63) - open/close 10000 (time 1734259876.97 total 18.06 last 545.18) - open/close 15693 (time 1734259886.98 total 28.06 last 569.29) - open/close 20000 (time 1734259894.81 total 35.90 last 549.37) - open/close 25658 (time 1734259904.82 total 45.90 last 565.73) - open/close 30000 (time 1734259912.48 total 53.56 last 566.82) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 58.18 seconds: 559.67 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9756 (time 1734259927.97 total 10.00 last 975.54) - /unlink 19779 (time 1734259937.97 total 20.00 last 1002.30) - /unlink 30000 (time 1734259947.83 total 29.86 last 1036.51) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.31 seconds: 1007.85 ops/second Waiting for MDT destroys to complete oleg424-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg424-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2