== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:03:03 (1787828583) Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg214-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server oleg214-server: oleg214-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg214-server: detected 4 online CPUs by sysfs oleg214-server: MODOPTS_LIBCFS= oleg214-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg214-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg214-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 659 (time 1787828816.54 total 10.01 last 65.84) - open/close 1289 (time 1787828826.57 total 20.04 last 62.81) - open/close 1834 (time 1787828836.59 total 30.05 last 54.43) - open/close 2452 (time 1787828846.59 total 40.06 last 61.75) - open/close 3202 (time 1787828856.61 total 50.08 last 74.84) - open/close 3888 (time 1787828866.62 total 60.09 last 68.54) total: 4500 open/close in 69.53 seconds: 64.72 ops/second - /unlink 1269 (time 1787828891.61 total 10.00 last 126.89) - /unlink 2477 (time 1787828901.61 total 20.01 last 120.71) - /unlink 3751 (time 1787828911.62 total 30.01 last 127.37) total: 4500 /unlink in 36.20 seconds: 124.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 730 (time 1787828933.50 total 10.01 last 72.94) - open/close 1449 (time 1787828943.50 total 20.01 last 71.87) - open/close 1988 (time 1787828953.50 total 30.02 last 53.88) - open/close 2570 (time 1787828963.52 total 40.03 last 58.12) - open/close 3001 (time 1787828973.53 total 50.05 last 43.03) - open/close 3542 (time 1787828983.54 total 60.05 last 54.06) - open/close 4159 (time 1787828993.56 total 70.07 last 61.61) total: 4500 open/close in 76.77 seconds: 58.62 ops/second - /unlink 1045 (time 1787829015.29 total 10.01 last 104.44) - /unlink 2289 (time 1787829025.29 total 20.01 last 124.37) - /unlink 3481 (time 1787829035.29 total 30.01 last 119.18) total: 4500 /unlink in 38.18 seconds: 117.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 679 (time 1787829059.84 total 10.01 last 67.82) - open/close 1435 (time 1787829069.84 total 20.02 last 75.58) - open/close 2171 (time 1787829079.86 total 30.03 last 73.47) - open/close 2893 (time 1787829089.87 total 40.04 last 72.16) - open/close 3582 (time 1787829099.87 total 50.05 last 68.85) - open/close 4290 (time 1787829109.88 total 60.05 last 70.76) total: 4500 open/close in 62.80 seconds: 71.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1469 (time 1787829127.17 total 10.00 last 146.89) - /unlink 3001 (time 1787829137.18 total 20.01 last 153.10) - /unlink 4384 (time 1787829147.19 total 30.01 last 138.22) total: 4500 /unlink in 31.07 seconds: 144.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 845 (time 1787829163.46 total 10.05 last 84.08) - open/close 1571 (time 1787829173.47 total 20.06 last 72.55) - open/close 2283 (time 1787829183.48 total 30.07 last 71.13) - open/close 2963 (time 1787829193.48 total 40.07 last 67.96) - open/close 3603 (time 1787829203.49 total 50.08 last 63.97) - open/close 4236 (time 1787829213.49 total 60.08 last 63.27) total: 4500 open/close in 63.23 seconds: 71.16 ops/second - /unlink 1285 (time 1787829231.27 total 10.01 last 128.35) - /unlink 2560 (time 1787829241.27 total 20.01 last 127.48) - /unlink 4029 (time 1787829251.28 total 30.02 last 146.84) total: 4500 /unlink in 34.03 seconds: 132.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 518 (time 1787829272.02 total 10.04 last 51.60) - open/close 1171 (time 1787829282.02 total 20.04 last 65.30) - open/close 1868 (time 1787829292.03 total 30.04 last 69.66) - open/close 2575 (time 1787829302.03 total 40.05 last 70.69) - open/close 3213 (time 1787829312.03 total 50.05 last 63.77) - open/close 3901 (time 1787829322.05 total 60.06 last 68.70) - open/close 4463 (time 1787829332.06 total 70.08 last 56.13) total: 4500 open/close in 70.51 seconds: 63.82 ops/second - /unlink 1298 (time 1787829349.65 total 10.01 last 129.73) - /unlink 2592 (time 1787829359.66 total 20.01 last 129.29) - /unlink 4008 (time 1787829369.66 total 30.02 last 141.54) total: 4500 /unlink in 34.04 seconds: 132.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 668 (time 1787829389.38 total 10.01 last 66.73) - open/close 1373 (time 1787829399.39 total 20.02 last 70.45) - open/close 2092 (time 1787829409.40 total 30.03 last 71.81) - open/close 2761 (time 1787829419.40 total 40.03 last 66.88) - open/close 3456 (time 1787829429.40 total 50.04 last 69.48) - open/close 4037 (time 1787829439.44 total 60.08 last 57.86) total: 4500 open/close in 68.62 seconds: 65.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1232 (time 1787829462.61 total 10.00 last 123.17) - /unlink 2734 (time 1787829472.61 total 20.00 last 150.20) - /unlink 4248 (time 1787829482.62 total 30.01 last 151.28) total: 4500 /unlink in 32.32 seconds: 139.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 619 (time 1787829500.53 total 10.01 last 61.86) - open/close 1199 (time 1787829510.54 total 20.01 last 57.96) - open/close 1833 (time 1787829520.54 total 30.02 last 63.34) - open/close 2491 (time 1787829530.56 total 40.03 last 65.73) - open/close 3188 (time 1787829540.56 total 50.04 last 69.65) - open/close 3893 (time 1787829550.56 total 60.04 last 70.50) total: 4500 open/close in 67.86 seconds: 66.31 ops/second - /unlink 1398 (time 1787829572.37 total 10.00 last 139.75) - /unlink 2669 (time 1787829582.37 total 20.01 last 127.08) - /unlink 4000 (time 1787829592.37 total 30.01 last 133.05) total: 4500 /unlink in 33.61 seconds: 133.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 730 (time 1787829611.50 total 10.00 last 72.97) - open/close 1446 (time 1787829621.51 total 20.01 last 71.55) - open/close 2112 (time 1787829631.51 total 30.01 last 66.59) - open/close 2766 (time 1787829641.52 total 40.02 last 65.34) - open/close 3462 (time 1787829651.53 total 50.04 last 69.50) - open/close 4037 (time 1787829661.54 total 60.04 last 57.48) total: 4500 open/close in 66.75 seconds: 67.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1075 (time 1787829683.75 total 10.01 last 107.41) - /unlink 2211 (time 1787829693.75 total 20.01 last 113.56) - /unlink 3414 (time 1787829703.76 total 30.02 last 120.19) total: 4500 /unlink in 38.32 seconds: 117.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 666 (time 1787829727.43 total 10.01 last 66.56) - open/close 1360 (time 1787829737.44 total 20.02 last 69.31) - open/close 1893 (time 1787829747.48 total 30.06 last 53.07) - open/close 2414 (time 1787829757.49 total 40.07 last 52.07) - open/close 3006 (time 1787829767.49 total 50.07 last 59.19) - open/close 3711 (time 1787829777.49 total 60.07 last 70.49) - open/close 4305 (time 1787829787.49 total 70.07 last 59.40) total: 4500 open/close in 73.36 seconds: 61.34 ops/second - /unlink 1432 (time 1787829805.85 total 10.00 last 143.17) - /unlink 2754 (time 1787829815.85 total 20.01 last 132.15) - /unlink 4151 (time 1787829825.85 total 30.01 last 139.67) total: 4500 /unlink in 32.85 seconds: 136.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 649 (time 1787829843.11 total 10.01 last 64.85) - open/close 1294 (time 1787829853.11 total 20.01 last 64.49) - open/close 2016 (time 1787829863.12 total 30.02 last 72.15) - open/close 2660 (time 1787829873.12 total 40.02 last 64.34) - open/close 3235 (time 1787829883.13 total 50.03 last 57.44) - open/close 3918 (time 1787829893.14 total 60.04 last 68.26) total: 4500 open/close in 68.60 seconds: 65.60 ops/second - /unlink 1284 (time 1787829916.22 total 10.01 last 128.33) - /unlink 2589 (time 1787829926.23 total 20.02 last 130.34) - /unlink 3895 (time 1787829936.24 total 30.02 last 130.52) total: 4500 /unlink in 35.47 seconds: 126.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 735 (time 1787829957.18 total 10.01 last 73.40) - open/close 1391 (time 1787829967.19 total 20.02 last 65.55) - open/close 2044 (time 1787829977.19 total 30.02 last 65.29) - open/close 2777 (time 1787829987.20 total 40.04 last 73.21) - open/close 3556 (time 1787829997.21 total 50.04 last 77.85) - open/close 4153 (time 1787830007.21 total 60.05 last 59.67) total: 4500 open/close in 64.89 seconds: 69.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1171 (time 1787830026.63 total 10.00 last 117.05) - /unlink 2373 (time 1787830036.63 total 20.01 last 120.14) - /unlink 3705 (time 1787830046.63 total 30.01 last 133.19) total: 4500 /unlink in 36.15 seconds: 124.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 620 (time 1787830068.33 total 10.01 last 61.93) - open/close 1222 (time 1787830078.34 total 20.02 last 60.16) - open/close 1730 (time 1787830088.39 total 30.07 last 50.52) - open/close 2096 (time 1787830098.40 total 40.08 last 36.58) - open/close 2402 (time 1787830108.43 total 50.11 last 30.50) - open/close 2875 (time 1787830118.44 total 60.12 last 47.26) - open/close 3374 (time 1787830128.45 total 70.14 last 49.82) - open/close 4027 (time 1787830138.46 total 80.14 last 65.26) total: 4500 open/close in 86.42 seconds: 52.07 ops/second - /unlink 1284 (time 1787830160.14 total 10.00 last 128.38) - /unlink 2578 (time 1787830170.14 total 20.01 last 129.31) - /unlink 3776 (time 1787830180.15 total 30.02 last 119.66) total: 4500 /unlink in 36.44 seconds: 123.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 611 (time 1787830202.29 total 10.00 last 61.08) - open/close 1317 (time 1787830212.29 total 20.01 last 70.58) - open/close 1880 (time 1787830222.30 total 30.01 last 56.26) - open/close 2584 (time 1787830232.30 total 40.02 last 70.35) - open/close 3235 (time 1787830242.31 total 50.03 last 65.05) - open/close 3950 (time 1787830252.31 total 60.03 last 71.49) total: 4500 open/close in 67.80 seconds: 66.37 ops/second - /unlink 1094 (time 1787830275.29 total 10.02 last 109.21) - /unlink 2360 (time 1787830285.30 total 20.02 last 126.52) - /unlink 3607 (time 1787830295.30 total 30.02 last 124.69) total: 4500 /unlink in 37.15 seconds: 121.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 761 (time 1787830317.39 total 10.01 last 76.05) - open/close 1397 (time 1787830327.41 total 20.02 last 63.52) - open/close 2027 (time 1787830337.42 total 30.03 last 62.93) - open/close 2720 (time 1787830347.42 total 40.03 last 69.29) - open/close 3455 (time 1787830357.44 total 50.05 last 73.38) - open/close 4123 (time 1787830367.44 total 60.05 last 66.77) total: 4500 open/close in 66.04 seconds: 68.14 ops/second - /unlink 1618 (time 1787830388.37 total 10.00 last 161.73) - /unlink 2627 (time 1787830398.37 total 20.01 last 100.87) - /unlink 3702 (time 1787830408.38 total 30.01 last 107.44) total: 4500 /unlink in 35.65 seconds: 126.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 753 (time 1787830428.02 total 10.00 last 75.28) - open/close 1481 (time 1787830438.02 total 20.00 last 72.79) - open/close 2118 (time 1787830448.03 total 30.01 last 63.66) - open/close 2898 (time 1787830458.03 total 40.02 last 77.94) - open/close 3656 (time 1787830468.04 total 50.03 last 75.74) - open/close 4424 (time 1787830478.05 total 60.03 last 76.74) total: 4500 open/close in 61.01 seconds: 73.76 ops/second - /unlink 1471 (time 1787830493.47 total 10.00 last 147.05) - /unlink 2775 (time 1787830503.48 total 20.01 last 130.28) - /unlink 4136 (time 1787830513.49 total 30.02 last 135.97) total: 4500 /unlink in 32.53 seconds: 138.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 802 (time 1787830530.17 total 10.00 last 80.18) - open/close 1590 (time 1787830540.18 total 20.01 last 78.77) - open/close 2346 (time 1787830550.18 total 30.01 last 75.59) - open/close 3042 (time 1787830560.23 total 40.06 last 69.26) - open/close 3712 (time 1787830570.26 total 50.09 last 66.79) - open/close 4395 (time 1787830580.27 total 60.10 last 68.24) total: 4500 open/close in 61.22 seconds: 73.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1492 (time 1787830597.10 total 10.00 last 149.16) - /unlink 2973 (time 1787830607.10 total 20.00 last 148.07) - /unlink 4150 (time 1787830617.11 total 30.01 last 117.64) total: 4500 /unlink in 33.61 seconds: 133.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 607 (time 1787830636.68 total 10.01 last 60.67) - open/close 1311 (time 1787830646.68 total 20.01 last 70.36) - open/close 2035 (time 1787830656.69 total 30.02 last 72.36) - open/close 2788 (time 1787830666.69 total 40.02 last 75.29) - open/close 3573 (time 1787830676.69 total 50.02 last 78.48) - open/close 4201 (time 1787830686.70 total 60.02 last 62.77) total: 4500 open/close in 64.75 seconds: 69.50 ops/second - /unlink 1138 (time 1787830707.19 total 10.00 last 113.79) - /unlink 2378 (time 1787830717.20 total 20.00 last 123.95) - /unlink 3678 (time 1787830727.21 total 30.02 last 129.84) total: 4500 /unlink in 36.32 seconds: 123.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 769 (time 1787830748.73 total 10.00 last 76.89) - open/close 1444 (time 1787830758.74 total 20.01 last 67.44) - open/close 2110 (time 1787830768.75 total 30.01 last 66.57) - open/close 2807 (time 1787830778.75 total 40.02 last 69.65) - open/close 3551 (time 1787830788.76 total 50.03 last 74.33) - open/close 4129 (time 1787830798.77 total 60.04 last 57.73) total: 4500 open/close in 66.87 seconds: 67.29 ops/second - /unlink 1400 (time 1787830820.30 total 10.00 last 139.98) - /unlink 2743 (time 1787830830.30 total 20.00 last 134.26) - /unlink 4139 (time 1787830840.31 total 30.01 last 139.53) total: 4500 /unlink in 32.37 seconds: 139.00 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95541 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: modules unloaded.