== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:40:37 (1781221237) Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg423-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg423-server oleg423-server: oleg423-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg423-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg423-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 785 (time 1781221424.54 total 10.00 last 78.48) - open/close 1612 (time 1781221434.54 total 20.01 last 82.67) - open/close 2330 (time 1781221444.55 total 30.01 last 71.76) - open/close 2903 (time 1781221454.55 total 40.02 last 57.27) - open/close 3449 (time 1781221464.56 total 50.02 last 54.56) - open/close 4093 (time 1781221474.57 total 60.03 last 64.33) total: 4500 open/close in 65.55 seconds: 68.65 ops/second - /unlink 1394 (time 1781221494.17 total 10.00 last 139.34) - /unlink 2869 (time 1781221504.17 total 20.01 last 147.48) - /unlink 4279 (time 1781221514.18 total 30.01 last 140.88) total: 4500 /unlink in 31.82 seconds: 141.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 719 (time 1781221530.32 total 10.01 last 71.82) - open/close 1469 (time 1781221540.34 total 20.03 last 74.83) - open/close 2252 (time 1781221550.36 total 30.05 last 78.21) - open/close 2976 (time 1781221560.36 total 40.05 last 72.38) - open/close 3629 (time 1781221570.36 total 50.05 last 65.29) - open/close 4278 (time 1781221580.37 total 60.06 last 64.85) total: 4500 open/close in 63.20 seconds: 71.21 ops/second - /unlink 1270 (time 1781221598.21 total 10.00 last 126.95) - /unlink 2563 (time 1781221608.21 total 20.01 last 129.24) - /unlink 3924 (time 1781221618.21 total 30.01 last 136.09) total: 4500 /unlink in 34.14 seconds: 131.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 824 (time 1781221636.35 total 10.01 last 82.32) - open/close 1543 (time 1781221646.35 total 20.01 last 71.87) - open/close 2322 (time 1781221656.35 total 30.01 last 77.90) - open/close 3147 (time 1781221666.36 total 40.02 last 82.42) - open/close 3969 (time 1781221676.39 total 50.05 last 81.99) total: 4500 open/close in 56.58 seconds: 79.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1581 (time 1781221696.47 total 10.01 last 158.00) - /unlink 3082 (time 1781221706.47 total 20.01 last 150.04) total: 4500 /unlink in 29.25 seconds: 153.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 833 (time 1781221730.09 total 10.01 last 83.19) - open/close 1612 (time 1781221740.10 total 20.02 last 77.82) - open/close 2430 (time 1781221750.10 total 30.03 last 81.77) - open/close 3187 (time 1781221760.11 total 40.04 last 75.61) - open/close 3704 (time 1781221770.12 total 50.05 last 51.63) - open/close 4414 (time 1781221780.15 total 60.08 last 70.79) total: 4500 open/close in 61.50 seconds: 73.18 ops/second - /unlink 1333 (time 1781221796.41 total 10.00 last 133.25) - /unlink 2539 (time 1781221806.41 total 20.01 last 120.58) - /unlink 4028 (time 1781221816.41 total 30.01 last 148.87) total: 4500 /unlink in 33.36 seconds: 134.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 732 (time 1781221834.77 total 10.01 last 73.14) - open/close 1510 (time 1781221844.78 total 20.01 last 77.76) - open/close 2247 (time 1781221854.78 total 30.02 last 73.68) - open/close 3006 (time 1781221864.78 total 40.02 last 75.87) - open/close 3835 (time 1781221874.79 total 50.02 last 82.87) total: 4500 open/close in 58.70 seconds: 76.67 ops/second - /unlink 1463 (time 1781221897.41 total 10.00 last 146.30) - /unlink 2898 (time 1781221907.41 total 20.00 last 143.48) - /unlink 4243 (time 1781221917.42 total 30.01 last 134.38) total: 4500 /unlink in 32.12 seconds: 140.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 667 (time 1781221934.48 total 10.00 last 66.68) - open/close 1286 (time 1781221944.48 total 20.00 last 61.89) - open/close 2034 (time 1781221954.48 total 30.01 last 74.78) - open/close 2778 (time 1781221964.48 total 40.01 last 74.38) - open/close 3463 (time 1781221974.50 total 50.02 last 68.40) - open/close 3984 (time 1781221984.51 total 60.04 last 52.02) total: 4500 open/close in 69.29 seconds: 64.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1242 (time 1781222008.07 total 10.00 last 124.17) - /unlink 2623 (time 1781222018.07 total 20.00 last 138.10) - /unlink 3966 (time 1781222028.07 total 30.01 last 134.25) total: 4500 /unlink in 34.52 seconds: 130.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 736 (time 1781222047.08 total 10.00 last 73.60) - open/close 1516 (time 1781222057.10 total 20.02 last 77.83) - open/close 2280 (time 1781222067.12 total 30.04 last 76.27) - open/close 2985 (time 1781222077.13 total 40.05 last 70.41) - open/close 3697 (time 1781222087.14 total 50.06 last 71.17) - open/close 4356 (time 1781222097.14 total 60.06 last 65.85) total: 4500 open/close in 62.04 seconds: 72.54 ops/second - /unlink 1338 (time 1781222114.32 total 10.00 last 133.74) - /unlink 2628 (time 1781222124.32 total 20.01 last 128.95) - /unlink 4022 (time 1781222134.32 total 30.01 last 139.34) total: 4500 /unlink in 33.61 seconds: 133.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 704 (time 1781222153.19 total 10.02 last 70.23) - open/close 1460 (time 1781222163.20 total 20.03 last 75.56) - open/close 2202 (time 1781222173.21 total 30.04 last 74.15) - open/close 2979 (time 1781222183.22 total 40.05 last 77.61) - open/close 3792 (time 1781222193.22 total 50.05 last 81.28) - open/close 4478 (time 1781222203.23 total 60.06 last 68.57) total: 4500 open/close in 60.33 seconds: 74.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1236 (time 1781222218.41 total 10.00 last 123.56) - /unlink 2370 (time 1781222228.42 total 20.02 last 113.24) - /unlink 3558 (time 1781222238.42 total 30.02 last 118.78) total: 4500 /unlink in 39.36 seconds: 114.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 649 (time 1781222263.12 total 10.02 last 64.78) - open/close 1119 (time 1781222273.13 total 20.02 last 46.98) - open/close 1714 (time 1781222283.13 total 30.03 last 59.45) - open/close 2374 (time 1781222293.15 total 40.04 last 65.92) - open/close 2894 (time 1781222303.15 total 50.05 last 51.98) - open/close 3354 (time 1781222313.15 total 60.05 last 45.98) - open/close 3967 (time 1781222323.16 total 70.06 last 61.27) total: 4500 open/close in 78.70 seconds: 57.18 ops/second - /unlink 1207 (time 1781222346.93 total 10.01 last 120.62) - /unlink 2435 (time 1781222356.93 total 20.01 last 122.79) - /unlink 3524 (time 1781222366.93 total 30.01 last 108.89) total: 4500 /unlink in 39.22 seconds: 114.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 623 (time 1781222390.96 total 10.02 last 62.20) - open/close 1274 (time 1781222400.97 total 20.03 last 65.03) - open/close 1907 (time 1781222410.98 total 30.03 last 63.28) - open/close 2503 (time 1781222420.99 total 40.04 last 59.55) - open/close 3096 (time 1781222430.99 total 50.05 last 59.26) - open/close 3743 (time 1781222441.00 total 60.05 last 64.67) - open/close 4284 (time 1781222451.01 total 70.06 last 54.06) total: 4500 open/close in 73.94 seconds: 60.86 ops/second - /unlink 938 (time 1781222470.97 total 10.01 last 93.72) - /unlink 1994 (time 1781222480.97 total 20.01 last 105.54) - /unlink 3196 (time 1781222490.97 total 30.01 last 120.19) - /unlink 4377 (time 1781222500.98 total 40.02 last 118.05) total: 4500 /unlink in 41.16 seconds: 109.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 631 (time 1781222517.11 total 10.01 last 63.02) - open/close 1204 (time 1781222527.13 total 20.04 last 57.15) - open/close 1615 (time 1781222537.14 total 30.05 last 41.07) - open/close 2084 (time 1781222547.16 total 40.07 last 46.81) - open/close 2546 (time 1781222557.17 total 50.08 last 46.14) - open/close 3186 (time 1781222567.19 total 60.09 last 63.91) - open/close 3868 (time 1781222577.19 total 70.10 last 68.19) - open/close 4474 (time 1781222587.19 total 80.10 last 60.57) total: 4500 open/close in 80.50 seconds: 55.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1167 (time 1781222602.26 total 10.00 last 116.64) - /unlink 2229 (time 1781222612.26 total 20.01 last 106.19) - /unlink 3296 (time 1781222622.27 total 30.02 last 106.56) - /unlink 4439 (time 1781222632.28 total 40.02 last 114.26) total: 4500 /unlink in 40.62 seconds: 110.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 586 (time 1781222647.31 total 10.02 last 58.49) - open/close 1202 (time 1781222657.33 total 20.03 last 61.53) - open/close 1775 (time 1781222667.34 total 30.05 last 57.21) - open/close 2348 (time 1781222677.34 total 40.05 last 57.30) - open/close 2965 (time 1781222687.35 total 50.05 last 61.66) - open/close 3560 (time 1781222697.35 total 60.05 last 59.50) - open/close 4225 (time 1781222707.36 total 70.06 last 66.45) total: 4500 open/close in 74.58 seconds: 60.33 ops/second - /unlink 1220 (time 1781222726.69 total 10.00 last 121.97) - /unlink 2316 (time 1781222736.70 total 20.01 last 109.54) - /unlink 3455 (time 1781222746.70 total 30.01 last 113.85) total: 4500 /unlink in 38.76 seconds: 116.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 677 (time 1781222770.43 total 10.00 last 67.68) - open/close 1277 (time 1781222780.44 total 20.01 last 59.96) - open/close 1796 (time 1781222790.44 total 30.01 last 51.88) - open/close 2386 (time 1781222800.46 total 40.03 last 58.93) - open/close 3002 (time 1781222810.46 total 50.03 last 61.59) - open/close 3558 (time 1781222820.46 total 60.03 last 55.59) - open/close 4127 (time 1781222830.48 total 70.05 last 56.78) total: 4500 open/close in 75.94 seconds: 59.26 ops/second - /unlink 1213 (time 1781222851.43 total 10.00 last 121.24) - /unlink 2358 (time 1781222861.43 total 20.01 last 114.50) - /unlink 3471 (time 1781222871.44 total 30.01 last 111.25) total: 4500 /unlink in 39.92 seconds: 112.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 650 (time 1781222896.86 total 10.04 last 64.75) - open/close 1305 (time 1781222906.87 total 20.05 last 65.45) - open/close 2022 (time 1781222916.88 total 30.06 last 71.63) - open/close 2661 (time 1781222926.90 total 40.07 last 63.79) - open/close 3343 (time 1781222936.91 total 50.08 last 68.14) - open/close 3991 (time 1781222946.93 total 60.10 last 64.66) total: 4500 open/close in 68.57 seconds: 65.63 ops/second - /unlink 1134 (time 1781222970.97 total 10.00 last 113.35) - /unlink 2198 (time 1781222980.97 total 20.00 last 106.39) - /unlink 3248 (time 1781222990.97 total 30.01 last 104.99) - /unlink 4435 (time 1781223000.97 total 40.01 last 118.68) total: 4500 /unlink in 40.61 seconds: 110.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 650 (time 1781223017.08 total 10.01 last 64.92) - open/close 1345 (time 1781223027.09 total 20.02 last 69.47) - open/close 2065 (time 1781223037.09 total 30.02 last 71.95) - open/close 2786 (time 1781223047.10 total 40.03 last 72.04) - open/close 3602 (time 1781223057.10 total 50.04 last 81.57) - open/close 4348 (time 1781223067.11 total 60.04 last 74.54) total: 4500 open/close in 62.45 seconds: 72.06 ops/second - /unlink 1127 (time 1781223085.92 total 10.02 last 112.49) - /unlink 2214 (time 1781223095.93 total 20.02 last 108.66) - /unlink 3506 (time 1781223105.94 total 30.04 last 129.02) total: 4500 /unlink in 37.43 seconds: 120.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 626 (time 1781223128.29 total 10.02 last 62.48) - open/close 1290 (time 1781223138.30 total 20.03 last 66.31) - open/close 1905 (time 1781223148.31 total 30.04 last 61.48) - open/close 2639 (time 1781223158.31 total 40.04 last 73.37) - open/close 3394 (time 1781223168.31 total 50.04 last 75.49) - open/close 4104 (time 1781223178.32 total 60.05 last 70.96) total: 4500 open/close in 64.54 seconds: 69.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1313 (time 1781223196.58 total 10.00 last 131.28) - /unlink 2366 (time 1781223206.60 total 20.02 last 105.07) - /unlink 3352 (time 1781223216.60 total 30.02 last 98.59) total: 4500 /unlink in 38.86 seconds: 115.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 786 (time 1781223240.65 total 10.01 last 78.56) - open/close 1503 (time 1781223250.65 total 20.01 last 71.69) - open/close 2224 (time 1781223260.66 total 30.01 last 72.07) - open/close 2989 (time 1781223270.67 total 40.02 last 76.42) - open/close 3710 (time 1781223280.67 total 50.03 last 72.07) - open/close 4444 (time 1781223290.67 total 60.03 last 73.39) total: 4500 open/close in 60.91 seconds: 73.88 ops/second - /unlink 1126 (time 1781223306.18 total 10.01 last 112.50) - /unlink 2340 (time 1781223316.18 total 20.01 last 121.40) - /unlink 3821 (time 1781223326.19 total 30.01 last 148.04) total: 4500 /unlink in 34.40 seconds: 130.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 760 (time 1781223344.88 total 10.01 last 75.96) - open/close 1464 (time 1781223354.90 total 20.02 last 70.32) - open/close 2221 (time 1781223364.91 total 30.03 last 75.63) - open/close 2951 (time 1781223374.91 total 40.03 last 72.95) - open/close 3709 (time 1781223384.92 total 50.04 last 75.77) - open/close 4493 (time 1781223394.92 total 60.05 last 78.34) total: 4500 open/close in 60.13 seconds: 74.83 ops/second - /unlink 1352 (time 1781223409.44 total 10.01 last 135.10) - /unlink 2628 (time 1781223419.44 total 20.01 last 127.57) - /unlink 3893 (time 1781223429.44 total 30.01 last 126.50) total: 4500 /unlink in 34.61 seconds: 130.03 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 93989 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded.