== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:34:34 (1787398474) Stopping clients: oleg617-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg617-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg617-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg617-server oleg617-server: oleg617-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing load_modules_local oleg617-server: Loading modules from /home/green/git/lustre-release/lustre oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg617-server: detected 4 online CPUs by sysfs oleg617-server: MODOPTS_LIBCFS= oleg617-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg617-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg617-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg617-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock 192.168.206.117@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 727 (time 1787398666.25 total 10.01 last 72.64) - open/close 1456 (time 1787398676.25 total 20.01 last 72.87) - open/close 2219 (time 1787398686.26 total 30.02 last 76.21) - open/close 2946 (time 1787398696.27 total 40.03 last 72.67) - open/close 3537 (time 1787398706.27 total 50.03 last 59.09) - open/close 4170 (time 1787398716.29 total 60.05 last 63.16) total: 4500 open/close in 65.37 seconds: 68.84 ops/second - /unlink 1338 (time 1787398736.21 total 10.01 last 133.64) - /unlink 2635 (time 1787398746.22 total 20.02 last 129.66) - /unlink 3939 (time 1787398756.22 total 30.02 last 130.40) total: 4500 /unlink in 34.44 seconds: 130.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 720 (time 1787398775.37 total 10.02 last 71.84) - open/close 1488 (time 1787398785.38 total 20.03 last 76.76) - open/close 2189 (time 1787398795.38 total 30.03 last 70.08) - open/close 2867 (time 1787398805.39 total 40.04 last 67.73) - open/close 3588 (time 1787398815.40 total 50.05 last 72.06) - open/close 4363 (time 1787398825.40 total 60.05 last 77.46) total: 4500 open/close in 61.61 seconds: 73.05 ops/second - /unlink 1476 (time 1787398841.08 total 10.01 last 147.52) - /unlink 2845 (time 1787398851.09 total 20.02 last 136.76) - /unlink 4272 (time 1787398861.09 total 30.02 last 142.65) total: 4500 /unlink in 31.87 seconds: 141.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 753 (time 1787398876.86 total 10.01 last 75.24) - open/close 1549 (time 1787398886.87 total 20.01 last 79.59) - open/close 2334 (time 1787398896.87 total 30.02 last 78.44) - open/close 3129 (time 1787398906.88 total 40.02 last 79.47) - open/close 3887 (time 1787398916.88 total 50.03 last 75.75) total: 4500 open/close in 59.05 seconds: 76.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1370 (time 1787398940.66 total 10.00 last 136.94) - /unlink 2628 (time 1787398950.67 total 20.01 last 125.73) - /unlink 4002 (time 1787398960.67 total 30.02 last 137.32) total: 4500 /unlink in 33.50 seconds: 134.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 721 (time 1787398978.66 total 10.00 last 72.08) - open/close 1502 (time 1787398988.68 total 20.02 last 77.94) - open/close 2278 (time 1787398998.69 total 30.03 last 77.52) - open/close 3045 (time 1787399008.70 total 40.04 last 76.65) - open/close 3711 (time 1787399018.71 total 50.05 last 66.53) - open/close 4432 (time 1787399028.72 total 60.06 last 72.05) total: 4500 open/close in 61.24 seconds: 73.49 ops/second - /unlink 1338 (time 1787399044.78 total 10.01 last 133.72) - /unlink 2758 (time 1787399054.78 total 20.01 last 141.99) - /unlink 4145 (time 1787399064.79 total 30.01 last 138.60) total: 4500 /unlink in 32.77 seconds: 137.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 731 (time 1787399082.39 total 10.01 last 73.06) - open/close 1469 (time 1787399092.42 total 20.03 last 73.65) - open/close 2258 (time 1787399102.42 total 30.04 last 78.83) - open/close 3068 (time 1787399112.44 total 40.05 last 80.87) - open/close 3810 (time 1787399122.44 total 50.05 last 74.19) total: 4500 open/close in 58.48 seconds: 76.95 ops/second - /unlink 1363 (time 1787399145.30 total 10.00 last 136.24) - /unlink 2582 (time 1787399155.30 total 20.01 last 121.89) - /unlink 3705 (time 1787399165.30 total 30.01 last 112.28) total: 4500 /unlink in 36.13 seconds: 124.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 664 (time 1787399186.07 total 10.00 last 66.38) - open/close 1332 (time 1787399196.07 total 20.01 last 66.76) - open/close 1998 (time 1787399206.08 total 30.01 last 66.58) - open/close 2699 (time 1787399216.08 total 40.02 last 70.04) - open/close 3465 (time 1787399226.09 total 50.03 last 76.55) - open/close 4243 (time 1787399236.10 total 60.04 last 77.72) total: 4500 open/close in 63.49 seconds: 70.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1268 (time 1787399254.79 total 10.00 last 126.77) - /unlink 2601 (time 1787399264.79 total 20.01 last 133.20) - /unlink 3969 (time 1787399274.80 total 30.02 last 136.72) total: 4500 /unlink in 33.87 seconds: 132.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 762 (time 1787399293.77 total 10.02 last 76.05) - open/close 1539 (time 1787399303.78 total 20.03 last 77.63) - open/close 2308 (time 1787399313.78 total 30.03 last 76.88) - open/close 3063 (time 1787399323.79 total 40.04 last 75.44) - open/close 3820 (time 1787399333.79 total 50.04 last 75.67) total: 4500 open/close in 59.02 seconds: 76.24 ops/second - /unlink 1244 (time 1787399357.97 total 10.00 last 124.38) - /unlink 2646 (time 1787399367.98 total 20.00 last 140.16) - /unlink 4113 (time 1787399377.98 total 30.01 last 146.61) total: 4500 /unlink in 32.46 seconds: 138.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 779 (time 1787399395.54 total 10.02 last 77.77) - open/close 1569 (time 1787399405.55 total 20.02 last 78.97) - open/close 2366 (time 1787399415.56 total 30.03 last 79.62) - open/close 3040 (time 1787399425.57 total 40.04 last 67.30) - open/close 3586 (time 1787399435.59 total 50.06 last 54.53) - open/close 4173 (time 1787399445.60 total 60.07 last 58.64) total: 4500 open/close in 65.54 seconds: 68.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1229 (time 1787399467.52 total 10.01 last 122.72) - /unlink 2487 (time 1787399477.52 total 20.02 last 125.79) - /unlink 3575 (time 1787399487.53 total 30.02 last 108.74) total: 4500 /unlink in 37.63 seconds: 119.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 727 (time 1787399510.67 total 10.01 last 72.62) - open/close 1426 (time 1787399520.68 total 20.02 last 69.82) - open/close 2153 (time 1787399530.69 total 30.03 last 72.66) - open/close 2929 (time 1787399540.69 total 40.03 last 77.56) - open/close 3721 (time 1787399550.70 total 50.04 last 79.12) total: 4500 open/close in 59.86 seconds: 75.17 ops/second - /unlink 1329 (time 1787399575.07 total 10.00 last 132.86) - /unlink 2734 (time 1787399585.07 total 20.00 last 140.49) - /unlink 4087 (time 1787399595.08 total 30.02 last 135.13) total: 4500 /unlink in 33.07 seconds: 136.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 694 (time 1787399613.05 total 10.01 last 69.33) - open/close 1453 (time 1787399623.05 total 20.01 last 75.89) - open/close 2198 (time 1787399633.05 total 30.02 last 74.47) - open/close 2959 (time 1787399643.06 total 40.02 last 76.04) - open/close 3630 (time 1787399653.07 total 50.03 last 67.08) - open/close 4379 (time 1787399663.07 total 60.03 last 74.86) total: 4500 open/close in 61.53 seconds: 73.14 ops/second - /unlink 1291 (time 1787399680.05 total 10.00 last 129.04) - /unlink 2532 (time 1787399690.05 total 20.01 last 124.09) - /unlink 3782 (time 1787399700.05 total 30.01 last 124.97) total: 4500 /unlink in 35.34 seconds: 127.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 777 (time 1787399719.97 total 10.00 last 77.69) - open/close 1598 (time 1787399729.97 total 20.01 last 82.05) - open/close 2371 (time 1787399739.98 total 30.01 last 77.28) - open/close 2956 (time 1787399749.98 total 40.02 last 58.47) - open/close 3654 (time 1787399759.99 total 50.02 last 69.77) - open/close 4348 (time 1787399770.00 total 60.04 last 69.29) total: 4500 open/close in 62.62 seconds: 71.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1166 (time 1787399788.19 total 10.01 last 116.53) - /unlink 2141 (time 1787399798.21 total 20.02 last 97.32) - /unlink 3171 (time 1787399808.21 total 30.02 last 103.00) - /unlink 4340 (time 1787399818.21 total 40.03 last 116.86) total: 4500 /unlink in 41.57 seconds: 108.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 634 (time 1787399834.98 total 10.01 last 63.33) - open/close 1428 (time 1787399844.99 total 20.02 last 79.32) - open/close 2125 (time 1787399854.99 total 30.02 last 69.69) - open/close 2909 (time 1787399865.03 total 40.06 last 78.14) - open/close 3603 (time 1787399875.03 total 50.06 last 69.39) - open/close 4371 (time 1787399885.04 total 60.07 last 76.72) total: 4500 open/close in 61.61 seconds: 73.04 ops/second - /unlink 1285 (time 1787399901.06 total 10.00 last 128.45) - /unlink 2507 (time 1787399911.07 total 20.01 last 122.12) - /unlink 3733 (time 1787399921.07 total 30.01 last 122.56) total: 4500 /unlink in 36.20 seconds: 124.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 741 (time 1787399943.15 total 10.03 last 73.90) - open/close 1556 (time 1787399953.16 total 20.03 last 81.45) - open/close 2272 (time 1787399963.17 total 30.05 last 71.51) - open/close 2993 (time 1787399973.17 total 40.05 last 72.07) - open/close 3685 (time 1787399983.18 total 50.06 last 69.15) - open/close 4374 (time 1787399993.18 total 60.06 last 68.88) total: 4500 open/close in 61.57 seconds: 73.09 ops/second - /unlink 1359 (time 1787400008.92 total 10.00 last 135.85) - /unlink 2750 (time 1787400018.93 total 20.01 last 139.04) - /unlink 4065 (time 1787400028.93 total 30.01 last 131.44) total: 4500 /unlink in 33.73 seconds: 133.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 657 (time 1787400047.88 total 10.01 last 65.62) - open/close 1304 (time 1787400057.88 total 20.01 last 64.70) - open/close 1925 (time 1787400067.88 total 30.02 last 62.08) - open/close 2626 (time 1787400077.88 total 40.02 last 70.10) - open/close 3312 (time 1787400087.89 total 50.03 last 68.52) - open/close 4122 (time 1787400097.91 total 60.04 last 80.90) total: 4500 open/close in 64.44 seconds: 69.83 ops/second - /unlink 1377 (time 1787400116.83 total 10.00 last 137.68) - /unlink 2819 (time 1787400126.84 total 20.01 last 144.10) - /unlink 4215 (time 1787400136.85 total 30.01 last 139.54) total: 4500 /unlink in 32.48 seconds: 138.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 674 (time 1787400155.65 total 10.00 last 67.37) - open/close 1293 (time 1787400165.67 total 20.02 last 61.78) - open/close 1916 (time 1787400175.67 total 30.03 last 62.25) - open/close 2486 (time 1787400185.69 total 40.05 last 56.91) - open/close 3116 (time 1787400195.69 total 50.05 last 62.99) - open/close 3881 (time 1787400205.70 total 60.05 last 76.47) total: 4500 open/close in 68.06 seconds: 66.12 ops/second - /unlink 1373 (time 1787400228.27 total 10.00 last 137.26) - /unlink 2632 (time 1787400238.28 total 20.01 last 125.83) - /unlink 4026 (time 1787400248.28 total 30.01 last 139.35) total: 4500 /unlink in 33.29 seconds: 135.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 820 (time 1787400267.20 total 10.01 last 81.94) - open/close 1613 (time 1787400277.20 total 20.01 last 79.30) - open/close 2376 (time 1787400287.20 total 30.01 last 76.28) - open/close 3107 (time 1787400297.21 total 40.01 last 73.07) - open/close 3876 (time 1787400307.23 total 50.03 last 76.76) total: 4500 open/close in 58.75 seconds: 76.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1238 (time 1787400330.97 total 10.02 last 123.60) - /unlink 2404 (time 1787400340.99 total 20.04 last 116.37) - /unlink 3595 (time 1787400350.99 total 30.04 last 119.09) total: 4500 /unlink in 37.22 seconds: 120.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 815 (time 1787400373.09 total 10.03 last 81.27) - open/close 1584 (time 1787400383.10 total 20.03 last 76.85) - open/close 2226 (time 1787400393.10 total 30.04 last 64.18) - open/close 2997 (time 1787400403.11 total 40.04 last 77.07) - open/close 3680 (time 1787400413.11 total 50.04 last 68.29) - open/close 4432 (time 1787400423.12 total 60.06 last 75.09) total: 4500 open/close in 61.34 seconds: 73.36 ops/second - /unlink 1285 (time 1787400439.54 total 10.00 last 128.48) - /unlink 2641 (time 1787400449.56 total 20.02 last 135.39) - /unlink 4000 (time 1787400459.57 total 30.03 last 135.77) total: 4500 /unlink in 34.08 seconds: 132.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 779 (time 1787400478.45 total 10.01 last 77.85) - open/close 1491 (time 1787400488.46 total 20.02 last 71.10) - open/close 2292 (time 1787400498.46 total 30.02 last 80.09) - open/close 2895 (time 1787400508.46 total 40.02 last 60.29) - open/close 3595 (time 1787400518.47 total 50.03 last 69.98) - open/close 4285 (time 1787400528.48 total 60.04 last 68.93) total: 4500 open/close in 62.57 seconds: 71.92 ops/second - /unlink 1210 (time 1787400545.77 total 10.01 last 120.90) - /unlink 2490 (time 1787400555.78 total 20.02 last 127.92) - /unlink 3817 (time 1787400565.79 total 30.02 last 132.60) total: 4500 /unlink in 34.94 seconds: 128.78 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94876 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg617-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server unloading modules via unload_modules_local on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing unload_modules_local oleg617-server: modules unloaded.