== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:51:54 (1787838714) Stopping clients: oleg343-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg343-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg343-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg343-server oleg343-server: oleg343-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg343-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg343-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg343-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_loc=0x1312 fail_val=5 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 697 (time 1787838929.70 total 10.01 last 69.64) - open/close 1370 (time 1787838939.70 total 20.01 last 67.27) - open/close 2029 (time 1787838949.70 total 30.01 last 65.89) - open/close 2732 (time 1787838959.71 total 40.02 last 70.24) - open/close 3460 (time 1787838969.71 total 50.02 last 72.78) - open/close 4142 (time 1787838979.72 total 60.03 last 68.13) total: 4500 open/close in 65.41 seconds: 68.80 ops/second - /unlink 1245 (time 1787838999.34 total 10.01 last 124.41) - /unlink 2521 (time 1787839009.35 total 20.02 last 127.49) - /unlink 3830 (time 1787839019.35 total 30.02 last 130.81) total: 4500 /unlink in 34.78 seconds: 129.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 665 (time 1787839038.75 total 10.00 last 66.48) - open/close 1339 (time 1787839048.75 total 20.01 last 67.37) - open/close 1983 (time 1787839058.76 total 30.02 last 64.34) - open/close 2594 (time 1787839068.77 total 40.03 last 61.02) - open/close 3300 (time 1787839078.78 total 50.04 last 70.55) - open/close 3980 (time 1787839088.79 total 60.04 last 67.98) total: 4500 open/close in 67.76 seconds: 66.41 ops/second - /unlink 1346 (time 1787839110.86 total 10.01 last 134.53) - /unlink 2688 (time 1787839120.86 total 20.01 last 134.09) - /unlink 4074 (time 1787839130.87 total 30.02 last 138.57) total: 4500 /unlink in 33.31 seconds: 135.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 679 (time 1787839148.90 total 10.00 last 67.88) - open/close 1379 (time 1787839158.91 total 20.01 last 69.92) - open/close 2030 (time 1787839168.92 total 30.02 last 65.04) - open/close 2730 (time 1787839178.93 total 40.03 last 69.96) - open/close 3453 (time 1787839188.93 total 50.03 last 72.29) - open/close 4131 (time 1787839198.94 total 60.04 last 67.71) total: 4500 open/close in 64.92 seconds: 69.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1355 (time 1787839218.05 total 10.00 last 135.48) - /unlink 2629 (time 1787839228.05 total 20.00 last 127.37) - /unlink 4014 (time 1787839238.05 total 30.01 last 138.44) total: 4500 /unlink in 33.65 seconds: 133.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 712 (time 1787839256.68 total 10.02 last 71.08) - open/close 1419 (time 1787839266.69 total 20.03 last 70.64) - open/close 2053 (time 1787839276.70 total 30.04 last 63.31) - open/close 2732 (time 1787839286.71 total 40.05 last 67.83) - open/close 3477 (time 1787839296.72 total 50.05 last 74.47) - open/close 4106 (time 1787839306.72 total 60.06 last 62.88) total: 4500 open/close in 66.75 seconds: 67.42 ops/second - /unlink 1213 (time 1787839328.72 total 10.00 last 121.30) - /unlink 2453 (time 1787839338.72 total 20.00 last 124.00) - /unlink 3830 (time 1787839348.73 total 30.01 last 137.59) total: 4500 /unlink in 35.09 seconds: 128.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 712 (time 1787839368.16 total 10.02 last 71.09) - open/close 1420 (time 1787839378.17 total 20.02 last 70.73) - open/close 2092 (time 1787839388.17 total 30.03 last 67.19) - open/close 2819 (time 1787839398.19 total 40.05 last 72.54) - open/close 3522 (time 1787839408.20 total 50.05 last 70.26) - open/close 4243 (time 1787839418.21 total 60.06 last 72.02) total: 4500 open/close in 63.39 seconds: 70.99 ops/second - /unlink 1187 (time 1787839435.30 total 10.00 last 118.65) - /unlink 2143 (time 1787839445.31 total 20.02 last 95.49) - /unlink 3073 (time 1787839455.32 total 30.02 last 92.93) - /unlink 3911 (time 1787839465.33 total 40.03 last 83.73) total: 4500 /unlink in 44.93 seconds: 100.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 529 (time 1787839486.89 total 10.04 last 52.67) - open/close 1170 (time 1787839496.89 total 20.05 last 64.08) - open/close 1881 (time 1787839506.90 total 30.06 last 71.03) - open/close 2614 (time 1787839516.90 total 40.06 last 73.27) - open/close 3347 (time 1787839526.91 total 50.07 last 73.23) - open/close 4004 (time 1787839536.92 total 60.07 last 65.67) total: 4500 open/close in 68.26 seconds: 65.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1241 (time 1787839559.47 total 10.00 last 124.09) - /unlink 2596 (time 1787839569.47 total 20.00 last 135.47) - /unlink 4019 (time 1787839579.48 total 30.01 last 142.26) total: 4500 /unlink in 33.33 seconds: 135.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 702 (time 1787839598.01 total 10.01 last 70.15) - open/close 1427 (time 1787839608.02 total 20.02 last 72.40) - open/close 2108 (time 1787839618.04 total 30.04 last 68.00) - open/close 2756 (time 1787839628.04 total 40.04 last 64.78) - open/close 3427 (time 1787839638.05 total 50.05 last 67.05) - open/close 4138 (time 1787839648.06 total 60.06 last 71.02) total: 4500 open/close in 65.57 seconds: 68.62 ops/second - /unlink 1283 (time 1787839667.97 total 10.02 last 128.06) - /unlink 2555 (time 1787839677.98 total 20.03 last 127.06) - /unlink 3848 (time 1787839687.98 total 30.03 last 129.29) total: 4500 /unlink in 35.31 seconds: 127.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 723 (time 1787839707.63 total 10.01 last 72.25) - open/close 1364 (time 1787839717.64 total 20.01 last 64.05) - open/close 1989 (time 1787839727.65 total 30.02 last 62.47) - open/close 2621 (time 1787839737.66 total 40.03 last 63.14) - open/close 3315 (time 1787839747.67 total 50.05 last 69.28) - open/close 3996 (time 1787839757.69 total 60.06 last 68.03) total: 4500 open/close in 67.86 seconds: 66.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1145 (time 1787839780.76 total 10.00 last 114.50) - /unlink 2571 (time 1787839790.77 total 20.01 last 142.46) - /unlink 3954 (time 1787839800.77 total 30.01 last 138.28) total: 4500 /unlink in 34.16 seconds: 131.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 665 (time 1787839819.72 total 10.01 last 66.43) - open/close 1357 (time 1787839829.72 total 20.02 last 69.17) - open/close 2010 (time 1787839839.73 total 30.03 last 65.23) - open/close 2703 (time 1787839849.75 total 40.04 last 69.22) - open/close 3391 (time 1787839859.75 total 50.04 last 68.80) - open/close 4103 (time 1787839869.75 total 60.05 last 71.14) total: 4500 open/close in 65.43 seconds: 68.78 ops/second - /unlink 1306 (time 1787839889.26 total 10.00 last 130.58) - /unlink 2399 (time 1787839899.26 total 20.01 last 109.26) - /unlink 3472 (time 1787839909.26 total 30.01 last 107.29) total: 4500 /unlink in 38.56 seconds: 116.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 740 (time 1787839933.04 total 10.01 last 73.94) - open/close 1509 (time 1787839943.05 total 20.01 last 76.85) - open/close 2228 (time 1787839953.06 total 30.02 last 71.85) - open/close 2947 (time 1787839963.06 total 40.02 last 71.89) - open/close 3604 (time 1787839973.06 total 50.03 last 65.68) - open/close 4337 (time 1787839983.07 total 60.03 last 73.28) total: 4500 open/close in 62.16 seconds: 72.40 ops/second - /unlink 1330 (time 1787839999.95 total 10.00 last 132.99) - /unlink 2608 (time 1787840009.95 total 20.00 last 127.76) - /unlink 3969 (time 1787840019.96 total 30.01 last 136.02) total: 4500 /unlink in 34.15 seconds: 131.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 687 (time 1787840038.99 total 10.01 last 68.61) - open/close 1407 (time 1787840049.00 total 20.02 last 71.93) - open/close 2121 (time 1787840059.01 total 30.04 last 71.28) - open/close 2786 (time 1787840069.02 total 40.04 last 66.47) - open/close 3466 (time 1787840079.03 total 50.05 last 67.94) - open/close 4157 (time 1787840089.04 total 60.07 last 69.00) total: 4500 open/close in 64.61 seconds: 69.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1355 (time 1787840107.97 total 10.01 last 135.40) - /unlink 2543 (time 1787840117.97 total 20.01 last 118.73) - /unlink 3711 (time 1787840127.97 total 30.02 last 116.78) total: 4500 /unlink in 36.70 seconds: 122.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 679 (time 1787840149.63 total 10.00 last 67.89) - open/close 1330 (time 1787840159.65 total 20.02 last 64.97) - open/close 1997 (time 1787840169.65 total 30.02 last 66.69) - open/close 2679 (time 1787840179.67 total 40.04 last 68.07) - open/close 3387 (time 1787840189.69 total 50.05 last 70.71) - open/close 4110 (time 1787840199.70 total 60.07 last 72.22) total: 4500 open/close in 67.39 seconds: 66.78 ops/second - /unlink 1189 (time 1787840221.78 total 10.00 last 118.88) - /unlink 2329 (time 1787840231.78 total 20.01 last 113.93) - /unlink 3581 (time 1787840241.79 total 30.02 last 125.10) total: 4500 /unlink in 37.64 seconds: 119.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 696 (time 1787840264.63 total 10.01 last 69.56) - open/close 1400 (time 1787840274.65 total 20.02 last 70.30) - open/close 2056 (time 1787840284.66 total 30.04 last 65.48) - open/close 2737 (time 1787840294.67 total 40.04 last 68.08) - open/close 3402 (time 1787840304.68 total 50.05 last 66.40) - open/close 4103 (time 1787840314.68 total 60.06 last 70.07) total: 4500 open/close in 66.06 seconds: 68.12 ops/second - /unlink 1170 (time 1787840336.93 total 10.00 last 116.95) - /unlink 2432 (time 1787840346.94 total 20.01 last 126.13) - /unlink 3724 (time 1787840356.94 total 30.02 last 129.12) total: 4500 /unlink in 35.75 seconds: 125.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 649 (time 1787840377.51 total 10.00 last 64.88) - open/close 1348 (time 1787840387.52 total 20.01 last 69.84) - open/close 2062 (time 1787840397.53 total 30.02 last 71.32) - open/close 2761 (time 1787840407.53 total 40.03 last 69.87) - open/close 3442 (time 1787840417.54 total 50.04 last 68.03) - open/close 4133 (time 1787840427.54 total 60.04 last 69.10) total: 4500 open/close in 66.12 seconds: 68.06 ops/second - /unlink 1350 (time 1787840448.40 total 10.00 last 134.94) - /unlink 2410 (time 1787840458.41 total 20.01 last 105.99) - /unlink 3418 (time 1787840468.41 total 30.01 last 100.75) total: 4500 /unlink in 39.55 seconds: 113.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 665 (time 1787840493.27 total 10.00 last 66.49) - open/close 1311 (time 1787840503.28 total 20.01 last 64.57) - open/close 1981 (time 1787840513.28 total 30.01 last 66.99) - open/close 2695 (time 1787840523.29 total 40.02 last 71.28) - open/close 3412 (time 1787840533.31 total 50.04 last 71.59) - open/close 4133 (time 1787840543.32 total 60.05 last 72.03) total: 4500 open/close in 64.88 seconds: 69.36 ops/second - /unlink 1393 (time 1787840562.21 total 10.00 last 139.27) - /unlink 2703 (time 1787840572.21 total 20.00 last 130.99) - /unlink 3963 (time 1787840582.22 total 30.01 last 125.97) total: 4500 /unlink in 34.78 seconds: 129.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 654 (time 1787840602.32 total 10.00 last 65.39) - open/close 1221 (time 1787840612.33 total 20.01 last 56.66) - open/close 1941 (time 1787840622.35 total 30.03 last 71.85) - open/close 2647 (time 1787840632.36 total 40.04 last 70.54) - open/close 3310 (time 1787840642.36 total 50.04 last 66.29) - open/close 4008 (time 1787840652.36 total 60.04 last 69.79) total: 4500 open/close in 66.30 seconds: 67.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1298 (time 1787840672.44 total 10.00 last 129.76) - /unlink 2215 (time 1787840682.45 total 20.01 last 91.66) - /unlink 3119 (time 1787840692.46 total 30.02 last 90.31) - /unlink 4369 (time 1787840702.46 total 40.02 last 124.98) total: 4500 /unlink in 41.00 seconds: 109.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 700 (time 1787840717.37 total 10.00 last 69.97) - open/close 1435 (time 1787840727.38 total 20.01 last 73.44) - open/close 2098 (time 1787840737.40 total 30.03 last 66.20) - open/close 2786 (time 1787840747.41 total 40.04 last 68.72) - open/close 3441 (time 1787840757.41 total 50.04 last 65.47) - open/close 4173 (time 1787840767.42 total 60.05 last 73.16) total: 4500 open/close in 64.92 seconds: 69.31 ops/second - /unlink 1257 (time 1787840786.78 total 10.01 last 125.60) - /unlink 2328 (time 1787840796.78 total 20.01 last 107.06) - /unlink 3267 (time 1787840806.79 total 30.02 last 93.82) - /unlink 4182 (time 1787840816.79 total 40.02 last 91.49) total: 4500 /unlink in 42.56 seconds: 105.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 712 (time 1787840834.05 total 10.02 last 71.06) - open/close 1306 (time 1787840844.08 total 20.05 last 59.22) - open/close 1888 (time 1787840854.09 total 30.06 last 58.13) - open/close 2533 (time 1787840864.10 total 40.07 last 64.47) - open/close 3151 (time 1787840874.11 total 50.08 last 61.72) - open/close 3826 (time 1787840884.12 total 60.09 last 67.45) total: 4500 open/close in 69.41 seconds: 64.83 ops/second - /unlink 1023 (time 1787840908.74 total 10.01 last 102.24) - /unlink 2189 (time 1787840918.74 total 20.01 last 116.57) - /unlink 3382 (time 1787840928.74 total 30.01 last 119.23) total: 4500 /unlink in 39.32 seconds: 114.43 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95282 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: modules unloaded.