日志收集探讨-PHP Tutorial-php.cn

Home

Backend Development

PHP Tutorial

日志收集探讨

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 06, 2016 pm 08:37 PM

golangphppythonredis

在项目上线后，往往需要监控一些日志，但是日志分散在各个服务器的不同角落，查看时需要找到某应用在那台服务器上，然后翻apache或nginx配置找日志路径，耗时耗力很麻烦。
所以想将这些日志收集到一个地方，做一套web界面的日志查看工具
需要的日志主要有

apache日志
apache虚拟主机access,error日志
nginx日志
nginx虚拟主机access,error日志
php错误日志
mysql日志
php应用日志(如yii的/runtime/application.log)

基本实现实时查看即可（一分钟内延迟）
看了下Scribe和Flume，似乎太过重量级，
我的想法是做个简单的文本变化检测，将新增内容发送到统一redis队列，后端脚本处理入库。
不知道大家是如何解决这个问题的？探讨下吧！

回复内容：

apache日志
apache虚拟主机access,error日志
nginx日志
nginx虚拟主机access,error日志
php错误日志
mysql日志
php应用日志(如yii的/runtime/application.log)

你的这个需求直接使用elasticsearch、logstash和kibana结合即可，部署配置都比较简单，实时性较高，又不用开发代码~

ELK全部搞定妥妥的 https://www.elastic.co/

说一个常用方案：很多公司都是专门布了脚本，定时去抓，抓完了索引存到数据平台，数据平台提供查询，接口确实是略重。

来点前沿科技——systemd 的 journald～不用检测文本变化，它本身就是类似数据库那种二进制存储，直接用它给的 API 读取就可以了。当然目前 journald 不支持搜索，只能根据一些域来取数据，不过满足你的需求够了。

当然我猜测你们服务器上目前是不会用 systemd 的。journald 大概也不能不依赖 systemd 自己跑……

看一下logstash吧。 http://logstash.net/

负责任地说，成熟的日志处理和监控平台很多很多很多很多。。手机用户就不方便写了改天用电脑的时候给你列出来十几样

参考：在Linux上使用logwatch分析监控日志文件

看看Sentry？https://github.com/getsentry/sentry

go语言的话可以考虑一下heka

楼主可以关注下开源分布式搜索平台ELK(Elasticsearch+Logstash+Kibana)

http://finallygo.iteye.com/blog/2170588
建议你看看这个,非常轻量

成熟的商业方案Splunk不错。

fluentd + MongoDB

可以试试阿里云的日志服务，即开即用，无需运维。
1）提供客户端快速采集日志（https://yq.aliyun.com/articles/3228?spm=5176.team4.teamshow1.37.faH93Z）
2）日志收集到服务端后提供多种消费方式：数据通道（类Kafka）/数据投递（ODPS/OSS）/实时搜索
具体可以参考：https://help.aliyun.com/document_detail/sls/user-guide/overview.html

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

How to make PHP applications fasterMay 12, 2025 am 12:12 AM

TomakePHPapplicationsfaster,followthesesteps:1)UseOpcodeCachinglikeOPcachetostoreprecompiledscriptbytecode.2)MinimizeDatabaseQueriesbyusingquerycachingandefficientindexing.3)LeveragePHP7 Featuresforbettercodeefficiency.4)ImplementCachingStrategiessuc

PHP Performance Optimization Checklist: Improve Speed NowMay 12, 2025 am 12:07 AM

ToimprovePHPapplicationspeed,followthesesteps:1)EnableopcodecachingwithAPCutoreducescriptexecutiontime.2)ImplementdatabasequerycachingusingPDOtominimizedatabasehits.3)UseHTTP/2tomultiplexrequestsandreduceconnectionoverhead.4)Limitsessionusagebyclosin

PHP Dependency Injection: Improve Code TestabilityMay 12, 2025 am 12:03 AM

Dependency injection (DI) significantly improves the testability of PHP code by explicitly transitive dependencies. 1) DI decoupling classes and specific implementations make testing and maintenance more flexible. 2) Among the three types, the constructor injects explicit expression dependencies to keep the state consistent. 3) Use DI containers to manage complex dependencies to improve code quality and development efficiency.

PHP Performance Optimization: Database Query OptimizationMay 12, 2025 am 12:02 AM

DatabasequeryoptimizationinPHPinvolvesseveralstrategiestoenhanceperformance.1)Selectonlynecessarycolumnstoreducedatatransfer.2)Useindexingtospeedupdataretrieval.3)Implementquerycachingtostoreresultsoffrequentqueries.4)Utilizepreparedstatementsforeffi

Simple Guide: Sending Email with PHP ScriptMay 12, 2025 am 12:02 AM

PHPisusedforsendingemailsduetoitsbuilt-inmail()functionandsupportivelibrarieslikePHPMailerandSwiftMailer.1)Usethemail()functionforbasicemails,butithaslimitations.2)EmployPHPMailerforadvancedfeatureslikeHTMLemailsandattachments.3)Improvedeliverability

PHP Performance: Identifying and Fixing BottlenecksMay 11, 2025 am 12:13 AM

PHP performance bottlenecks can be solved through the following steps: 1) Use Xdebug or Blackfire for performance analysis to find out the problem; 2) Optimize database queries and use caches, such as APCu; 3) Use efficient functions such as array_filter to optimize array operations; 4) Configure OPcache for bytecode cache; 5) Optimize the front-end, such as reducing HTTP requests and optimizing pictures; 6) Continuously monitor and optimize performance. Through these methods, the performance of PHP applications can be significantly improved.

Dependency Injection for PHP: a quick summaryMay 11, 2025 am 12:09 AM

DependencyInjection(DI)inPHPisadesignpatternthatmanagesandreducesclassdependencies,enhancingcodemodularity,testability,andmaintainability.Itallowspassingdependencieslikedatabaseconnectionstoclassesasparameters,facilitatingeasiertestingandscalability.

Increase PHP Performance: Caching Strategies & TechniquesMay 11, 2025 am 12:08 AM

CachingimprovesPHPperformancebystoringresultsofcomputationsorqueriesforquickretrieval,reducingserverloadandenhancingresponsetimes.Effectivestrategiesinclude:1)Opcodecaching,whichstorescompiledPHPscriptsinmemorytoskipcompilation;2)DatacachingusingMemc

See all articles