search
HomeBackend DevelopmentPHP TutorialPHP Tokenizer 学习笔记_PHP

简述

在某个项目中需要分析 PHP 代码,分离出对应的函数调用(以及源代码对应的位置)。虽然这使用正则也可以实现,但无论从效率还是代码复杂度方面考虑,这都不是最优的方式。

查询了 PHP 手册,发现其实 PHP 已经内置解析器的接口,那就是 PHP Tokenizer,这工具正是我想要的。使用 PHP Tokenizer 能简单、高效、准确的分析出 PHP 源代码的组成。

实例

官方站点对 Tokenizer 的文档很少,不过这不影响我们理解它。Tokenizer 组件仅仅包含两个函数:token_get_all 以及 token_name,它们分别用于分析 PHP 代码以及获取代码对应的标识符名称。

下面是个简单的实例,说明如何使用这两个函数:

以下为引用的内容:

$code = '<?php echo "string1"."string2"; ?>';
$tokens = token_get_all($code);
foreach ($tokens as $token) {
    if (is_array($token)) {
        // 行号、标识符字面量、对应内容
        printf("%d - %s\t%s\n", $token[2], token_name($token[0]), $token[1]);
    }
}

对应的输出为

以下为引用的内容:

1 - T_OPEN_TAG    <?php 1 - T_ECHO    echo
1 - T_WHITESPACE     
1 - T_CONSTANT_ENCAPSED_STRING    "string1"
1 - T_CONSTANT_ENCAPSED_STRING    "string2"
1 - T_WHITESPACE     
1 - T_CLOSE_TAG    ?>

这里顺便说明下,$token 如果为数组,那么分别对应的三个数组成员为 token 标识符(可以用 token_name 获得字面量)、对应的源代码内容、以及对应的行号。

还有中情况就是 $token 为字符串,这可能的情况之一就是为 T_CONSTANT_ENCAPSED_STRING 等常量,在分析代码时要注意。如果对这点很在意,可以考虑使用这里的代码。

是的,调用方式非常的简单,我们的野心当然远远要比写个简单的循环要大得多。我们可以利用这个组件做写实事,例如下面的代码用于“压缩” PHP 代码,去除不不要的换行、空白以及注释

以下为引用的内容:

/**
 * “压缩”PHP 源代码
 *
 * @see http://c7y.phparch.com/c/entry/1/art,practical_uses_tokenizer
 */
class CompactCode
{
    static protected $out;
    static protected $tokens;

    static public function compact($source)
    {
        // 解析 PHP 源代码
        self::$tokens = token_get_all($source);   
        self::$out = '';

        reset(self::$tokens);

        // 递归判断每个标记符的类型
        while ($t = current(self::$tokens)) {
            if (is_array($t)) {
                // 过滤空白、注释
                if ($t[0] == T_WHITESPACE || $t[0] == T_DOC_COMMENT || $t[0] == T_COMMENT) {
                    self::skipWhiteAndComments();
                    continue;
                }       
                self::$out .= $t[1];
            } else {
                self::$out .= $t;
            }

            next(self::$tokens);
        }

        return self::$out;
    }

    static private function skipWhiteAndComments()
    {
        // 增加个空格,用于分割关键字
        self::$out .= ' ';
        while ($t = current(self::$tokens)) {
            // 再次贪婪查找
            if (is_array($t) && ($t[0] == T_WHITESPACE || $t[0] == T_DOC_COMMENT || $t[0] == T_COMMENT)) {
                next(self::$tokens);
            } else {
                return;
            }
        }
    }
}

调用方式很简单,只需要使用

以下为引用的内容:

CompactCode::compact($source_code);

即可,返回的字符串就是压缩以后的内容。在这里还有更多使用 Tokenizer 的实例,推荐阅读。

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
11 Best PHP URL Shortener Scripts (Free and Premium)11 Best PHP URL Shortener Scripts (Free and Premium)Mar 03, 2025 am 10:49 AM

Long URLs, often cluttered with keywords and tracking parameters, can deter visitors. A URL shortening script offers a solution, creating concise links ideal for social media and other platforms. These scripts are valuable for individual websites a

Introduction to the Instagram APIIntroduction to the Instagram APIMar 02, 2025 am 09:32 AM

Following its high-profile acquisition by Facebook in 2012, Instagram adopted two sets of APIs for third-party use. These are the Instagram Graph API and the Instagram Basic Display API.As a developer building an app that requires information from a

Working with Flash Session Data in LaravelWorking with Flash Session Data in LaravelMar 12, 2025 pm 05:08 PM

Laravel simplifies handling temporary session data using its intuitive flash methods. This is perfect for displaying brief messages, alerts, or notifications within your application. Data persists only for the subsequent request by default: $request-

Build a React App With a Laravel Back End: Part 2, ReactBuild a React App With a Laravel Back End: Part 2, ReactMar 04, 2025 am 09:33 AM

This is the second and final part of the series on building a React application with a Laravel back-end. In the first part of the series, we created a RESTful API using Laravel for a basic product-listing application. In this tutorial, we will be dev

Simplified HTTP Response Mocking in Laravel TestsSimplified HTTP Response Mocking in Laravel TestsMar 12, 2025 pm 05:09 PM

Laravel provides concise HTTP response simulation syntax, simplifying HTTP interaction testing. This approach significantly reduces code redundancy while making your test simulation more intuitive. The basic implementation provides a variety of response type shortcuts: use Illuminate\Support\Facades\Http; Http::fake([ 'google.com' => 'Hello World', 'github.com' => ['foo' => 'bar'], 'forge.laravel.com' =>

cURL in PHP: How to Use the PHP cURL Extension in REST APIscURL in PHP: How to Use the PHP cURL Extension in REST APIsMar 14, 2025 am 11:42 AM

The PHP Client URL (cURL) extension is a powerful tool for developers, enabling seamless interaction with remote servers and REST APIs. By leveraging libcurl, a well-respected multi-protocol file transfer library, PHP cURL facilitates efficient execution of various network protocols, including HTTP, HTTPS, and FTP. This extension offers granular control over HTTP requests, supports multiple concurrent operations, and provides built-in security features.

12 Best PHP Chat Scripts on CodeCanyon12 Best PHP Chat Scripts on CodeCanyonMar 13, 2025 pm 12:08 PM

Do you want to provide real-time, instant solutions to your customers' most pressing problems? Live chat lets you have real-time conversations with customers and resolve their problems instantly. It allows you to provide faster service to your custom

Announcement of 2025 PHP Situation SurveyAnnouncement of 2025 PHP Situation SurveyMar 03, 2025 pm 04:20 PM

The 2025 PHP Landscape Survey investigates current PHP development trends. It explores framework usage, deployment methods, and challenges, aiming to provide insights for developers and businesses. The survey anticipates growth in modern PHP versio

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function