token_get_all

(PHP 4 >= 4.2.0, PHP 5)

token_get_all — 将提供的源码按 PHP 标记进行分割

说明

array token_get_all ( string $source )

token_get_all() 解析提供的 source 源码字符，然后使用 Zend 引擎的语法分析器获取源码中的 PHP 语言的解析器代号

解析器代号列表见解析器代号列表, 或者使用 token_name() 翻译获取这个代号的字符串表示.

参数

source: 需要解析的 PHP 源码.

返回值

An array of token identifiers. Each individual token identifier is either a single character (i.e.: ;, ., or a three element array containing the token index in element 0, the string content of the original token in element 1 and the line number in element 2.

范例

Example #1 token_get_all() examples

  <?php
$tokens  =  token_get_all ( '<?php echo; ?>' );  


 $tokens  =  token_get_all ( '' );  // => array(array(T_INLINE_HTML, ''));
 ?>

更新日志

版本	说明
5.2.2	Line numbers are returned in element 2

用户评论:

[#1] gomodo at free dot fr [2009-08-02 10:08:03]

Yes, some problems (On WAMP, PHP 5.3.0 ) with get_token_all()

1 : bug line numbers
Since PHP 5.2.2 token_get_all()  should return Line numbers in element 2..
.. but for instance (5.3.0 on WAMP), it work perfectly only with PHP code (not HMTL miwed), but if you have some T_INLINE_HTML detected by token_get_all() ,  sometimes you find wrongs line numbers  (return next line)... :(

2: bug warning message can impact loops
Warning with php code uncompleted (ex : php code line by line) :
for example if a comment tag is not closed  token_get_all()  can block loops on this  warning :
Warning: Unterminated comment starting line

This problem seem not occur in CLI mod (php command line), but only in web mod.

Waiting more stability, used token_get_all()  only on PHP code (not HMTL miwed) :
First extract entirely PHP code (with open et close php tag),
Second use token_get_all()  on the pure PHP code.

3 : Why there not function to extract PHP code (to extract HTML, we have Tidy..)?

Waiting, I used a function :

The code at end this post :
http://www.developpez.net/forums/d786381/php/langage/
fonctions/analyser-fichier-php-token_get_all/

This function not support :
- Old notation :  "<?php  ?>" and "<% %>"
- heredoc syntax
- nowdoc syntax (since PHP 5.3.0)

[#2] Dennis Robinson from basnetworks dot net [2009-06-28 21:24:40]

I wanted to use the tokenizer functions to count source lines of code, including counting comments. Attempting to do this with regular expressions does not work well because of situations where

// ...

<a href="<?php echo $PHP_SELF,'?',$_SERVER['QUERY_STRING'],'&action=email' ?>">email to us</a>

// ...

<?php if ($action=='email') include('emailForm.htm'); ?>

[#7] jrg45 at pantheon dot yale dot edu [2002-07-10 14:14:55]

Note that $_SERVER["HTTP_REFERER"] may not include GET data that was included in the referring address, depending on the browser. So if you rely on GET variables to generate a page, it's not a good idea to use HTTP_REFERER to smoothly "bounce" someone back to the page he/she came from.

[#8] postmaster at asmatic dot ch [2002-04-29 09:15:25]

If you want to get the filename requested on a global error page like a 404, just use this code...

// get the full var...
$page = $HTTP_SERVER_VARS["QUERY_STRING"];

// part[1] is the url...
// part[0] is the http code (404, etc).
if(strpos($page,";")>0) {
$pageParts = explode(";",$page);
$page = $pageParts[1];
}

// get only the filename...
$page = basename($page);

[#9] stephane-wantiez at tiscalinet dot be [2002-02-07 06:29:13]

if you do this, it will be easier :
echo "http://{$HTTP_HOST}{$REQUEST_URI}";

[#10] verdy_p at wanadoo dot fr [2001-05-26 10:47:23]

Note also that the URL shown in $HTTP_REFERER is not always the URL of the web page where the user clicked to invoke the PHP script.
This may instead be a document of your own web site, which contains an HTML element whose one attribute references the script. Note also that the current page fragment (#anchor) may be transmitted or not with the URL, depending on the browser.
Examples:
<FRAME src="your-page-script.php"8>
<IMAGE src="your-image-script.php">

In such case, browsers should transmit the URL of the container document, but some still persist in using the previous document in the browser history, and this could cause a different $HTTP_REFERER value be sent when the user comes back to the document referencing your script. If you wanna be sure that the actual current document or previous document in the history is sent, use client-side JavaScript to send it to your script:

<SCRIPT language="JavaScript"></SCRIPT><NOSCRIPT>
<FRAME src="your-page-script.php?js=0">
</NOSCRIPT>

And then check the value of $js in your page script to generate appropriate content when the remote user agent does not support client-side scripts (such as most index/scan robots, some old or special simplified browsers, or browsers with JavaScript disabled by their users).

[#11] tumor at kkt dot bme dot hu [2000-04-01 10:18:49]

To check if a URL is valid, try to fopen() it. If fopen() results an error (returns false), then PHP cannot open the URL you asked. This is usually because it is not valid...