☞☞☞AI 智能聊天, 问答助手, AI 智能搜索, 多模态理解力帮你轻松跨越从0到1的创作门槛☜☜☜
{\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"attrs\":{\"id\":\"9f02a959-f008-471e-a4a1-c26adebf8187\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"摘要:从文档图像中准确恢复表格结构是文档智能的核心挑战之一。针对复杂无边框表格及跨行跨列单元格难以解析的问题,本文提出一种以ocr文本行为原语的多模态图transformer识别框架。该框架综合编码文本行的几何、视觉与语义特征,通过空间关系偏置的图注意力网络推断行间、列间及单元格合并关系,并引入可微行列分配损失保证全局结构一致性。在pubtabnet及icdar 2019 ctdar数据集上的评估表明,该方法在树编辑距离相似度指标上显著优于纯几何图方法,尤其对无线表格的鲁棒性增强明显。\"}]},{\"type\":\"image\",\"attrs\":{\"id\":\"aaac1970-73f4-4477-a1b8-f3fcb4d7c462\",\"src\":\"https://developer.qcloudimg.com/http-save/audit-7081184/59e791d0019609de0ecb21f37ef8dce5.png\",\"extension\":\"png\",\"align\":\"center\",\"alt\":\"\",\"showalt\":false,\"href\":\"\",\"boxshadow\":\"\",\"width\":842,\"aspectratio\":\"1.342903\",\"status\":\"success\",\"showtext\":true,\"ispercentage\":false,\"percentage\":0,\"ishoverdraghandle\":false}},{\"type\":\"orderedlist\",\"attrs\":{\"id\":\"727b5242-5292-4aa7-8b0b-bf81b20f333f\",\"start\":1,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"listitem\",\"attrs\":{\"id\":\"dd73e022-cc20-4bcf-99ea-72ef27e7c016\"},\"content\":[{\"type\":\"paragraph\",\"attrs\":{\"id\":\"56f38a22-7138-401f-9a21-3fcddd9405ff\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"引言\"},{\"type\":\"hardbreak\",\"attrs\":{\"id\":\"329361ad-49c0-4e24-9bff-16c28da9e4f0\"}},{\"type\":\"text\",\"text\":\"表格结构识别旨在从图像中提取单元格的行列坐标与合并关系,生成机器可读的层次化表示。典型管线为:先定位表格区域,再对该区域内的文字块赋予逻辑结构。本文聚焦于结构识别环节,并假设已由前端ocr引擎(如paddleocr或tesseract)提供了较为准确的文本行边界框和文本内容。现实中的表格常具有部分缺失的框线、不规则的居中对齐与密集排列,使基于投影直方图或连通域分析的启发式方法失效。近年来,基于深度学习的工作分为像素级分割(如tablenet、lgpma)和基于图的关系推理(如graphtsr、ties)两大流派。纯分割方案易产生单元格断裂或粘连,而现有图方法大多仅利用坐标几何,忽略了相邻文本行的视觉连贯性和语义信息,难以可靠处理跨页表头或同类格式的合并单元。\"}]}]}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"a4c1d5f6-9fbb-40a1-90d5-474e3068b2a5\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"我们观察到,判断两个文本块是否属于同一行、同一列或同一单元格,不仅依赖于坐标重叠,还与字体、背景、对齐线等视觉线索高度相关。为此,本文提出一种多模态图transformer结构,将每一ocr文本行视为图节点,动态构建空间感知邻域,并通过关系偏置的自注意力机制联合推断两两之间的结构关系,最终解码为逻辑表格。\"}]},{\"type\":\"orderedlist\",\"attrs\":{\"id\":\"39576f80-393f-4ea7-aab6-31c55122b159\",\"start\":2,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"listitem\",\"attrs\":{\"id\":\"d2ebbab3-ad34-4a0d-a21c-1b3474fe7c66\"},\"content\":[{\"type\":\"paragraph\",\"attrs\":{\"id\":\"08900e85-1f8a-40cd-b194-ad47325c6735\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"方法\"},{\"type\":\"hardbreak\",\"attrs\":{\"id\":\"a27b5244-e156-4342-9e4c-335d9534b1ee\"}},{\"type\":\"text\",\"text\":\"给定表格区域图像 i\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\" 及ocr输出的 n\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"n\"},{\"type\":\"text\",\"text\":\" 个文本行 b={bi}\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"b\"},{\"type\":\"text\",\"text\":\"={b\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\"},bi=(xi,yi,wi,hi,ti)b\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\"=(x\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\",\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"yi\"},{\"type\":\"text\",\"text\":\",\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"wi\"},{\"type\":\"text\",\"text\":\",h\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\",t\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\"),tit\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\" 为文字串。模型输出关系图,从中解析单元格的行号、列号与跨行跨列属性。\"}]}]}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"b4b22467-cddd-4c5a-8ee2-742c74305c02\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"2.1 多模态节点特征\"},{\"type\":\"hardbreak\",\"attrs\":{\"id\":\"86cbb3de-0e51-44fa-8991-bf759a0cc064\"}},{\"type\":\"text\",\"text\":\"为每个文本行提取三种模态的嵌入。几何特征 gi\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"gi\"},{\"type\":\"text\",\"text\":\" 由归一化中心坐标、宽高及面积组成,经过两层mlp映射为64维。视觉特征 vi\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"vi\"},{\"type\":\"text\",\"text\":\" 通过roialign从共享cnn特征图中裁剪对应区域,再经一个轻量resnet块得到128维。文本语义特征 sis\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\" 利用预训练的轻量语言模型将 tit\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i\"},{\"type\":\"text\",\"text\":\" 转化为64维向量。三者拼接后经线性投影得到节点初始状态 hi(0)∈r256h\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i(0)\"},{\"type\":\"text\",\"text\":\"∈r\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"256\"},{\"type\":\"text\",\"text\":\"。\"}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"dc2d2161-f36a-4cec-a32a-6cdf8878c67f\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"2.2 空间感知图构建\"},{\"type\":\"hardbreak\",\"attrs\":{\"id\":\"d9d786a3-ff6e-4eb8-a517-ee3412580db4\"}},{\"type\":\"text\",\"text\":\"为避免对全连接图进行平方级复杂度的计算,边集 e\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"e\"},{\"type\":\"text\",\"text\":\" 由两个准则构成:对每个节点选取欧氏距离最近的 k=16\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"k\"},{\"type\":\"text\",\"text\":\"=16 个邻居;并加入水平或垂直投影交并比超过0.2的文本行对。对边 (u,v)(u,\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"v\"},{\"type\":\"text\",\"text\":\"),显式编码其相对几何向量 ruv=[Δx,Δy,Δw,Δh,iouh,iouv]r\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"uv\"},{\"type\":\"text\",\"text\":\"=[Δx,Δ\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"y\"},{\"type\":\"text\",\"text\":\",Δ\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"w\"},{\"type\":\"text\",\"text\":\",Δh,iou\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"h\"},{\"type\":\"text\",\"text\":\",iou\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"v\"},{\"type\":\"text\",\"text\":\"],为注意力提供空间先验。\"}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"29f54cd8-8dd1-476c-b267-468208c15b30\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"2.3 关系推理与结构解码\"},{\"type\":\"hardbreak\",\"attrs\":{\"id\":\"585dbcec-0e19-48cd-a4a9-700527ae3a65\"}},{\"type\":\"text\",\"text\":\"采用 l=3l=3 层多头图注意力进行信息传播。第 l\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"l\"},{\"type\":\"text\",\"text\":\" 层的节点更新式为:\"}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"a233d027-d958-495a-8fc2-54a820da4404\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"hu(l+1)=ffn(∥m=1m∑v∈n(u)αuvmwvmhv(l))h\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"u(l+1)\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"=ffn\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"m=1m\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"v∈n(u)∑\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"αuvm\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"wvm\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"h\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"v(l)\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"\"}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"2454704f-4de3-4a3e-b62e-8abf44a1aa71\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"注意力系数引入关系偏置:\"}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"ae8c7f20-cf2d-4d2d-8c9a-ebd57172481c\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"αuvm=softmax((wqmhu(l))t(wkmhv(l))d+mlp(ruv))\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"αuvm\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"=softmax\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"(d(wqmhu(l))t(wkmhv(l))\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\"+mlp(r\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"uv\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"rgb(15, 17, 21)\",\"background\":\"\"}}],\"text\":\")\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\")\"}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"0d584b11-1e3e-4132-9098-66bfef69e117\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"该设计使模型能根据空间相对位置调整消息权重,垂直邻近的节点自然强化行关系的判别。\"}]},{\"type\":\"image\",\"attrs\":{\"id\":\"43d3b815-c4f1-4a4f-988c-d0db5e467214\",\"src\":\"https://developer.qcloudimg.com/http-save/audit-7081184/d10ead34da2d7d62bbf4bc793e241273.png\",\"extension\":\"png\",\"align\":\"center\",\"alt\":\"\",\"showalt\":false,\"href\":\"\",\"boxshadow\":\"\",\"width\":1016,\"aspectratio\":\"2.175589\",\"status\":\"success\",\"showtext\":true,\"ispercentage\":false,\"percentage\":0,\"ishoverdraghandle\":false}},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"89de2370-305f-44ad-92c7-e202e368835e\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"最终节点特征 hi(l)h\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"i(l)\"},{\"type\":\"text\",\"text\":\" 用于边级关系预测。对于每条边 (u,v)(u,\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"v\"},{\"type\":\"text\",\"text\":\"),双线性分类器输出四种关系的概率:同行不同列、同列不同行、同一单元格以及无关。训练时采用交叉熵损失。并行地,为每个节点分配行列索引,并利用匈牙利匹配损失解决行列置换不变性,加速收敛并提升全局一致性。\"}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"b342f27c-21b9-4f79-8077-a902325a2a61\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"解码阶段,依据“同列”关系构建列图、依据“同行”关系构建行图,通过连通分量分析得到初步行列划分;将“同一单元格”边连接的节点合并,形成跨行跨列单元。对空穴与冲突进行后处理,最终输出html结构的表格。\"}]},{\"type\":\"orderedlist\",\"attrs\":{\"id\":\"53f90861-7424-42b8-9409-c57e0e39f698\",\"start\":3,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"listitem\",\"attrs\":{\"id\":\"41092976-3cc9-4f9b-b463-7e23e254cdb5\"},\"content\":[{\"type\":\"paragraph\",\"attrs\":{\"id\":\"df58ea25-b86b-4d7c-a528-3ef3187c7714\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"实验\"},{\"type\":\"hardbreak\",\"attrs\":{\"id\":\"545bd94d-5bdd-4885-a7b7-b97dcd0b1395\"}},{\"type\":\"text\",\"text\":\"在pubtabnet及icdar 2019 ctdar track b2数据集上评估。前端ocr采用paddleocr,表格区域检测使用faster r-cnn。骨干网络为resnet-50+fpn,优化器adamw,初始学习率10−410\"},{\"type\":\"text\",\"marks\":[{\"type\":\"textstyle\",\"attrs\":{\"color\":\"\",\"background\":\"\"}}],\"text\":\"−4\"},{\"type\":\"text\",\"text\":\"。主要指标采用树编辑距离相似度(teds)。我们的方法在pubtabnet上达到96.5%的teds,相较仅使用几何坐标的图基线提高2.1个百分点;无边框子集上提升更为显著(+3.8%)。消融实验表明,视觉特征与语义嵌入分别贡献约0.9%和1.1%的teds提升,且空间关系偏置使训练收敛步数减少约30%。\"}]}]},{\"type\":\"listitem\",\"attrs\":{\"id\":\"53759e58-063b-4c06-8865-b30f2bba0ec4\"},\"content\":[{\"type\":\"paragraph\",\"attrs\":{\"id\":\"63832a9a-3d54-499f-95a5-bfb9a7ded06b\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false},\"content\":[{\"type\":\"text\",\"text\":\"结论\"},{\"type\":\"hardbreak\",\"attrs\":{\"id\":\"5a19b6a9-ad9b-44ef-86c7-83d2fc564e91\"}},{\"type\":\"text\",\"text\":\"本文提出了融合几何、视觉与语义的多模态图注意力表格结构识别方法。利用空间关系偏置的自注意力机制和可微行列分配,在复杂表格上实现了稳健的结构恢复。未来工作将探索与表格检测的端到端联合学习,并拓展至弯曲文档与移动端应用。\"}]}]}]},{\"type\":\"paragraph\",\"attrs\":{\"id\":\"e6d7ac69-4a3f-4ab5-93b7-1a0cfe7970f0\",\"textalign\":\"inherit\",\"indent\":0,\"color\":null,\"background\":null,\"ishoverdraghandle\":false}}]}