How to implement text deduplication in python (without disturbing the original order)

This article mainly introduces the python method to achieve text deduplication without disturbing the original order. Friends who need it can refer to it.

The code was also found online, and the efficiency is quite good. , especially suitable for deduplication of dictionary files

#coding=utf-8
import sys
def open_txt(): #打开TXT文本写入数组
try:
xxx = file(sys.argv[1], 'r')
for xxx_line in xxx.readlines():
passlist.append(xxx_line)
xxx.close()
except:
return 0

def write_txt(): #打开TXT文本写入数组
try:
yyy = file(sys.argv[2], 'w')
for i in list_passwed:
yyy.write(i)
yyy.close()
except:
return 0

global passlist #声明全局变量
passlist = [] #用户名：anonymous 密码为空
open_txt() #TXT导入数组
#passlist = list(set(passlist)) #python 列表去重
global list_passwed #列表去重，不打乱原来的顺序
list_passwed=[]
for i in passlist:
if i not in list_passwed:
list_passwed.append(i)
write_txt()

python reads TXT into an array list to deduplicate without disturbing the original order

####################################################################
#qq:316118740
#BLOG:http://www.php.cn/
# python 读取TXT到数组  列表去重，不打乱原来的顺序
# 刚学写的不好请大家见谅
####################################################################
 
 
 
def open_txt(): #打开TXT文本写入数组
  try:
    infile = file('admin.txt', 'r')
    xxx = file('admin.txt', 'r')
    for xxx_line in xxx.readlines():
      passlist.append(xxx_line)
    xxx.close()
  except:
    return 0
 
def list_del(): #清空list列表
  try:
    i = 0 #得到list的第一个元素
    while i < len(passlist):
      del passlist[i]
      del list_passwed[i]
  except:
    return 0
 
 
 
###################################### 
 
  global passlist #声明全局变量
  passlist = []  #用户名：anonymous 密码为空
  www_cj(www) #域名拆解
  open_txt()  #TXT导入数组
  #passlist = list(set(passlist))  #python 列表去重
  global list_passwed #列表去重，不打乱原来的顺序
  list_passwed=[]
  for i in passlist:
    if i not in list_passwed:
      list_passwed.append(i)
######################################  遍历数组组合出 密码
  I1 = 0 #得到list的第一个元素
  while I1 < len(list_passwed):
    print "WWWWWWWWWWW",I1
    if I1==len(list_passwed):
      break #退出循环
    I2 = 0 #得到list的第一个元素
    while I2 < len(list_passwed):
      print "1111:",list_passwed[I1],"2222:",list_passwed[I2]
      I2 = I2 + 1 #二层
    I1 = I1 + 1  #一层
######################################

The example in this article describes how python reads TXT into an array and sorts the list in the original order after deduplication. Share it with everyone for your reference.

The above is the detailed content of How to implement text deduplication in python (without disturbing the original order). For more information, please follow other related articles on the PHP Chinese website!