Home > Article > Backend Development > Django batch imports xml data
Django background batch import data
In a production environment, there are often not a few or hundreds of pieces of data. So for example, if you import the employee numbers or account passwords of all the company's employees into the background, it is not recommended. Records are added one by one in the background
How to batch import svn records from xml
Step 1:
Build a model for the data
@python_2_unicode_compatible class SVNLog(models.Model): vision = models.IntegerField(verbose_name=u"修订版本", blank=False, null=False,) author = models.CharField(verbose_name=u"作者", max_length=60, blank=True, null=True) date = models.DateTimeField(verbose_name=u"修订时间",null=True ) msg = models.TextField(verbose_name=u"注释消息", blank=False, null=False, default=u"") paths = models.TextField(verbose_name=u"影响的文件", blank=False, null=False, default=u"") created_time = models.DateTimeField(verbose_name=u"创建时间", auto_now_add=True, ) update_time = models.DateTimeField(verbose_name=u"修改时间", auto_now=True, ) class Meta: ordering = ['revision'] def __str__(self): return u'r%s' % (self.revision or u"", )
Now that the model has been established, let’s go Create models that accept our xml files
@python_2_unicode_compatible class ImportLogFile(models.Model): LogFile = models.FileField(upload_to='LogFile') FileName = models.CharField(max_length=50, verbose_name=u'文件名') class Meta: ordering = ['FileName'] def __str__(self): return self.FileName
ok. In the above code, we have defined the model of data and uploaded files
Synchronize the database
python manage.py makemigrations python manage.py migrate
Then we modify admin.py Let us upload files from the background,
class ImportLogAdmin(admin.ModelAdmin): list_display = ('LogFile','FileName',) list_filter = ['FileName',] def save_model(self, request, obj, form, change): re = super(YDImportLogAdmin,self).save_model(request, obj, form, change) update_svn_log(self, request, obj, change) return re
Pay attention to save_model in the above code, here is the key, here I rewrite the save_model method in ModelAdmin
Because we need to upload files and read files, Parse the file and operate the database in one step. You can turn on debug. When uploading a file, the return parameter obj includes the path to upload the file. This path is also the key to our next step of parsing the file. Okay, let's do this Create a new utils.py under this app folder to operate the tool class we use to operate files and databases. For simplicity, I wrote the function as follows
First, paste the xml file we want to test
qwert2016-09-27T07:16:37.396449Z/aaa/README20160927 151630VisualSVN Server2016-09-20T05:03:12.861315Z/branches/tags/trunkhello word
Output result format
r2 | qwer | 2016-09-27 15:16:37 +0800 (二, 27 9 2016) | 1 line Changed paths: A /xxx/README 20160927 151630 ------------------------------------------------------------------------ r1 | VisualSVN Server | 2016-09-20 13:03:12 +0800 (二, 20 9 2016) | 1 line Changed paths: A /branches A /tags A /trunk Initial structure. from .models import SVNLog import xmltodict def update_svn_log(self, request, obj, change): headers = ['r','a','d','m','p'] filepath = obj.LogFile.path xmlfile = xmltodict.parse(open(filepath, 'r')) xml_logentry = xml.get('log').get('logentry') info_list = [] pathlist = [] sql_insert_list = [] sql_update_list = [] for j in xml: data_dict = {} # get path paths = j.get('paths').get('path') if isinstance(paths,list): for path in paths: action = path.get('@action') pathtext = path.get('#text') pathtext = action + ' ' + pathtext pathlist.append(pathtext) _filelist = u'\n'.join(pathlist) _paths = u"Changed paths:\n {}".format(_filelist) print _paths else: _filelist = paths.get('@action') + ' ' + paths.get('#text') _paths = u"Changed paths:\n {}".format(_filelist) print _paths # get revision vision = j.get('@vision') # get auth author = j.get('author') #get date date = j.get('date') #get msg msg = j.get('msg') data_dict[headers[0]] = int(vision) data_dict[headers[1]] = author data_dict[headers[2]] = date data_dict[headers[3]] = msg data_dict[headers[4]] = _paths info_list.append(data_dict) _svnlog = SVNLog.objects.filter().order_by('-vision').first() _last_version = _svnlog.vision if _svnlog else 0 for value in info_list: vision = value['r'] author = value['a'] date = value['d'] msg = value['m'] paths = value['p'] print vision,author _svnlog = YDSVNLog.objects.filter().order_by('-revision').first() _last_version = _svnlog.revision if _svnlog else 0 if vision > _last_version: sql_insert_list.append(SVNLog(revision=revision, author=author, date=date, msg = msg, paths = paths)) else: sql_update_list.append(SVNLog(revision=revision, author=author, date=date, msg = msg, paths = paths)) SVNLog.objects.bulk_create(sql_insert_list) SVNLog.objects.bulk_create(sql_update_list)
We use the third-party library xmltodict to parse xml. It parses the content into an efficient orderdict type, which is a sequenced dictionary
The more complicated thing in this xml is the path in the paths, because This xml contains two elements. The path of the first element only contains one path, and the paths in the second element contain three paths. Therefore, we need to judge when parsing and obtaining.
paths = j.get('paths').get('path') if isinstance(paths,list): pass
We judge Is this path a list type? If so, then we will process it in a list way. If not, then we will process it in a single way. After obtaining it, we will process the result according to the output result format and then get other content
revision = j.get('@vision') # get auth author = j.get('author') #get date date = j.get('date') #get msg msg = j.get('msg')
Finally, we will store the obtained elements in the dictionary
Judge the current version number and the version number in the database in the loop,
If it is smaller than the original one, then we will perform the update operation, otherwise we will perform the insertion operation
Finally, bulk_create is used to operate the database, which avoids the waste of resources caused by database operations every time in the loop