Learn GIT warehouse slimming and GIT LFS migration instructions in ten minutes

This article brings you relevant knowledge about GIT warehouse slimming and GIT LFS migration instructions. I hope it will be helpful to you.

Background

I used CI/CD on some git projects and found that jenkins git clone failed, and setting depth and clone time failed. We can only consider strategies such as warehouse downsizing. It was found that the warehouse has a lot of binary files, and these binary files change quite frequently. This operation will cause the git warehouse to grow exponentially and expand rapidly. Git itself is only suitable for managing text files.

Let me tell you another interesting past story. I once had a colleague who was into graphics programming. The source code of this language was in the form of pictures, and the file was very large. I had to use git to manage it. Small company projects were subject to frequent changes, which resulted in no How long did it take for the company's internal git server hard disk to be filled up by several of his git warehouses?

GIT LFS (Large File Storage)

Although git has never been suitable for managing binary files, git now seems to provide git lfs, a plug-in specifically for managing large files, by default.

The basic principle is simply to use a file pointer (text) instead of actual file storage. Git only stores the change history of the file pointer instead of the entire binary file, and automatically provides hooks when used. This is convenient for operations such as clone, pull, reset, etc. to automatically obtain the source binary files of these file pointers. Similarly, when updating the binary file commit, git will automatically convert the source file into a file pointer and enter it into git log, and at the same time, the source file will be uploaded to lfs. So at the user level, the use of GIT LFS is actually senseless.

Migration

The above briefly introduces GIT LFS, and then we will directly talk about how to migrate. As for why we talk about migration directly instead of how to use LFS from scratch.
This is because often when using the git warehouse, you find that the warehouse is very big and clone is very slow, and then you think about using LFS.

Migration requires us to have administrator rights of the warehouse, and to unprotect protected branches;
The specific LFS migration is mainly divided into the following steps.

It is best to make a backup before migrating and communicate well with team colleagues. After all, the operation involves -f high-risk operations, and it is easy to take the blame.

GIT LFS server configuration

If you build some self-built git services, you may need to enable LFS on the server side, such as gitlab.

GIT LFS client installation

The git installation package for windows comes with this plug-in. No additional installation is required. Other platforms can install it themselves and link.

Try the following commands on the command line.

git lfs

If there is information output similar to the help document, it means there is already a git lfs client.

git-lfs/2.11.0 (GitHub; windows amd64; go 1.14.2; git 48b28d97)git lfs <command> [<args>]Git LFS is a system for managing and versioning large files in
association with a Git repository.  Instead of storing the large files
within the Git repository as blobs, Git LFS stores special "pointer
files" in the repository, while storing the actual file contents on a
Git LFS server.  The contents of the large file are downloaded
automatically when needed, for example when a Git branch containing
the large file is checked out.Git LFS works by using a "smudge" filter to look up the large file
contents based on the pointer file, and a "clean" filter to create a
new version of the pointer file when the large file's contents change.It also uses a pre-push hook to upload the large file contents to
the Git LFS server whenever a commit containing a new large file
version is about to be pushed to the corresponding Git server.</args></command>

Then you need to execute the following command to configure the LFS global environment. It only needs to be configured once, and the hooks of the current warehouse will also be updated.

git lfs install

Migrate local history warehouse

The basic idea of ​​lfs migration: lfs rewrites the local history—>force push overwrites the remote end to achieve the migration effect.
So we'd better synchronize the local warehouse with the remote one, and create local branches for all remote branches;
Then cd to your local warehouse and execute the following command, -include contains the glob expression, Add the file name you want LFS to manage by yourself, –everything represents all local branches

git lfs migrate import --include="*.bin,*.lib,*.so,*.dll,*.a,*.param,*.zip,*.gz" --everything

migrate: Sorting commits: ..., done.
migrate: Rewriting commits: 100% (193/193), done.
  develop                       bacb490a80ea46d73bd3866c2e7cf7ad199ce5eb -> 72884bcb4629417bad73ea3d485d08a0708909cd
  feature/npu-platform          a3645632756becc527c7f4d58514b3c479f824d3 -> e227900a3903b3a6955e4dffee48daeceac6cdff
  master                        1ccdecdcb4b5d6224a6e24c6f87793bfcc15ee4c -> 1d9fc2139600ef3d92a20d65bb5db89021b8c488
  0.1.0                         07c6b2aa732506f1cc88cedb551f37f376b6efa6 -> 8e55193221dfca9f6bb28ccd9cca85af9c5958c9
  1.0.0                         0f694efcd7aa9df641836e1ea6eebbb730b940b5 -> 3f9e77575120b6e56b34790c998a362116da75f5
migrate: Updating refs: ..., done.

After rewriting local branches, tags, etc.,

We can execute git lfs here first ls-filesCheck which files have been converted to lfs management and check if there are any omissions

At this time, no matter which branch you are in, the .gitattributes file will appear and will be added. Something similar to the following.

*.bin filter=lfs diff=lfs merge=lfs -text
*.lib filter=lfs diff=lfs merge=lfs -text
*.so filter=lfs diff=lfs merge=lfs -text
*.dll filter=lfs diff=lfs merge=lfs -text
*.a filter=lfs diff=lfs merge=lfs -text
*.param filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text

At the same time, you can see that all our binary files have been converted into the following form of text

version https://git-lfs.github.com/spec/v1
oid sha256:9171c8350d72ccca6ad60ac80b577157ad1f9fd44ca05744216e02ccbfcdf491
size 10260

Confirm that it is correct, and then it can be pushed to the remote end;
Since the migration of lfs will be rewritten All commits, and the hash value is modified, so we need to add –froce
This step requires canceling the protected branch (protected branches cannot be -f)

git push --force --all

In this way, the lfs migration of the remote warehouse is completed

迁移一些补充说明

1. 迁移者的本地仓库lfs文件转源文件：经过以上步骤，由于我们将所有文件都已经转成文件指针，我们需要将文件下载回来才能正常使用该仓库。
   需要注意，其他人重新clone 或者同步 lfs迁移过的remote仓库 是不需要该步，只针对迁移作者本地的仓库。

git lfs pull

2. 团队中其他成员迁移前的本地仓库同步: 由于远程仓库的历史已经被全部重写，所以无法直接同步，最好是删除本地分支，重新拉取远程分支，如果本地已经有部分commit需要提交，可以重名本地分支，拉取远程再做cherry pick。git tag 同理，删除迁移前的tag。
3. 本地仓库清理：上面的迁移成功将二进制文件迁移成git lfs 对象，git log 也不在存储源文件文件变更而是指针变更，但是在本地.git文件夹中仍存在之前不再需要的git log 缓存，执行以下命令做清理。

git reflog expire --expire-unreachable=now --all
git gc --prune=now

清理前后仓库对比

lfs直观来讲更多的是针对仓库大clone慢的问题，我这边lfs迁移前后各备份各一个小型远程仓库做测试，
用的测试仓库二进制文件比较小，总大50m内，且变更次数也在个位数。
clone下来的仓库大小对比。
和我预估差不多，总的来说更适合二进制文件频繁变更，如果单纯是文件大，但文件不变更的话，在clone的时候区别不大，毕竟lfs在clone仍有下载源文件的步骤，除开下载，操作文件指针对git来说理论仍会有性能提升，但是可能感知不强。

推荐学习：《Git教程》

The above is the detailed content of Learn GIT warehouse slimming and GIT LFS migration instructions in ten minutes. For more information, please follow other related articles on the PHP Chinese website!