By “End user” I mean a user who compiles from source, and/or package maintainer

gitLFS is seamless, it works by replacing the binary blob (e.g. a png file) with a link to the blob on the remote repository, thus the local git clone will not add the blob, but just the link. When a revision is checked out, the blob is downloaded remotely.

For this to work, the end user will simply need to do two steps (once):

1. Not only git needs to be installed, but also the git-lfs module.

   • On Arch linux:

     sudo pacman -S git-lfs
     

   • On Debian/Ubuntu:

     sudo apg-get install git-lfs
     

2. If you’re making a fresh clone of pioneer’s repo, git-lfs will bootstrap itself, and download the lfs files needed for the current branch.

   git clone https://github.com/pioneerspacesim/pioneer.git
   

   Note, there is significant speedup, especially on Windows, by explicitly using lfs:

   git lfs clone
   

   as this will first do a normal checkout, then batch download all lfs files, dramatically reducing the number of HTTP requests and processes spawned.

   Similarly, git pull works seamlessly, but if any file isn’t downloaded properly (for some reason?), try again with git lfs pull. Same with git push, which is seamless, except there will be some additional status output, as lfs tracked files are pushed to the lfs-server.

   If you already have a clone, I assume the new path to the new repo must be added?

   Get latest lfs objects on remote repo, e.g. from origin branch:

   git lfs fetch origin master
   

For future clones of the repositories, gitLFS will bootstrap itself when you clone a repo.

To not download all lfs files, e.g. when configuring a CI build for unit tests, we can exclude lfs objects using --exclude or -X flag:

git lfs fetch -X "data/models/**"

Similarly, for only including special files, use --include or -I, e.g. for an audio engineer:

git lfs fetch -I "*.ogg,*.wav"

Or combine, fetch everything in data directory, except

git lfs fetch -I "data/models/**" -X "*.dds"

Pattern is the same as for track and .gitignore. These can be permanent

git config lfs.fetchinclude "data/models/**"
git config lfs.fetchexclude "*.dds"

By “owner” I mean the person on the dev team making the actual transition

Idea is to migrate our existing git repository (on github), to use an externally hosted LFS (hosted on gitlab, since they allow up to 10 GB for free).

Initiate git-lfs in the repo (assumes git-lfs package is installed on the system):

git lfs install

mark which files to use lfs for, e.g. (note: quotes important! ²):

git lfs track "data/models/*.dae"

Now, when pushing files, there is a pre-commit hook that pushes the large files to a separate server.

The track command creates/updates the .gitattributes files (in same folder the command was issued under ³), which needs to be tracked by git, or else new clones of the repo will not get the lfs-files, so remember to check it in. Otherwise Git LFS will not be working properly for people cloning the project:

git add .gitattributes

Issuing git lfs track without any arguments shows summary of all .gitattributes-files.

To remove a pattern, simply edit the .gitattributes-file or

git lfs untrack <pattern>

(Obviously, the files should not be in .gitignore, or they’re not tracked)

Now we want to set up github to put (selected) binary files into a repository on gitlab.

“…simply adding the large files that are already in your repository to Git LFS, will not actually reduce the size of your repository because the files are still referenced by previous commits”

Solution: Follow this guide (based on a bitbucket-guide) to:

1. Clean git commit history: bfg, e.g. with (see linked guide for full info), will scan all the history, looking for any files with specified extension, and convert them to an LFS pointer:

bfg --convert-to-git-lfs "*.{png,mp4,jpg,gif}" --no-blob-protection pioneer.git

1. Then add git lfs tracking rules to new binary files
2. All developer’s branches are now borked! (And that’s a bad thing!). Possibly, one could still cherry pick? Or convert old commits to patches at the very least?

Most likely, we don’t want to do this due to it’s brütalitÿ. Just do something like in this github guide: Configuring Git Large File Storage to use a third party server where third party is gitlab.

• Disable git lfs on github settings–>options page (here)

• Check out created repo on gitlab that is to hold the lfs files, and check it’s Endpoint:

  git lfs env
  

• Switch over to github repo, and create .lfsconfig that points to third party server (Endpoint). E.g. as gitlab doc:

  git config -f .lfsconfig lfs.url https://third-party-lfs-server/path/to/repo
  

• Inspect .lfsconfig:

  [remote "origin"]
      lfsurl = https://third-party-lfs-server/path/to/repo
  

• Finally, to have same configuration for all users:

  git add .lfsconfig
  git commit -m "Adding LFS config file for third party server"
  

WHAT REMAINS?

• Find lfs server url (git lfs env on gitlab repo?)
• migrate the files: https://docs.github.com/enterprise/2.16/admin/guides/installation/migrating-to-a-different-git-large-file-storage-server

https://docs.gitlab.com/ee/user/project/repository/reducing_the_repo_size_using_git.html

skim? https://docs.gitlab.com/ee/administration/lfs/index.html

To download lfs copies for all branch tips that are more recent than 7 days (default):

git lfs fetch --recent

This is useful for e.g. when you anticipate working off line, or want to easily cherry pick commits across branches, or rewrite history.

To change default meaning of “recent” to 10 days:

git config lfs.fetchrecentrefsdays 10

Or to be bloody brütal, just fetch all:

git lfs fetch --all

these could then be removed from your local cache by:

git lfs prune

which removes all “old” lfs files, meaning files not in one of:

• currently checked out commit
• a commit that has not yet been pushed (to origin, or whatever lfs.pruneremotetocheck is set to)

• a “recent” commit (default: 10 days, by summing lfs.fetchrecentrefsdays (7) with lfs.pruneoffsetday). Can be change with:

  # don't prune commits younger than four weeks (7 + 21)
  git config lfs.pruneoffsetdays 21
  

NOTE:

“Unlike Git’s built-in garbage collection, Git LFS content is not pruned automatically, so running git lfs prune on a regular basis is a good idea to keep your local repository size down.”

Worth pruning, perhaps exercise caution:

git lfs prune --dry-run
git lfs prune --dry-run --verbose

and/or make sure the files have remote copy, before they’re pruned:

git lfs prune --verify-remote

and for peace of mind, make this the default behavior for pruning (note: takes longer time):

git config lfs.pruneverifyremotealways true

The Git LFS command-line client doesn’t support pruning files from the server, so how you delete them depends on your hosting provider.

Gitlab writes: To remove objects from LFS:

• Use git filter-repo to remove the objects from the repository.
• Delete the relevant LFS lines for the objects you have removed from your .gitattributes file and commit those changes

Some useful commands, if one knows the sha-256 OID of and LFS object:

# Find which commit references a lfs object,
: git log --all -p -S <OID>
# find a particular object by OID in HEAD
git grep <OID> HEAD
# find a particular object by OID on the "power-ups" branch
git grep <OID> power-ups

There is no easy way to resolve merge conflicts with binary files. One solution is to lock files by extension or name, to prevent them being overwritten during merge.

Need to tell git lfs which files are lockable, e.g.

git lfs track  "data/models/*.dae" --lockable

then add to .gitattributes

"data/models/*.dae" filter=lfs diff=lfs merge=lfs -text lockable

when making changes to an lfs file:

git lfs lock data/models/ships/ac33.dae Locked data/models/ships/ac33.dae

and to unlock:

git lfs unlock data/models/ships/ac33.dae

Like with push --force we can force unlocking, (be sure you know what you’re doing):

git lfs unlock data/models/ships/ac33.dae --force

Example of how to increase timeout for authentication when pushing, to one hour (src)

git config --global credential.helper 'cache --timeout=3600'