Using Git for Version Control in Audio Projects - Page 2
You are Unregistered, please register to gain Full access.    
Advertisements


Workstation Center Digital audio workstations (DAW), trackers, sequencers and programming.

Reply
Thread Tools
Old 30-03-2017, 09:16 PM   #21
camusny
Learning The Ropes
camusny is currently spamming hard
NYC
Posts: 8
MC Status: 10
Thanks: 0
Thanked 0 Times in 0 Posts
Re: Using Git for Version Control in Audio Projects

Isn't this what splice.com provides?

Workflow and Project Version Tracking...

Advertisements

camusny is offline   Thanks Reply With Quote
Old 30-03-2017, 09:31 PM   #22
Artificer
Deep Underground
Artificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MC
Artificer's Avatar
Southern Hell
Posts: 716
MC Status: 40660
Thanks: 295
Thanked 813 Times in 476 Posts
Re: Using Git for Version Control in Audio Projects

Quote:
Originally Posted by camusny View Post
Isn't this what splice.com provides?

Workflow and Project Version Tracking...
Probably. I don't use Splice, but that's my understanding. I assume it costs money?

I run my audio storage on ZFS and use snapshots for local version control. It's been working fine so far. I guess it depends on the volume of data, whether you want to work via cloud and how much control you need.
Artificer is online now   Thanks Reply With Quote
Old 30-03-2017, 11:37 PM   #23
camusny
Learning The Ropes
camusny is currently spamming hard
NYC
Posts: 8
MC Status: 10
Thanks: 0
Thanked 0 Times in 0 Posts
Re: Using Git for Version Control in Audio Projects

Quote:
Originally Posted by Artificer View Post
Probably. I don't use Splice, but that's my understanding. I assume it costs money?

I run my audio storage on ZFS and use snapshots for local version control. It's been working fine so far. I guess it depends on the volume of data, whether you want to work via cloud and how much control you need.
Damn you're using a ZFS pool for audio storage?

That's great. Is this a NAS setup w/ FreeNAS ?
camusny is offline   Thanks Reply With Quote
Old 30-03-2017, 11:49 PM   #24
Artificer
Deep Underground
Artificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MC
Artificer's Avatar
Southern Hell
Posts: 716
MC Status: 40660
Thanks: 295
Thanked 813 Times in 476 Posts
Re: Using Git for Version Control in Audio Projects

Nope, not a big fan of FreeNAS (BSD in general drives me a bit crazy). I'm running ZoL on CentOS. It's just part of my larger storage system. The setup's a bit complex, but basically audio (along with VMs that need high iops) get written to a btrfs mirror of SSDs then offloaded to the ZFS array of spinning disks. The process is fed over 10G direct connect between the workstation and NAS. The speed of the SSDs means no dropouts, and I can have my ZFS array be as slow as it wants to be. It's like very large, very flexible cache for the larger array.

Technically there's a failure point there but the chance of losing the SSD mirror before unloading to the z1 array is pretty slim given the uptime of both SSDs and a mirror, and once it's on the ZFS array it's pretty fail proof.

But yeah, snapshots work pretty well on ZFS.
Artificer is online now   Thanks Reply With Quote
Old 31-03-2017, 04:45 AM   #25
camusny
Learning The Ropes
camusny is currently spamming hard
NYC
Posts: 8
MC Status: 10
Thanks: 0
Thanked 0 Times in 0 Posts
Re: Using Git for Version Control in Audio Projects

Quote:
Originally Posted by Artificer View Post
Nope, not a big fan of FreeNAS (BSD in general drives me a bit crazy). I'm running ZoL on CentOS. It's just part of my larger storage system. The setup's a bit complex, but basically audio (along with VMs that need high iops) get written to a btrfs mirror of SSDs then offloaded to the ZFS array of spinning disks. The process is fed over 10G direct connect between the workstation and NAS. The speed of the SSDs means no dropouts, and I can have my ZFS array be as slow as it wants to be. It's like very large, very flexible cache for the larger array.

Technically there's a failure point there but the chance of losing the SSD mirror before unloading to the z1 array is pretty slim given the uptime of both SSDs and a mirror, and once it's on the ZFS array it's pretty fail proof.

But yeah, snapshots work pretty well on ZFS.
I think I've found my home in this sub-group.

So the SSD is your fast tier storage and you migrate down to slower-tier via 10G interconnect onto a zfs pool?
sounds pretty fucking cool.

how do you integrate the snapshots into your workflow? do you save a file then open your browser to a gui or is there a tigger that does it automatically for yoU?
camusny is offline   Thanks Reply With Quote
Old 31-03-2017, 05:53 AM   #26
Artificer
Deep Underground
Artificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MCArtificer is a savage MC
Artificer's Avatar
Southern Hell
Posts: 716
MC Status: 40660
Thanks: 295
Thanked 813 Times in 476 Posts
Re: Using Git for Version Control in Audio Projects

The SSDs provide iops for everything that needs them. Obviously 450MB/s write is wasted for stuff like streaming music or books or whatever so that all goes directly on the large slow storage. Most of the SSD storage is there to serve the oVirt (kvm) cluster, which is my desktop and a couple of servers right now. Everything is virtualized including my desktop, because as much as I love Linux I don't love making music on it, so Windows it is. The VMs use shared storage (the SSDs) so no data goes away unless I flush it and the VMs are set to mount and use the networked ZFS store for writing audio.

The 10G connection is what makes it possible. Recording audio over the network in real time is pretty hairy (much like network video editing) and standard ethernet gets a lot of dropouts. I actually rebuilt my storage from a three node Ceph cluster to the current ZFS setup because Ceph, while most robust, isn't really performant the way ZFS is at the scale I was running. I may play around with a Gluster setup at some point as it seems more suited to a home setting than Ceph is right now. I moved my SSD mirror from ZFS to btrfs at the same time. It's all working ok thusfar.

The SSDs and ZFS array all live in the same machine for right now, so I guess that's my NAS. I decommed one of my Ceph cluster to act as an onsite backup and then just stuffed everything else into a single case for the time being. That means there's basically no latency between the fast storage and the ZFS array so I can mirror one to the other without an issue. The SSDs are only 1TB each (so 1TB mirrored) and there's currently ~30TB on the ZFS array so as long as I clean up my messes every now and then space isn't too big an issue. Everything gets incrementally backed up to Amazon Cloud Drive nightly for a little extra security.

The snapshotting is currently just set to every 15 minutes. The way ZFS snapshots work is that they're really just a changelog of blocks that differ from the time of snapshot. So the initial snapshot doesn't actually take any extra space (as actual blocks = snapshot blocks) and only grows as those blocks change (which means that while the computer isn't doing much those snapshots don't grow in size, which is nice). It blows up when I'm writing a lot of data (like recording audio) so I occasionally have to flush the older snapshots. I've got a little script I can run if I want to make a snapshot on the fly. I'll use that when I think I'm doing something really stupid to a song and might want to go back. ZFS will also let you clone data if you want a more permanent snapshot.

If I'm being honest I really just did this as a proof of concept because I had the equipment and I wanted to see if it'd work. It does, but it's a bit janky - I've had the 10G cards stall mid-stream and lose data or crash the network, I've had some weird quorum issues with oVirt and I've had all kinds of things go wrong getting the audio stuff to virtualize. The plan was initially to make it work and then tear it all down and go back to something more reasonable (like a non-virtualized desktop) but it just seemed like a lot of trouble to undo. Now that I've got my storage pretty well set I'm likely going to tear down and tinker with the actual cluster and get something a little more standard.

And that's my setup. Ridiculous, expensive, pretty fun, moderately effective, way more trouble than it's worth. As much as git makes my tear my hair out, it's probably easier than this.

Advertisements

Artificer is online now   Thanks Reply With Quote
Reply


Thread Tools

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How do you backup your projects? Xenon Workstation Center 26 27-09-2013 12:06 PM
Embarassing Old Projects Spyderbyte Music Scene Chat 30 20-06-2013 06:57 PM
[Breaks/Techno] Online Mixing Service version Vs. My version modescmoesc Listening Booth 8 07-12-2011 11:31 PM
[Electro] G.M.U.F. - Hallway (Version 1 & Version 2) GodMadeUsFonky Listening Booth 1 08-11-2011 10:43 PM
My Projects Chromasonic Introduce Yourself 0 02-10-2011 08:08 PM


All times are GMT +1. The time now is 02:38 PM.


Electronic Music Forums

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2018, vBulletin Solutions, Inc.